GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost

Abstract

Large pre-trained models have revolutionized natural language processing (NLP) research and applications, but high training costs and limited data resources have prevented their benefits from being shared equally amongst speakers of all the world's languages. To address issues of cross-linguistic access to such models and reduce energy consumption for sustainability during large-scale model training, this study proposes an effective and energy-efficient framework called GreenPLM that uses bilingual lexicons to directly ``translate'' pre-trained language models of one language into another at almost no additional cost. We validate this approach in 18 languages' BERT models and show that this framework is comparable to, if not better than, other heuristics with high training costs. In addition, given lightweight continued pre-training on limited data where available, this framework outperforms the original monolingual language models in six out of seven tested languages with up to 200x less pre-training efforts. Aiming at the Leave No One Behind Principle (LNOB), our approach manages to reduce inequalities between languages and energy consumption greatly. We make our codes and models publicly available at https://github.com/qcznlp/GreenPLMs.

Contribution highlights

• To address issues of cross-linguistic access to such models and reduce energy consumption for sustainability during large-scale model training, this study proposes an effective and energy-efficient framework called GreenPLM that uses bilingual lexicons to directly ``translate'' pre-trained language models of one language into another at almost no additional cost.

Extracted from the public abstract; no claims have been added.

Code and data

Publication metadata

DOI

10.24963/ijcai.2023/698

Pages / article

6290-6298

Version relationship

• Preprint: https://arxiv.org/abs/2211.06993
• Version of record: https://doi.org/10.24963/ijcai.2023/698

BibTeX

@inproceedings{Zeng2023Greenplmcross,
  title = {GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost},
  author = {Qingcheng Zeng and Lucas Garay and Peilin Zhou and Dading Chong and Yining Hua and Jiageng Wu and Yikang Pan and Han Zhou and Rob Voigt and Jie Yang},
  booktitle = {International Joint Conference on Artificial Intelligence (IJCAI)},
  year = {2023},
  url = {https://arxiv.org/abs/2211.06993},
  doi = {10.24963/ijcai.2023/698},
  pages = {6290--6298}
}