Proceedings of the 19th Conference on Natural Language Processing (KONVENS 2023)
Yes
19th Conference on Natural Language Processing (KONVENS 2023)
from 18-09-2023 to 22-09-2023
Ingolstadt
Germany
[en] natural language processing ; luxembourgish ; NLP ; BERT ; pre-training ; language model ; computational linguistics ; datasets ; low-resource language ; luxembert
[en] Despite the widespread use of pre-trained models in NLP, well-performing pre-trained models for low-resource languages are scarce. To address this issue, we propose two novel BERT models for the Luxembourgish language that improve on the state of the art. We also present an empirical study on both the performance and robustness of the investigated BERT models. We compare the models on a set of downstream NLP tasks and evaluate their robustness against different types of data perturbations. Additionally, we provide novel datasets to evaluate the performance of Luxembourgish language models. Our findings reveal that pre-training a pre-loaded model has a positive effect on both the performance and robustness of fine-tuned models and that using the German GottBERT model yields a higher performance while the multilingual mBERT results in a more robust model. This study provides valuable insights for researchers and practitioners working with low-resource languages and highlights the importance of considering pre-training strategies when building language models.
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > TruX - Trustworthy Software Engineering
[University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > TruX >]
