Jadwin, A., & Huang, C. Improving minBERT Performance on Multiple Tasks through In-domain Pretraining, Negatives Ranking Loss Learning, and Hyperparameter Optimization.
|
 |
|
Abstract
|
This study employs an in-domain pretraining strategy in which minBERT is pretrained on a Masked Language Model (MLM) objective on the datasets which it performs tasks on, which significantly improved minBERT’s performance. BERT models have seen a recent explosion in use cases, but an understanding of how to optimize BERT for various tasks is developing. The present study aims to improve the performance of minBERT, a smaller version of the original BERT model, on a variety of sentence-level tasks (sentiment classification, paraphrase detection, and semantic contextual similarity) simultaneously. To do so, we employ an in-domain pretraining strategy in which minBERT is pretrained on a Masked Language Model (MLM) objective on the datasets which it performs tasks on. We also employ Negatives Ranking Loss Learning to improve baseline BERT embed-dings. Both strategies, along with optimal learning rate selection, significantly improved minBERT’s performance.
|
| Notes |
[Online; accessed 1. Jun. 2024]
|
