Cs224n, S., Project, D., Xia, S., & Jiang, Y. minBERT and extensions for downstream tasks.
|
 |
|
Abstract
|
Detailed analysis reveals that the model for sentiment analysis is capable at predicting polarity, but sometimes faces difficulty in predicting specific extents of polarity, and indicates the importance of analysis beyond simple numerical metrics, and the necessity for data quality control. Sentiment classification, paraphrase identification and semantic textual similarity are among those main-stream NLP tasks. The goal of this project is to implement BERT models from basic building blocks and to develop a model which performs well on these three tasks. The baseline method is to use pre-trained BERT encoders and add three task-specific heads at the end for prediction. Following that, six extensions on top of the baseline method were explored, including revising the training scheme, loss functions, model scales, sentence fusion method and adding in-domain training. Results show fine-tuning the entire model significantly improves performances on all three tasks, compared with freezing the BERT-based encoder. Also, concatenating the two sentences at the beginning to use as input is very effective for sentence pair tasks, compared with concatenating the two sentence embeddings at the end for task-specific heads. Furthermore, it was shown that scaling up helps only if other design choices such as the sentence integration method are sound. Detailed analysis reveals that the model for sentiment analysis is capable at predicting polarity, but sometimes faces difficulty in predicting specific extents of polarity. It also indicates the importance of analysis beyond simple numerical metrics, and the necessity for data quality control. As a pedagogical project, these main findings and analysis are very useful for future projects as they can inform future project planning and design choices.
|
| Notes |
[Online; accessed 1. Jun. 2024]
|
