Sufleta, G. J., & Taori, R. Integrating Cosine Similarity into minBERT for Paraphrase and Semantic Analysis.
|
 |
|
Abstract
|
This paper highlights the architecture and performance evaluation of miniBERT across various tasks, including sentiment analysis, paraphrase detection, and semantic textual analysis, with the upgrades proposed by the SBERT model and suggests directions for future research to enhance model robustness and performance across diverse NLP tasks. In Natural Language Processing (NLP), efficient and semantically rich sentence embeddings has been a driving force for innovation. This paper investigates harnesses miniBERT, a scaled down version of the default BERT architecture, while trying to enhance it with sentence embeddings through Siamese BERT-networks (SBERT). The objective was to have a balance between computational efficiency and performance in downstream tasks. Leveraging insights from Reimers and Gurevych (2019), miniBERT inherits the efficiency enhancements of SBERT while striving to maintain high performance. This paper highlights my attempts to har-nass the architecture and performance evaluation of miniBERT across various tasks, including sentiment analysis, paraphrase detection, and semantic textual analysis, with the upgrades proposed by the SBERT model. Experimental results discussed herein highlight the challenges of overfitting, performance plateauing, and task antagonism, particularly in the fine-tuning phase. Strategies such as progressive layer unfreezing, task-specific optimization heads, selective backpropagation, and early stopping are discussed as potential avenues for mitigating these challenges. The study provides insights into the nuanced dynamics of model optimization and suggests directions for future research to enhance model robustness and performance across diverse NLP tasks.
|
| Notes |
[Online; accessed 25. May 2024]
|
