Shee, P., Kundu, S., Bhar, A., Ghosh, M., & Tech, B. Rebert-an enhanced bert.
|
 |
|
Abstract
|
This article shows that BERT can be a competitive lexical normalization model without the need of any UGC resources aside from 3,000 training sentences, and adapt and analyze the ability of this model to handle noisy UGC data. - BERT (Bidirectional Encoder Representations from Transformers) is a transformer-based language model that comprehends the context of words by considering surrounding words in both directions. It revolutionized natural language processing by capturing rich contextual information, enhancing performance in various language understanding tasks like sentiment analysis, text classification. In this article, focusing on User Generated Content (UGC) in a resource-scarce scenario, we study the ability of BERT (Devlinet al., 2018) to perform lexical normalization. by enhancing its architecture and by carefully finetuning it, we show that BERT can be a competitive lexical normalization model without the need of any UGC resources aside from 3,000 training sentences. The enhanced BERT model features a hierarchical contextualization module for improved long-range dependency understanding, a domain-specific adaptation layer for specialized language contexts, and efficiency optimization through dynamic attention head pruning and weight sharing. Fine-tuned pre-training broadens language comprehension, while task-specific heads enable fine-tuning. Rigorous evaluation and iterative refinement ensure performance enhancement across tasks, addressing limitations and advancing language understanding. It will be our first work done in adapting and analyzing the ability of this model to handle noisy UGC data.
|
| Notes |
[Online; accessed 31. May 2024]
|
