Li, X., Li, Z., Li, J., Xie, H., & Li, Q. ESE: Espresso Sentence Embeddings.
|
 |
|
Abstract
|
Extensive experiments on STS and RAG suggest that ESE can effectively produce high-quality embeddings with less model depth and embedding size, enhancing embedding inference efficiency. High-quality sentence embeddings are fundamental in many natural language processing (NLP) tasks, such as semantic textual similarity (STS) and retrieval-augmented generation (RAG). Nevertheless, most existing methods leverage fixed-length embeddings from full-layer language models, which lack the scalability to accommodate the diverse available resources across various applications. Viewing this gap, we propose a novel sentence embedding model \$\textbackslash mathrm\{Espresso\}\$ \$\textbackslash mathrm\{Sentence\}\$ \$\textbackslash mathrm\{Embeddings\}\$ (ESE) with two learning processes. First, the learn-to-express process encodes more salient representations to lower layers. Second, the learn-to-compress process compacts essential features into the initial dimensions using Principal Component Analysis (PCA). This way, ESE can scale model depth via the former process and embedding size via the latter. Extensive experiments on STS and RAG suggest that ESE can effectively produce high-quality embeddings with less model depth and embedding size, enhancing embedding inference efficiency.
|
| Notes |
[Online; accessed 25. May 2024]
|
