Provided by

arXiv

arxiv.org
𝕏gh
0.0/5
0 views
0 saved
FreeadvancedText Generation
ResearchersAI EngineersData Scientists
Introduces BERT, a groundbreaking model using deep bidirectional Transformers for state-of-the-art language understanding tasks.

Overview

BERT (Bidirectional Encoder Representations from Transformers) is a pioneering language representation model that revolutionized NLP. It leverages deep bidirectional Transformers, pre-trained on vast unlabeled text using novel tasks like Masked Language Model and Next Sentence Prediction. This approach allows BERT to jointly condition on left and right contexts, enabling it to achieve state-of-the-art results across various NLP benchmarks, including GLUE, MultiNLI, and SQuAD, with minimal task-specific fine-tuning. Its profound impact laid the foundation for modern large language models.

Abstract

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike previous work, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), and SQuAD v1.1 F1 score to 93.2 (1.5% absolute improvement), outperforming the previous best systems by a large margin.