We've upgraded! Visit our new site at www.undermind.ai. The new deep search tool covers all of science, not just ArXiv, and is 10x faster. Your previous data is still available at app.undermind.ai.

Search Topic:

To what extent do large language models (LLMs) based on transformers learn to represent grammar in their attention mechanism?

Additional Context Provided:

Many language models use attention to process text. I want to understand how attention mechanisms learn to link certain word types, like nouns, to other word types, like verbs. I want papers that explicitly look at how this sort of connection is present in certain heads in the transformer attention mechanism.


Deep search found 32 relevant papers. This is ~72% of all relevant papers that exist on the arXiv database (see comprehensiveness analysis for details).

Highly Relevant References
 🟢 [1]
Do Attention Heads in BERT Track Syntactic Dependencies? Phu Mon Htut, ..., Samuel R. Bowman (2019)

We investigate the extent to which individual attention heads in pretrained transformer language models, such as BERT and RoBERTa, implicitly capture syntactic dependency relations. We employ two methods---taking the maximum attention weight and computing the maximum spanning tree---to extract implicit dependency relations from the attention weights of each layer/head, and compare them to the ground-truth Universal Dependency (UD) trees. We show that, for some UD relation types, there exist heads that can recover the dependency type significantly better than baselines on parsed English text, suggesting that some self-attention heads act as a proxy for syntactic structure. We also analyze BERT fine-tuned on two datasets---the syntax-oriented CoLA and the semantics-oriented MNLI---to investigate whether fine-tuning affects the patterns of their self-attention, but we do not observe substantial differences in the overall dependency relations extracted using our methods. Our results suggest that these models have some specialist attention heads that track individual dependency types, but no generalist head that performs holistic parsing significantly better than a trivial baseline, and that analyzing attention weights directly may not reveal much of the syntactic knowledge that BERT-style models are known to learn.

The paper directly examines the relationship between the attention heads of transformer-based language models (e.g., BERT, RoBERTa) and syntactic dependency relations. The methods used in the study, such as comparing attention weights to Universal Dependency trees, are well-suited to understanding how grammar is captured within the attention mechanism. The paper even tests for differences in syntactic tracking pre- and post-fine-tuning. Given that the paper's focus is squarely on the syntactic information captured by attention heads, it is likely to be of high relevance to your colleague's research question.

 🟢 [2]
From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentions David Mareček, ..., Rudolf Rosa (2019)

We inspect the multi-head self-attention in Transformer NMT encoders for three source languages, looking for patterns that could have a syntactic interpretation. In many of the attention heads, we frequently find sequences of consecutive states attending to the same position, which resemble syntactic phrases. We propose a transparent deterministic method of quantifying the amount of syntactic information present in the self-attentions, based on automatically building and evaluating phrase-structure trees from the phrase-like sequences. We compare the resulting trees to existing constituency treebanks, both manually and by computing precision and recall.

This paper appears to be highly relevant to the specific topic of interest. The authors have investigated the self-attention patterns in transformer-based NMT (neural machine translation) encoders, with an aim to find syntactic structures within those patterns. They have developed a quantitative method to evaluate the syntactical content of the attention heads by building constituency trees from the attention outputs and comparing these to standard treebanks. This approach seems to specifically target how grammatical relationships may be represented in attention heads, which is precisely what the researcher is interested in. Furthermore, since the study involves a manual examination of the resulting structure as well as precision and recall comparisons, it suggests a thorough analysis of attention heads in capturing syntactic information.

 🟢 [3]
Attention Can Reflect Syntactic Structure (If You Let It) Vinit Ravishankar, ..., Joakim Nivre (2021)

Since the popularization of the Transformer as a general-purpose feature encoder for NLP, many studies have attempted to decode linguistic structure from its novel multi-head attention mechanism. However, much of such work focused almost exclusively on English -- a language with rigid word order and a lack of inflectional morphology. In this study, we present decoding experiments for multilingual BERT across 18 languages in order to test the generalizability of the claim that dependency syntax is reflected in attention patterns. We show that full trees can be decoded above baseline accuracy from single attention heads, and that individual relations are often tracked by the same heads across languages. Furthermore, in an attempt to address recent debates about the status of attention as an explanatory mechanism, we experiment with fine-tuning mBERT on a supervised parsing objective while freezing different series of parameters. Interestingly, in steering the objective to learn explicit linguistic structure, we find much of the same structure represented in the resulting attention patterns, with interesting differences with respect to which parameters are frozen.

The paper 'Attention Can Reflect Syntactic Structure (If You Let It)' closely aligns with the desired topic of interest, as it explicitly examines the capability of multi-head attention mechanisms within transformers, specifically multilingual BERT, to reflect dependency syntax. The authors not only present decoding experiments across multiple languages but also delve into the subtleties of the attention mechanism as it relates to understanding linguistic structure. Moreover, the paper provides experimental evidence suggesting that single attention heads can encode full syntactic trees and that certain syntactic relations are consistently captured by the same heads across different languages. The study also contributes to the debate about the explanatory power of attention by exploring the impact of fine-tuning with various parameters frozen, thereby shedding light on the model's learning of explicit linguistic structure.

 🟢 [4]
Have Attention Heads in BERT Learned Constituency Grammar? Ziyang Luo (2021)

With the success of pre-trained language models in recent years, more and more researchers focus on opening the "black box" of these models. Following this interest, we carry out a qualitative and quantitative analysis of constituency grammar in attention heads of BERT and RoBERTa. We employ the syntactic distance method to extract implicit constituency grammar from the attention weights of each head. Our results show that there exist heads that can induce some grammar types much better than baselines, suggesting that some heads act as a proxy for constituency grammar. We also analyze how attention heads' constituency grammar inducing (CGI) ability changes after fine-tuning with two kinds of tasks, including sentence meaning similarity (SMS) tasks and natural language inference (NLI) tasks. Our results suggest that SMS tasks decrease the average CGI ability of upper layers, while NLI tasks increase it. Lastly, we investigate the connections between CGI ability and natural language understanding ability on QQP and MNLI tasks.

The paper directly investigates a critical aspect of the researcher's inquiry, focusing on the extent to which the attention heads of specific language models, BERT and RoBERTa, have internalized constituency grammar. Utilizing the syntactic distance method to analyze the implicit grammar within attention weights indicates a targeted approach to understanding the nuances of grammar representation. The paper also extends this analysis to evaluate how fine-tuning tasks influence this representation, which could provide depth to the understanding of attention mechanism's adaptability regarding grammatical relations. Moreover, exploring the connection between grammatical induction and general language understanding tasks aligns well with the overarching interest in the functional significance of grammar representation in LLMs.

 🟢 [5]
A Primer in BERTology: What we know about how BERT works Anna Rogers, ..., Anna Rumshisky (2020)

Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue and approaches to compression. We then outline directions for future research.

The selected parts of the paper discuss studies focusing on the interpretability of BERT's self-attention heads in relation to encoding syntactic relationships, which aligns well with the interest in understanding how LLMs based on transformers represent grammar. Specific studies mentioned in the text, such as Clark et al. (2019) and Htut et al. (2019), have found evidence of attention heads attending to words in specific syntactic roles and the analysis of whether attention weights can serve as indicators for linguistic structures like subject-verb agreement and reflexive anaphora. Additionally, the paper touches on the specialization of attention heads in tracking semantic relations, their contribution to model performance, and the ablation studies that test their necessity. This level of detail suggests that the paper directly addresses the research topic in question and would contribute significantly to an understanding of how grammatical relationships are handled by attention mechanisms in transformers. This makes the paper highly relevant and worthy of detailed examination.

 🟢 [6]
Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads Bowen Li, ..., Frank Keller (2020)

Transformer-based pre-trained language models (PLMs) have dramatically improved the state of the art in NLP across many tasks. This has led to substantial interest in analyzing the syntactic knowledge PLMs learn. Previous approaches to this question have been limited, mostly using test suites or probes. Here, we propose a novel fully unsupervised parsing approach that extracts constituency trees from PLM attention heads. We rank transformer attention heads based on their inherent properties, and create an ensemble of high-ranking heads to produce the final tree. Our method is adaptable to low-resource languages, as it does not rely on development sets, which can be expensive to annotate. Our experiments show that the proposed method often outperform existing approaches if there is no development set present. Our unsupervised parser can also be used as a tool to analyze the grammars PLMs learn implicitly. For this, we use the parse trees induced by our method to train a neural PCFG and compare it to a grammar derived from a human-annotated treebank.

This paper appears to be highly relevant to the specific research interest. The authors address the syntactic knowledge learned by pre-trained language models (PLMs), like BERT, which is central to the inquiry of whether these models incorporate grammar into their attention mechanisms. Specifically, the method proposed for unsupervised constituency parsing directly evaluates the grammatical structures that can be extracted from attention heads, thus offering insight into how grammar might be represented within the attention mechanism. Furthermore, the research is aligned with BERTology, which is mentioned as a critical reference point for this line of inquiry. The approach to analyze induced parse trees to understand the implicit grammars learned by PLMs is consistent with the goal of identifying connections between word types as learned by individual attention heads in a transformer model.

 🟢 [7]
Open Sesame: Getting Inside BERT's Linguistic Knowledge Yongjie Lin, ..., Robert Frank (2019)

How and to what extent does BERT encode syntactically-sensitive hierarchical information or positionally-sensitive linear information? Recent work has shown that contextual representations like BERT perform well on tasks that require sensitivity to linguistic structure. We present here two studies which aim to provide a better understanding of the nature of BERT's representations. The first of these focuses on the identification of structurally-defined elements using diagnostic classifiers, while the second explores BERT's representation of subject-verb agreement and anaphor-antecedent dependencies through a quantitative assessment of self-attention vectors. In both cases, we find that BERT encodes positional information about word tokens well on its lower layers, but switches to a hierarchically-oriented encoding on higher layers. We conclude then that BERT's representations do indeed model linguistically relevant aspects of hierarchical structure, though they do not appear to show the sharp sensitivity to hierarchical structure that is found in human processing of reflexive anaphora.

This paper presents a directly relevant analysis to the topic of interest. It explores whether BERT's self-attention vectors encode hierarchical (syntactic structure) and linear (word order) information. The authors use diagnostic classifiers and a new quantitative method to evaluate BERT's attention patterns and their reflection of syntactic knowledge, particularly in terms of structural relationships like subject-verb agreement and anaphor-antecedent dependencies. Consequently, the paper assesses exactly how BERT's attention heads may relate to grammatical structures. It appears to address the core aspects of your colleague's topic, delving into the specifics of grammar representation within the attention mechanism of a large language model.

 🟢 [8]
Probing LLMs for Joint Encoding of Linguistic Categories Giulio Starace, ..., Ekaterina Shutova (2023)

Large Language Models (LLMs) exhibit impressive performance on a range of NLP tasks, due to the general-purpose linguistic knowledge acquired during pretraining. Existing model interpretability research (Tenney et al., 2019) suggests that a linguistic hierarchy emerges in the LLM layers, with lower layers better suited to solving syntactic tasks and higher layers employed for semantic processing. Yet, little is known about how encodings of different linguistic phenomena interact within the models and to what extent processing of linguistically-related categories relies on the same, shared model representations. In this paper, we propose a framework for testing the joint encoding of linguistic categories in LLMs. Focusing on syntax, we find evidence of joint encoding both at the same (related part-of-speech (POS) classes) and different (POS classes and related syntactic dependency relations) levels of linguistic hierarchy. Our cross-lingual experiments show that the same patterns hold across languages in multilingual LLMs.

The paper in question appears to extensively investigate how LLMs encode syntactic features, such as parts of speech and syntactic dependency relations, which are core components of grammar. The authors explore how linguistic categories are jointly encoded, which suggests an examination of attention mechanisms' role in linking word types like nouns to verbs. This directly relates to the researcher's interest in understanding the attentional linking across grammatical categories within LLMs. Furthermore, the cross-lingual aspect adds value by showing that these findings are not language-specific but generalize across languages, indicating a deeper understanding of the grammatical learning process of transformers. The research questions align well with the desired topic, specifically looking at syntactic category encoding and interactions at different linguistic hierarchy levels within the model.

 🟢 [9]
Acceptability Judgements via Examining the Topology of Attention Maps Daniil Cherniavskii, ..., Evgeny Burnaev (2022)

The role of the attention mechanism in encoding linguistic knowledge has received special interest in NLP. However, the ability of the attention heads to judge the grammatical acceptability of a sentence has been underexplored. This paper approaches the paradigm of acceptability judgments with topological data analysis (TDA), showing that the geometric properties of the attention graph can be efficiently exploited for two standard practices in linguistics: binary judgments and linguistic minimal pairs. Topological features enhance the BERT-based acceptability classifier scores by $8$%-$24$% on CoLA in three languages (English, Italian, and Swedish). By revealing the topological discrepancy between attention maps of minimal pairs, we achieve the human-level performance on the BLiMP benchmark, outperforming nine statistical and Transformer LM baselines. At the same time, TDA provides the foundation for analyzing the linguistic functions of attention heads and interpreting the correspondence between the graph features and grammatical phenomena.

This paper is highly relevant to the specific topic of interest. It explores the role of the attention mechanism in encoding linguistic knowledge, with a particular focus on grammatical acceptability judgements. The authors utilize topological data analysis to examine the attention maps of transformer models, seeking to understand the correspondence between attention patterns and grammatical phenomena. This aligns well with the desired topic of exploring how LLMs based on transformers might learn to represent grammar within their attention mechanism. Moreover, the paper contributes novel insights by applying a specific methodological approach (TDA) to the problem, potentially offering a unique perspective on the relationship between attention head activity and grammatical structures.

 🟢 [10]
Physics of Language Models: Part 1, Context-Free Grammar Zeyuan Allen-Zhu, ..., Yuanzhi Li (2023)

We design controlled experiments to study HOW generative language models, like GPT, learn context-free grammars (CFGs) -- diverse language systems with a tree-like structure capturing many aspects of natural languages, programs, and logics. CFGs are as hard as pushdown automata, and can be ambiguous so that verifying if a string satisfies the rules requires dynamic programming. We construct synthetic data and demonstrate that even for difficult (long and ambiguous) CFGs, pre-trained transformers can learn to generate sentences with near-perfect accuracy and impressive diversity. More importantly, we delve into the physical principles behind how transformers learns CFGs. We discover that the hidden states within the transformer implicitly and precisely encode the CFG structure (such as putting tree node information exactly on the subtree boundary), and learn to form "boundary to boundary" attentions resembling dynamic programming. We also cover some extension of CFGs as well as the robustness aspect of transformers against grammar mistakes. Overall, our research provides a comprehensive and empirical understanding of how transformers learn CFGs, and reveals the physical mechanisms utilized by transformers to capture the structure and rules of languages.

The paper conducts controlled experiments utilizing generative language models, such as GPT, to learn and generate sentences based on context-free grammars (CFGs), which are relevant to understanding grammatical structure. The authors analyze how transformers learn and represent the CFG structure within the hidden states and attention mechanism, specifically focusing on 'boundary to boundary' attentions that resemble a dynamic-programming approach. This examination of attention patterns as they correspond to the CFG's syntactic structure aligns closely with the desired topic of understanding how attention links word types and represents grammar. The distinction made between generative and encoder-based models, like BERT and DeBERTa, indicates a nuanced approach that could be highly pertinent for understanding the specific capabilities and limitations of attention mechanisms in different transformer models with regard to grammatical representation.

Closely Related References
 🟡 [11]
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned Elena Voita, ..., Ivan Titov (2019)

Multi-head self-attention is a key component of the Transformer, a state-of-the-art architecture for neural machine translation. In this work we evaluate the contribution made by individual attention heads in the encoder to the overall performance of the model and analyze the roles played by them. We find that the most important and confident heads play consistent and often linguistically-interpretable roles. When pruning heads using a method based on stochastic gates and a differentiable relaxation of the L0 penalty, we observe that specialized heads are last to be pruned. Our novel pruning method removes the vast majority of heads without seriously affecting performance. For example, on the English-Russian WMT dataset, pruning 38 ou