Imagine asking an LLM to predict the sentiment of a movie review. How does it decide whether the tone is positive or negative? Large Language Models (LLMs), including transformer-based systems such as GPT-style models and BERT, have transformed natural language processing, allowing us to ask conversational agents to summarise text, answer questions, and classify sentiment. However, their impressive outputs can often feel opaque, leaving users and developers to wonder: why did the model make this prediction? This is where Explainable AI (XAI) steps in, offering tools to make model predictions more transparent. In this post, we will explore four key approaches to understanding the behaviour of LLMs.
Feature Attribution: Pinpointing Influential Inputs
One way to explain a model’s output is through feature attribution. These methods aim to identify which parts of the input, such as words or tokens, had the greatest influence on the model’s prediction. A common starting point is the attention mechanism, which is a foundational component of many LLMs.1 Attention weights can be visualised as heat maps, showing how strongly different tokens attend to one another during processing.
Figure 1 shows an example of how attention weights can be visualised. The input “query”, represented by a row token, is shown on the y-axis, and its attention weight is mapped to each “key/value” column token on the x-axis. Darker cells indicate stronger attention, meaning that the column token contributes more to updating the row token’s representation. The weights are typically normalised so that each row sums to approximately 1, making it easier to compare the relative strength of different connections.
Attention maps can provide a useful starting point for interpretation because they show which tokens the model is focusing on internally. However, they should be treated with caution: high attention does not always mean that a token caused the final prediction. For this reason, attention-based explanations are often most useful when combined with other XAI methods.
Post-Hoc Explanations: Decoding Predictions After the Fact
While attention maps use information from inside the model, post-hoc explanation methods analyse a prediction after it has been made. These methods are especially useful because they can often be applied without needing full access to the model’s internal architecture.
Important examples include model-agnostic tools such as SHAP2 and LIME3, which work by systematically perturbing the input and observing how the output changes. This allows us to estimate which tokens or features most influenced the prediction. If changing part of the input causes a large change in the output, this suggests that it had a strong impact on the model’s decision. Figure 2 shows an example of a LIME-style explanation, where the size of each bar corresponds to the influence of each word. In practice, this can reveal why an email was classified as spam, or why a customer support query was routed to a specific department.
Counterfactual analysis4 builds on a similar idea by asking: what is the smallest change to the input that would change the model’s output? For example, if switching “he” to “she” in a job application significantly alters the result, this may indicate bias in the system. Counterfactuals can therefore help identify whether a model is relying on sensitive or inappropriate features.
Concept-Based Explanations: Linking Models to Human Ideas
LLMs do not simply process words one by one. They map text into high-dimensional mathematical representations that can capture relationships between words, phrases, and concepts. Concept-based explanations aim to connect these internal representations back to human-understandable ideas.
For example, a model might represent words such as “happy”, “joyful”, and “excited” in nearby regions of its internal representation space. This suggests that the model has learned a broader concept related to positivity. This idea is central to concept vectors,5 which allow researchers to test whether a model has encoded a particular concept and how strongly that concept influences its predictions.
Concept-based explanations can be especially valuable because they move beyond individual words and instead ask whether models are using higher-level ideas that make sense to humans. This can help researchers understand not only what a model predicts, but also what kinds of concepts may be shaping its behaviour.
Global Model Understanding: Seeing the Bigger Picture
Unlike feature attribution or concept-based explanations, which often focus on a specific prediction or concept, global interpretation methods aim to summarise a model’s overall behaviour across many inputs. These techniques help researchers identify broader patterns in how a model organises information.
However, the internal representations used by LLMs are usually high-dimensional, making them difficult to interpret directly. Dimensionality reduction methods such as t-SNE6 and UMAP7 offer a way to visualise these representations in two or three dimensions. For example, they can show whether texts with similar topics, writing styles, or sentiments cluster together, and where the model may be confusing one category with another.
Global analysis can also reveal how representations change from layer to layer, or whether the model makes recurring types of errors. In this way, global methods complement local explanations by providing a system-level view of model behaviour.
The Path Forward: Building Transparent AI
Interpreting LLMs is no simple task. Their size and complexity make explanations computationally challenging, and the explanations produced by XAI methods can sometimes be ambiguous or incomplete. Models may also appear confident even when their predictions are unreliable, which makes it important to interpret explanations carefully.
Explainable AI methods are helping us better understand LLMs by providing partial, task-specific insights into their behaviour. Taken together, feature attribution, post-hoc explanations, concept-based analyses, and global model understanding can make models less opaque and more trustworthy, even if full transparency remains an open challenge. As LLMs become increasingly important in fields such as healthcare, law, and education, the demand for interpretability will only grow. XAI is not just a tool for researchers; it is a foundation for building AI systems we can better understand, evaluate, and trust.
References
1. Vaswani, Ashish, et al. “Attention Is All You Need.” Advances in Neural Information Processing Systems (2017).
2. Lundberg, Scott M., and Su-In Lee. “A Unified Approach to Interpreting Model Predictions.” Advances in Neural Information Processing Systems (2017).
3. Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. “Why Should I Trust You? Explaining the Predictions of Any Classifier.” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144, 2016.
4. Wachter, Sandra, Brent Mittelstadt, and Chris Russell. “Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR.” Harvard Journal of Law & Technology 31 (2017): 841.
5. Kim, Been, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, and Fernanda Viégas. “Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV).” International Conference on Machine Learning, pp. 2668–2677. PMLR, 2018.
6. van der Maaten, Laurens, and Geoffrey Hinton. “Visualizing Data Using t-SNE.” Journal of Machine Learning Research 9, no. 11 (2008).
7. McInnes, Leland, John Healy, and James Melville. “UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction.” arXiv preprint arXiv:1802.03426 (2018).
About the Author
Molly Kennedy, BA, MSc is a doctoral researcher specializing in Explainable AI for understanding and mitigating media bias in Large Language Models, under the supervision of Prof. Dr. Hinrich Schütze at the Chair for Computational Linguistics (LMU Munich). As a member of relAI, her work focuses on making AI-generated content more transparent, interpretable, and accountable.
