Causality Papers I Should Read
A running list of papers on causality, language models and machine learning that I want to read (or reread).
Foundations of causal inference
- From Tokens to Policy: Causal and Interpretable Heterogeneous Treatment Effects Identification
- Causal Inference: A Tale of Three Frameworks
- What is causal about causal models and representations?
- Prediction-Powered Causal Inferences
- Smoke and Mirrors in Causal Downstream Tasks
- Comparing Causal Frameworks: Potential Outcomes, Structural Models, Graphs, and Abstractions
- Counterfactual Identifiability of Bijective Causal Models
- From Statistical to Causal Learning
- A Topological Perspective on Causal Inference
- Causal Consistency of Structural Equation Models
Causal representation learning and invariance
- Causal Learning with the Invariance Principle
- Causal Fine-Tuning under Latent Confounded Shift
- Identifiable Exchangeable Mechanisms for Causal Structure and Representation Learning
- Unifying Causal Representation Learning with the Invariance Principle
- Do Finetti: On Causal Effects for Exchangeable Data
- Causal de Finetti: On the Identification of Invariant Causal Structure in Exchangeable Data
- Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
Causal reasoning in language models
- Synergizing Large Language Models and Theory for Human-like Causal Reasoning
- CauSim: Scaling Causal Reasoning with Increasingly Complex Causal Simulators
- Towards a Universal Causal Reasoner
- Compared to What? Baselines and Metrics for Counterfactual Prompting
- The Illusion of Intervention: Your LLM-Simulated Experiment is an Observational Study
- Language Models as Causal Effect Generators
- Prompting Fairness: Integrating Causality to Debias Large Language Models
- Preference Learning for AI Alignment: a Causal Perspective
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- Passive learning of active causal strategies in agents and language models
LLMs as (causal) scientists
- CauScientist: Teaching LLMs to Respect Data for Causal Discovery
- CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists
- TabPFN-3: Technical Report
- Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systems
- Training AI Co-Scientists Using Rubric Rewards
How models learn, and interpretability
- What do Language Models Learn and When? The Implicit Curriculum Hypothesis
- LittleLearner: Language Models Under Pedagogically-Controlled Knowledge Exposure
- Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning
- The Fragile Foundations of CoT Monitoring
- Causality is Key for Interpretability Claims to Generalise
- Position: Don’t Just “Fix it in Post”: A Science of AI Must Study Training Dynamics
- Bridging Fairness and Explainability: Can Input-Based Explanations Promote Fairness in Hate Speech Detection?
- From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence
- Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models
- The emergence of sparse attention: impact of data distribution and benefits of repetition
- How Do Large Language Models Acquire Factual Knowledge During Pretraining?
- Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task
- What Can Transformers Learn In-Context? A Case Study of Simple Function Classes
- The MultiBERTs: BERT Reproductions for Robustness Analysis
- Measuring Causal Effects of Data Statistics on Language Model’s `Factual’ Predictions
Agents, games and world models
- What Must Generalist Agents Remember?
- From Observation to Insight: Mechanistic World Models and the Quest for Autonomous Discovery
- Code World Models for General Game Playing
- SPARTAN: A Sparse Transformer World Model Attending to What Matters
- Reasoning about Causality in Games
- Agent Incentives: A Causal Perspective
Philosophy of science and learning theory
- Understanding Generalization and Robustness in Modern Deep Learning
- The Falsificationist View of Machine Learning
- Popper, Falsification and the VC-dimension
- Statistical Learning Theory: Models, Concepts, and Results
- A Falsificationist Account of Artificial Neural Networks