AI Qazaqstan
Исследования

How Different are Different Differences-in-Differences? A Large-Scale Meta-Analysis -- by Peter Hull

I use AI-assisted replication to assess the empirical importance of negative weights in two-way fixed effects (TWFE) estimation, drawing on 1,346 staggered-adoption specifications from 108 economics papers. TWFE estimates are highly correlated with estimates from modern difference-in-differences methods, with observed sign reversals in 10.8–14.5% of comparisons and greater precision for TWFE. Hierarchical models that account for estimation noise imply a 91–98% correlation between estimands and a

7 November 2026 · Source: NBER new working papers

I use AI-assisted replication to assess the empirical importance of negative weights in two-way fixed effects (TWFE) estimation, analyzing 1,430 staggered-adoption specifications from 116 economics papers. TWFE estimates are highly correlated with modern difference-in-differences estimates, with sign reversals in 12.0–14.9% of comparisons and greater precision for TWFE. Hierarchical models accounting for estimation noise imply estimand correlations of 91–97% and a 0.7–2.4% probability of a statistically significant TWFE estimate opposing the modern estimand’s sign. These findings suggest negative weights may be a less serious problem in practice and illustrate how AI can support systematic, large-scale empirical assessments of new econometric methods. Acknowledgements and Disclosures I thank Jason Abaluck, Zach Bleemer, Kirill Borusyak, Jiafeng Chen, Xavier Jaravel, Lihua Lei, Jonathan Roth, Pedro Sant’Anna, Jann Spiess, and Amanda Starc for helpful comments; Samuel Bazzi, Judson Boomhower, Enrico Cantoni, Pedro Dal Bó, Kyle Emerick, Luiz Felipe Fontes, Sebastian Galiani, Álvaro García-Marín, Matthew Gibson, Ricard Gil, Alex Hollingsworth, Aljoscha Janssen, Dean Karlan, Victor Lavy, Monica Martinez-Bravo, Benjamin Marx, Mathilde Muñoz, Peter Nencka, Mounu Prem, Paul Raschky, Jonah Rexer, Edson Severnini, Joakim Weill, and Thomas Wollmann for help replicating their papers; and OpenAI’s ChatGPT for excellent research assistance. Refine.ink was used to check for consistency and clarity. The views expressed herein are those of the author and do not necessarily reflect the views of the National Bureau of Economic Research. I thank Jason Abaluck, Zach Bleemer, Kirill Borusyak, Jiafeng Chen, Xavier Jaravel, Lihua Lei, Jonathan Roth, Pedro Sant’Anna, Jann Spiess, and Amanda Starc for helpful comments; Samuel Bazzi, Judson Boomhower, Enrico Cantoni, Pedro Dal Bó, Kyle Emerick, Luiz Felipe Fontes, Sebastian Galiani, Álvaro García-Marín, Matthew Gibson, Ricard Gil, Alex Hollingsworth, Aljoscha Janssen, Dean Karlan, Victor Lavy, Monica Martinez-Bravo, Benjamin Marx, Mathilde Muñoz, Peter Nencka, Mounu Prem, Paul Raschky, Jonah Rexer, Edson Severnini, Joakim Weill, and Thomas Wollmann for help replicating their papers; and OpenAI’s ChatGPT for excellent research assistance. Refine.ink was used to check for consistency and clarity. The views expressed herein are those of the author and do not necessarily reflect the views of the National Bureau of Economic Research. Citation and Citation Data Copy Citation Peter Hull, "How Different are Different Differences-in-Differences? A Large-Scale Meta-Analysis," NBER Working Paper 35974 (2026), https://doi.org/10.3386/w35974. Copy to Clipboard Download Citation MARC RIS BibTeΧ Download Citation Data Other Versions October 28, 2026 In addition to working papers, the NBER disseminates affiliates’ latest findings through a range of free periodicals—the NBER Reporter, the NBER Digest, the Bulletin on Health, the Bulletin on the Economics of Alzheimer's, and the Bulletin on Entrepreneurship—as well as research spotlights, lectures, and panel videos. Presenters: Melissa Dell & Ashesh Rambachan Presenter: Paul Goldsmith-Pinkham

Read at the source →

More in this topic

ИсследованияSame Facts, Different Remedy: Linguistic Profiling and Arbitrariness in LLM and System One DecisionsarXiv:2611.08995v1 Announce Type: new Abstract: Large language models are increasingly used to resolve consumer complaints, and the way complainants write varies with age, education and socioeconomic status (SES). We audit three remedy-decision systems using sociolinguistic rewrites of US consumer financial complaints that are instructed to preserve every fact: two generative LLMs that return a single label (GPT-4o-mini and GPT-5-mini) and a probabilistic "System One" model that returns a distriarXiv cs.CY (Computers and Society)
ИсследованияAI Exposure and the Changing Responsibilities in Job PostingsarXiv:2611.09870v1 Announce Type: new Abstract: The growing adoption of artificial intelligence (AI) in the workplace has raised questions about how employer demand for different work activities is changing. We examine how the prevalence and content of work activities in online job postings change in association with activity-level measures of AI exposure, derived from Copilot and Claude usage. To enable systematic comparisons, we develop a pipeline that extracts the responsibilities from job adarXiv cs.CY (Computers and Society)
ИсследованияSeeing Is Not Knowing: The Epistemic Limits of LLM-Based Mental Health Inference from Youth Social Media ConversationsarXiv:2611.08565v1 Announce Type: cross Abstract: Adolescents increasingly use social media for private conversations, sharing experiences and seeking support. These interactions generate digital traces, prompting efforts to infer mental health and psychosocial states. As large language models become more capable of reasoning about conversations and interpersonal dynamics, a fundamental question remains nonetheless: what can models actually "know" about an individual's psychological experience farXiv cs.CY (Computers and Society)
ИсследованияEntering a New Domain in the Era of Generative AI: An Autoethnography of Solo Game Development as a Bioinformatics EducatorarXiv:2611.09623v1 Announce Type: cross Abstract: The rapid improvement of generative AI (GenAI) not only increased the efficiency of practice but also empowered individuals with capabilities to explore new domains. However, identity formation in the wake of such empowerment remains understudied. We use an autoethnography to understand identity formation in the context of solo game development as a bioinformatics educator. With the goal of developing a create-your-own-adventure (CYOA) game, we carXiv cs.CY (Computers and Society)