CAU · Data science project
What happened to Stack Overflow?
How much of the fall in new questions since ChatGPT goes beyond Stack Overflow's old decline, and beyond what similar sites saw
- Questions vs. comparable sites
- −72%
- Fewer questions, Dec 2022 – Sep 2026
- 1.6M
- Robustness checks within −60% to −74%
- 15/15
The problem
New Stack Overflow questions have collapsed since ChatGPT came out in November 2022. But they were already falling slowly from 2014, and other Q&A sites fell too. The real question is how much of the drop goes beyond that old decline and beyond what similar sites saw, and what it did to the community that remains.
The data
Stack Exchange data dump, 2026-Q3 (27 sites) ↗
Every question on Stack Overflow and 26 other Stack Exchange sites to September 30, 2026, parsed from XML to about 10 GB of Parquet. Published as weekly and monthly panels (CC BY-SA 4.0).
Approach
- Wrote down the hypotheses, methods and decision rules in a pre-registration before any model saw post-ChatGPT data, and logged every later change with its date.
- Built a synthetic Stack Overflow from a weighted mix of non-programming sites that tracked it closely from 2017 to 2022, then measured the gap after ChatGPT.
- Checked the result with three other methods (an interrupted time series and two forecasts), placebo tests on all 16 sites, and 15 robustness checks.
- Modelled all 297 of the largest tags with partial pooling, and followed newcomers and answerers by cohort.
- Corrected the four secondary hypotheses for multiple tests (Holm), as pre-registered.
How it holds up
- Pre-registered before any model saw post-ChatGPT data
- Synthetic control; 3 other methods agree
- Holm-corrected: 1 of 5 hypotheses supported
Of the five pre-registered hypotheses, only H1 (questions fell beyond the trend and the comparison sites) is supported. H2–H5 (tags, question mix, newcomers, answerers) are not, after the pre-registered Holm correction. H1's interval was read as the average of monthly intervals, a choice made after seeing both readings and logged as a deviation; under the other reading H1 would fail its interval test, though every other check points the same way.
Results
From December 2022 to September 2026, new questions averaged 72% below the synthetic Stack Overflow (95% interval −76% to −68%), about 1.6 million fewer questions. The gap grew every year: −34% in 2023, −61% in 2024, −81% in 2025 and −92% in 2026. It is the largest gap of all 16 sites, the three other methods give −81% to −83%, and every robustness check stays between −60% and −74%. All 297 large tags fell, but older, well-documented tags fell less, not more, the opposite of the "thin training data" idea. Newcomers who post again within a year fell from 44% to 29%, and questions answered within 24 hours from 67% to 55%.
What this doesn't show
That ChatGPT alone caused the fall. Coding assistants and Stack Overflow's own changes came at the same time.
The comparison sites fell too (−31% against their own trends), so −72% is Stack Overflow's fall beyond similar sites, and its total loss to AI is probably larger.
Tools
- Python
- DuckDB
- Polars
- pandas
- statsmodels
- pysyncon
- Prophet
- PyMC
- Bambi
- lifelines
- Quarto