The Automation Blind Spot

Before automated testing, developers reviewed test code alongside production code. Tests received comments, discussions, and scrutiny during code review. The review process was imperfect, but test quality was at least subject to human attention.

Sun, Wu, Assunção, and Stolee (arXiv:2603.15935) replicate earlier studies of test code review in the GitHub Actions era and find a striking shift. After projects adopt GitHub Actions for automated testing, the probability of test code receiving any review comments drops to a median of zero. Comment density on test files drops to zero. The tests still exist — they run automatically on every pull request — but nobody reads them anymore.

The mechanism is displacement, not neglect. GitHub Actions provides a green checkmark when tests pass. The checkmark substitutes for the review that previously evaluated test quality. Developers trust the automation to validate the tests, but the automation only validates that tests pass — not that they test the right things, cover the right cases, or will catch future regressions. The signal (“tests pass”) displaces the evaluation (“tests are good”).

This is the automation paradox applied to quality assurance itself. The tool designed to improve testing quality reduces the human attention that maintained testing quality. The green checkmark answers a narrow question (do tests pass?) but is interpreted as answering a broad question (are tests adequate?). The gap between these questions is invisible precisely because the automation’s confidence is so legible.

The same pattern operates wherever a measurement displaces the judgment it was supposed to support. Hospital safety checklists that replace clinical reasoning. Compliance metrics that replace ethical evaluation. Performance reviews that replace actual observation of work. The instrument meant to support a human process ends up replacing it, and what gets lost is exactly what the instrument cannot measure.