British Police Assessed the Danger Level of Hundreds of Thousands of People Using AI Algorithms. However, the Forecasts Turned Out to Be Unreliable - Wired

Avon and Somerset Police created a large-scale data collection and algorithmic risk assessment system. It contained information on almost half a million residents, and based on this data, the authorities tried to predict who might commit a crime, become a victim of violence, go missing, or fall into other "risk groups." However, at least two AI models had to be shut down due to low accuracy and the impossibility of independent verification of results. This is reported in a Wired investigation. 🔵 A Database for Half the City. At the center of the investigation is the Think Family Database, launched in 2016 by Bristol City Council and the police. It contained various sensitive data: reports, information on housing, mental health, teenage pregnancies, participation in parenting courses, and free school meals. On top of this information, officials built machine learning models that assigned risk scores to adults and children. The idea was to get an overall "picture of threats, harm, and risk" in the region. Journalists are aware of at least 23 models - from predicting burglaries and failure to appear in court to the risk of a person going missing and the probability of becoming a victim of domestic violence. 🔵 Data Was Collected Without People's Knowledge. How exactly the police developed these tools remained unclear for a long time. One of the figures in the material, activist John Pigram, learned about the existence of the database only in 2023. When he tried to find out if the police were using his data, the department initially refused to answer. Later, the police confirmed the information but did not disclose details. Pigram did not know if the algorithm had assigned him any risk score, what it might be, and how it affected his interaction with the authorities. 🔵 Unreliable Models. Wired found that at least two risk assessment models were quietly discontinued after Bristol City Council employees deemed them unreliable. These were systems related to the risks of sexual and criminal exploitation of children. Council employees complained that the algorithms had become inaccurate: children who should have been considered vulnerable did not appear in the results, and the origin of the scores was opaque. 🔵 Errors. The police provided Wired with a dataset on the performance of 13 risk models used from 2017 to 2024. The independent company Eticas, which studied this data, concluded that many models showed low accuracy - a significant portion of people they flagged as risky were identified incorrectly. One model, related to burglary prediction, showed less than 10% accuracy for over three years. The police stated that some developed models were never implemented, and problematic tools were updated or disabled. The department also claims that models undergo expert review before being launched. 🔵 Context. In April 2025, The Guardian reported that the UK Ministry of Justice was working on a program to "predict murders." According to the authorities' plan, AI algorithms would identify potential murderers among already convicted criminals. Human rights activists then called the project dystopian and stated that the algorithm would begin to discriminate against ethnic minorities and the poor.

Avon and Somerset police developed an AI system analyzing data on nearly 500,000 residents to predict risks like crime or becoming a victim. This system, built on sensitive personal information, faced criticism for its lack of transparency and low accuracy. At least two AI models, designed to identify risks for children, were discontinued due to unreliability and opaque results, while a burglary prediction model operated with less than 10% accuracy for over three years.

  • Avon and Somerset police deployed an AI system analyzing data of nearly 500,000 residents to predict various risks.
  • The Think Family Database collected sensitive information without explicit knowledge of the individuals involved.
  • At least two AI models related to child exploitation risks were discontinued due to low accuracy and lack of transparency.
  • A burglary prediction model reportedly had an accuracy rate below 10% for over three years.
  • Previous reports indicate the UK Ministry of Justice is developing an AI program to “predict murders.”
Write a comment