Table of contents
Three research groups spent this spring studying roughly 250,000 real Claude conversations from April and May 2026. Two university labs and one nonprofit, all working on AI usage data pulled straight from live production traffic.
None of them read a single conversation.
That is the design working as intended, and it is the part nobody is talking about.
What Anthropic shipped
The tool is called Anthropic Insights, and it used to be called Clio. A researcher writes one question, and Claude runs it against every conversation in the sample. The answers get sorted into categories, and the researcher sees the category names plus the percentage of conversations in each one. That is the whole output, with no transcripts and no raw text. Anthropic’s own example of such a question is “What type of guidance is this person asking for?”
The AI usage data never leaves Anthropic in raw form. Only the counts do.
Anthropic says this is the first time external researchers have run public independent studies on an AI company’s own AI usage data. That claim holds up.
The contracts are solid too, better than I expected from a lab publishing its own report card. Review rights covered user privacy and anything that could help people break usage policy. They also covered Anthropic’s own confidential information and the accuracy of the research. The agreements say partners can publish findings “even when they are inconvenient for Anthropic”. Imperial College London ran a third-party privacy audit. The aggregate data is on HuggingFace already, so the thing is real.
The privacy guarantee is the audit hole
Anthropic wrote this themselves, which is to their credit, because they could have left it out.
“Because we are relying on Claude’s judgments, the tool is sensitive to a question’s wording; a poorly phrased one can place conversations into categories that misrepresent them.”
Then comes the sentence that should have been the headline.
“Because no one can read the underlying conversations, these errors are hard to catch.”
Read that twice, because the loop it describes is closed. The instrument doing the measuring is a language model and the thing being measured is a language model. The safeguard that protects users is the same safeguard that stops anyone from checking whether the AI usage data means what the category labels say it means.
If you lean on a model’s judgment, you should have a prior. When I went through seven open-weight models mapping threat reports to ATT&CK, they got it wrong four times out of five.
Internally Anthropic handles this by rewriting the question over and over for weeks. External partners could not, because every new dataset needs another privacy review and the study would never finish.
Why the practice sandbox does not match production
The workaround was to have the researchers tune their questions on WildChat, a public dataset of human-AI conversations where they could read the underlying text and check whether the categories made sense. Then they took the tuned question and pointed it at live Claude traffic.
Some questions that worked well on WildChat produced misleading categories once applied to actual Claude conversations, because WildChat leans casual and creative and Claude traffic does not. Two pools of AI usage data, two different populations.
Anyone who has written a detection rule knows this shape. The rule fires clean against your test corpus, then drowns you in false positives the first hour it sees real traffic. It is the same failure in a different field.
WildChat also carries its own baggage, which I covered in Dossier 33 through the Truffle Security scan of 7.6 petabytes of HuggingFace training data. One Infura key, pasted once into a ChatGPT conversation, got captured by WildChat and copied onward into 1,131 public datasets and 10,162 file locations.
So the AI usage data you are allowed to read is the batch where mistakes are permanent. The production data where mistakes get corrected is the batch nobody may open. That is one trade, made twice.
What it means when you quote AI usage data
The findings themselves are worth having. Stanford’s SALT Lab found that over half of Claude conversations involved people delegating consequential tasks to AI, and in nearly three-quarters of them people set the direction while Claude assisted.
Before you drop that into a board deck, know what you are holding. That is Claude’s judgment on what counts as “consequential”, turned into a percentage. The researcher who published it cannot go back and check a single case. Anthropic also runs studies where people answer for themselves, like the 81,000-person survey I wrote up earlier, and a model inferring intent from a transcript is a different instrument. The number can still be true. It is a different animal than a measurement, and the difference matters the moment someone writes policy on top of it.
There is one more filter on this AI usage data. Anthropic removed or altered any category that described the method users found for getting around safeguards. Categories covering what users attempted stayed in. Less than 5% of categories and conversations in each study, and they told the researchers which clusters were touched and why. That is honest handling. It is still the lab editing the dataset before the auditor sees it.
Four questions before you cite an AI usage data study
Steal these. They work on any usage research that reaches you, this pilot included.
1. Find out who read the primary records. If a model did the reading, say so out loud when you quote the number.
2. Get the exact wording of the question the model answered. The wording sets the categories, and the categories are the finding.
3. Ask what was removed before publication, and whether anyone told you. Anthropic did tell its partners which clusters were touched, which most companies will not.
4. Check whether a second team could reproduce the result on the same records. On live lab traffic today, nobody can.
The part that outlives this pilot
Watch where this goes next. Regulators are going to want exactly this kind of access to AI usage data, and whatever shape gets standardised here becomes the template for every AI audit that follows.
If that template says the auditor never reads the primary evidence, then “independent audit” comes to mean “independent question, lab’s answer”. Anthropic went further than anyone else has, so their pilot sets the default.
I would still take this over nothing, easily. The direction is spot on.
If I were a researcher I would fill in the form. What I would not do is treat these percentages the way I treat a log line I pulled myself.
Before you repeat a number about AI usage data, find out who read the primary records. A model doing that reading leaves you with a hypothesis. Cite it like one.
Source | https://www.anthropic.com/research/enabling-independent-research
