We got extraordinarily lucky with pretraining data. Training large language models required a truly vast corpus of text, and it was just kind of lying around.
Not exactly by accident. For the last 50 or so years, humanity has been busily creating and digitizing an enormous public record of human knowledge. Most obviously, Wikipedia’s editors have assembled the most comprehensive encyclopedia humanity has ever produced. A vigorous open source movement created a couple petabytes of software source code. Communities on Stack Exchange spent more than a decade asking questions, arguing about answers, and carefully curating the results. Scientists uploaded their papers to ArXiv. Project Gutenberg—almost as old as the internet itself—produced digital editions of out-of-copyright books.
And of course, billions of people made websites, commented, had arguments both petty and profound, uploaded photos and paintings, wrote fiction, and—believe it or not—sometimes went and told lies on the internet.
Efforts like Gutenberg, Internet Archive, Wikipedia, ArXiv, and Stack Exchange were self-conscious efforts at creating a digital knowledge commons. And then there’s Common Crawl. Since 2007, Common Crawl has been regularly archiving snapshots of a sizable subset of the internet, and then making it freely available for anyone to access and use. Today, there’s something like 10 petabytes of data archived by Common Crawl.
This vast commons of human knowledge formed the basis for the LLM revolution. GPT-3 was trained on about 60% Common Crawl-derived data. The original Llama data mix was almost entirely extracted from the internet commons.1
That’s not to say that refining the raw material of the web into LLM-ready chunks was simple. Web pages are extraordinarily messy creatures, and a large fraction of the web is spam or junk intended to fool Google’s crawler. People are still writing papers on how to do filtering, HTML-to-text extraction, old document OCR, and synthetic rewriting.
Though the incumbent large players have seemingly moved on with their own custom web crawlers (Cloudflare reports that over half of crawler traffic is for AI), modern open LLMs like Nemotron and our own still rely on this commons for the vast majority of their pretraining data. Without continued access to this commons, truly open LLMs cannot exist.
These days, the focus of frontier AI research has shifted to posttraining, particularly so-called “agentic” AI. Like pretraining before it, posttraining requires vast amounts of compute and vast amounts of data. However, there isn’t the same ready-made trove of raw data lying around for posttraining. In short, there is no posttraining data commons.
The fundamental difference is simple: Pretraining consumes documents, while posttraining consumes tasks.2 But “task” here is already an abstraction. The tasks used by reinforcement learning for posttraining are not the same tasks that humans perform in the world.
Human tasks are situated in a particular moment. Someone wants something; they act in the world to get it; the world responds; the goal changes as they learn; and, eventually, they succeed, or fail, or change their goals. Once attempted, the task is gone. The same conditions will never quite occur again.
In this sense, human tasks echo oral culture: An oral story is as much an event as it is an object. The telling of a story changes with the teller, the audience, and the occasion. Writing the story down makes it durable and repeatable, but it freezes a single telling.
Encoding a human activity as an RL task does something similar. RL tasks are frozen, schematized, and legible, while human-world tasks are adaptive, conditional, and ephemeral. An episode of situated action is transmuted into an object that can be performed again and again.
For an RL task, we have to preserve at least three components: a world, an intent, and a set of tools for interacting with the world. For reinforcement learning, we also need a grader that looks at the results of the agent’s actions and determines how well the intent was satisfied.
Humans have spent millennia building institutions around durable knowledge stored in the form of documents: books, libraries, archives, the internet. We have had much less time, developed much less infrastructure, for making action durable.
Except for one very important exception: software. Software engineering is, in large part, the practice of writing tasks down. Software engineers have spent decades turning work into durable artifacts: preserving the world in which the work happens, recording the intent behind it, and leaving a trace of the actions taken. It is no wonder that agentic AI came for software first.
Computers are built around determinism. Restart a computer, and its state is more or less a clean slate, aside from any information—any documents—you've chosen to write down. (Mom, this is why I keep telling you to reboot your computer.) This determinism is perfect for reinforcement learning. Reinforcement learning algorithms work best when an agent can attempt a task, have the changes discarded, and try again from the exact same starting point.
Over the last few decades, software has undergone a series of transformations to double down on the determinism of computers. Version control preserves the history of programs. Continuous integration and formalized unit testing libraries record procedures for exercising them. Virtual machines and containers encapsulate a controlled snapshot of the world a program needs to function in. None of this is perfect. Software too flows: system clocks move, versions change, packages disappear. Bits rot. Debian has been working systematically on reproducible builds since 2013, and more than a decade later still does not describe the distribution as fully reproducible.
Despite these limitations, open-source software has created a commons of preserved worlds and recorded actions. But turning the software commons into tasks that can be consumed by machine learning is not easy. And it’s hard in a way that is qualitatively different from cleaning web text: the missing information may never have been recorded at all. Turning software repositories into posttraining data therefore looks less like curation and more like manufacturing.
To build a task, we need the world, tools, intent, and a grader. In theory, the open source software commons gives us all of these:
- The world consists of the software repository at a point in time, its dependencies and the container and build machinery.
- The tools are just the tools of software engineering and computer use: the shell, compiler, browser, etc.
- The intent we can recover from GitHub issues or pull requests.
- The grader is whether the tests pass.
It turns out that this is basically the framing used by the original SWE-bench. And it turns out to be both surprisingly difficult and much too simple. They crawled GitHub issues and pull requests from 12 Python repositories, yielding a bit over 90,000 PRs. Of these, only about 1 in 8 could be readily mapped to the frame: they either didn’t get merged, or weren’t linked to an issue, or didn’t change a test. They lacked a clear mapping for intent or the grader.
Of the remaining 11,407, about 80% couldn’t be turned into executable tasks. For example, the package could not be reconstructed at that state, or there were weird runtime errors. These could not have their worlds reconstructed or their graders failed. This left the 2,294 tasks that comprised the original SWE-bench.
A year later, OpenAI set about creating SWE-bench Verified. OpenAI hired 93 experienced Python developers to manually review 1,699 randomly selected SWE-bench tasks. Of these, more than two thirds were flagged for removal due to underspecified prompts, unfair tests, or other major issues. After filtering, they produced a pool of 500 “Verified” tasks. Then, a later audit of a problematic subset of the tasks found that another sixth or so of the overall 500 were also broken, despite the extensive set of checks leading to that point.
I’m not trying to pick on SWE-bench. It’s one of the better constructed public coding datasets available. When examining the compilation of tasks in TaskTrove, we found that in one subset of 5,000 converted Java tasks, more than 90% had to be discarded during repair: tests did not parse, checked nothing, or depended on missing project context and external services. In another Python subset, almost 98% of the inspected verifier bundles could not even start their tests because dependencies were missing. And executability is only the beginning. An audit by Vals found that two thirds of the 2,698 coding environments in Xiaomi’s MiMo v2.6 public release still contained reference fixes recoverable from Git. In one demonstrated case, the agent copied the fix and passed all five hidden tests. Making good tasks is hard.
And again, software is the easy case! And if even software is so difficult to get right—I didn’t even talk about building sandboxes that your agents can’t hack their way out of, and tasks that don’t have a predefined unit test for a grader—then how difficult must the rest of the economy be? How can the frontier AI companies reliably encode the rest of so-called “economically valuable human activity,” the work done by lawyers, scientists, salespeople, doctors, etc., as the kind of formal repeatable tasks that RL needs?
For the rest of the economy, there is not even the beginning of a public “task commons.” So, what’s a frontier lab to do? Pay those lawyers, scientists, salespeople, and doctors to fit their work into the RL task frame, to inscribe the ephemeral tasks of their human world into the ones consumed by agents. They create the tasks, environments, rubrics, verifiers, simulators, etc. All of this work itself must be coordinated and verified. And thus we have the raison d’être of the rapidly expanding posttraining data industry.
So, what are we, the open models community, to do? We don’t have a Common Crawl for posttraining because there is no commons to crawl. Instead, a posttraining commons must begin with large-scale and systematic inscription, with finding and transmuting human tasks into RL tasks. (Or we figure out a better paradigm. That would be better.)
This is not an easy lift. A large fraction of the pretraining commons was already being produced in roughly the right form. People wanted to publish webpages, source code, papers, and books for reasons that had nothing to do with training language models. But the human activity that produced those artifacts—the goals, decisions, revisions, negotiations and false starts—is mostly gone.
Luckily, we aren’t starting from zero. The open source commons is a treasure trove of partially inscribed tasks. Other parts of the economy, other forms of human activity, leave traces too: documents with tracked changes, logs, procedures. And, increasingly, agents are capable of helping us reconstruct tasks from outputs and traces, and to inscribe them into RL tasks.
In one reading, this all reduces to “we just need to make a whole lot of RL environments” as quickly as possible. And at a surface level that is true. But the deeper task—the human task—is deciding which parts of human activity we want to inscribe into a durable form legible to reinforcement learning, and then building the machinery and institutions to do so systematically. The approach taken in the construction of GDPval points at one way of achieving that aim.
At this point, the automation of legible tasks in the economy seems inevitable, if only through the sheer weight of capital being poured in. The question is whether the resulting inscriptions of human work become private assets held by a few companies, or whether we do the work to make some meaningful fraction a commons that everyone can build on.
As AI becomes capable of performing more and more human work, the ability to develop and direct that capability becomes an enormous source of economic power. If only a handful of companies can amass the tasks and environments necessary to train frontier models, then those companies will have an outsized say in what kinds of work AI can do, and in who benefits from that work.
A posttraining commons would let universities, nonprofits, independent researchers, and companies develop capabilities that the large AI companies might never prioritize. A commons would give all of us the ability to participate meaningfully in shaping how AI is developed and deployed, and a stronger voice in the policies and norms that govern it. Otherwise, we risk ceding enormous power over perhaps the most consequential technology in generations to the commercial interests of a few enormously wealthy firms.
We got lucky with pretraining. The people who built Wikipedia and Stack Exchange weren’t making training data; they were making things they wanted to exist. But no one is going to inscribe the work of a tax attorney into a reproducible environment for fun. If there’s going to be a posttraining commons, someone has to build it deliberately, knowing what it’s for, and what it might cost.
That’s harder. But it also means that, for once, the commons is something we get to choose.
Cite this post
@misc{hall2026_there_is_no_posttraining_data_commons,
author = {Hall, David},
title = {There is no posttraining data commons},
year = {2026},
month = {oct},
howpublished = {\url{https://www.openathena.ai/blog/there-is-no-posttraining-data-commons/}},
note = {Open Athena Blog}
}