In April, GitHub flipped a switch. Individual Copilot users — Free, Pro, Pro+ — were opted into training data collection by default. The interaction data now feeding model training includes, per GitHub's own documentation: prompts, suggestions, and accept/reject signals.
That last item is the one that matters. Accept/reject signals are not just telemetry. They are training labels. When a developer accepts a suggestion, the model learns: this completion, in this context, was correct. When a developer rejects, the model learns: try something different. Microsoft confirmed that training on employee interaction data produced "meaningful improvements — increased acceptance rates in multiple languages."
This is a clean, well-understood training loop. There's just one problem.
The accept button was designed for a product that no longer exists.
What the Button Was Built For
When GitHub designed Copilot's data policy, the product was an inline suggestion engine. You typed, ghost text appeared, you pressed Tab or Escape. One suggestion, one evaluation, one label. The mapping was 1:1 — each accept/reject signal corresponded to a single, inspectable code completion that the developer had read and judged.
That product is no longer what most developers use.
A Microsoft Research paper published this month provides the first production-scale analysis of what Copilot actually looks like in practice: 3.2 million users, 13 million sessions, 761 million LLM calls, 95 trillion tokens. The core finding:
87% of LLM calls are agent-initiated. The average agentic session unfolds 6.6 LLM calls per user prompt.
The developer types one instruction. The agent autonomously plans, edits files, runs terminal commands, reads error output, iterates. Six or seven LLM calls happen in a chain before the developer sees the result and decides: accept or reject.
The label covers all of them.
The Mismatch
This is the structural finding: the April 24 data policy blog post makes zero mention of agent mode, autonomous tool calls, or agent-initiated actions. The policy was written for inline suggestions — binary accept/reject per completion. But the product it applies to now generates 87% of its LLM calls autonomously, in multi-step chains the developer never inspects.
GitHub's own documentation confirms that agent mode falls under the same data collection toggle. Same policy. Same labels. Different product.
In suggestion mode, "accept" meant: I read this code and it's correct. In agent mode, "accept" means: the outcome looks about right and I want to move on. The label hasn't changed. What it measures has.
Triple Duty
This connects to something I wrote last week. In "66.3%", I covered the Approval Game — a browser experiment where 40,000 people played as AI agent approvers. They caught two-thirds of malicious actions. One in three slipped through.
The accept button in agent mode now serves three functions simultaneously:
| Function | Requires Evaluation? | Works? |
|---|---|---|
| Security consent — authorizing tool calls, file writes, command execution | Yes — must evaluate what the agent is doing | No (66.3% accuracy) |
| Training label — accept/reject as RLHF signal for model improvement | Yes — must evaluate code quality | Degrades with agent ratio |
| UX confirmation — "apply these changes to my codebase" | No — just needs intent | Yes |
When a single action serves multiple functions, the function requiring the least effort wins. Security consent requires reading and understanding every tool call. Training labels require evaluating code quality. UX confirmation just requires intent — yes, apply this. In the time pressure of an agentic session averaging 6.6 LLM calls, the button becomes a confirmation click, not an evaluation.
The functions that need evaluation degrade. The functions that don't need evaluation — applying changes, collecting data — continue working perfectly.
Three Strategies, One Pipeline
GitHub's opt-in-by-default is one data acquisition strategy. It's not the only one.
Meta made the price visible. Muse Code launched August 5 with two tiers. Standard: $1.25 per million input tokens, $4.25 per million output. Contributor: $0.10 input, $0.20 output. Same model. Same capabilities. The contributor tier costs 92.8% less — in exchange for Meta using your prompts and completions to train its models. The rate limit drops from 3,000 to 60 requests per minute. And early reports indicate Muse Code defaults to the contributor tier after install.
Meta's approach is honest in a way GitHub's isn't. There's a price tag. You can see what your training data is worth: the difference between $4.25 and $0.20 per million output tokens. That gap — 21x — is the value of real-world coding data to a model trainer.
Microsoft closed the loop. MAI-Code-1-Flash, announced at Build on June 2, is a 5-billion-parameter model trained directly on Copilot's production workflows — not just code, but the agentic tooling, file editing patterns, terminal integrations, and multi-step task loops that developers use daily. Microsoft can do this because it now owns every layer: the IDE (VS Code), the agent (Copilot), the data pipeline (April 24 opt-in), and the model (Polaris). Data collected via Copilot trains MAI-Code-1, which powers Copilot. Closed feedback loop. GPT-4 Turbo has been demoted to a three-month fallback before removal.
The temporal sequence matters:
Establish the data pipeline. Then end the subsidy. The order isn't accidental. Flat-rate pricing subsidized agentic usage that generated training data. Once the pipeline was established, usage-based billing shifted the cost of token-heavy workflows back to the developer — while the data collection continued.
What the Label Doesn't See
There's a subtler problem. In suggestion mode, the developer evaluates a specific piece of code. The accept/reject signal has high information content — it tells the model whether this particular completion was right for this particular context.
In agent mode, the developer evaluates an outcome. The code might compile and pass tests, so they click accept. But any individual step in the 6.6-call chain might have been suboptimal — a workaround instead of a clean solution, an unnecessary file creation, a fragile approach that happens to work. The developer accepts the result, and the model receives a positive training signal for every intermediate step in the chain, including the ones nobody evaluated.
The MSR paper documents this architecture: agentic sessions consist of "sparse user-initiated turns, each unfolding into an autonomous agent loop of LLM calls coupled nearly 1:1 with tool execution." The user sees the beginning and the end. The model records every step as training data. The label applies to all of it.
This is a known problem in reinforcement learning — reward sparsity. When reward signals are sparse relative to the action sequence, the model can't distinguish which actions in the chain actually contributed to the positive outcome. In suggestion mode, reward was dense: one action, one signal. In agent mode, reward is sparse: one signal per 6.6 actions. The label quality degrades proportionally.
The Pattern
The accept button in 2021 did one thing: confirm a suggestion. By 2026, it does three things — security consent, training label, workflow confirmation — and the functions that require human evaluation are failing while the functions that don't are working fine.
This is how multi-function interfaces degrade. Not through any single failure, but through the quiet victory of the lowest-effort function. Security consent requires reading tool calls. Training labels require evaluating code quality. Workflow confirmation requires a click. In a 6.6-call chain, the click wins.
Three companies have built data acquisition strategies around this button. One made it opt-out. One priced the gap at 21x. One closed the loop entirely. All three depend on the same mechanism: a human pressing accept on work they haven't fully evaluated, generating a training label that says they did.