Protect the Decision, Not the Data

AI is learning how organizations think. Who owns what it learns?

The next enterprise AI governance challenge isn't just controlling who can access your data. It's controlling who gets to learn from your decisions.

1. Data was the asset. Judgment is becoming the moat.

For decades, companies competed on what they knew. Increasingly, they will compete on how they decide. AI makes both learnable, but only one has historically been treated as a protected asset.

Organizations spent twenty years building infrastructure for information: data warehouses, data lakes, analytics platforms, customer data platforms and governance programs. All of it collects, organizes and protects data exhaust, the record of what happened.

AI-assisted work produces a second stream: decision exhaust, the record of how the organization chose.

• A developer accepts one implementation and rejects another.
• A product manager prioritizes one feature over five alternatives.
• An architect rejects a technically valid design because it violates an internal principle.
• A sales leader overrides a forecast.
• An executive passes on an otherwise attractive opportunity.

Each of these encodes how the organization weighs alternatives. Historically, such choices were scattered across meetings, documents and hallway conversations, and mostly lost. AI agents now make more of them observable, structured and reusable.

The distinction is worth stating carefully. Data tells a model what the world looks like. Decision traces can teach it how an organization responds to that world. Whoever learns from your decisions may gain access to something your data alone cannot reveal: the patterns behind your judgment.

2. AI makes judgment learnable

Models learn many things at once: facts, representations, correlations, procedures, preferences and behavioral patterns. The relevant point is that the last two are learnable too. When a training example is a record of someone working, predicting the next step means predicting a choice.

Three mechanisms carry decisions into models:

• Fine-tuning on traces. Records of experts working, such as accepted code, edited drafts and resolved tickets, show a model what a good next move looks like.
• Reinforcement from feedback. Thumbs-ups, accepted suggestions, rejected alternatives and merged pull requests can become preference signals. When collected, curated and incorporated into training, they help optimize a model toward those preferences.
• Distillation. A model is trained on another model's outputs, including any written-out reasoning, and approximates its behavior without seeing its training data.

Two cautions keep this honest. First, training on decisions approximates aspects of judgment; it does not automatically reproduce the underlying expertise, context or reasoning. Second, fluency is not infallibility. A model trained deeply on a workflow is more reliable inside it, but it still makes errors, and can make confident ones at the edges.

3. AI labs already understand the value of learned behavior

The labs offer the clearest evidence that behavior is trainable and worth guarding. Each example proves something specific, and none proves everything.

Canvas: a decision policy can be trained, and beats prompting. OpenAI post-trained GPT-4o for Canvas on synthetic data distilled from o1-preview (OpenAI). The trained behaviors were decisions: when to open the canvas, when to make a targeted edit versus a rewrite, when to comment. OpenAI reported 83% and 94% accuracy on the open-canvas decision for writing and coding, and a 30% accuracy gain on comments over a carefully prompted GPT-4o. This shows that a bounded behavioral policy can be learned more reliably than it can be instructed. It does not show that strategic judgment, with its context and conflicting goals, can be captured the same way.

Distillation: capability transfers through outputs. In February 2026, Anthropic reported that three labs ran over 16 million exchanges with Claude through about 24,000 fraudulent accounts, targeting reasoning, agentic coding and tool use (Anthropic). In October, it said its defenses now focus almost entirely on making Claude's written-out reasoning harder to access, and estimated they cut the data distillers harvest by roughly 90 to 99% (Anthropic). The same post shows why reasoning is the prize. Anthropic trained a pretrained Claude model on Claude Opus 4.8 transcripts: with full reasoning it scored 80.4% on SWE-bench Verified, with summarized reasoning 71.8%. Same base model and benchmark; only the visibility of the reasoning changed. Two qualifications apply. Distillation shows capability moves through outputs, not that an enterprise's judgment can be extracted from ordinary workflow records as effectively. And written-out reasoning is an explanation the model produces, not necessarily the internal process by which it reached an answer.

The lab feedback loop: a hypothesis, not a finding. Labs use their own models intensively. Claude Code's product lead has said about 90% of Anthropic's code is now written with it (Let's Data Science, summarizing O'Reilly Radar), and each lab's model often feels most at home in the kinds of products that lab builds. The interesting question is whether that closeness creates a compounding advantage, with internal decisions becoming training signal. The environment for such a loop clearly exists. How much internal work actually flows into training is not publicly established, and deliberate training investment, benchmarks, reinforcement learning and target customers could explain the same strengths.

4. The enterprise blind spot

Every accepted suggestion, rejected alternative and corrected AI response is a small lesson in how your organization thinks. Individually, these signals look insignificant. Collectively, they may become a transferable approximation of institutional judgment.

Enterprise AI governance has traditionally focused on data access, privacy and security. Far less attention has gone to the decision traces generated through human-AI interactions. That gap creates two distinct risks, and the defenses differ.

Risk	What happens	Where it occurs	Primary defenses
Learning risk	A vendor or third party uses your decision traces to train or improve models that serve others	Vendor training rights, feedback and telemetry, observability tools, connected platforms	Contract terms, training opt-outs, data residency, control over feedback and eval data
Access risk	An AI system exposes or acts on proprietary judgment at runtime, with no training involved	Agents reading chats, strategy docs, negotiations and approvals; retrieval; tool calls	Least-privilege agent permissions, retrieval access controls, prompt-injection defenses, audit logs

Training restrictions alone do not solve the second risk. An agent with broad access to product strategy, architecture debates and customer negotiations can surface that judgment to the wrong person, or be manipulated into leaking it, without its weights ever changing.

In an October 10, 2026 essay, Microsoft CEO Satya Nadella argued that enterprises should treat frontier models as insider risks, and that we need to "separate the supply of intelligence from the authority over it" (Redmond). There is a complementary boundary worth protecting: learning. The observability, audit trails and human feedback needed to make AI accountable also create rich records of organizational judgment. The mechanisms built to make AI trustworthy can become valuable repositories of institutional intelligence, and they need protection of their own. Controlling what intelligence can do is one challenge. Controlling who benefits from what it learns is another.

The platform layer matters too. In 2025, Salesforce changed Slack's API terms to bar bulk export, persistent indexes and LLM use of data obtained through the API, and Glean warned customers this would limit their use of their own Slack data in other AI tools (Hunton). Salesforce's motives may include platform economics, privacy, security or competition. Whatever the reason, the episode shows that control over where enterprise conversations accumulate has become strategically important.

This leads to the question most enterprises have not asked: who owns the intelligence created when employees continuously teach AI systems how the organization makes decisions? The organization, the employee, the model provider and the platform capturing the interactions all have a plausible claim. In practice, contracts and technical access will decide, and they may not agree.

5. Protect the learning advantage

The goal is not to stop AI from learning how your organization works. That learning is what makes AI useful. The goal is to keep the benefit of that learning inside the organization. Three questions matter more than whether you own a model: who controls the learning process, who can access the traces, and who can reuse the resulting behavior.

1. Map your decision exhaust. Inventory where judgment is recorded: code reviews, approval chains, chat threads, CRM overrides, prompt libraries, agent configurations and AI feedback logs. Treat these as sensitive assets alongside customer data.
2. Control where organizational judgment accumulates. Keep decision traces, feedback signals, evaluations and adaptations (retrieval indexes, fine-tunes, system prompts) in a clearly governed environment. You can use a third-party foundation model and still keep all of this under your control. Owning a fine-tuned model, by contrast, protects little if traces leak through observability tools or connected apps.
3. Set vendor terms on reuse. Ensure vendors cannot use your prompts, outputs, feedback or telemetry to improve models serving competitors without explicit authorization. Review terms on renewal, since platforms can change them unilaterally.
4. Govern agent access as tightly as training. Give agents least-privilege permissions, scope retrieval to the user's own entitlements, log what agents read and do, and test for prompt injection.
5. Keep institutional knowledge portable. Store decision history, evals and adaptations in forms you can move between models, so your learned advantage is not locked to one vendor.
6. Keep accountable people on defining decisions. Use AI broadly for execution, but keep humans on the calls that express strategy. This preserves the tacit judgment models cannot yet capture.

Limits of the argument

• Data still matters. Decisions are made on data, and privacy law and customer trust still require protecting it. The claim is "not only data," not "data no longer matters."
• Bounded behavior is not strategy. The strongest evidence involves narrow policies like when to open an editor. Whether rich, contextual judgment transfers as well is still an open empirical question.
• Judgment decays. Decisions are tied to a market and a moment. A model trained on last year's choices may apply them confidently to a world that has moved on, which limits how long a copied policy stays valuable.
• Tacit judgment resists capture. Much expertise is never written down. That limits the risk, and points to where human advantage will persist.
• Copied judgment copies blind spots. Synthetic and distilled training can carry a teacher's errors and biases forward (TechCrunch).

Conclusion

Organizations must start treating decision-making as an intellectual asset, and that creates a tension. Companies want AI systems to learn how their best people work, because that is how AI becomes genuinely useful. But the better those systems learn, the more valuable, and more transferable, the resulting decision traces become.

The challenge is not preventing AI from learning your organization's judgment. It is ensuring your organization captures the benefit of that learning without unintentionally handing its competitive advantage to a vendor, a platform or a rival.

The next enterprise AI governance challenge isn't just controlling who can access your data. It's controlling who gets to learn from your decisions.

Sources

• Introducing canvas, OpenAI
• Detecting and preventing distillation attacks, Anthropic (Feb 2026)
• Strengthening our defenses against distillation attacks, Anthropic (Oct 2026)
• Microsoft expands Project Perception, citing Nadella's insider-risk essay, Redmond (Oct 2026)
• Salesforce locks down Slack data, Hunton
• Anthropic's Claude Code accelerates developer workflows, Let's Data Science
• Tech giants embrace synthetic data, TechCrunch