The Routing Decision You Cannot Inspect

On-device SLMs have not displaced cloud inference in 2026. They have created a hybrid routing layer whose quality thresholds are set by Apple and Google, never published, and impossible for practitioners to audit.

The routing decision you cannot inspect

At WWDC in June 2026, Craig Federighi described Apple's system orchestrator as "key to the privacy architecture of our entire system." The orchestrator makes a binary call on every query: local or cloud. That framing positions the routing decision as a privacy mechanism, which is accurate as far as it goes. What it does not position it as is a quality mechanism, and that omission is worth examining closely.

The criteria for the routing call are not published. There is no named eval harness, no held-out benchmark, no quality floor expressed as a number on a specific task distribution. Google's LiteRT-LM, launched at I/O in May 2026 as production infrastructure for Gemma 4 on mobile and edge, describes itself as "production-proven" and cites a 2.2x speedup from Multi-Token Prediction. That is a latency claim. It names a mechanism. It is measurable. There is no equivalent claim about where the quality threshold sits or what happens when a query lands near the boundary.

For practitioners building on top of these systems, that asymmetry matters. You can verify the speedup. You cannot verify whether the routing decision is calibrated to user-facing task quality, to infrastructure cost, to liability exposure, or to something cheaper to measure like perplexity on a clean distribution.

Cloud inference grew 60% quarter-over-quarter while on-device shipped demos

Before addressing what the routing threshold should reflect, it is worth being precise about what has actually been displaced. The answer is: not much, not yet.

Google processed 16 billion tokens per minute in Q1 2026, up 60% quarter-over-quarter. Google Cloud revenue grew 63% in the same quarter. Azure grew 40%, AWS 28%, all three primarily driven by cloud inference demand. OpenAI CFO Sarah Fryer characterized the environment as "a vertical wall of demand with compute being the bottleneck" in May 2026. The five largest US cloud and AI infrastructure providers collectively committed $660–690 billion in capital expenditures for 2026, nearly doubling 2025 levels. These are not the numbers of an industry watching its core workload migrate to endpoints.

IDC's March 2026 forecast projects that 50% of enterprise AI inference will run on endpoints by 2030. That is a four-year projection, not a current state. Intel's IEEE ICCE 2026 paper on hybrid LLM inference is more direct: "the cloud remains essential for large-scale and highly complex workloads." A June 2026 piece from Callstack asked the more useful question — why aren't local LLMs production-ready on mobile yet, despite real improvements from Llama and Mistral? — and found the answer is still mostly hardware and reliability constraints rather than model quality alone.

What has emerged is not displacement but structural bifurcation. On-device handles latency-sensitive, privacy-critical, and context-light queries. Cloud handles multi-turn reasoning, agentic pipelines, and anything requiring a context window larger than what fits in mobile DRAM. The split is real. "Hybrid" is accurate. "Displaced" is not.

What the hardware gap actually explains, and what it doesn't

The strongest argument for conservative cloud routing thresholds is architectural, not financial. Vikas Chandra and Raghuraman Krishnamoorthi's January 2026 analysis put the memory bandwidth gap between mobile devices (50–90 GB/s) and data center GPUs (2–3 TB/s) at 30–50x. Their conclusion: "For LLM inference, this gap is decisive because decode is memory-bound." This is not a policy claim. It is a statement about silicon, and it is correct.

LLM decode throughput scales with memory bandwidth. A model that generates 150–260 tokens per second on an RTX 5090 with 32GB VRAM, confirmed in April 2026 for a 7B Q4 model, will generate substantially fewer tokens on mobile hardware running at a fraction of that bandwidth. The gap does not disappear with better quantization; it shrinks. Routing a complex, multi-step query to a mobile model with one-thirtieth the memory bandwidth of a data center GPU is not vendor conservatism. It is physics.

This counterargument deserves honest engagement. If the bandwidth gap is that large, what critics might describe as routing to cloud beyond what task difficulty requires becomes genuinely hard to distinguish from "the routing is correct and we just don't like the hardware ceiling." Demanding auditable thresholds could simply be asking Apple and Google to publish an embarrassing quality ceiling rather than exposing any other motive.

That argument is strong. It is not complete. The bandwidth gap explains why a threshold exists. It does not explain where the threshold is set, or whether its location is calibrated to user-facing task quality rather than to something else. A 30–50x bandwidth disadvantage means on-device models will underperform on decode throughput. It does not specify accuracy on a coding task, factual recall rate on a document Q&A benchmark, or task completion rate for a scheduling request — the metrics practitioners actually care about.

The evidence from Apple in May 2026 complicates the "the routing is just correct" reading further. Apple acknowledged that Siri improvements announced in 2024 would take "longer than we thought," a public admission that its on-device floor had not met the standard its routing policy implicitly assumed. Meanwhile, Apple held meetings with startup PrismML, which has reportedly shrunk Alibaba's Qwen 3.6, a 27B-parameter model substantially larger than Apple's existing on-device models, to run entirely on an iPhone 17 Pro. If the hardware ceiling were simply fixed, neither development would be particularly interesting. The ceiling is moving, and moving faster than Apple's public quality claims anticipated. That suggests the threshold's current location is not purely a function of hardware constraints.

The threshold's location is the claim that needs an eval, and none exists

The bandwidth gap argument, fully accepted, establishes that some queries should go to the cloud. It says nothing about where exactly the boundary sits, or whether it is drawn where quality degrades or where something else gets optimized.

Apple's orchestrator is described explicitly as privacy architecture. That framing implies the routing variable is data sensitivity, not task difficulty. The two are not equivalent. A highly sensitive but simple query, "remind me to take my medication," probably runs fine on-device. A non-sensitive but complex query, "summarize and compare these three research papers I just attached," probably does not. If the routing weights privacy framing over quality estimation, the threshold's location reflects a different objective than the one practitioners assume when they accept cloud fallback as a quality guarantee.

Google's LiteRT-LM documentation, as of its May 2026 launch, provides latency numbers and a speedup claim. No named eval harness appears in the public materials. No held-out benchmark is cited. "Production-proven" is a phrase, not a measurement. The Gartner prediction from April 2025 that organizations will use SLMs three times more than general-purpose LLMs by 2027 is a forecast about adoption, not a quality audit of routing decisions. None of these constitute evidence that the threshold is calibrated to the metric practitioners need it calibrated to.

The May 2026 Siri admission sharpens this. Apple acknowledged, publicly, that its on-device quality ceiling was lower than its routing policy assumed. That is not a hardware statement. That is a calibration statement. The routing had been sending queries to on-device inference that on-device inference was not handling well. Users experienced the gap. Apple acknowledged it. The mechanism that was supposed to prevent this, the orchestrator Federighi called "key to the privacy architecture of our entire system," did not prevent it.

The bandwidth argument cannot close this gap. A threshold can be architecturally motivated and still be miscalibrated. The distance between "a physical constraint exists" and "this specific threshold correctly reflects that constraint" is exactly the distance a published eval would need to cover.

What a falsifiable routing claim would look like

The Intel IEEE ICCE 2026 paper describes its "on-device-first" routing logic in mechanistic terms: what the device handles, what triggers cloud escalation, what the fallback condition is. That framing, a specification rather than a promise, is the minimum a practitioner needs to reason about whether a hybrid system is behaving correctly.

Here is the test I would actually run. Take a task set spanning three difficulty tiers: single-turn factual recall, single-turn instruction following, and multi-step document reasoning. Run each query through the orchestrator and log which tier gets routed where. Then run the same queries directly against the on-device model and the cloud model separately, scoring on task completion rate. If the orchestrator's escalation boundary tracks quality degradation on the on-device path, the routing is doing what it claims. If the boundary sits well above the quality cliff, something else is being optimized. That experiment is reproducible, the metric is interpretable, and the result would be falsifiable. Neither Apple nor Google has published anything in this form as of July 2026.

LiteRT-LM's Multi-Token Prediction speedup is a useful reference for what transparency looks like in the latency domain: a named mechanism, a measured number, a reproducible benchmark. The quality threshold needs the same structure. Not a promise that on-device is "good enough." A named eval, a specific task distribution, a score, and the condition under which the orchestrator escalates. That specification would let practitioners decide whether the task distribution is the right one, whether the cutoff is appropriate, and whether the metric correlates with the quality their users actually experience.

Neither vendor has published this as of July 2026. The orchestrator routes. The criteria are private. The quality floor is implicit. The May 2026 Siri admission is evidence that the gap between implicit floor and user-experienced quality is not merely theoretical. Until the eval exists, practitioners cannot distinguish a threshold calibrated to experience from one calibrated to something cheaper to measure.