# Apple Foundation Models on-device: measured capabilities and limits | PTLLM > We scored Apple's on-device Foundation Models framework on real agentic gates. Measured verdict: a free, private floor for lightweight turns — it cannot… Canonical: https://posttrainllm.com/apple-foundation-models We scored Apple's on-device Foundation Models framework on real agentic gates. Measured verdict: a free, private floor for lightweight turns — it cannot ground tool actions and its 4096-token context cannot hold a real tool catalog. Apple's FoundationModels framework (macOS/iOS 26+) exposes an on-device ~3B model through `LanguageModelSession` with structured output (`@Generable`), guided generation (`DynamicGenerationSchema`), and a `Tool` protocol. We scored it on our own agentic gates to answer a concrete question: can it ground actions well enough to serve as an agent's brain? The measured verdict: **a free, private, battery-cheap floor for lightweight turns — not an agentic action model.** ## The measured results | Gate | Result | Why | | --- | --- | --- | | BFCL agentic breadth, full catalog (n=8 VehicleControl) | 25% | Schemas present, but the ~3–4.4k-token catalog nearly overflows context | | BFCL agentic breadth, compact catalog (52 tasks) | ~0% | Fits context, but stripped parameter schemas produce wrong arguments | | Planner gate (action-grounding) | 13% | Picks the right tool name, cannot ground the arguments | | Planner gate (out-of-scope refusal) | ~95% | Judgment-light classification is its genuine strength | Three findings explain the numbers: - **It cannot ground actions.** It selects the right tool name under enum constraints, then fills arguments wrong. In one probe the gold call was `lockDoors(unlock=True, door=['driver','passenger','rear_left','rear_right'])`; the model emitted `lockDoors(unlock=false, door="all")` — inverted the boolean and guessed a string for an enum list. Some gold arguments are unguessable without the schema. - **The context catch-22.** The full catalog carries schemas but overflows the 4096-token context; the compact catalog fits but drops the schemas. Either way, the on-device context cannot host a real agentic tool catalog. - **It is not faster.** Roughly 3–4s per step — the same ballpark as a 4B MLX model with about 8× the context. Its real advantages are ANE perf-per-watt and zero setup, RAM, or cost. ## How we measured it The on-device model does not speak the OpenAI function-calling API our harness uses, so we built a standalone Swift HTTP bridge (`fm_agent_bridge.swift`) that exposes it behind an OpenAI-compatible chat-completions endpoint. Tool calling runs through guided generation: each request builds a `DynamicGenerationSchema` of `{tool_calls:[{name, arguments_json}], message}` and reads the generated JSON back out. With that bridge in place, the same BFCL multi-turn scorer runs unchanged — the on-device model is just another backend. ## Where it fits — and where it does not Use it as the free floor tier in a routing setup: lightweight, private, offline, battery-sensitive turns such as classification, refusal, and short answers stay on-device; grounded agentic work escalates to a distilled specialist; frontier jobs escalate further. Apple commoditized the plumbing — on-device LLM, tools, and structured output as an OS API — not the model capability your product is differentiated on. Two paths we evaluated and declined: adapter-tuning Apple's model (`.fmadapter`) locks the product to Apple's model, format, and OS; Core ML–compiling your own weights to the ANE remains a possible future battery optimization, not a capability bet. ## Related - [Evaluate a local LLM honestly](https://posttrainllm.com/evaluate-local-llm) - [The eval leaderboard](https://posttrainllm.com/eval-leaderboard) - [Build a routed specialist](https://posttrainllm.com/build-small-language-model-specialist) - [Fine-tune an LLM on a Mac](https://posttrainllm.com/fine-tune-llm-on-mac) **CTA:** Read the full measurement notes in the repository's `docs/learn/apple-on-device-foundation-models.md`, including the bridge source and per-gate tables. ## Continue through the factory - [evaluate local llm](https://posttrainllm.com/evaluate-local-llm) - [eval leaderboard](https://posttrainllm.com/eval-leaderboard) - [fine tune llm on mac](https://posttrainllm.com/fine-tune-llm-on-mac) - [build small language model specialist](https://posttrainllm.com/build-small-language-model-specialist) - [experiments](https://posttrainllm.com/experiments) - [artifacts](https://posttrainllm.com/artifacts)