Skip to content
posttrainllm docs
Esc
↑↓navigate↵open⌘Jpreview
On this page

vllm-mlx Investigation

vllm-mlx Investigation

Date: 2026-06-06

Recommendation: partial-proceed. Do not wrap vllm-mlx into posttrainllm serve yet. Keep native serve as the default and as the eval/logprob backend. Treat vllm-mlx as a production-chat candidate after an approved local install and benchmark pass.

Sources Checked

1. Maturity

The project is real and active enough to watch:

  • Public Apache-2.0 repo, Python package, Apple Silicon target.
  • GitHub API on 2026-06-06 showed 1,302 stars, 182 forks, 67 open issues/PRs, and latest main HEAD 015e080ff2c2ab95fb4368c736a12f58bcde63a9.
  • Latest pushed commit was 2026-05-31.
  • Repo docs advertise OpenAI and Anthropic APIs, continuous batching, paged KV cache, prefix cache, model acquisition/conversion, and bench-serve.

But it is not mature enough to become the default posttrainllm serving backend:

  • pyproject.toml marks it as Development Status :: 3 - Alpha.
  • Recent open issues include continuous-batching crashes and MLX stream failures. Examples checked from the open issue feed:
    • #592 reports NoneType iteration crashes under continuous batching.
    • #593 reports /v1/chat/completions returning 500 because generation runs on a thread without the expected MLX stream context.
  • It is moving quickly, which is good for upstream velocity but bad for a stable wrapper contract today.

Decision: do not default to it. If used, keep it opt-in and isolated from eval infrastructure.

2. Model Formats

vllm-mlx expects an MLX/HuggingFace-style model identifier or local model directory:

  • vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit
  • vllm-mlx serve /path/to/local/model
  • vllm-mlx model convert <hf-model> --output <dir> ...

It does not load .tinygpt checkpoints directly.

posttrainllm already has posttrainllm to-safetensors, but that currently writes a single model.safetensors with HF-Llama-ish names. That is not yet a complete MLX/HF model directory contract for vllm-mlx because serving needs compatible config.json, tokenizer files, and architecture expectations. posttrainllm’s from-scratch byte-level models also do not map cleanly to an upstream HF model family without a wrapper config.

Decision: a direct .tinygpt -> vllm-mlx adapter is not a small flag. The minimum honest bridge is:

  1. Export weights.
  2. Emit a compatible model directory (config.json, tokenizer, generation config, model.safetensors).
  3. Verify vllm-mlx serve <dir> loads it.

Until that exists, vllm-mlx can serve external MLX/HF models, not arbitrary posttrainllm checkpoints.

3. OpenAI Surface Parity

vllm-mlx documents:

  • POST /v1/chat/completions
  • POST /v1/completions
  • GET /v1/models
  • streaming chat/completion responses
  • Anthropic /v1/messages
  • embeddings, health, metrics, status

That is good enough for production chat clients.

The critical eval question is logprobs + echo for lm-eval-harness/local-completions. This remains unproven. The server docs and CLI reference checked today document chat/completions and streaming, but do not document the exact completions echo: true plus logprobs behavior that posttrainllm’s native Serve.scoreLogprobs supports for eval scoring.

Decision: keep native serve as the eval backend. Even if vllm-mlx is adopted for production chat, eval code should continue to route through native serve until logprobs + echo is curl-verified.

4. Performance Reality Check

No local benchmark was run in this pass. Installing vllm-mlx and benchmarking it against a checkpoint would require Python package installation, model download/conversion, and sustained inference loops. The repo’s macOS safety rules require explicit approval before that class of workload.

Upstream benchmark data is still useful for triage:

  • M4 Max results claim 402.3 tok/s for Qwen3-0.6B-8bit and 463.6 tok/s for Llama-3.2-1B-Instruct-4bit.
  • Continuous batching docs claim 1.5x to 3.39x throughput improvement for five concurrent requests depending on model.
  • Paged-cache docs show workload-dependent gains: meaningful memory savings, modest or no throughput gain in some M1 Max paged-cache tests.

Decision: performance upside is plausible, but not proven for posttrainllm. The next step is an approved local benchmark, not code integration.

Recommendation

Partial-proceed:

  1. Do not modify posttrainllm serve yet.
  2. Keep native serve as the default and as the eval backend.
  3. Add a future integration only after a local benchmark answers:
    • Can vllm-mlx load a posttrainllm-exported model directory?
    • Does /v1/completions support echo + logprobs well enough for lm-eval-harness?
    • What are throughput, startup time, and memory deltas on the same Mac and same prompts?
  4. If the local run passes, implement --backend native|vllm-mlx as an explicit opt-in. Do not switch defaults until stability data exists.

Proposed Local Benchmark Plan

Run only with explicit approval:

uv tool install vllm-mlx
vllm-mlx --help

Then use a small external MLX model first:

vllm-mlx serve mlx-community/Qwen3-0.6B-8bit --port 8001 --continuous-batching
vllm-mlx bench-serve --url http://127.0.0.1:8001 --prompts short --concurrency 1,4 --max-tokens 64 --format json

Compare with native serve on an existing posttrainllm checkpoint using the same prompt set and max-token cap. Stop all spawned servers immediately after the run.

Was this page helpful?