# Parked: the four "multi-model" directions

User decision (this session): hold these four options and focus on the
HF-compat capabilities (SwiGLU + RoPE + GQA + BPE tokenizer) first.
Once those land, the Mac app can load any modern open-weight model
and LoRA-fine-tune it. Then these four become more interesting
because we'd have real models to apply them to.

Each entry below is a brief — what it is, what it costs, what it buys,
and where the wiring would go.

## 1. Multi-modal (vision + text)

Add image input to the Mac app. The standard recipe (LLaVA-style):

  - Pre-trained vision encoder: CLIP-ViT-B/32 or SigLIP-B. Loaded
    once at startup (~150 MB int8).
  - Tiny projector: 2-layer MLP from vision encoder dim → LLM token
    embedding dim. Trained briefly on image-caption pairs.
  - At inference: image → encoder → projector → "vision tokens" →
    prepend to text prompt → LLM processes the joined sequence.

**Engineering**: ~1-2 focused weeks.
**Dependency**: image-caption training data (~100 MB of paired data
for the projector). LAION-400M's filtered subsets are accessible.
**Demo value**: high. "Train your model to describe images" reads as
flagship multi-modal capability.
**Quality bar**: passable. Real multi-modal models need instruction-
tuning on multi-modal chat data (LLaVA-Instruct, ShareGPT4V) — we'd
land "describes images" not "answers questions about images well."

## 2. Multiple models in one session

Two slightly different things:

a) Multiple bases simultaneously: load Llama-3-3B AND Phi-3-mini at
   once in the same process. 48 GB easily holds 3-4. Each is its
   own `TinyGPTModel` instance; the SwiftUI app shows a "Model A vs
   Model B" side-by-side view. ~1 day of UI work — purely additive.

b) Multiple LoRA adapters over one base: ALREADY SHIPPED. Tonight's
   `LoraStackInjection` composes any number of adapters over a single
   base. Use `--lora` multiple times on the sample CLI.

So the only un-shipped piece is (a)'s UI. Tag this as a v0.2 polish item.

## 3. Mixture of Experts (MoE)

Architecturally the most interesting of the four. Replace each MLP
with N parallel "expert" MLPs (typically 8) + a small "router" that
picks the top-2 experts per token based on a learned gating signal.

Why: a model with 8 experts × 7B params each = 56B parameters total,
but only 14B *active* per token (since each token picks 2 of 8).
Compute cost matches a 14B dense model; quality matches a 56B dense.
Mixtral 8x7B is the open-weight reference.

**Engineering**: ~3-5 focused days.
**Cost**: ~200 lines (router + top-k expert dispatch).
**Constraint**: training MoE requires a load-balancing auxiliary loss
to keep experts from collapsing onto the same routing. Adds a hyper-
parameter and complicates the training loop slightly.
**Demo value**: medium. The pitch ("47B params on a laptop") is real
but doesn't tweet as well as multi-modal.
**Combines with HF loading**: yes. Once we can load HF models, we
could load Mixtral and exercise the MoE path.

## 4. Model ensembling

Run 2-3 small models in parallel, average logits before sampling.
Cheap (~50 lines) but doubles/triples compute for modest quality
gain. Mostly a research technique these days; production stacks
prefer one bigger model over averaged smaller ones.

**Engineering**: ~50 lines.
**Cost**: real-time perf gets divided by N.
**Demo value**: low. Hard to convey why this matters.

Recommend: skip unless we find a specific use case (e.g.,
ensembling LoRA adapters trained on different domains).

## When to revisit

After the HF-compat pieces land. At that point:
- Multi-modal becomes "add vision to your LoRA-fine-tuned Llama"
- MoE becomes "train your own Mixtral-style architecture"
- Multi-base UI becomes a polish item once people are actually using
  the Mac app

If forced to pick one: **multi-modal**. It's the highest visual-impact
and most "wow" feature. MoE is more technically interesting but harder
to convey to anyone who isn't deep in the field.