typed

typed / typed-max (free tiers, run on your machine)

typed and typed-max are customer-facing model ids whose inference runs on your own machine, on a llama.cpp llama-server you start yourself. The typed CLI sends those turns straight to your server and never contacts api.typed.cloud for them.

That is what makes it free: you supply the compute, so there is no upstream bill to pass on. It is priced at zero, consumes no monthly quota, and needs no typed login -- typed --model typed works before you have an account. They are also private by construction: the prompt never leaves the box.

typed is the default tier -- an unconfigured client lands here -- and runs the mixture-of-experts model. typed-max runs the dense one; it is a different tradeoff, not a higher rung. Both are advertised like any other tier: listed in /v1/models, selectable via typed --model typed / typed use typed-max, and named in the unknown-model 400 body.

typed-local is a back-compat ALIAS for typed. It still resolves, so an existing pin keeps working, but it is not advertised anywhere and new writing should use typed. (typed-xhigh, the previous spelling of typed, is an alias on the same footing.)

The tradeoffs, all consequences of running on one machine: a much smaller context window, no failover if your server is down, slower responses, and no support for the prefix layers -- skill, codebase, or an active prefix bundle. Each surfaces explicitly (a 413, a 503, a 400, or a response header naming what was withheld) rather than being papered over by silently re-routing onto a model somebody has to pay for.

Requirements

32 GB of RAM. This is the one typed tier with a hardware requirement, and it is a hard floor rather than a recommendation:

GiB
model weights (UD-Q4_K_S) 19.46
llama-server runtime overhead 2.25
KV cache at the 16K minimum context 0.31
server process minimum 22.02
held back for your OS 6.00
this machine needs 28.02

GiB, not GB, and the distinction is load-bearing rather than pedantic. A machine sold as "32 GB" reports 33,893,679,104 bytes -- 31.566 GiB -- so a check written as "at least 32" in these units would refuse every machine that meets the stated requirement. typed local publishes 32 GB and compares against the 28.02 above -- 28.0225 GiB before the table rounds it, so a machine reporting exactly 28.02 is short by a hair and is refused; when it turns a machine away it prints this whole table with your machine's own figure at the bottom, so you can check the subtraction. Every figure in it comes from a constant in apps/cli/src/local-setup.ts and the CLI computes the same sum, so this table disagreeing with what typed local prints is a bug on this page.

The 16K in the third row is derived, not picked to make the table tidy. The CLI's own system prompt measures ~9,548 tokens and it never asks for fewer than 1,024 output tokens, so a window under 10,572 cannot complete a single turn -- and one at exactly 10,572 holds the prompt and a minimum answer with nothing left for your question, the files, or a tool result. 16,384 is the first power-of-two --ctx-size above that, and it is the smallest window typed local will size a machine for.

A 24 GB machine reports ~23.75 GiB and cannot clear that before the OS takes its share; 16 GB cannot hold the weights at all. Every other typed model runs on typed's infrastructure and works on any machine -- the local tiers are free precisely because they run on yours, and that is the trade.

You also need ~23 GB free on the volume holding your model cache for the download.

The model and quantization are fixed: unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_S. Not a default to tune -- the validated config. "Fits in RAM" and "actually runs" are different questions (IQ4_XS fits the same budget and does not work), and every quant is a separate surface to validate for tool-call grammar and template behavior. One tested config beats a ladder of untested ones.

Let typed size it for you

typed local

Reads your RAM, cores and free disk, then prints the exact launch command for that machine -- or tells you plainly if it is under the floor. Everything below is the same command explained flag by flag; three of its values are derived from the machine it was printed on, so take those from typed local rather than from here.

1. Start llama-server

llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_S \
  --device none --jinja \
  --ctx-size 32768 -np 1 \
  --port 8080 --threads 10 --threads-batch 12 --flash-attn on \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --spec-type ngram-cache \
  --batch-size 2048 --ubatch-size 1024

Three of those values are per-machine, not settings to copy: --ctx-size 32768 (what your RAM allows, capped for responsiveness), --threads 10 and --threads-batch 12 (derived from your core count -- the numbers above are a 12-core machine). Everything else -- the model, the quant, --jinja, -np 1, the cache types, --flash-attn, --spec-type, the batch sizes -- is fixed and identical on every machine. typed local fills in the three and leaves the rest alone.

Five flags carry more weight than they look:

  • --jinja is required for tool calling; without it the server has no tool-call parser.

  • -np 1 pins the slot count. --ctx-size is the total SHARED across parallel slots, so leaving it to a default other than 1 silently divides your usable context and reads as the model getting dumber.

  • --ctx-size is bounded by RAM and by the model's trained 262144, whichever is lower -- and the 32768 above is neither. The KV cache is modest for a model this size -- 20,480 bytes per token, the figure typed local sizes against -- so 32768 costs ~0.63 GiB. The WEIGHTS are what eat the RAM: 19.46 GiB of them, plus 2.25 GiB of runtime overhead and the 6 GiB typed local holds back for the OS. On a 32 GB machine (31.566 GiB) that leaves 3.86 GiB for KV, which at ~20 KiB/token works out to 202162 tokens -- what the planner reports as the largest window that machine could hold, still short of the model's own ceiling. It is just over 6x the window printed above. So the recommended size is a LATENCY choice, not a RAM limit: prompt eval is what every extra token costs you (see --threads-batch), and 200K tokens of prompt eval at the measured rate is hours. RAM is what binds on a 32 GB machine; the trained ceiling only starts to bind above ~32.71 GiB of total RAM, about 4.7 GiB clear of the floor -- 262144 is hard, and typed local will not suggest a window past it however much memory you have. typed local prints the largest window your machine could hold whenever it caps you below it, so you can see what you are giving up. If you raise --ctx-size yourself, watch free RAM anyway: CPU inference that swaps is unusable.

  • --threads-batch above --threads helps where the time actually goes: prompt eval dominates every turn -- measured at ~46 tok/s on one 12-core Snapdragon X Elite. Of every number on this page that is the one that travels worst: prompt eval tracks core count and memory bandwidth, so read it as an order of magnitude for your own machine, not a spec.

  • --spec-type ngram-cache roughly doubles generation on the shape this CLI produces. It drafts from n-grams already in the context, and an Edit tool call's old_string is a verbatim span of a file the model just read. Measured on build 10333 (scripts/probe-local-speculative.mjs): 6.01 -> 13.28 tok/s at 90% draft acceptance, output unchanged. Free-form prose is flat -- acceptance drops to 3-9% and the wasted passes are too few to measure. Unlike a -md draft model it loads no second set of weights, so it costs none of your RAM. Note the build offers four other n-gram variants (ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod) and none of them were measured -- ngram-cache is a tested choice, not a winner of a comparison. It also needs llama.cpp b10333 or newer: llama-server rejects an unknown argument rather than ignoring it (verified -- error: invalid argument: <flag>, exit 1), so an older binary does not start at all. b10333 is the build this was measured on, not a bisected first-supported build, so the floor is conservative.

    One more caveat on the numbers: they come from a single 12-core machine. The copy-shaped win should hold anywhere, since an accepted draft skips a forward pass regardless of hardware. The free-form result -- flat, within noise -- is the one that may not generalize, because a rejected draft costs a wasted pass and that costs proportionally more with fewer cores.

--cache-reuse is deliberately absent. It looks like the obvious companion -- reuse KV chunks after a mid-prompt change instead of re-prefilling -- and on this model it does nothing. llama-server disables it at load (KV cache shifting is not supported for this context) because only 10 of 40 layers are full-attention and linear-attention state cannot be repositioned. Measured with and without: a one-line change at 30% prompt depth cost a full re-prefill either way (prompt_n 3493, cache_n 0). An identical prompt still reuses everything, so ordinary turn-over-turn prefix caching is fine -- it is only mid-prompt divergence that has no remedy here, which is worth knowing when reasoning about context folding.

Use a native build for your architecture. On Windows ARM64 the x86_64 binary runs under emulation and is dramatically slower.

Expect it to be slow

This is CPU inference. A measured golden-path turn (typed -p with the full system prompt and tool schemas) took 3m37s end to end, nearly all of it prompt eval before the first token. That is normal, not a fault: the CLI's stream watchdogs are widened for this tier so a long silent stretch is not mistaken for a stall, and the "still working" line on stderr is a liveness signal rather than a warning.

One build to avoid on ARM, for now. A KleidiAI-enabled llama.cpp build measured ~0.2 tok/s generation on the Snapdragon X Elite box above -- roughly 35x slower than the same machine's ordinary CPU build, while prompt eval stayed normal. That is a pathology, not a tradeoff, and it is unresolved: nothing here explains why the kernels help prefill and destroy decode. Recorded so a slow session is not mistaken for the tier being slow. If generation is measured in seconds per token rather than tokens per second, check which build you are running before anything else.

GPU offload (optional)

Two different questions hide behind "is the GPU worth it", and they have different answers. Both measured on a Snapdragon X Elite / Adreno X1-85.

Offloading part of the 35B-A3B preset: worth ~19% on prompt eval, nothing on generation. scripts/probe-local-inference.mjs, cold prompt each run:

CPU only -ngl 24
prompt eval 46.3 tok/s 55.0 tok/s
generation 7.1 tok/s 7.2 tok/s
wait before first token, 8K prompt 173s 145s

Prompt eval is the number that compounds -- every turn re-reads the conversation -- so the gain is real for agentic use and irrelevant if you are just watching output stream.

Running a smaller model that fits on the GPU is a different proposition. A 9B dense model at Q8_0 on the same Adreno measured 117 tok/s prompt eval and 6.0 tok/s generation -- prompt eval more than double the partly-offloaded 35B above, on a model small enough not to fight the CPU for the same RAM. Generation is in the same range as everything else on this hardware, which is the honest summary of the whole section: on this box the GPU buys prefill, never decode.

Those two rows are not a like-for-like comparison and are not presented as one -- different models, different quants, different offload fractions. The point is that the 19% figure is a property of THAT preset and does not predict what a GPU-resident model does.

To try it you need a llama.cpp build with a backend for your GPU; the default win-cpu-* release has none compiled in. Then drop --device none and add -ngl.

Find -ngl by climbing. Do not compute it. The binding limit is OpenCL's per-buffer allocation cap, not total or free memory, so nothing about your specs predicts it: on the machine above -ngl 99 died with failed to allocate OpenCL buffer of size 2122702080 while -ngl 24 ran fine. Try 8, then 16, then 24, and keep the last value that loads.

Two things not to expect. It will not free RAM -- the GPU draws on the same physical memory, and OpenCL simply exposes a slice of it as a device budget (16 GB on the machine above, which is what caps -ngl at 24). And it will not speed up generation; that stayed flat within measurement noise.

Note llama-server --list-devices prints to the console handle rather than stdout, so it shows your GPU in a terminal but returns nothing to any script trying to capture it. typed local therefore mentions offload unconditionally instead of detecting it.

2. Use it

typed --model typed                # or: typed use typed
typed --model typed-max            # the dense model instead

That is the whole setup. The CLI defaults to http://127.0.0.1:8080, which is llama-server's default bind, so a server on the standard port needs no configuration at all.

Running somewhere else -- a different port, or another box on your LAN?

export TYPED_LOCAL_ENDPOINT=http://192.168.1.50:9090

Origin only, no /v1. The CLI appends /v1/messages itself.

What the tier actually requires of your server

Everything above assumes llama-server, because that is the easy path and the one this page documents end to end: install it, run one command, point typed at 127.0.0.1:8080, done. Nothing about the tier is bound to it, though. What the CLI needs is two endpoints:

Method Path What typed does with it
POST /v1/messages the turn itself, Anthropic Messages shape, streamed
GET /props reads default_generation_settings.n_ctx to learn your real context window

llama-server serves both natively, which is why it needs no adapter. Any other server that speaks them is equally valid: there is no allowlist of models or engines (local-match.ts retired the one that used to exist; bring-your-own is the normal case, disclosed rather than refused), and the served model name is READ from /props and reported at startup, never validated against a list. What typed probes is CAPABILITY, never identity -- it does not care what is behind the endpoint, only what that thing can do.

A worked example of a non-llama.cpp server on the same wiring: a small server running a 4B model on the Snapdragon NPU through Qualcomm's Genie runtime, speaking those same two endpoints. typed reaches it with the same TYPED_LOCAL_ENDPOINT you would use for a llama-server on another port -- no shim, no OpenAI adapter, no flag.

There is one more thing typed asks your server at startup, and it is worth knowing because it decides whether the session can use tools at all. typed POSTs a single tools payload to /v1/messages and reads the answer:

The probe gets typed concludes You see
a normal response grammars build; the session is agentic and ships tool schemas a startup note saying tool calls are enabled
a 4xx naming the limitation, or a dropped connection no tool calling; the schemas are dropped and the tokens reclaimed a startup note saying so
a timeout, a 401/403, or a 5xx nothing provable, so it stays toolless nothing

That third row is the one to watch when bringing your own server. An authenticating or slow-to-start endpoint lands there, and the session runs chat-only with no note explaining why -- the probe took the request, which is not evidence of a fault, so typed does not accuse it of one. If your server means to support tool calling, make sure it answers that probe.

Three things a bring-your-own server still owes you, because typed cannot infer them:

  • An honest /props. If n_ctx is wrong, typed plans every turn against a window your server does not have and the server truncates your prompt silently.
  • An honest stop_reason. A reply cut off at the output cap should report max_tokens, not end_turn. When a server gets this wrong typed now notices anyway (see the truncation notice below), but the notice is a backstop, not a substitute.
  • A definite answer to the tool probe, per the table above -- either "yes" or a clean refusal. Silence and ambiguity both resolve to a chat-only session.

The other path: a self-hosted API

There is a second, unrelated way to reach a local model: a deployment that runs typed's own API next to a model server, and sets TYPED_LOCAL_BASE_URL in the API's environment. Most people want the CLI path above instead.

export TYPED_LOCAL_BASE_URL=http://127.0.0.1:8080/v1   # NOTE: includes /v1
export TYPED_LOCAL_API_KEY=...                          # only if started with --api-key

The two variables use opposite URL conventions -- TYPED_LOCAL_ENDPOINT is an origin, TYPED_LOCAL_BASE_URL includes /v1 -- because the two code paths append different suffixes. Swapping one for the other 404s every request, which is why they are separate names.

Unset on the API side, the slot stays unwired and a typed / typed-max request that reaches api.typed.cloud returns 503 config_error / router_misconfigured rather than being silently rerouted onto a paid carrier.

What this tier does differently

free local (typed, typed-max) paid typed++ rungs
Cost $0, free (your hardware) metered per token
Quota accrues nothing counts against the monthly ceiling
Auth no typed login needed API key required
Where it runs your machine typed's upstreams
Context cap 7,500 fallback, or 92% of the probed llama-server window 950K
Oversized request 413 413
Breaker open 503, no failover fails over to another carrier (except typed++high, which 503s -- it is an eval slot with no failover)
Knowledge prefix skipped injected
x-typed-skill / x-typed-codebase 400 honored
Active prefix bundle skipped, with a warning header injected

Several of those rows are the same decision seen from different angles: every failover or prefix target is a PAID carrier, and a request you chose because it was free must never quietly become a billed one. Failing honestly with a 413 or a 503 is what a free tier owes its users; silently upgrading onto a metered model is not. The prefix layers are also simply too large for the window -- the skill prefix alone reserves 16,896 tokens and the codebase prefix 8,192, against a default cap of 7,500. Neither fits even before your prompt is counted, which is why the headers are rejected outright rather than quietly dropped.

Two notes on the context-cap row. The 7,500 figure is a FALLBACK, not the tier's real window: the API adopts the server's own number when it can probe it, taking 92% of it to absorb tokenizer drift and the prefix injections that land after the gate (MAX_CONTEXT_TOKENS_LOCAL and LOCAL_WINDOW_SAFETY_FRACTION in apps/api/src/routes/messages.ts), and the first-party CLI probes /props at startup and uses the real number directly. On this tier max_tokens also comes OUT of that budget before your prompt is measured, because llama.cpp reserves the output allocation up front -- a hosted-sized max_tokens will squeeze the prompt out of a small window. And the paid column says 413 rather than "upgrades" because the long-context upgrade band is zero-width today: the primary cap and the long-context cap are both 950K with the same reserves subtracted, so an oversized paid request 413s too. Both long-context carriers stay wired as failover targets.

An active prefix bundle is the one layer that is neither server-configured nor asked for on the request -- you register it ahead of time on your account -- so it is skipped rather than rejected: a 400 would fail every free-local request until you deactivated the bundle account-wide. Because the withheld text is your own, the skip is never silent. The response carries x-typed-prefix-bundle-warning: bundle_skipped_on_local, in the same slot where an applied bundle sets x-typed-prefix-bundle: <bundleId>.

Timeouts are much longer than the hosted defaults (2min TTFT, 15min non-streaming response) and retries are off. Local inference is slow, and re-submitting a multi-minute prompt eval only queues the same work behind itself.

Guardrails specific to this tier

The free local tiers run a smaller model with a much smaller window, so the harness enforces limits the hosted tiers do not need. These are mechanical -- they do not depend on the model following an instruction, which a 4-bit 35B does unreliably.

Guardrail Why
output cap scaled to the window llama.cpp reserves max_tokens out of n_ctx before generating; a hosted-sized 16K cap on a 32K window squeezes out the prompt, so the model answers about code it can no longer see
compaction trigger scaled to the window the 120K default is ~4x an entire local context and never fires there
Edit refuses an unread file an old_string the model never observed was imagined, and a lucky match applies cleanly to real code
turn cap lowered a turn is minutes, and a small model that starts confabulating spirals rather than recovering
subagents disabled a single-slot server queues them behind the main turn, and they start without the parent's context
truncation notice when a reply fills the output cap not every server reports stop_reason: max_tokens when it cuts a reply off at the cap -- some report end_turn, which makes a truncated answer indistinguishable from a finished one

That last row is the one guardrail written for servers rather than for models, so it is worth saying what it does and does not claim. When a turn ends on any reason other than max_tokens or tool_use but the reply consumed every output token that turn was allowed, typed emits a notice saying the answer may have been cut off. It reads the output token count, which most servers report, rather than the stop reason, which not every server gets right -- so it costs a correct server nothing (llama-server reports max_tokens properly and never trips it) and catches the case on one that does not (the Snapdragon NPU's Genie runtime signals a normal sentence-end at the cap). It stays a notice: the turn still completes and the stop reason is still reported verbatim, because a model CAN legitimately finish on its last allowed token and typed has no way to tell the two apart.

Two limits on it, both deliberate. A server that reports no output tokens never trips it -- the check needs a count to compare, and failing quiet beats a notice on every turn. And unlike every other row in this table it is not gated to the local tier: it is written for servers whose stop reasons cannot be trusted, and that is a property of a server rather than of a tier. A hosted turn reports max_tokens correctly and so never reaches the check, which makes the gate unnecessary rather than missing -- do not "fix" it by adding one.

Tool-call output is already grammar-constrained by llama.cpp whenever tools are sent, so malformed tool JSON is structurally impossible rather than merely discouraged. That needs no configuration.

An untested lever

TYPED_LOCAL_MODE=precise lowers sampling entropy (temperature 0.3 vs the default 0.7) on the theory that a confidently wrong tool argument costs more than a dull phrasing. It is opt-in and unmeasured: the model card publishes no non-thinking coding row, so this is reasoning about sampling rather than a finding. Try it on a real task and compare. It is deliberately not greedy -- its authors warn that temperature 0 degenerates into repetition.

Troubleshooting

Requests hang, then the connection drops (ECONNRESET, or HTTP 000 from curl). If the request carried tools, this is a llama.cpp tool-call grammar failure, not a typed bug. Confirm by sending the same request with "tool_choice": "none" -- if that succeeds, grammar construction is the fault, and the fix is a newer llama.cpp build or a different model/template pairing. Observed on build b10326 with a 27B dense build of the same family and gone on b10331+ with the 35B-A3B MoE.

Do NOT use chat_format from /props as the diagnostic. It reads "Content-only" on a server where tool calling demonstrably works -- it reflects the DEFAULT generation params (no tools in the request), not whether tool grammar can be built. The reliable signal is the connection reset itself, reproducible at max_tokens: 1.

Also note --chat-template <name> only accepts the built-in legacy template names, which disable jinja tool calling entirely -- it is not a fix. The per-request chat_template_kwargs field IS honored, which is how typed drives enable_thinking per session without restarting the server.

Every request 413s, including tiny ones. The prefix reserves exceeded the tier's window. Should be impossible now (the knowledge reserve and any active prefix bundle are skipped, and the customer-requested layers 400 on this tier), but if it recurs the API returns a config_error with code context_budget_exceeds_window naming the cause instead of a misleading "request too large".

503 config_error / router_misconfigured. The request reached api.typed.cloud instead of your own server. Through the typed CLI that should be impossible -- if you see it, the turn went out over the hosted path, so check that the model really resolved to typed or typed-max (typed status).

ECONNREFUSED / connection refused. The CLI reached for your model server and found nothing there. Confirm llama-server is up (curl http://127.0.0.1:8080/health) and that TYPED_LOCAL_ENDPOINT matches its port.

Boot warns about n_ctx. The API-side path enforces a constant (MAX_CONTEXT_TOKENS_LOCAL) that is smaller than what your server serves. Harmless for CLI sessions -- the CLI reads the real n_ctx from /props at startup and sizes itself to whatever you launched with -- but the API path would 413 earlier than necessary.