typed-local (the free tier, run on your machine)

typed-local is the free tier: a customer-facing model id whose inference runs on your own machine, on a model server you start yourself: a llama.cpp llama-server (the path this page documents end to end), LM Studio, Ollama, an MLX server such as vllm-mlx on an Apple Silicon Mac, or the GenieX Server (Snapdragon). "Set up your server" below has copy-paste steps for each one on Windows, macOS and Linux, and "LM Studio, Ollama and MLX servers" says what typed reads from each. The typed CLI sends those turns straight to your server and never contacts api.typed.cloud for them.

That is what makes it free: you supply the compute, so there is no upstream bill to pass on. It is priced at zero, consumes no monthly quota, and needs no typed login -- typed --model typed-local works before you have an account. It is also private by construction: the prompt never leaves the box.

typed-local is the default tier without a paid plan -- a client with no typed.cloud key on the machine, or a key whose account holds no plan, lands here -- and the only free id. With a paid plan the CLI's default is typed, the hosted economy tier; typed use typed-local pins the free tier regardless. A session runs whatever your server is serving (llama-server ignores the model field; on the engines that need one, typed sends the name the server reports), so a second local id could only ever name the same thing.

Until 23 September 2026 this tier was spelled typed. Since that date a bare typed is the hosted economy tier (paid, served by api.typed.cloud), and typed-local -- until then an alias of the free tier -- is its canonical id. A .typed or ~/.config/typed/model pin that says typed and was written before the rename meant this tier; the launcher and the CLI rewrite such a pin to typed-local once and say so. An ANTHROPIC_MODEL=typed in your shell is taken at its new meaning, with a one-time note that names typed-local as the free tier. typed-xhigh (the spelling before 19 August 2026) and typed-max are aliases of typed-local. typed-max used to make typed local size a dense 27B preset instead of the default; that preset is retired, so a pin that names it now gets exactly what typed-local gets.

The tradeoffs, all consequences of running on one machine: a much smaller context window, no failover if your server is down, slower responses, and no support for the prefix layers -- skill, codebase, or an active prefix bundle. Each surfaces explicitly (a 413, a 503, a 400, or a response header naming what was withheld) rather than being papered over by silently re-routing onto a model somebody has to pay for.

Requirements

32 GB of RAM. This is the one typed tier with a hardware requirement, and it is a hard floor rather than a recommendation:

	GiB
model weights (UD-Q4_K_S)	19.46
llama-server runtime overhead	2.25
KV cache at the 16K minimum context	0.17
server process minimum	21.88
held back for your OS	6.00
this machine needs	27.88

GiB, not GB, and the distinction is load-bearing rather than pedantic. A machine sold as "32 GB" reports 33,893,679,104 bytes -- 31.566 GiB -- so a check written as "at least 32" in these units would refuse every machine that meets the stated requirement. typed local publishes 32 GB and compares against 27.876015625 GiB, which the bottom row above rounds UP to 27.88 -- so the printed figure sits a hair ABOVE the real gate and a machine reporting exactly 27.88 clears it; when it turns a machine away it prints this whole table with your machine's own figure at the bottom, so you can check the subtraction. Every figure in it comes from a constant in apps/cli/src/local-setup.ts and the CLI computes the same sum, so this table disagreeing with what typed local prints is a bug on this page.

The runtime overhead row is the one with real memory hiding inside it, so here is what it holds. Measured on 2026-09-15 at a 65536 window: 62.81 MiB of recurrent state for the 30 linear-attention layers (which is not part of the KV cache row and does not grow with the window), 272.30 MiB of compute buffer (176.30 MiB at 16384 -- the only one of the three that does grow with it), and the server's context checkpoints, which it enables by default: copies of that recurrent state, 62.813 MiB each, up to 32 (about 1.96 GiB). At idle the whole non-weight, non-KV set measured 1.35 GiB (1.31 GiB in a second run with --no-repack), 0.90 GiB inside the 2.25 GiB the row reserves, and that is the only slack: the weights row counts the model file once, which the --no-repack flag in the command is there to make true (see below). The checkpoints are what can use it. Reading llama.cpp's source at b10333: the server keeps an earlier request's checkpoints only when they are more than 8,192 tokens apart -- at most 8 in a 65536 window -- while the request it is serving keeps all of its own, one where its last user message starts and up to three more near the end of its prompt, and speculative decoding holds at most one more copy while it checks a draft. That is at most 13 copies, 0.80 GiB, which the row covers at 65536 with 0.10 GiB to spare. Above 65536 it does not: every further 8,193 tokens of window allows another checkpoint and the compute buffer grows too, so from 73,739 tokens the worst case outgrows the row, by 0.52 GiB at 131072 and 1.50 GiB at 262144 (from the source and a straight-line compute buffer, not measured in a session that long). So when typed local --ctx-size or --max-window sizes a window past 73,738 tokens, it counts that window's own overhead in place of the row -- 2.77 GiB at 131072, 3.75 GiB at 262144 -- and refuses a window that does not fit beside it, with the arithmetic. The table above, the default window and the 27.88 GiB gate do not change: at 73,738 tokens and below, the row is what is counted.

The 16K in the third row is derived, not picked to make the table tidy. The CLI's own static prefix on this tier measures ~8,135 tokens by the model's own tokenizer on a reference project (harness, compact tool schemas, the project's CLAUDE.md chain, memory index and skill catalog -- the startup line that begins (prefix: shows your own figure, estimated from character counts, which reads a few percent lower) and it counts 1,024 output tokens as the smallest reply that can finish a turn, so a window under 9,159 cannot complete a single turn -- and one at exactly 9,159 holds the prompt and a minimum answer with nothing left for your question, the files, or a tool result. 16,384 is the first power-of-two --ctx-size above that, and it is the smallest window typed local will size a machine for.

A 24 GB machine reports ~23.75 GiB and cannot clear that before the OS takes its share; 16 GB cannot hold the weights at all. Every other typed tier runs on typed's infrastructure and works on any machine -- typed-local is free precisely because it runs on yours, and that is the trade.

You also need ~23 GB free on the volume holding your model cache for the download.

This floor, and everything typed local sizes, is the plan for llama-server running the preset below. With LM Studio, Ollama or an MLX server the engine loads the model and chooses where it lives, so typed does not size or refuse the machine; what it still reads from the server at startup is described in "LM Studio, Ollama and MLX servers".

The model and quantization are fixed: typed local sizes exactly one preset, unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_S, and it is a validated config rather than a default to tune. "Fits in RAM" and "actually runs" are different questions (IQ4_XS fits the same budget and does not work), and every quant is a separate surface to validate for tool-call grammar and template behavior. One tested config beats a ladder of untested ones.

Let typed size it for you

typed local

Reads your RAM, cores and free disk, then prints the exact launch command for that machine -- or tells you plainly if it is under the floor. Everything below is the same command explained flag by flag; three of its values are derived from the machine it was printed on, so take those from typed local rather than from here.

To size it differently, ask typed local rather than editing the command by hand:

typed local --ctx-size 131072        # this window instead of the default
typed local --max-window             # the largest window your RAM plan holds
typed local --cache-ram auto         # add a host-RAM prompt cache (or a size in MiB)
typed local --slots 2                # one slot per typed pane on this server (or auto)

--ctx-size <tokens> (16384 to 262144) and --max-window choose the window -- pass one of them -- --cache-ram <MiB|auto> adds llama-server's host-RAM prompt cache, and --slots <N|auto> (1 to 8, or TYPED_LOCAL_SLOTS in the environment) gives the server one slot per typed pane (see -np below). Every value is checked against the same RAM arithmetic as the default: a size that fits prints its command with the cold-prompt cost for that window, and a size that does not is refused with the arithmetic and the largest size that would fit. Nothing is quietly shrunk to make it fit.

1. Start llama-server

llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_S \
  --device none --jinja \
  --ctx-size 65536 -np 1 \
  --cache-ram 0 \
  --no-repack \
  --port 9931 --threads 10 --threads-batch 12 \
  --flash-attn on \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --spec-type ngram-cache \
  --min-p 0.0 --presence-penalty 1.5 \
  --batch-size 2048 --ubatch-size 1024

Three of those values are per-machine, not settings to copy: --ctx-size 65536 (what your RAM allows, capped to bound a cold prefill), --threads 10 and --threads-batch 12 (derived from your core count -- the numbers above are a 12-core machine). One more depends on the mode rather than the machine: --presence-penalty 1.5 is the default instruct mode's value, and TYPED_LOCAL_MODE=thinking needs --presence-penalty 0.0 (see the sampling flags below). Everything else -- the model, the quant, --jinja, -np 1, --cache-ram 0, --no-repack, the cache types, --flash-attn, --spec-type, --min-p 0.0, the batch sizes -- is fixed and identical on every machine. typed local fills in the machine's three and the mode's value from your environment, and leaves the rest alone.

Eight flags carry more weight than they look:

• --jinja is required for tool calling; without it the server has no tool-call parser. Current builds (b10333) turn it on by default and the command spells it out anyway; a server started with --no-jinja answers every request carrying tools with 500 tools param requires --jinja flag, and typed's startup check then says tool calls are off for the session and quotes that message.

• -np 1 pins the slot count. --ctx-size is the total SHARED across parallel slots, so leaving it to a default other than 1 silently divides your usable context and reads as the model getting dumber. One slot holds one conversation's prompt, so several typed panes on one server evict each other: every switch makes the next turn back re-read its whole prompt, about 3 minutes for the system prompt alone at the measured rate and about 15 at 40,000 tokens. typed local --slots <N> prints -np N --kv-unified --no-cache-idle-slots instead, so each pane keeps its own slot. --kv-unified gives every slot the whole --ctx-size as its window, drawn from one shared KV pool, so each pane still gets the window above and the KV cache does not grow; when the panes' prompts together outgrow the pool, the server clears an idle slot and that pane re-reads, as it would with one slot. --no-cache-idle-slots stops the server from clearing idle slots whenever another one starts work, which it otherwise does when a --cache-ram is set. Each slot past the first keeps its own recurrent state and context checkpoints, 0.86 GiB at 65536, and typed local counts that in the RAM plan: a 32 GB machine fits 4 slots at 65536, and a count that does not fit is refused with the most that do. --slots auto takes the most that fit (up to 8). Multi-slot serving was sized from the llama.cpp source, not measured on a running server.

• --ctx-size is bounded by RAM and by the model's trained 262144, whichever is lower -- and the 65536 above is neither. The KV cache is modest for a model this size -- 10,880 bytes per token, the figure typed local sizes against, measured off the server's own llama_kv_cache line on 2026-09-15 -- so 65536 costs 0.66 GiB. The WEIGHTS are what eat the RAM: 19.46 GiB of them, plus 2.25 GiB of runtime overhead and the 6 GiB typed local holds back for the OS -- 28.37 GiB in all at 65536, which a 32 GB machine (31.566 GiB) clears with ~3.2 GiB to spare. That machine has 3.86 GiB for KV, which at 10.6 KiB/token works out to 380541 tokens -- but a window that large also takes more runtime overhead (the checkpoints and compute buffer under Requirements), so the largest window the planner reports that machine could hold is 237077 tokens, about 3.6x the window printed above. So the recommended size bounds a COLD prefill, not RAM. A warm turn costs only what is new: the server reuses the prefix it already holds (the --cache-reuse note below says exactly how far that goes), so a bigger window costs an ordinary tool turn nothing. What scales with the window is the cold prompt -- the first turn, the turn after a compaction or an eviction, a prefix-cache miss -- which prompt eval reads from the start (see --threads-batch). typed holds a prompt to 85% of the window, so the largest cold prompt at 65536 is 55,705 tokens, about 20 minutes at the measured rate; 200K tokens of it would be hours. The cap was 32768 until the smaller window proved to cause the wait it was meant to avoid: a coding session filled it every tool turn or two, and each compaction that followed was a cold prefill of its own. The trained ceiling starts to bind above ~31.87 GiB of total RAM, just above the 31.566 GiB a 32 GB machine reports -- so on the floor machine it is RAM that limits how far you can raise this. 262144 is hard, and typed local will not suggest a window past it however much memory you have. typed local prints the largest window your machine could hold whenever it caps you below it, so you can see what you are giving up. To raise it, run typed local --ctx-size <tokens> or typed local --max-window: both check the size against the RAM plan and print the cold-prompt cost at that window. A hand-edited --ctx-size skips that check, and CPU inference that swaps is unusable. The default does not grow on a faster machine: no server is running when typed local sizes it, so there is no prompt-eval rate to read, and the reason the window is not smaller (compaction every tool turn or two) would only ever let a rate raise it, which --ctx-size already does.

• --cache-ram 0 turns off llama-server's host-RAM prompt cache, which is on by default with an 8192 MiB limit. That cache copies a slot's whole saved state into ordinary memory whenever a new request would replace more than half of it -- up to 8 GiB that no figure on this page counts, on a 32 GB machine with ~3.2 GiB to spare. On CPU inference that is not slower, it is swap. The cache exists so several conversations can share one slot; with -np 1 and one client there is nothing to share, and what keeps a warm turn cheap is the prefix the slot itself holds, not this copy of it. If you want one anyway, typed local --cache-ram <MiB> counts it in the RAM plan beside the window and refuses a size that does not fit. --cache-ram auto takes the smaller of the RAM left beside the window and one full slot state. A saved slot is larger than its KV cache: llama-server stores the slot's KV rows, its 62.81 MiB of recurrent state and a copy of every context checkpoint it holds, 62.813 MiB each. typed sizes a full slot at the server's maximum of 32 checkpoints, about 2.69 GiB for a 65536 window -- more than a slot at that window holds on b10333, but a newer build can place checkpoints differently, and the server skips an entry bigger than the limit rather than truncating it. A --cache-ram below that size, auto or not, prints a line saying so. The default stays 0 because no benefit has been measured for one client on one slot, and one full entry is most of a 32 GB machine's spare RAM.

• --no-repack keeps one copy of the weights. By default llama-server repacks them into the layout its CPU kernels prefer, and measured on b10333 with this model and command that kept them twice: the model file mapped into memory (19,516.80 MiB) plus a 19,299.60 MiB private repacked copy. The requirements table counts the weights once, so the second copy had nowhere to go but the pagefile -- on a 32 GB machine (31.57 GiB) system commit charge sat within about 2 GB of the Windows limit, and the server took 436.5 s to load. With the flag the server keeps only the file mapping, which the OS can drop and re-read from the model file under pressure instead of paging weights out: commit charge about 19 GB lower, the server's private memory 2.19 GiB instead of 21.06 GiB after the same prompts, and a 10.7 s load. Prompt eval could not be told apart under the load that machine carried during both runs. Decode speed was not compared, so whether repacked kernels generate faster on your CPU is not known; the flag is there for memory, and taking it out puts a second copy of the weights outside the RAM plan.

• --threads-batch above --threads helps where the time actually goes: prompt eval dominates every turn -- measured at ~46 tok/s on one 12-core Snapdragon X Elite. Of every number on this page that is the one that travels worst: prompt eval tracks core count and memory bandwidth, so read it as an order of magnitude for your own machine, not a spec.

• --spec-type ngram-cache roughly doubles generation on the shape this CLI produces. It drafts from n-grams already in the context, and an Edit tool call's old_string is a verbatim span of a file the model just read. Measured on build 10333 (scripts/probe-local-speculative.mjs): 6.01 -> 13.28 tok/s at 90% draft acceptance, output unchanged. Free-form prose is flat -- acceptance drops to 3-9% and the wasted passes are too few to measure. Unlike a -md draft model it loads no second set of weights, so it costs none of your RAM. Note the build offers four other n-gram variants (ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod) and none of them were measured -- ngram-cache is a tested choice, not a winner of a comparison. It also needs llama.cpp b10333 or newer: llama-server rejects an unknown argument rather than ignoring it (verified -- error: invalid argument: <flag>, exit 1), so an older binary does not start at all. b10333 is the build this was measured on, not a bisected first-supported build, so the floor is conservative.

  One more caveat on the numbers: they come from a single 12-core machine. The copy-shaped win should hold anywhere, since an accepted draft skips a forward pass regardless of hardware. The free-form result -- flat, within noise -- is the one that may not generalize, because a rejected draft costs a wasted pass and that costs proportionally more with fewer cores.

• --min-p 0.0 --presence-penalty 1.5 are the only sampling settings in the command, and the only way these two reach the sampler. llama-server's /v1/messages endpoint builds its request from a fixed list of fields and drops min_p and presence_penalty: measured on b10333 (and the same list is in upstream's current source), they ran at the server defaults, presence_penalty 0.0 and min_p 0.05, whatever the request carried, while a server started with these flags ran at the flag values even for a request asking otherwise. typed sends temperature, top_p and top_k with every request, and those are honored, so they stay out of the command. One server serves every mode, so the presence penalty is the mode's: 1.5 for the default instruct mode and for precise, 0.0 for TYPED_LOCAL_MODE=thinking. Run typed local with the mode set and it prints the matching value. A session reads both values from /props when it starts, says nothing when they match, and prints one line naming the server's values and the command to restart it with when they do not -- the one typed local prints, which sets both along with the flags /props cannot show; on a server whose /props does not report them it says it could not confirm them and names the same command (typed local install refreshes an older build). A build patched to take both per request runs the session's values whatever its flags say, and /props reports only the flags: when a turn's /slots reading, taken while the prompt is read, shows the session's values ran where /props reported others, a second line says the first one does not apply. That line needs such a reading, so it never appears on a server started with --no-slots, and a turn whose prompt is read before typed's next /slots poll (every 5 s) cannot show it; the first line stands until a turn does.

--cache-reuse is deliberately absent. It looks like the obvious companion -- reuse KV chunks after a mid-prompt change instead of re-prefilling -- and on this model it does nothing. llama-server disables it at load (KV cache shifting is not supported for this context) because only 10 of 40 layers are full-attention and linear-attention state cannot be repositioned. Measured with and without: a one-line change at 30% prompt depth cost a full re-prefill either way (prompt_n 3493, cache_n 0). An identical prompt still reuses everything, so turn-over-turn prefix caching within one prompt's tool run is fine -- it is only mid-prompt divergence that has no remedy here, which is worth knowing when reasoning about context folding.

A real divergence costs a little more than the text after it: this build restores the server's newest context checkpoint at or before the divergence and re-reads from there, rather than resuming at the divergence itself. Measured on b10333, checkpoints were dense enough that the extra re-read was 155 tokens; with no checkpoint before the change it is the whole prompt.

One divergence used to come with the model's chat template rather than with anything typed changed, and typed now turns it off. The template renders a reply's reasoning block only for replies after the newest user message unless the request sets preserve_thinking, so without it the first request of every new prompt -- and the summarization request compaction sends -- re-rendered the replies since the previous prompt and the server re-read from the first of them. typed sends "chat_template_kwargs": {"enable_thinking": false, "preserve_thinking": true} on every local request ("enable_thinking": true in thinking mode; preserve_thinking in every mode), and compaction keeps the replies' reasoning in its request so they render the same way there too. Measured on b10333 with the command typed local prints, prompt tokens the server evaluated, with the flag against without it:

request	instruct (8-turn history)	thinking (4-turn history)
first turn of a new prompt	152 vs 6,647	149 vs 4,814
compaction of the whole history	218 vs 6,713	1,243 keeping the reasoning vs 4,896 stripping it

The price is 4 prompt tokens per earlier reply in instruct mode (an empty reasoning wrapper) and, in thinking mode, the earlier reasoning itself staying in the prompt -- 132 tokens over that 4-turn run, paid once into the cache instead of re-read at every prompt. Four full generations on those histories, both modes with and without the flag, showed no change in the model's output: no leaked tags, valid tool calls, the same next call. A history that already carries reasoning renders it in instruct mode too, so a thinking-mode session resumed in instruct mode keeps its earlier reasoning in the prompt. typed leaves it there: the template renders the current prompt's reasoning in any mode anyway, the instruct reply still does not reason, and the reasoning counts against the window like any other text until a compaction summarizes it away.

What the flag cannot fix is a compaction that has to cut the oldest turns to fit its own request: that request starts at a later message than the conversation, diverges right after the tool schemas, and re-reads the kept tail. A compaction the in-loop guard triggers is almost always that case. typed cannot see whether a server honors preserve_thinking -- a different model's template may ignore it -- so a slow turn after a new prompt is named prefix cache miss: ... (server restarted, another client used the slot, or the server ignored preserve_thinking) rather than blamed on any one of them, and a cut compaction as (history was rewritten (eviction or compaction)).

Use a native build for your architecture. On Windows ARM64 the x86_64 binary runs under emulation and is dramatically slower.

Expect it to be slow

This is CPU inference. A measured golden-path turn (typed -p with the full system prompt and tool schemas) took 3m37s end to end, nearly all of it prompt eval before the first token. That is normal, not a fault: the CLI's stream watchdogs are widened for this tier so a long silent stretch is not mistaken for a stall, and the "still working" line on stderr is a liveness signal rather than a warning. On a machine whose model weights page to disk, a long prompt can take hours before the first token (below).

How long typed waits for the first token. While nothing has streamed, typed asks llama-server's GET /slots how far through the prompt it is. llama-server answers only between batches, and its count moves one --batch-size (2,048 tokens) at a time, so the "still working" line can show the same "(4.2k prompt tokens evaluated so far)" for minutes at a time on a slow machine: that is a batch in progress. For scale: measured on 2026-09-17 with build b10333 and the command typed local prints, a 32 GB machine whose model weights were paging to disk evaluated a prompt at about 10.8 tokens per second, so an 8K-token prompt took about 12 minutes before the first token. A long prompt is much slower than that, because each batch takes longer the deeper into the prompt it is: on the same machine on 2026-09-18, the first 45,170 tokens of a longer prompt took 11,135 seconds, so a 45K-token prompt waits about 3 hours for its first token, and deep into it the count sat still for up to about 17 minutes per batch. What typed does while it waits depends on what /slots shows:

• The count keeps moving. typed keeps waiting, however long the prompt takes. Any request's progress counts, so a request of yours queued behind another client's (or a parallel subagent's) keeps waiting while that one moves.
• The count stops after typed has timed a full batch. The turn fails when the count stops moving for about 8 minutes, or for 2.5 times that prompt's longest batch when that is longer; if the last turn that timed a batch saw a longer one, that batch is used instead, though never so as to wait past the fixed ceiling described below. A full batch is a step of the count the same size as the step before it, on the prompt llama-server is working on: a prompt's first chunk and its last few shorter steps are not full batches, and neither is one report that covers two batches because an answer came back late (the next pair of equal steps times the prompt again).
• The count stops before typed has timed a full batch. That is the case before the prompt at work has shown two equal batch-size steps in a row; again after one report covers two batches, until the next equal pair; and once the prompt reaches its last steps or starts answering. A request of yours queued behind another client's is in this case from the time the one ahead reaches its last steps until your own prompt has a batch timed. Here typed does not end the turn while llama-server shows it is working (a slot busy, or a /slots request not yet answered), so batches longer than 8 minutes (a larger --batch-size, or a machine under about 4.3 tokens per second at 2,048) run on until one is timed, bounded meanwhile by the fixed ceiling described below, which restarts each time the count moves.
• The server shows no work. The latest /slots reading since your request went out shows every slot idle, or there has been no such reading and /slots answers with an error instead of a slot list: a server that shows no work at all fails after about 8 minutes. An error after a reading that showed a slot busy does not count as no work; that reading still stands, and the wait goes on.
• The server is gone. A turn also fails at once, naming the server, if llama-server stops accepting connections; start it again with the command typed local prints.
• The server cannot report progress. A server started with --no-slots, or one that is not llama-server, gets one line saying progress is not visible; its wait is then bounded by a fixed ceiling that grows with the context window (about 50 minutes at 65,536 tokens).

One build to avoid on ARM, for now. A KleidiAI-enabled llama.cpp build measured ~0.2 tok/s generation on the Snapdragon X Elite box above -- roughly 35x slower than the same machine's ordinary CPU build, while prompt eval stayed normal. That is a pathology, not a tradeoff, and it is unresolved: nothing here explains why the kernels help prefill and destroy decode. Recorded so a slow session is not mistaken for the tier being slow. If generation is measured in seconds per token rather than tokens per second, check which build you are running before anything else.

GPU offload (optional)

Two different questions hide behind "is the GPU worth it", and they have different answers. Both measured on a Snapdragon X Elite / Adreno X1-85.

Offloading part of the 35B-A3B preset: worth ~19% on prompt eval, nothing on generation. scripts/probe-local-inference.mjs, cold prompt each run:

	CPU only	-ngl 24
prompt eval	46.3 tok/s	55.0 tok/s
generation	7.1 tok/s	7.2 tok/s
wait before first token, 8K prompt	173s	145s

Prompt eval is the number that compounds -- every turn re-reads the conversation -- so the gain is real for agentic use and irrelevant if you are just watching output stream.

Running a smaller model that fits on the GPU is a different proposition. A 9B dense model at Q8_0 on the same Adreno measured 117 tok/s prompt eval and 6.0 tok/s generation -- prompt eval more than double the partly-offloaded 35B above, on a model small enough not to fight the CPU for the same RAM. Generation is in the same range as everything else on this hardware, which is the honest summary of the whole section: on this box the GPU buys prefill, never decode.

Those two rows are not a like-for-like comparison and are not presented as one -- different models, different quants, different offload fractions. The point is that the 19% figure is a property of THAT preset and does not predict what a GPU-resident model does.

To try it you need a llama.cpp build with a backend for your GPU; the default win-cpu-* release has none compiled in. Then drop --device none and add -ngl.

Find -ngl by climbing. Do not compute it. The binding limit is OpenCL's per-buffer allocation cap, not total or free memory, so nothing about your specs predicts it: on the machine above -ngl 99 died with failed to allocate OpenCL buffer of size 2122702080 while -ngl 24 ran fine. Try 8, then 16, then 24, and keep the last value that loads.

Two things not to expect. It will not free RAM -- the GPU draws on the same physical memory, and OpenCL simply exposes a slice of it as a device budget (16 GB on the machine above, which is what caps -ngl at 24). And it will not speed up generation; that stayed flat within measurement noise.

Note llama-server --list-devices prints to the console handle rather than stdout, so it shows your GPU in a terminal but returns nothing to any script trying to capture it. typed local therefore mentions offload unconditionally instead of detecting it.

2. Use it

typed                              # no --model needed without a paid plan: the free tier is the default
typed --model typed-local          # or: typed use typed-local, to pin it explicitly (needed with a key on a paid plan)

That is the whole setup. The CLI defaults to http://127.0.0.1:9931, which is llama-server's default bind on current builds (older builds bind 8080 unless told otherwise, which is why the command above passes --port 9931), so a server started that way needs no configuration at all. With nothing pinned, typed also checks the other supported engines on their own default ports at launch (LM Studio on 1234, Ollama on 11434, an MLX server on 9000, the GenieX Server (Snapdragon) on 8123) and uses the one that answers; when more than one does, it asks which to use.

Running somewhere else -- a different port, or another box on your LAN?

export TYPED_LOCAL_ENDPOINT=http://192.168.1.50:9090

Origin only, no /v1. The CLI appends /v1/messages itself.

What the tier actually requires of your server

Everything above assumes llama-server, because that is the easy path and the one this page documents end to end: install it, run one command, point typed at 127.0.0.1:9931, done. Nothing about the tier is bound to it, though. typed reads each engine through that engine's own endpoints; for llama-server and the GenieX Server (Snapdragon) that is two endpoints (LM Studio, Ollama and MLX servers are covered in their own section below):

Method	Path	What typed does with it
POST	/v1/messages	the turn itself, Anthropic Messages shape, streamed
GET	/props	reads default_generation_settings.n_ctx to learn your real context window, and default_generation_settings.params.presence_penalty and .min_p to check the sampler against the session's mode

llama-server serves both natively, which is why it needs no adapter. Any other server that speaks them is equally valid: there is no allowlist of models or engines (local-match.ts retired the one that used to exist; bring-your-own is the normal case, disclosed rather than refused), and the served model name is READ from /props and reported at startup, never validated against a list. What typed probes is CAPABILITY, never identity -- it does not care what is behind the endpoint, only what that thing can do.

A worked example of a non-llama.cpp server on the same wiring: the GenieX Server (Snapdragon), a small server running a 4B model on the Snapdragon NPU through Qualcomm's Genie runtime, speaking those same two endpoints. typed reaches it with the same TYPED_LOCAL_ENDPOINT you would use for a llama-server on another port -- no shim, no OpenAI adapter, no flag.

There is one more thing typed asks your server at startup, and it is worth knowing because it decides whether the session can use tools at all. typed POSTs a single STREAMED request carrying one tool (stream: true, max_tokens: 1) to /v1/messages, reads only the status line, and lets go of the stream without reading it:

The probe gets	typed concludes	You see
a 2xx status line	grammars build; the session is agentic and ships tool schemas	a startup note saying tool calls are enabled
a 4xx other than 401, 403, 408 or 429, or a connection closed before any status line	no tool calling; the schemas are dropped and the tokens reclaimed	a ! note: tool calls are off for this session, the server's answer in its own words, and the fix for your engine
no status line within 5 s, a 401/403, a 408/429, or a 5xx	nothing provable, so tools stay off	a ! note: tool calls are off for this session, which of these happened (quoting the server's message for a status), and what to do

Every row but the first leaves the session chat-only -- the model cannot read a file, run a command or edit -- and every one of them says so at startup: in the interactive session, on stderr under typed -p, and in typed acp's stderr log. Setting TYPED_CLI_NO_TOOL_PROBE skips the check, and a dim line (also on typed -p stderr) says the session has no tools because of it. A server that was down at launch gets the unreachable warning instead, which also says to restart typed once the server answers: the check runs only at launch (with TYPED_CLI_NO_TOOL_PROBE set it says instead that the session has no tools even once the server is up).

"Down" means /props did not answer within 500 ms. (On LM Studio, Ollama and an MLX server it is the endpoints typed reads there, each read within the same limit.) That is plenty for a server on the same machine, but a server on another host, or one on a machine too busy to answer quickly, can miss it and be read as down while it is up. Set TYPED_CLI_LOCAL_PROBE_TIMEOUT_MS to wait longer (a whole number of milliseconds from 500 to 60000; anything else refuses to start).

Why a streamed request, and why the status line is enough: llama-server b9426 and later send a streamed request's 200 text/event-stream headers when the slot starts the task -- after the chat template rendered the tools and the tool-call grammar was built and parsed, and before a single prompt token is read. Anything that cannot handle tools fails before that point with a 400 (b10333: "Unable to generate parser for this template", or "JSON schema conversion failed" for a schema it cannot convert). So the verdict arrives in tens of milliseconds whatever the machine's prefill speed. The probe used to be a non-streamed request with a 2 s deadline, whose answer waited for the prefill of its own ~252-token prompt: 3.0-24.5 s on a 32 GB floor machine that pages the weights, so that machine got a toolless session on every launch, and no note said so.

The timeout row is the one to watch when bringing your own server. It covers a slot still busy with an earlier request (a turn from a session you just closed runs until its current batch ends -- start typed again once the server is idle), a llama.cpp build older than b9426, which sends its headers only after reading the prompt (typed local install fetches a current one), and any server that holds its response headers until the first token. typed cannot turn tools on against a server of that last kind today, so restarting will not help; the note says so when typed has not identified the engine (a TYPED_LOCAL_ENDPOINT with no TYPED_LOCAL_ENGINE or engine pin, such as vLLM, a proxy, or LM Studio, Ollama or an MLX server moved off its default port without TYPED_LOCAL_ENGINE set). A 408 or 429 reads as a busy server, and a 503 (llama-server answers Loading model until its model is ready) as one still loading: restart typed once it is idle or loaded.

Three things a bring-your-own server still owes you, because typed cannot infer them:

• An honest window. If the window it reports (n_ctx in /props on llama-server) is wrong, typed plans every turn against a window your server does not have and the server truncates your prompt silently.
• An honest stop_reason. A reply cut off at the output cap should report max_tokens, not end_turn. When a server gets this wrong typed now notices anyway (see the truncation notice below), but the notice is a backstop, not a substitute.
• A definite answer to the tool probe, per the table above -- either "yes" or a clean refusal. Silence and ambiguity both resolve to a chat-only session, with a startup note naming which one it was.

LM Studio, Ollama and MLX servers

Three more engines are supported by name. None of them serves /props, so typed reads each one through its own endpoints instead: which model it is serving, and, where the engine reports it, the context window and whether the model can call tools. Every turn still goes to POST /v1/messages, which all three serve natively. The install and start steps for each, per operating system, are in "Set up your server" below; this section is what typed does with them.

Engine	TYPED_LOCAL_ENGINE	Default endpoint	Runs on
LM Studio	lmstudio	http://127.0.0.1:1234	macOS 14+ on Apple Silicon, Windows, Linux
Ollama	ollama	http://127.0.0.1:11434	macOS, Windows, Linux
an MLX server (vllm-mlx, MTPLX, oMLX)	mlx	http://127.0.0.1:9000	Apple Silicon Macs

Port 9000 is typed's choice, not the MLX servers': vllm-mlx and MTPLX both listen on 8000 by default, so start them with --port 9000 as below (or point TYPED_LOCAL_ENDPOINT at the port you used).

Choosing the engine. A server on its default port is found at launch with no configuration, as described under "Use it". To pin one, set TYPED_LOCAL_ENGINE=lmstudio (or ollama, mlx), put the engine name in a .typed-engine file in the project, or run /local <engine> in a session, which saves the choice for this machine (/local auto removes it, so each launch detects again). A server on another port or host needs both variables: TYPED_LOCAL_ENDPOINT for where it is, and TYPED_LOCAL_ENGINE for what it is -- without the engine typed treats the endpoint as llama-server and asks it for /props. With an engine selected, typed local prints that engine's setup instead of the llama-server command.

Which model typed asks for. Unlike llama-server, these engines route by the model field, so typed sends the name the server reports it is serving. To choose one yourself, set TYPED_LOCAL_MODEL to the server's name for it. That variable wins over what the server reports; blank counts as unset. typed never picks among several models on its own: when the server has more than one candidate and TYPED_LOCAL_MODEL is unset, the startup warning lists them and names the variable, and when TYPED_LOCAL_MODEL names a model the server does not have, the warning lists the names it does have. (llama-server and the GenieX Server (Snapdragon) serve one model and ignore the field.)

A server that speaks only the OpenAI API. typed talks to a local server in the Anthropic Messages shape (POST /v1/messages) and nothing else; it has no OpenAI adapter. Some servers, such as mlx_lm.server or a plain vLLM, answer only /v1/chat/completions. When a local /v1/messages answers 404, typed asks the server for GET /v1/models (nothing is generated), and when that answers with an OpenAI model list it says so once: the server answers the OpenAI API but not /v1/messages, so use one that speaks the Anthropic API -- llama-server, LM Studio 0.4.1 or newer, Ollama, or an MLX server such as vllm-mlx.

LM Studio

lms load <model-key> --context-length 65536
lms server start --port 1234

typed reads GET /api/v1/models and uses the LLM you have LOADED: its id is what typed sends as model, and the context length it was loaded with is the window typed plans against. Load the model before starting typed. With nothing loaded, typed warns and tells you to run lms load <key> --context-length 65536; it does not lean on LM Studio's load-on-first-request, which would load the model at a default window typed cannot see. More than one LLM loaded is the several-candidates case above.

Load a model trained for tool use. LM Studio reports that per model, and when the loaded model is not, typed turns tools off with a note saying so instead of sending its tool check. If you turned on the server's "Require Authentication" setting, set TYPED_LOCAL_API_KEY to the token; typed sends it as both Authorization: Bearer and x-api-key, and LM Studio accepts either. To change the window, load the model again with a different --context-length (or change it in the model's load settings in the app). LM Studio does not report its sampler; typed sends the session's values, and a startup note says so.

Ollama

OLLAMA_CONTEXT_LENGTH=65536 ollama serve
ollama pull <model>

Pull a model with tool support (ollama show <model> lists its capabilities). typed identifies Ollama by GET /api/version, then reads the models it has loaded (GET /api/ps) and, when none is loaded yet, the models you have pulled (GET /api/tags). One loaded model is used with its real window. With none loaded and exactly one pulled, typed uses that one, and warns that the window is learned only after the model's first load; with none pulled it tells you to ollama pull one. Several loaded, or several pulled and none loaded, is the several-candidates case above.

Set the window when you start Ollama. Ollama picks a default window from your GPU memory -- 4,096 tokens below 23 GiB, 32,768 from 23 GiB, 262,144 from 47 GiB -- and a request cannot change it. 4,096 cannot hold even typed's own prefix, so when the window Ollama reports is under 16,384 tokens typed warns and tells you to restart it with OLLAMA_CONTEXT_LENGTH=65536 ollama serve (a Modelfile num_ctx works too). "Set up your server" shows how to make that setting stick on each operating system, where Ollama usually runs as a desktop app or a service rather than from your shell.

Ollama ignores min_p and presence_penalty; it samples with temperature, top_p and top_k only, and a startup note says so. It also ignores the setting typed uses to turn the model's thinking off, and turns thinking ON for a model that supports it unless told otherwise, so in the default instruct mode (and precise) typed also sends thinking: {"type": "disabled"} to Ollama. The tool check works as on llama-server: a model without tool support is refused with does not support tools before anything streams, and the note says to check ollama show <model>. A model not found refusal means the name is not pulled: check ollama list and TYPED_LOCAL_MODEL.

MLX servers (Apple Silicon)

The mlx engine is any MLX server on port 9000 that serves POST /v1/messages: vllm-mlx (what typed local prints, and the one for several sessions or parallel subagents), MTPLX (faster for a single session), or oMLX. typed identifies it by GET /v1/models, the OpenAI-style model list these servers publish, and sends the one chat model listed there (or TYPED_LOCAL_MODEL; more than one listed is the several-candidates case above). Flags below that belong to one server are named with it.

uv tool install vllm-mlx
vllm-mlx serve mlx-community/Qwen3.6-35B-A3B-4bit --port 9000 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --default-min-p 0.0 --default-presence-penalty 1.5

That is the MLX build of the same model the llama-server command runs, and typed local with TYPED_LOCAL_ENGINE=mlx prints this command with the sampler values for the session's mode filled in (--default-presence-penalty 0.0 under TYPED_LOCAL_MODE=thinking), followed by the MTPLX alternative. vllm-mlx takes min_p and presence_penalty only from those two launch flags and ignores them per request, the same reason the llama-server command carries its own pair. MTPLX also ignores a request's presence_penalty and has no min_p at all. A startup note says which applies. vllm-mlx answers a request for a model name it does not serve with a 404; restart it with --served-model-name, or set TYPED_LOCAL_MODEL to the name it serves. MTPLX answers any name with its loaded model.

The window. MTPLX reports its window and typed plans against it: the window this Mac's memory fits (execution_window in /health) when it reports one, else context_length in /v1/models. vllm-mlx does not report one anywhere, so typed assumes 65,536 tokens and says so at startup; set TYPED_CLI_CONTEXT_WINDOW to the window you want typed to plan against, and the warning goes away. vllm-mlx has a --max-kv-size flag, and it is deliberately not in the command: it bounds memory by switching to a rotating cache that silently drops the start of a long context, which is exactly the truncation typed's own context guard is there to prevent. If you set it anyway, set TYPED_CLI_CONTEXT_WINDOW to the same value, so typed compacts before the server starts dropping. typed also caps each request's output at 32,768 tokens, which is vllm-mlx's own default per-request limit (--max-request-tokens) -- a larger request is refused by that server outright.

Tools need two vllm-mlx flags. vllm-mlx accepts a request carrying tools even when it was started without a tool parser, and then returns the model's tool calls as plain text. typed's tool check cannot tell the difference, so when it passes, a note reminds you that a vllm-mlx server must be started with --enable-auto-tool-choice --tool-call-parser <parser>, as in the command above. MTPLX parses tool calls with no flag.

What these engines do not have

• No prompt-progress readout. typed follows a long prompt evaluation through llama-server's /slots, which none of the three serves, so a slow first token shows no running count of prompt tokens read.
• No RAM plan. typed local sizes llama-server only; with these engines the memory the model needs is the engine's business (for Macs, see "Apple Silicon Macs: memory and context" under "Set up your server").
• No --ctx-size. When typed says the window is too small, the fix it names is the engine's own: lms load <key> --context-length <N> on LM Studio, a restart with OLLAMA_CONTEXT_LENGTH=<N> ollama serve on Ollama, and on an MLX server a relaunch with a larger window (MTPLX: --context-window <N>) -- or, on vllm-mlx, which reports no window to read, relaunching typed with TYPED_CLI_CONTEXT_WINDOW=<tokens>.

The other path: a self-hosted API

There is a second, unrelated way to reach a local model: a deployment that runs typed's own API next to a model server, and sets TYPED_LOCAL_BASE_URL in the API's environment. Most people want the CLI path above instead.

export TYPED_LOCAL_BASE_URL=http://127.0.0.1:9931/v1   # NOTE: includes /v1
export TYPED_LOCAL_API_KEY=...                          # only if started with --api-key

The two variables use opposite URL conventions -- TYPED_LOCAL_ENDPOINT is an origin, TYPED_LOCAL_BASE_URL includes /v1 -- because the two code paths append different suffixes. Swapping one for the other 404s every request, which is why they are separate names.

Unset on the API side, the slot stays unwired and a typed-local request that reaches api.typed.cloud returns 503 config_error / router_misconfigured rather than being silently rerouted onto a paid carrier. (Before 23 September 2026 that was also what a bare typed request got there; since the rename a bare typed is the hosted economy tier and is served.)

Set up your server

typed needs one model server running on your machine that answers the Anthropic Messages API (POST /v1/messages) with tool calls. Pick one from the table, follow its steps for your operating system, and start typed. Each guide ends the same way: a curl that proves the server answers, and the lines typed prints at startup.

Server	Windows	macOS	Linux	Port typed expects
llama-server (llama.cpp)	yes	yes	yes	9931
LM Studio (0.4.1 or newer)	yes (x64, ARM64)	Apple Silicon, macOS 14+	yes (x64, ARM64)	1234
Ollama	yes	yes (macOS 14+)	yes	11434
MLX servers (vllm-mlx, MTPLX)	no	Apple Silicon only	no	9000
GenieX Server (Snapdragon)	Snapdragon X only	no	no	8123

Three things hold for all of them:

• The model needs tool support. Without it the session can only chat; typed checks at launch and says so in a ! line.
• The window should be 65,536 tokens. That is the window typed local prints for llama-server, and the one each guide below sets. typed's own prompt is about 8,000 tokens before your first word, so a server's small default window is unusable.
• Pointing typed at the server is automatic when it is on the port in the table and nothing else answers: start typed and it finds it. To choose explicitly, run /local <engine> in a session (saved for this machine) or set TYPED_LOCAL_ENGINE (llama-server, lmstudio, ollama, mlx or snapdragon). On another port, set TYPED_LOCAL_ENDPOINT as well. Then run typed --model typed-local (or plain typed without a paid plan).

What typed prints when it works. At startup a local session prints the server it reached and the model it is serving, tier typed-local -- the server at http://127.0.0.1:<port> is serving <model>, and a tool line that starts (local tool calls: enabled. typed status prints the same server with its window on its local server: line. A ! line instead names what is missing and the fix.

On Windows, run the curl checks below as curl.exe in PowerShell: in Windows PowerShell 5.1 a bare curl is an alias for Invoke-WebRequest. Multi-line commands here end lines with \, which PowerShell does not read as a continuation: paste them as one line, or replace each \ with a backtick.

llama-server (llama.cpp)

This is the server the rest of this page describes: typed local sizes its command to your machine.

Let typed install it. typed local install downloads the newest CPU build of llama.cpp for your platform into ~/.local/share/typed/llama.cpp (on Windows, %USERPROFILE%\.local\share\typed\llama.cpp) and does not add that folder to your PATH: either add it, or run llama-server by the full path the install prints. Then run typed local for the command.

Or install llama.cpp with your platform's package below, then run typed local for the launch command. The model is not part of the install; the command's -hf downloads it from Hugging Face the first time the server starts (about 19.5 GiB). Run the command typed local prints, as printed: it carries the window, thread counts and sampler settings for your machine and session mode (see "1. Start llama-server" above), and it passes --port 9931, the port typed dials. Build b10333 or newer is required; older builds refuse its flags.

Windows.

winget install --id ggml.llamacpp -e

That package is the Vulkan (GPU) build; the command typed local prints keeps the model on the CPU with --device none, which is the configuration typed measured. For the plain CPU build, download llama-<build>-bin-win-cpu-x64.zip (or -win-cpu-arm64.zip on Snapdragon and other ARM64 machines) from the llama.cpp releases page. Those builds are marked pre-release, so pick from the list rather than the "Latest" badge. On ARM64, use the ARM64 build: the x64 one runs under emulation and is far slower.

macOS.

brew install llama.cpp

MacPorts (sudo port install llama.cpp) and Nix (nix profile install nixpkgs#llama-cpp) work too. Mac builds include Metal, but typed local's command passes --device none, so the model runs on the CPU as measured; see "GPU offload" above for what dropping it buys.

Linux.

brew install llama.cpp

Homebrew on Linux has builds for x86_64 and arm64. Otherwise: Debian testing and unstable package it as sudo apt install llama.cpp-tools, which installs llama-server; Fedora's package is current only in Rawhide (Fedora 43's is older than b10333, so do not use it); Nix is nix profile install nixpkgs#llama-cpp; and the releases page has llama-<build>-bin-ubuntu-x64.tar.gz and -ubuntu-arm64.tar.gz builds, plus Vulkan, CUDA and ROCm variants.

Start it and point typed at it. Run the command typed local prints. llama-server is the default engine, so nothing else needs setting.

Check it.

curl http://127.0.0.1:9931/health
curl http://127.0.0.1:9931/props

/health answers once the model has loaded. In /props, default_generation_settings.n_ctx is the window the server is running with, which is what typed will plan against.

The window. llama-server runs at the --ctx-size it was started with; the command typed local prints sets 65,536 when your RAM allows it. Change it with typed local --ctx-size <tokens> or typed local --max-window, which check the size against your RAM, rather than by editing the command.

LM Studio

LM Studio 0.4.1 or newer serves /v1/messages. It runs on Apple Silicon Macs with macOS 14 or newer (not Intel Macs), on Windows x64 (with AVX2) and ARM64, and on Linux x64 and ARM64 (Ubuntu 20.04 or newer).

Windows.

winget install --id ElementLabs.LMStudio -e

macOS.

brew install --cask lm-studio

Linux. Download the AppImage and run it:

curl -L -o LM-Studio.AppImage https://lmstudio.ai/download/latest/linux/x64
chmod +x LM-Studio.AppImage
./LM-Studio.AppImage

On ARM64, use https://lmstudio.ai/download/latest/linux/arm64 instead.

Get the lms command. It ships with the app: launch LM Studio once, open a new terminal, and check lms --help. If it is not found, npx lmstudio install-cli adds it to your PATH.

No desktop (servers). LM Studio's headless build installs lms without the app. On Windows, irm https://lmstudio.ai/install.ps1 | iex; on macOS and Linux, curl -fsSL https://lmstudio.ai/install.sh | bash. The installer does not start the service, so open a new terminal and run lms daemon up before the steps below.

Get a model with tool support, load it at 65,536, and start the server (the same on every system):

lms get openai/gpt-oss-20b
lms load openai/gpt-oss-20b --context-length 65536
lms server start --port 1234
lms ps

openai/gpt-oss-20b is one model LM Studio marks as trained for tool use (it wants at least 12 GB of memory); any model with that mark works. lms ls lists the keys of the models you have downloaded, and lms ps should show exactly one model loaded. Always pass --port 1234: without it LM Studio reuses the last port it used. In the app, the context length is also in each model's load settings.

Point typed at it. On port 1234 typed finds it at launch; to pin it, run /local lmstudio or set TYPED_LOCAL_ENGINE=lmstudio. With more than one model loaded, set TYPED_LOCAL_MODEL to the one to use. If you turned on "Require Authentication" in the server settings, create a token there and set TYPED_LOCAL_API_KEY to it.

Check it.

curl http://127.0.0.1:1234/api/v1/models

In the loaded model's entry, loaded_instances[0].config.context_length should read 65536 and capabilities.trained_for_tool_use should be true.

The window. LM Studio loads a model at the window you load it with. A model loaded on the first request instead (just-in-time loading) gets a default window typed cannot see, so always lms load with --context-length 65536 before starting typed. To change it, load the model again with a different --context-length.

Ollama

Windows.

winget install -e --id Ollama.Ollama

or irm https://ollama.com/install.ps1 | iex, or the installer from ollama.com/download. Ollama runs in the tray and serves on port 11434 from then on.

Set the window for good: quit Ollama from the tray icon, set a user environment variable, then start Ollama again from the Start menu (and open a new terminal, so the ollama command sees the variable too):

[Environment]::SetEnvironmentVariable("OLLAMA_CONTEXT_LENGTH", "65536", "User")

That is PowerShell. In the Settings app the same thing is "Edit environment variables for your account": add OLLAMA_CONTEXT_LENGTH with the value 65536.

macOS. Install the app from ollama.com/download (drag it to Applications; on first launch it offers to install the ollama command), or with Homebrew:

brew install --cask ollama-app

Set the window, then quit and reopen the Ollama app:

launchctl setenv OLLAMA_CONTEXT_LENGTH 65536

The app's settings also have a context-length slider. Ollama needs macOS 14 or newer; on an Intel Mac it runs on the CPU only.

Linux.

curl -fsSL https://ollama.com/install.sh | sh

The script installs Ollama as a systemd service named ollama. A variable exported in your shell does not reach that service, so set the window in a service override:

sudo systemctl edit ollama.service

and add:

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=65536"

then:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Without systemd, start it yourself with OLLAMA_CONTEXT_LENGTH=65536 ollama serve. (Homebrew's Linux ollama formula is CPU-only.)

Get a model with tool support (the same on every system):

ollama pull gpt-oss:20b
ollama show gpt-oss:20b

ollama show lists tools under Capabilities when the model can call tools. gpt-oss:20b is one such model (a 14 GB download); ollama.com's model search can filter by tool support.

Point typed at it. On port 11434 typed finds it at launch; to pin it, run /local ollama or set TYPED_LOCAL_ENGINE=ollama. When typed's startup warning lists several models, set TYPED_LOCAL_MODEL to the one to use (a name ollama list shows).

Check it.

curl http://127.0.0.1:11434/api/version
curl http://127.0.0.1:11434/v1/messages -H "Content-Type: application/json" -H "x-api-key: ollama" -H "anthropic-version: 2023-06-01" -d '{"model":"gpt-oss:20b","max_tokens":64,"messages":[{"role":"user","content":"Say hello."}]}'
ollama ps

After that request has loaded the model, ollama ps shows its window in the CONTEXT column (65536) and where it runs in PROCESSOR: 100% GPU is what you want, and a CPU/GPU split means the window did not fit in GPU memory and turns will be slow -- pick a smaller model rather than a smaller window. In PowerShell, the JSON body needs its inner quotes escaped (or put it in a file and pass -d "@body.json").

The window. Ollama picks its default from GPU memory -- 4,096 tokens below 23 GiB, 32,768 from 23 GiB, 262,144 from 47 GiB -- and a request cannot change it, which is why each system above sets OLLAMA_CONTEXT_LENGTH. typed warns at startup when the window Ollama reports is under 16,384 tokens.

MLX servers (Apple Silicon Macs)

MLX runs only on Apple Silicon Macs (M1 or newer). On Windows and Linux there is no MLX server: use llama-server, LM Studio or Ollama.

typed's mlx engine is whichever MLX server answers on port 9000. Pick one:

• vllm-mlx, the default: what typed local prints, and the one to use for several typed sessions at once or parallel subagents.
• MTPLX, for the fastest single session: it runs one request at a time (others queue), using its own builds of the model.

vllm-mlx. Install it (Python 3.10 or newer), then start it with the command typed local prints for your session mode:

uv tool install vllm-mlx
TYPED_LOCAL_ENGINE=mlx typed local

For the default instruct mode, that command is:

vllm-mlx serve mlx-community/Qwen3.6-35B-A3B-4bit --port 9000 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --default-min-p 0.0 --default-presence-penalty 1.5

The first start downloads the model (about 20 GB) from Hugging Face. --port 9000 matters: vllm-mlx's own default is 8000. The two tool flags make its tool calls arrive as tool calls rather than plain text. Options worth knowing, none needed for one session:

• --continuous-batching (off by default, and slower for a single user) serves several typed sessions or parallel subagents at once. Prefix caching is on by default either way.
• --gpu-memory-utilization (default 0.90): lower it if macOS starts swapping.
• --kv-cache-quantization shrinks the cache that holds the conversation.
• --max-kv-size is left out on purpose: it makes the server drop the start of a long conversation without saying so. If you set it, also set TYPED_CLI_CONTEXT_WINDOW to the same number.

vllm-mlx does not report its window, so tell typed the window to plan against:

TYPED_CLI_CONTEXT_WINDOW=65536 typed --model typed-local

MTPLX. Install it with Homebrew, or with pip on Python 3.11 or newer (macOS 14 or newer):

brew install youssofal/mtplx/mtplx

or python3 -m pip install -U mtplx. Then, for the default instruct mode (the command typed local prints):

mtplx serve --model Youssofal/Qwen3.6-35B-A3B-MTPLX-Optimized-Speed \
  --download --port 9000 --no-auth \
  --default-presence-penalty 1.5

In thinking mode the last line is --default-presence-penalty 0.0.

--download fetches the model on first start (about 21 GB; without it an uncached model fails), --port 9000 replaces its default 8000, and --no-auth keeps a key from a config file or the environment from refusing typed's requests. That 35B build peaks at about 28 GiB of memory, measured on short prompts (long ones need more), so it wants a Mac with at least 32 GB, ideally more. MTPLX reports its window, so typed needs no TYPED_CLI_CONTEXT_WINDOW; --context-window <tokens> sets it. MTPLX ignores a request's presence penalty and has no min_p, which is why the command sets --default-presence-penalty for the session mode.

MTPLX publishes each memory tier as its own model, chosen with --model. Peak memory at short prompts:

Model	Tier	Peak
35B-A3B (the one above)	Optimized Speed, 4-bit	28 GiB
35B-A3B	Optimized Balance, 6-bit	32 GiB
27B	Bare Speed	20 GiB
27B	Optimized Speed, 4-bit	25 GiB
27B	Optimized Quality, 8-bit	33 GiB

On a 36 GB Mac stay at or under about 25-28 GiB: the 27B Optimized Speed build is Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed.

oMLX also serves /v1/messages (macOS 15 or newer). It listens on 8000, so point typed at it with TYPED_LOCAL_ENDPOINT=http://127.0.0.1:8000 and TYPED_LOCAL_ENGINE=mlx. typed recognizes it by the model list its /v1/models serves; oMLX documents that endpoint, but typed has not been tested against oMLX.

Point typed at it. On port 9000 typed finds it at launch; to pin it, run /local mlx or set TYPED_LOCAL_ENGINE=mlx.

Check it.

curl http://127.0.0.1:9000/v1/models

The list should hold one chat model; its id is the name typed sends. (On MTPLX the entry also carries context_length; typed reads execution_window in /health first, the window this Mac's memory fits.)

Apple Silicon Macs: memory and context

These apply to the servers that run the model on the Mac's GPU: MLX servers, LM Studio and Ollama. They do nothing for llama-server as typed local prints it, which runs on the CPU (--device none; see "GPU offload" above) -- only if you removed that flag.

Let the GPU use more of the unified memory. By default macOS keeps part of the memory away from the GPU. Raise the limit (this is the value for a 36 GB Mac):

sudo sysctl iogpu.wired_limit_mb=30720

Generally: your total memory in MB minus a few GB for macOS. Leaving macOS too little can crash the machine. The setting takes effect at once (restart an app such as LM Studio so it sees the new limit) and resets when the Mac restarts. To go back to the default:

sudo sysctl iogpu.wired_limit_mb=0

Size the context to the memory. On a 36 GB Mac with a 4-bit 27B build, a 32K-token context is comfortable, 64K needs the raised limit above, and 128K does not fit -- nor does any 8-bit 27B build. Those are estimates for that pairing, not measurements; a bigger Mac has room for more.

M5 Macs. MLX uses the M5's neural accelerators only on macOS 26.2 or newer.

GenieX Server (Snapdragon)

The GenieX Server (Snapdragon) runs a small model on the Snapdragon NPU. It runs only on Windows 11 on a Snapdragon X Elite or X Plus (ARM64). There is no macOS or Linux version; Linux or WSL is used only to build a model bundle, which you then copy to the Snapdragon machine. It is a research project, tested on one machine.

Setup is on the project's own page, which is the authority: github.com/YawLabs/snapdragon-genie-server. In short:

1. Install a native ARM64 Python (the server needs no pip packages):

   winget install --id Python.Python.3.14 --architecture arm64
   

2. Clone the server:

   git clone https://github.com/YawLabs/snapdragon-genie-server
   

3. Get Qualcomm's QAIRT 2.45 runtime and a model bundle built for your chip, laid out as the project page describes. A bundle is tied to one chip generation and one QAIRT version.

4. Start it from the clone with the launcher:

   powershell -File src\run-genie-server.ps1
   

   It prints port 127.0.0.1:8123 is reserved at once, refuses connections while the model loads (11-35 s), then prints endpoint on http://127.0.0.1:8123. Use the launcher rather than running the Python file directly, which listens on a different port. If PowerShell's execution policy blocks the script, run it as powershell -ExecutionPolicy Bypass -File src\run-genie-server.ps1.

Point typed at it. On port 8123 typed finds it at launch; to pin it, run /local snapdragon or set TYPED_LOCAL_ENGINE=snapdragon.

Check it.

curl.exe http://127.0.0.1:8123/health
curl.exe http://127.0.0.1:8123/props

/health answers 200 when the server can actually generate. In /props, default_generation_settings.n_ctx is the window, which typed reads as it does on llama-server.

The window. It is compiled into the model bundle when the bundle is built: 8,192 tokens for the default bundle, and no setting changes it at run time. 65,536 is not possible on this server. A larger window needs a bundle built for it, or llama-server. Tool calls work when the bundle's model supports them; otherwise the server refuses tools and typed runs the session chat-only, with a startup line saying so.

What this tier does differently

	free local (typed-local)	paid hosted (typed++, typed)
Cost	$0, free (your hardware)	metered per token
Quota	accrues nothing	counts against the monthly ceiling
Auth	no typed login needed	API key required
Where it runs	your machine	typed's upstreams
Context cap	7,500 fallback, or 92% of the probed llama-server window	1M on all hosted tiers
Oversized request	413	413
Breaker open	503, no failover	fails over to another carrier (except typed, which 503s -- that tier has no failover carrier)
Knowledge prefix	skipped	injected
x-typed-skill / x-typed-codebase	400	honored
Active prefix bundle	skipped, with a warning header	injected

Several of those rows are the same decision seen from different angles: every failover or prefix target is a PAID carrier, and a request you chose because it was free must never quietly become a billed one. Failing honestly with a 413 or a 503 is what a free tier owes its users; silently upgrading onto a metered model is not. The prefix layers are also simply too large for the window -- a skill prefix alone can reserve up to 16,896 tokens and the codebase prefix 8,192, against a default cap of 7,500. Neither fits even before your prompt is counted, which is why the headers are rejected outright rather than quietly dropped.

Two notes on the context-cap row. The 7,500 figure is a FALLBACK, not the tier's real window: the API adopts the server's own number when it can probe it, taking 92% of it to absorb tokenizer drift and the prefix injections that land after the gate (MAX_CONTEXT_TOKENS_LOCAL and LOCAL_WINDOW_SAFETY_FRACTION in apps/api/src/routes/messages.ts), and the first-party CLI probes /props at startup and uses the real number directly. On this tier max_tokens also comes OUT of that budget before your prompt is measured, because llama.cpp reserves the output allocation up front -- a hosted-sized max_tokens will squeeze the prompt out of a small window. And the paid column says 413 rather than "upgrades" because no paid tier has anywhere to upgrade to: all hosted tiers carry the same 1M window, so an oversized paid request 413s too.

An active prefix bundle is the one layer that is neither server-configured nor asked for on the request -- you register it ahead of time on your account -- so it is skipped rather than rejected: a 400 would fail every free-local request until you deactivated the bundle account-wide. Because the withheld text is your own, the skip is never silent. The response carries x-typed-prefix-bundle-warning: bundle_skipped_on_local, in the same slot where an applied bundle sets x-typed-prefix-bundle: <bundleId>.

Timeouts are much longer than the hosted defaults (2min TTFT, 15min non-streaming response) and retries are off. Local inference is slow, and re-submitting a multi-minute prompt eval only queues the same work behind itself.

Guardrails specific to this tier

The free tier runs a smaller model with a much smaller window, so the harness enforces limits the hosted tiers need far less. These are mechanical -- they do not depend on the model following an instruction, which a 4-bit 35B does unreliably. Rows marked (every tier) run on the hosted tiers too; this is the tier where they decide whether a session can keep going at all. Every figure below is at the 65536 window typed local prints.

Guardrail	Why
output cap sized per turn to the room the prompt leaves (every tier)	llama.cpp reserves max_tokens out of n_ctx before generating; a hosted-sized 16K cap beside a nearly full window squeezes out the prompt, so the model answers about code it can no longer see. When the cap drops, a notice says why and what to do: output limited to 15293 tokens this turn (of 16384): the prompt is ~45000 of the 65536-token window -- nothing older has been evicted this run; /compact frees room, or restart llama-server with a larger --ctx-size .... That is the interactive session's wording; outside it (-p, an editor session over ACP, a subagent) there is no /compact to type, so the line names what the context guard does on its own instead, and it says which of eviction and compaction has already run. It prints on the first reduction of a run and after that only when the cap has halved again or reached the 1,024 floor, not on every turn of a climbing prompt. The requested cap is 16,384 unless TYPED_CLI_MAX_TOKENS sets it (a whole number from 1024 to 64000; anything else refuses to start). On a 16K window the default request never fits, so the notice prints on the first turn of every run; a lower value such as 4096 makes it print only when a reply is really squeezed
tool results spilled to the room left (every tier)	a turn's results are sized to what the window still has after the prompt and the reply, not to a fixed share. A result over that keeps its head AND its tail (a command's exit code sits at the end) with a marker naming the lines kept and the ReadOutput(id: "...", offset, limit) call that pages the dropped lines (ReadOutput(id: "...") for the whole text when the tool had already cut what it returned); you get a tool results spilled to fit the context window line: in the interactive session it names the /show ids, elsewhere (-p, ACP) the ReadOutput ids the model can page, since there is no /show to type. A subagent's spill line reaches you too, labelled with its task. With no ReadOutput tool on the wire the marker says the rest is not retrievable and to re-run the call narrower. A Read page is the exception: it keeps an unbroken run of whole lines from the top, and its marker names the file lines kept and the Read(file_path, offset, limit) call that returns the rest of the page, in the file's own line numbers. The room is deliberately not capped lower to shorten prompt eval: it is the same line eviction uses, so a lower cap would only cut more of what the model asked for and send it back through later turns, where it is evaluated as prompt again
older tool results evicted before anything is summarized (every tier)	past 85% of the window less the reply's room (~47.5K tokens), the oldest tool-result bodies become one-line stubs naming the call and how to get it back (its ReadOutput id, which still pages after --resume because outputs are saved beside the transcript, or a re-run), oldest first, down to 40% of the window (~26K). Both lines are in real tokens: the stop line is divided by the tokenizer drift the session last measured (real prompt tokens over the char estimate), which is carried across runs, evictions and compactions, so a dense CJK or base64 session is not stopped at a line it is still twice over. No model turn is spent, where a summary costs a whole one. An evicted Read makes Edit and Write refuse that file until the model reads it again, unless another successful read of it is still in context or the model Wrote the file after that read; an in-loop compaction does the same for every file. A stub offers a plain re-run only for a read-only call -- anything else is to be re-run only if repeating it is safe. The rewrite changes the prompt partway through, so the next turn re-reads it from the server's newest context checkpoint at or before the first stub (not from the stub itself; from the start if no checkpoint precedes it) -- the notice says to expect a longer prompt eval -- and the low line sits at 40% so each batch buys several turns
the summarizer reuses the session prefix (every tier)	when eviction leaves nothing to stub, compaction sends the session's own system block and tool schemas ahead of the history, so the server keeps that prefix instead of re-reading the whole prompt for the summary and again for the turn after it. On this tier it keeps the conversation warm too when the whole history fits the request: preserve_thinking stops the chat template re-rendering earlier replies, and the request keeps their reasoning as it was sent (218 prompt tokens evaluated in instruct mode where 6,713 were re-read without the flag; see the --cache-reuse note). When the tail has to be cut to fit -- which a compaction the 85% line triggers almost always is -- the request diverges right after the tool schemas and the kept tail is re-read. The summarized tail is sized to the smaller of the window and that request's own input budget, scaled by the session's measured tokenizer drift. A summarizer that answers with a tool call gets one retry with that call refused, appended after the first attempt's prompt rather than rebuilt, so the server can resume from its newest checkpoint before the appended turns instead of re-reading the history (inferred from the render order, not yet measured); if it calls a tool again, auto-compaction is switched off for the rest of that run and a notice says so. A summary cut off at its 2,000-token limit is marked in the summary and announced
compact tool descriptions, no Progress tool	the tool block rides in every cold prefill; names and parameters are unchanged and only the prose is shorter (the startup prefix ledger gives both sizes). The Progress task list is not offered because each update costs a whole local turn; TYPED_CLI_LOCAL_PROGRESS=1 offers it
hook context budgeted by whole sections	one hook event's injected context gets 10% of the window, at most 24,000 chars (every tier). Over that, whole ### sections are kept in the order the hooks ranked them and the rest are named in the prompt with their size and ReadOutput(id: "..."), spilled to the session's output store; a section is cut mid-way only when none fits whole. On every tier (TYPED_CLI_HOOK_DEDUPE=0 turns it off) a section that an earlier user turn in the last 65,536 tokens of the conversation (an image counts as 1,600 tokens) already carries whole, ending where a section ends, is sent as a one-line pointer instead of a second copy
startup prefix ledger	a line like (prefix: harness ~0.8K + tools ~2.7K (compact; ~3.9K full) + CLAUDE.md ~3.3K + memory ~0.2K + skills ~0.6K + SessionStart hook ~0K = ~7.7K of 65536 tokens; per-turn hook context up to ~6K) shows what every cold prefill carries before your first word, so you can see which term to shrink
prefix-cache miss notice	/status shows how much of the last request the server reused (the status bar carries no cache figures). When a turn re-reads over half of what it could have reused (the smaller of its prompt and the previous one), and that is 4,096 tokens or more, a notice gives the size and the cause typed can see: prefix cache miss: the server re-read ~X of ~Y prompt tokens (...), where the cause is a history rewrite (an eviction, a compaction, or a summarization request whose tail was cut), a tool-list change, a model switch, or -- when nothing on typed's side changed -- a restarted server or another client on the slot. A miss on a request that added a user message (a new prompt, a queued comment, the summarization instruction) adds a third possibility typed cannot rule out, a server that ignored preserve_thinking, and says so: the template no longer re-renders earlier replies on the command typed local prints, so on that setup such a miss is not expected (see the --cache-reuse note). For a request that did not send the flag the cause reads the chat template can re-render earlier assistant turns after a new user message -- "can", because that re-render is measured for the shipped model's template only, and a server restarted between two prompts looks the same. The summarization requests of /compact and of a resumed -p run's compaction are observed and named the same way. A server that never reads anything back gets one note for the session, advising a smaller window: TYPED_CLI_CONTEXT_WINDOW, or on llama-server a smaller --ctx-size
compaction trigger scaled to the window	60% of what the window leaves after the static prefix (~34K here); the fixed 120K default is nearly twice the whole local window and would never fire. The hosted tiers (typed, typed++) do not scale it: they keep history to a fixed 450,000 tokens whatever the window, checked at every turn, with older tool results evicted before anything is summarized
Edit refuses an unread file	an old_string the model never observed was imagined, and a lucky match applies cleanly to real code
turn limit of 30 (hosted tiers: none unless --max-turns names one)	a turn is minutes, and a small model that starts confabulating spirals rather than recovering. A run that reaches it pauses with paused after 30 turns (safety limit) -- nothing is lost; type "continue" to keep going, or relaunch with --resume <id> --max-turns <n>, where <id> is this session's id; the count restarts on every prompt. --max-turns <n> (1-1000) is the only thing that raises it on this tier, and it is read at launch -- --resume <id> resumes the same session under the new limit
subagents disabled	a single-slot server queues them behind the main turn, and they start without the parent's context. When the ultracode marker is on or this project has subagent definitions in .claude/agents, a session that sends tools, with a --tools list (if any) that lets Agent through, says so in one line at launch and on a switch onto the tier (in the interactive session and the ACP log; typed -p prints only ! lines); a --tools list that names Agent gets its own ! --tools line instead, which says the same. The paid hosted tiers (typed, typed++) run them
truncation notice when a reply fills the output cap	not every server reports stop_reason: max_tokens when it cuts a reply off at the cap -- some report end_turn, which makes a truncated answer indistinguishable from a finished one

That last row is the one guardrail written for servers rather than for models, so it is worth saying what it does and does not claim. When a turn ends on any reason other than max_tokens or tool_use but the reply consumed every output token that turn was allowed, typed emits a notice saying the answer may have been cut off. It reads the output token count, which most servers report, rather than the stop reason, which not every server gets right -- so it costs a correct server nothing (llama-server reports max_tokens properly and never trips it) and catches the case on one that does not (the GenieX Server (Snapdragon) reports a normal sentence-end at the cap, and vllm-mlx's stream never reports max_tokens at all). It stays a notice: the turn still completes and the stop reason is still reported verbatim, because a model CAN legitimately finish on its last allowed token and typed has no way to tell the two apart.

Two limits on it, both deliberate. A server that reports no output tokens never trips it -- the check needs a count to compare, and failing quiet beats a notice on every turn. And like the rows marked (every tier) it is not gated to the local tier, for a reason of its own: it is written for servers whose stop reasons cannot be trusted, and that is a property of a server rather than of a tier. A hosted turn reports max_tokens correctly and so never reaches the check, which makes the gate unnecessary rather than missing -- do not "fix" it by adding one.

Tool-call output is already grammar-constrained by llama.cpp whenever tools are sent, so malformed tool JSON is structurally impossible rather than merely discouraged. That needs no configuration.

An untested lever

TYPED_LOCAL_MODE=precise lowers sampling entropy (temperature 0.3 vs the default 0.7) on the theory that a confidently wrong tool argument costs more than a dull phrasing. It is opt-in and unmeasured: the model card publishes no non-thinking coding row, so this is reasoning about sampling rather than a finding. Try it on a real task and compare. It is deliberately not greedy -- its authors warn that temperature 0 degenerates into repetition.

Troubleshooting

A request carrying tools gets a 400, or (older builds) the connection drops (ECONNRESET, or HTTP 000 from curl). This is a llama.cpp tool-call failure, not a typed bug, and typed's startup check reports it as "local tool calls are off for this session". On b10333 it is a 400 sent before any stream headers, and its message names the cause: "Unable to generate parser for this template" means the loaded model's chat template has no tool-call form (a different model, or a --chat-template-file that has one), and "JSON schema conversion failed" names a tool schema the build cannot convert (a newer build). Build b10326 dropped the connection instead -- observed with a 27B dense build of the same family, and gone on b10331+ with the 35B-A3B MoE. Confirm either shape by sending the same request with "tool_choice": "none" -- if that succeeds, grammar construction is the fault. A 500 reading "tools param requires --jinja flag" is a server started with --no-jinja: drop the flag.

Do NOT use chat_format from /props as the diagnostic. It reads "Content-only" on a server where tool calling demonstrably works -- it reflects the DEFAULT generation params (no tools in the request), not whether tool grammar can be built. The reliable signal is the refusal itself, reproducible at max_tokens: 1.

Also note --chat-template <name> only accepts the built-in legacy template names, which disable jinja tool calling entirely -- it is not a fix. The per-request chat_template_kwargs field IS honored, which is how typed drives enable_thinking per session without restarting the server, and sends preserve_thinking with it.

Every request 413s, including tiny ones. The prefix reserves exceeded the tier's window. Should be impossible now (the knowledge reserve and any active prefix bundle are skipped, and the customer-requested layers 400 on this tier), but if it recurs the API returns a config_error with code context_budget_exceeds_window naming the cause instead of a misleading "request too large".

503 config_error / router_misconfigured. The request reached api.typed.cloud instead of your own server. Through the typed CLI that should be impossible -- if you see it, the turn went out over the hosted path, so check that the model really resolved to typed-local (typed status). A bare typed is the hosted economy tier since 23 September 2026, so a session on it does not reach your server at all and bills instead.

ECONNREFUSED / connection refused. The CLI reached for your model server and found nothing there. Confirm llama-server is up (curl http://127.0.0.1:9931/health) and that TYPED_LOCAL_ENDPOINT matches its port. On another engine, start it with the steps in "Set up your server" (lms server start --port 1234, ollama serve or the Ollama app, or an MLX server with --port 9000) and check that TYPED_LOCAL_ENGINE or your /local choice names the engine you started.

"answers the OpenAI API but not /v1/messages". The server on that port speaks only OpenAI's /v1/chat/completions (for example mlx_lm.server or a plain vLLM). typed needs the Anthropic Messages API and has no adapter; run one of the servers in "Set up your server" instead.

Boot warns about n_ctx. The API-side path enforces a constant (MAX_CONTEXT_TOKENS_LOCAL) that is smaller than what your server serves. Harmless for CLI sessions -- the CLI reads the real n_ctx from /props at startup and sizes itself to whatever you launched with -- but the API path would 413 earlier than necessary.