002 / PRISM
CPU inference engine
A plugin-based runtime for inference and streaming models in C99: kernels, architectures, tokenizers, samplers, the generation loop and the frontend are all replaceable at runtime.
Measured, not marketed
- 20.9 tok/s
- TinyStories 15M, AVX28-core x86-64, greedy; 5.3 on reference kernels
- 4.2 tok/s
- TinyLlama 1.1B, Q4_0929 MB peak RSS
- 21
- test suitesall passing
- ~500
- lines of core registrythe entire framework
01Everything is a plugin
There is no framework to learn. The core is a registry of about five hundred lines — layered config, service lookup, an event bus, a dlopen loader and ordered composition — and every capability is a plugin behind it. Kernel, model, tokenizer, sampler, generation loop and frontend are each selected at boot, so swapping the backend is a config change rather than a fork.
That shape is what makes the engine legible: a plugin that does not implement an operation cannot silently inherit a default, and the reference kernels stay in the tree as the correctness oracle for the optimized ones.
02Formats and caches
Checkpoints come in as llama2.c, prism-pack q8_0 or GGUF v2/v3, and the pipeline plugin can detect, pack, benchmark and gate a model in one command. The KV cache is content-addressed and paged, with prefix sharing so a shared prompt is computed once.
For mixture-of-experts models, expert weights stream through a byte-budgeted LRU cache with a Markov-model prefetcher — the working set of a sparse model does not have to fit in RAM.
03The serving path
The HTTP plugin is OpenAI-compatible: chat completions, SSE streaming, per-request sampling parameters, chat templates, bearer authentication and token-bucket rate limiting. It is threaded per connection and bounded, with keep-alive.
Generation itself is serial by design — one model loop per process, behind one mutex — because the loop and its sampler are not reentrant. Requests that do not need the model (health, model listing, cache stats, cache hits) answer while a generation is in flight; a second generation waits its turn.
—What it does not do
- OpenAI compatibility is partial: n, best_of, logprobs, presence and frequency penalties, and partial mode are not implemented; unknown keys are ignored rather than rejected.
- Generation is serial. There is no mid-token cancellation, and a streaming request issued during a long generation waits for it.
- IPv4 only. TLS is compiled in only when built against mbedTLS.
- With no API keys configured the HTTP server is open to anyone who can reach it — keys belong in PRISM_HTTP_API_KEYS, not on argv.