Rare find

mini-infer. LLM inference engine from scratch — paged KV cache, continuous batching, chunked prefill, prefix caching, speculative decoding, CUDA graph, tensor parallelism, OpenAI-compatible serving

github.com/psmarter/mini-infer

Problem

Audience

Updates

No founder updates yet. Check back soon.