llama_cpp_router. Lightweight Node.js server that proxies requests to Llama Server instances, manages model loading/unloading based on VRAM availability

github.com/Quozul/llama_cpp_router
by @Quozul

Problem

Audience

Keep exploring

Updates

No founder updates yet. Check back soon.