turboderp

exllama. A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

alpaca_lora_4bit. Python

text-generation-webui. A Gradio web UI for Large Language Models. Supports transformers, GPTQ, llama.cpp (GGUF), Llama models.

tabbyAPI. An OAI compatible exllamav2 API that's both lightweight and fast

formatron. Formatron empowers everyone to control the format of language models' output with minimal overhead.

transformers. 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.