HandH1998

QQQ. QQQ is an innovative and hardware-optimized W4A8 quantization solution for LLMs.

vllm. A high-throughput and memory-efficient inference and serving engine for LLMs

marlin. FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.

sglang. SGLang is a fast serving framework for large language models and vision language models.