Rulin Shao

retrieval-scaling. Official repository for "Scaling Retrieval-Based Langauge Models with a Trillion-Token Datastore".

LightSeq. Official repository for DistFlashAttn: Distributed Memory-efficient Attention for Long-context LLMs Training

on-the-adversarial-robustness-of-visual-transformer. Code for the paper "On the Adversarial Robustness of Visual Transformers"

FastCkpt. Python package for rematerialization-aware gradient checkpointing

massive-serve. Python package for serving a local search engine. One command to download and serve a datastore---that's it 😎.

RAG-evaluation-harnesses. An evaluation suite for Retrieval-Augmented Generation (RAG).

VL-Instruct. Codes for vision-language instruction tuning. Currently support BLIP2-t5 and BLIP2-vicuna.

elastic-inference. (Agentic) Inference engine for poor PhD (myself) :P

rl_evolve. Python

Digital-Image-Processing. MATLAB

Llava-doremi. Automatic task-balancing for vision-language instruction tuning using group distributionally robust optimization (Group DRO, the technique used in Doremi)

sink-flash-attention-kernel. FlashAttention2 + learnable sink attention with training and inference support.