Mooler0410

MiroFish. A Simple and Universal Swarm Intelligence Engine, Predicting Anything. 简洁通用的群体智能引擎，预测万物

slime-n. A Multi-Policy, Multi-Agent RL Training Framework

concordia. A library for generative social simulation

LaProp-Optimizer. Codes accompanying the paper "LaProp: a Better Way to Combine Momentum with Adaptive Gradient"

LLMs-from-scratch. Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

MLQuestions. Machine Learning and Computer Vision Engineer - Technical Interview Questions

TorchCode. 🔥 LeetCode for PyTorch — practice implementing softmax, attention, GPT-2 and more from scratch with instant auto-grading. Jupyter-based, self-hosted or try online.

pyre-code. A self-hosted ML coding practice platform. 68 problems from ReLU to flow matching — attention, training, RLHF, diffusion, and more. Instant feedback in the browser.

ahxt.github.io. HTML

OpenSeeker. OpenSeeker: A search agent with open-source data and models

SteptronOss. A lightweight, AI-native training framework for large language models. Designed for fast iteration, reproducible experiments, and modular configuration across SFT, RLVR, and evaluation workflows.

strands-env. A framework for building agent environments for RL training and evaluation with Strands Agents.

strands-sglang. SGLang model provider of Strands Agents for on-policy agentic RL training.

nanochat. Pretraining Repository for Energy-based LM

dion. Dion optimizer algorithm

Megatron-Bridge. Training library for Megatron-based models with bidirectional Hugging Face conversion capability

EasyOffer. 《EasyOffer》（<大模型面经合集>）是针对LLM宝宝们量身打造的大模型暑期实习Offer指南，主要记录大模型暑期实习和秋招准备的一些常见大厂手撕代码、大厂面经经验、常见大厂思考题等；小白一个，正在学习ing......有问题各位大佬随时指正，希望大家都能拿到心仪Offer！

datatools. Common tools for data processing

OLMo-core. PyTorch building blocks for the OLMo ecosystem

megablocks. Python

grouped_gemm. PyTorch bindings for CUTLASS grouped GEMM.

flash-moba. C++

lighteval. Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Megatron-LM. Ongoing research training transformer models at scale

Decoding-Tree-Sketching. [ICML 2026] Decoding Tree Sketching (DTS): a training-free & model agonistic & plug-in framework for LLM parallel reasoning.

Kimi-Linear.

cwm. Research code artifacts for Code World Model (CWM) including inference tools, reproducibility, and documentation.

linghe. A high-performance kernel library for LLM training

LLM-Training-Puzzles. What would you do with 1000 H100s...

LlamaFactory. Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

Megatron-MoE-ModelZoo. Best practices for training DeepSeek, Mixtral, Qwen and other MoE models using Megatron Core.

oat. 🌾 OAT: A research-friendly framework for LLM online alignment, including reinforcement learning, preference learning, etc.

understand-r1-zero. Understanding R1-Zero-Like Training: A Critical Perspective

academic-homepage-typst. CSS

moe-recipes. Ongoing Research Project for Mixture of Expert models

Flash-Sparse-Attention. 🚀🚀 Efficient implementations of Native Sparse Attention

MiroMind-M1. MiroMind-M1 is a fully open-source series of reasoning language models built on Qwen-2.5, focused on advancing mathematical reasoning.

gpt-oss. gpt-oss-120b and gpt-oss-20b are two open-weight language models by OpenAI

AReaL. The RL Bridge for LLM-based Agent Applications. Made Simple & Flexible.

alfworld. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning

Flora. Official Implementation of Human/LLM-free SFT data construction strategy for long-context LLMs: "Flora: Effortless Context Construction to Arbitrary Length and Scale" [AAAI 2026 Oral]

LongGenBench. Jupyter Notebook

ms-swift. Use PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.6, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM4.5v, Gemma4, Llava, Phi4, ...) (AAAI 2025).

Kimi-K2. Kimi K2 is the large language model series developed by Moonshot AI team

verl. verl/HybridFlow: A Flexible and Efficient RL Post-Training Framework

dbgbench.github.io. DBGBench Website:

kvpress. LLM KV cache compression made easy

grouped-latent-attention. Python

s5cmd. Parallel S3 and local filesystem execution tool.

Star-Attention. Efficient LLM Inference over Long Sequences

vllm. A high-throughput and memory-efficient inference and serving engine for LLMs

NoLiMa. Official repository for "NoLiMa: Long-Context Evaluation Beyond Literal Matching"

triton-viz. Python

gamebench. Python

NLPCtlScalingAndBounds. [ICLR 2026 Oral] Intrinsic Entropy of Context Length Scaling in LLMs

DeepSick-R1. Reproduction of DeepSeek-R1

mini-llm-pretraining. Python

Qwen-Native-Sparse-Attention. qwen-nsa

mdy_triton. Jupyter Notebook

native-sparse-attention-pytorch. Implementation of the sparse attention pattern proposed by the Deepseek team in their "Native Sparse Attention" paper

native-sparse-attention. 🐳 Efficient Triton implementations for "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention"

MoBA. MoBA: Mixture of Block Attention for Long-Context LLMs

tilelang. Domain-specific language designed to streamline the development of high-performance GPU/CPU/Accelerators kernels

native-sparse-attention-triton. Efficient triton implementation of Native Sparse Attention.

open-gemini-deep-research. Python

ThinkPO. Python

s1. s1: Simple test-time scaling

deep-learning-pytorch-huggingface. Jupyter Notebook

open-r1. Fully open reproduction of DeepSeek-R1

DeepSeek-R1.

cute-flash-attention. Implement Flash Attention using Cute.

JetStream. JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).

ILearnDeepLearning.py. This repository contains small projects related to Neural Networks and Deep Learning in general. Subjects are closely linekd with articles I publish on Medium. I encourage you both to read as well as to check how the code works in the action.

LMCache. LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

alphafold3. AlphaFold 3 inference pipeline.

LongPPL. Code for ICLR 2025 Paper "What is Wrong with Perplexity for Long-context Language Modeling?"

viztracer. A debugging and profiling tool that can trace and visualize python code execution

LOGO. Code for paper: Long cOntext aliGnment via efficient preference Optimization

llama3.np. llama3.np is a pure NumPy implementation for Llama 3 model.

llama-from-scratch. Llama from scratch, or How to implement a paper without crying

modded-nanogpt. NanoGPT (124M) in 90 seconds

typst. A markup-based typesetting system that is powerful and easy to learn.

HolisticTraceAnalysis. A library to analyze PyTorch traces.

xformers. Hackable and optimized Transformers building blocks, supporting a composable construction.

minimind. 🧠「大模型」2小时完全从0训练64M的小参数LLM！Train a 64M-parameter LLM from scratch in just 2h!

granite-3.0-language-models.

lingua. Meta Lingua: a lean, efficient, and easy-to-hack codebase to research LLMs.

GrokkedTransformer. Code for NeurIPS'24 paper 'Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization'

OLMo. Modeling, training, eval, and inference code for OLMo

lolcats. Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"

Taylor-Unswift. Python

transformers_ssm_copy. Python

flex_head_fa. Fast and memory-efficient exact attention

vllm-kvcompress. KV cache compression for high-throughput LLM inference

LLM-Research-Scripts. Python

implicit-ins. Codebase for Instruction Following without Instruction Tuning

Triton-Puzzles-Solutions. Personal solutions to the Triton Puzzles

machine-learning-interview. Machine Learning Interviews from FAANG, Snapchat, LinkedIn. I have offers from Snapchat, Coupang, Stitchfix etc. Blog: mlengineer.io.

system-design-primer. Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.

ProLong. Homepage for ProLong (Princeton long-context language models) and paper "How to Train Long-Context Language Models (Effectively)"

HELMET. The HELMET Benchmark

Temp-LoRA. Python

youra. Repository for You Only Use Reactive Attention slice (YOURA) for long context retrieval

astralord.github.io. :ledger: A minimal, sidebar, responsive web design Jekyll theme that focuses on text presentation.

large_language_monkeys. Python

CacheBlend. Python

LCEG. [COLM'25] A Controlled Study on Long Context Extension and Generalization in LLMs

bAbI-tasks. Task generation for testing text understanding and reasoning

babilong. BABILong is a benchmark for LLM evaluation using the needle-in-a-haystack approach.

Attention-Mask-Patterns. Using FlexAttention to compute attention with different masking patterns

o1_inference_scaling_laws. Replicating O1 inference-time scaling laws

Awesome-LLM-Long-Context-Modeling. 📰 Must-read papers and blogs on LLM based Long Context Modeling 🔥

triton. Development repository for the Triton language and compiler

cola. Compositional Linear Algebra

RAM. A framework to study AI models in Reasoning, Alignment, and use of Memory (RAM).

DoubleSparse. 16-fold memory access reduction with nearly no loss

LongMIT. LongMIT: Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets

machine-learning-systems-design. A booklet on machine learning systems design with exercises. NOT the repo for the book "Designing Machine Learning Systems", which is `dmls-book`

kernels. Python

WordLlama. Things you can do with the token embeddings of an LLM

Awesome-LLM-Strawberry. A collection of LLM papers, blogs, and projects, with a focus on OpenAI o1 🍓 and reasoning techniques.

foundation-model-stack. 🚀 Collection of components for development, training, tuning, and inference of foundation models leveraging PyTorch native components.

longctx_bench. KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches. EMNLP Findings 2024

MemoRAG. Empowering RAG with a memory-based data interface for all-purpose applications!

GraphConv_is_Mixup. [TMLR] On the Equivalence of Graph Convolution and Mixup

alphafold3-pytorch. Implementation of Alphafold 3 from Google Deepmind in Pytorch

LinearLM. Code for the paper: https://arxiv.org/pdf/2309.06979.pdf

ml-sigmoid-attention. Python

context-cite. Attribute (or cite) statements generated by LLMs back to in-context information.

maskedmixers. Code for the paper 'Masked Mixers for Language Generation and Retrieval'

emnlp2024-attribute-or-abstain. This repository contains the code for the paper "Attribute or Abstain: LLMs as Long Document Assistants", published on arxiv.

LLMsPracticalGuide. A curated list of practical guide resources of LLMs (LLMs Tree, Examples, Papers)

MLR-Copilot. Python

writing-in-the-margins. Python

LongIns-Bench. Python

FinTextQA.

clapnq. Jupyter Notebook

kotaemon. An open-source RAG-based tool for chatting with your documents.

triton-index. Cataloging released Triton kernels.

boolean-questions.

RAG_Techniques. This repository showcases various advanced techniques for Retrieval-Augmented Generation (RAG) systems. Each technique has a detailed notebook tutorial.

Liger-Kernel. Efficient Triton Kernels for LLM Training

llm-numbers. Numbers every LLM developer should know

prompt-cache. Modular and structured prompt caching for low-latency LLM inference

attention-gym. Helpful tools and examples for working with flex-attention

TransformerLens. A library for mechanistic interpretability of GPT-style language models

flashattention2-custom-mask. Triton implementation of FlashAttention2 that adds Custom Masks.

CitationMap. A simple pip-installable Python tool to generate your HTML citation world map from your Google Scholar ID.

ml-superposition-prompting. Python

seq-test. Understand and test language model architectures on synthetic tasks.

MambaInLlama. [NeurIPS 2024] Official Repository of The Mamba in the Llama: Distilling and Accelerating Hybrid Models

torchtitan. A PyTorch native platform for training generative AI models

open_lm. A repository for research on medium sized language models.

HOMER. Official implementation of Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs (ICLR 2024).

annotated_deep_learning_paper_implementations. 🧑‍🏫 60+ Implementations/tutorials of deep learning papers with side-by-side notes 📝; including transformers (original, xl, switch, feedback, vit, ...), optimizers (adam, adabelief, sophia, ...), gans(cyclegan, stylegan2, ...), 🎮 reinforcement learning (ppo, dqn), capsnet, distillation, ... 🧠

LongMamba. Some preliminary explorations of Mamba's context scaling.

mamba2-minimal. Minimal Mamba-2 implementation in PyTorch

SparTA. Python

MInference. [NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

ThunderKittens. Tile primitives for speedy kernels

flash-attention-x. Python

baukit. Python

CD. [COLING'25] Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers?

loft. LOFT: A 1 Million+ Token Long-Context Benchmark

Quest. [ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference

DeciMamba. DeciMamba: Exploring the Length Extrapolation Potential of Mamba (ICLR 2025)

ltsm. LTSM-Bundle: A Toolbox and Benchmark on Large Language Models for Time Series Forecasting

cam. Pytorch implementation of our paper accepted by ICML 2024 -- CaM: Cache Merging for Memory-efficient LLMs Inference

ml-engineering. Machine Learning Engineering Open Book

know-dont-tell. Python

ArxivPaper. 定时爬取arXiv每日论文

epochraft-hf-fsdp. Example of using Epochraft to train HuggingFace transformers models with PyTorch FSDP

matmulfreellm. Implementation for MatMul-free LM.

tuning_playbook. A playbook for systematically maximizing the performance of deep learning models.

Triton-Puzzles. Puzzles for learning Triton

DPR. Dense Passage Retriever - is a set of tools and models for open domain Q&A task.

recurrentgemma. Open weights language model from Google DeepMind, based on Griffin.

flax. Flax is a neural network library for JAX that is designed for flexibility.

Samba. [ICLR 2025] Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling

CRUD_RAG. CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models

Awesome-RAG-Evaluation. The official repository for the paper: Evaluation of Retrieval-Augmented Generation: A Survey.

xRAG. [Neurips2024] Source code for xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token

LFQA-Verification. The repository for the paper Understanding Retrieval Augmentation for Long-Form Question Answering.

filco. [Preprint] Learning to Filter Context for Retrieval-Augmented Generaton

self-rag. This includes the original implementation of SELF-RAG: Learning to Retrieve, Generate and Critique through self-reflection by Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi.

BooookScore. A package to generate summaries of long-form text and evaluate the coherence of these summaries. Official package for our ICLR 2024 paper, "BooookScore: A systematic exploration of book-length summarization in the era of LLMs".

MiniCheck. MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents [EMNLP 2024]

repoqa. RepoQA: Evaluating Long-Context Code Understanding

MemoryLLM. The official implementation of the ICML 2024 paper "MemoryLLM: Towards Self-Updatable Large Language Models" and "M+: Extending MemoryLLM with Scalable Long-Term Memory"

contriever. Contriever: Unsupervised Dense Information Retrieval with Contrastive Learning

MultiHop-RAG. Repository for "MultiHop-RAG: A Dataset for Evaluating Retrieval-Augmented Generation Across Documents" (COLM 2024)

RGB. Python

tensor_parallel. Automatically split your PyTorch models on multiple GPUs for training & inference

PASTA. PASTA: Post-hoc Attention Steering for LLMs

FlashRAG. ⚡FlashRAG: A Python Toolkit for Efficient RAG Research (WWW2025 Resource)

transformer. Transformer: PyTorch Implementation of "Attention Is All You Need"

Parallel-Context-Windows. Python

sgpt. SGPT: GPT Sentence Embeddings for Semantic Search

fvcore. Collection of common code that's shared among different research projects in FAIR computer vision team.

GEAR. GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM