Max Ma

coda-kernels. CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs

wall-attention-release. Attention variant with per-channel multiplicative decay

GatedDeltaNet-2. Official PyTorch Implementation of Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

parameter-golf. Train the smallest LM you can that fits in 16MB. Best model wins!

raven. Python

tilelang. Domain-specific language designed to streamline the development of high-performance GPU/CPU/Accelerators kernels

FlashQLA. high-performance linear attention kernel library built on TileLang

cuLA. CUDA kernels for linear attention variants, written in CuTe DSL and CUTLASS C++.

quasar-attention. Quasar Attention Continuous-Time Memory Dynamics for Scaling Models Context to Millions of Tokens

MSA. Memory Sparse Attention - A scalable, end-to-end trainable latent-memory framework for 100M-token contexts.

Attention-Residuals.

SteptronOss. A lightweight, AI-native training framework for large language models. Designed for fast iteration, reproducible experiments, and modular configuration across SFT, RLVR, and evaluation workflows.

lectures. Python

gecko-llm. Gecko Architecture

Engram. Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

mHC.cu. mHC kernels implemented in CUDA

torchcomms. torchcomms: a modern PyTorch communications API

momoe-release. Memory optimized Mixture of Experts

grouped_gemm. PyTorch bindings for CUTLASS grouped GEMM.

ArchScale. Simple & Scalable Pretraining for Neural Architecture Research

llama3. The official Meta Llama 3 GitHub site

ml-cross-entropy. Python

mini_trainer. fast trainer for educational purposes

Liger-Kernel. Efficient Triton Kernels for LLM Training

uccl. UCCL is an efficient communication library for GPUs, covering collectives, P2P (e.g., KV cache transfer, RL weight transfer), and EP (e.g., GPU-driven)

reverse-engineering-gemma-3n. Reverse Engineering Gemma 3n: Google's New Edge-Optimized Language Model

DeepEP. DeepEP: an efficient expert-parallel communication library

MoBA. MoBA: Mixture of Block Attention for Long-Context LLMs

native-sparse-attention. 🐳 Efficient Triton implementations for "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention"

HIPIFY. HIPIFY: Convert CUDA to Portable C++ Code

hash-hop. Long context evaluation for large language models

flash-attention. Fast and memory-efficient exact attention

ttt-lm-pytorch. Official PyTorch implementation of Learning to (Learn at Test Time): RNNs with Expressive Hidden States

Infini-attention. Implementation of the paper: "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" from Google in pyTORCH

InfiniTransformer. Unofficial PyTorch/🤗Transformers(Gemma/Llama3) implementation of Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

cambrian. Cambrian-1 is a family of multimodal LLMs with a vision-centric design.

flash-linear-attention. 🚀 Efficient implementations for emerging model architectures

aisys-building-blocks. Building blocks for foundation models.

Awesome-LLM-Long-Context-Modeling. 📰 Must-read papers and blogs on LLM based Long Context Modeling 🔥

Convolutional-KANs. This project extends the idea of the innovative architecture of Kolmogorov-Arnold Networks (KAN) to the Convolutional Layers, changing the classic linear transformation of the convolution to learnable non linear activations in each pixel.

ThunderKittens. Tile primitives for speedy kernels

efficient-kan. An efficient pure-PyTorch implementation of Kolmogorov-Arnold Network (KAN).

ChebyKAN. Kolmogorov-Arnold Networks (KAN) using Chebyshev polynomials instead of B-splines.

pykan. Kolmogorov Arnold Networks

FourierKAN. Python

megalodon. Reference implementation of Megalodon 7B model

tokenizers-cpp. Universal cross-platform tokenizers binding to HF and sentencepiece

canmt-challenges. Python

fairseq-apollo. FairSeq repo with Apollo optimizer

mega. Sequence modeling with Mega.

mega-image. Implementation of Mega on Image Classification

awesome-multimodal-ml. Reading list for research topics in multimodal machine learning

Contrastive-Learning-NLP-Papers. Paper List for Contrastive Learning for Natural Language Processing

transformers. 🤗 Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX.

al-folio. A beautiful, simple, clean, and responsive Jekyll theme for academics

jekyll. :globe_with_meridians: Jekyll is a blog-aware static site generator in Ruby

rtg. Reader Translator Generator - NMT toolkit based on pytorch

luna-transformer. A PyTorch Implementation of the Luna: Linear Unified Nested Attention

swarm-distillation-zero-shot. Python

pytorch-optimizer. torch-optimizer -- collection of optimizers for Pytorch

Apollo-tf. Apollo optimizer in tensorflow

CausalNLP_Papers. A reading list for papers on causality for natural language processing (NLP)

unify-parameter-efficient-tuning. Implementation of paper "Towards a Unified View of Parameter-Efficient Transfer Learning" (ICLR 2022)

group-conditional-DRO. Group-conditional DRO to alleviate spurious correlations

fairseq-apollo. FairSeq repo with Apollo optimizer

adabelief-toy-examples-pytorch. numerical experiments in https://arxiv.org/pdf/2010.07468.pdf

pytorch-classification. Classification with PyTorch.

jmlr-style-file. LaTeX style file for the Journal of Machine Learning Research

NLP-progress. Repository to track the progress in Natural Language Processing (NLP), including the datasets and the current state-of-the-art for the most common NLP tasks.

apollo. Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization

website. Source Code for my website

wolf. Invertible Generative Flows

NeuroNLP2. Deep neural models for core NLP tasks (Pytorch version)

Style-Transfer-in-Text. Paper List for Style Transfer in Text

flowseq. Generative Flow based Sequence-to-Sequence Toolkit written in Python.

DeMa-BWE. NAACL 2019 paper: Density Matching for Bilingual Word Embedding (Zhou et al., 2019)

macow. Masked Convolutional Flow

texar. Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/

mae. MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders

MSVED-morph-reinflection. An implementation of semi-supervised VAE for morphology reinflection.

xnmt. eXtensible Neural Machine Translation

pytorch. Tensors and Dynamic neural networks in Python with strong GPU acceleration

NeuroNLP. Deep neural models for core NLP tasks

ReLiefParser. Reinforcement Learning Easy First Parser

LasagneNLP. NLP tools on Lasagne

cleverhans. An adversarial example library for constructing attacks, building defenses, and benchmarking both

caffe. Caffe

cmucoref. CMU Conference Resolution System developed by Max Ma

MaxParser. MaxParser in Java