Ruotian(RT) Luo

codex. Lightweight coding agent that runs in your terminal

DeepGEMM. DeepGEMM: clean and efficient BLAS kernel library on GPU

HunyuanVideo. HunyuanVideo: A Systematic Framework For Large Video Generation Model

Protenix. Toward High-Accuracy Open-Source Biomolecular Structure Prediction.

LLaMA-O1. Large Reasoning Models

coocmap. code for paper "Accessing higher dimensions for unsupervised word translation"

adaptive-length-tokenizer. Adaptive Length Image Tokenization via Recurrent Allocation | How many tokens is an image worth ?

Cosmos-Tokenizer. A suite of image and video neural tokenizers

Hunyuan3D-1. Tencent Hunyuan3D-1.0: A Unified Framework for Text-to-3D and Image-to-3D Generation

mochi. The best OSS video generation models, created by Genmo

lingua. Meta Lingua: a lean, efficient, and easy-to-hack codebase to research LLMs.

mle-bench. MLE-bench is a benchmark for measuring how well AI agents perform at machine learning engineering

vqgan-training. Train VAE like a boss

OpenRLHF. An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray (PPO & DAPO & REINFORCE++ & VLM & TIS & vLLM & Ray & Async RL)

PDF-Extract-Kit. A Comprehensive Toolkit for High-Quality PDF Content Extraction

MinerU. Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.

FluxMusic. Text-to-Music Generation with Rectified Flow Transformers

VAR. [NeurIPS 2024 Best Paper Award][GPT beats diffusion🔥] [scaling laws in visual generation📈] Official impl. of "Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction". An *ultra-simple, user-friendly yet state-of-the-art* codebase for autoregressive image generation!

claude-quickstarts. A collection of projects designed to help developers quickly get started with building deployable applications using the Claude API

Liger-Kernel. Efficient Triton Kernels for LLM Training

sapiens. High-resolution models for human tasks.

x-flux. Python

sglang. SGLang is a high-performance serving framework for large language models and multimodal models.

AI-Scientist. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery 🧑‍🔬

LawBench. Benchmarking Legal Knowledge of Large Language Models

Deep-Live-Cam. real time face swap and one-click video deepfake with only a single image

UHM. Official PyTorch implementation of "Authentic Hand Avatar from a Phone Scan via Universal Hand Model", CVPR 2024.

sam2. The repository provides code for running inference with the Meta Segment Anything Model 2 (SAM 2), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.

VLMEvalKit. Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

Queryable. Run OpenAI's CLIP and Apple's MobileCLIP model on iOS to search photos.

LlamaGen. Autoregressive Model Beats Diffusion: 🦙 Llama for Scalable Image Generation

LivePortrait. Bring portraits to life!

LLM101n. LLM101n: Let's build a Storyteller

claude-engineer. Claude Engineer is an interactive command-line interface (CLI) that leverages the power of Anthropic's Claude-3.5-Sonnet model to assist with software development tasks.This framework enables Claude to generate and manage its own tools, continuously expanding its capabilities through conversation. Available both as a CLI and a modern web interface

schedule_free. Schedule-Free Optimization in PyTorch

ShareGemini.

loft. LOFT: A 1 Million+ Token Long-Context Benchmark

Depth-Anything-V2. [NeurIPS 2024] Depth Anything V2. A More Capable Foundation Model for Monocular Depth Estimation

CleanDiffuser. CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Making

Ask-Anything. [CVPR2024 Highlight][VideoChatGPT] ChatGPT with video understanding! And many more supported LMs such as miniGPT4, StableLM, and MOSS.

odyssey-math. Jupyter Notebook

chameleon. Repository for Meta Chameleon, a mixed-modal early-fusion foundation model from FAIR.

Grounded_3D-LLM. Code&Data for Grounded 3D-LLM with Referent Tokens

titok-pytorch. Implementation of TiTok, proposed by Bytedance in "An Image is Worth 32 Tokens for Reconstruction and Generation"

SEED-Voken. SEED-Voken: A Series of Powerful Visual Tokenizers

Odysseus-Transformer. Odysseus: Playground of LLM Sequence Parallelism

LlamaFactory. Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

build-nanogpt. Video+code lecture on building nanoGPT from scratch

llm.c. LLM training in simple, raw C/CUDA

ml-4m. 4M: Massively Multimodal Masked Modeling

textgrad. TextGrad: Automatic ''Differentiation'' via Text -- using large language models to backpropagate textual gradients. Published in Nature.

translation-agent. Python

DeepStack-VL. [NeurIPS-24] This is the official implementation of the paper "DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs".

HunyuanDiT. Hunyuan-DiT : A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding

Open-LLaVA-NeXT. An open-source implementation for training LLaVA-NeXT.

ShareGPT4Video. [NeurIPS 2024] An official implementation of "ShareGPT4Video: Improving Video Understanding and Generation with Better Captions"

Video-LLaVA. 【EMNLP 2024🔥】Video-LLaVA: Learning United Visual Representation by Alignment Before Projection

Qwen3. Qwen3 is the large language model series developed by Qwen team, Alibaba Cloud.

Lumina-T2X. Lumina-T2X is a unified framework for Text to Any Modality Generation

random-phrase-generator. Python

Omost. Your image is almost there!

ChatTTS. A generative speech model for daily dialogue.

robocasa. RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots

DiffusionDPO. Code for "Diffusion Model Alignment Using Direct Preference Optimization"

rcg. PyTorch implementation of RCG https://arxiv.org/abs/2312.03701

generative-models. Generative Models by Stability AI

Vista. [NeurIPS 2024] A Generalizable World Model for Autonomous Driving

AlignProp. AlignProp uses direct reward backpropogation for the alignment of large-scale text-to-image diffusion models. Our method is 25x more sample and compute efficient than reinforcement learning methods (PPO) for finetuning Stable Diffusion

ddpo. Code for the paper "Training Diffusion Models with Reinforcement Learning"

InternVL. [CVPR 2024 Oral] InternVL Family: A Pioneering Open-Source Alternative to GPT-4o. 接近GPT-4o表现的开源多模态对话模型

MoMA. MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation

edm2. Minimal multi-gpu implementation of EDM2: "Analyzing and Improving the Training Dynamics of Diffusion Models"

easy-gpt4o. Easy-GPT4O opensource version

MiniCPM-V. A Pocket-Sized MLLM for Ultra-Efficient Image and Video Understanding on Your Phone

SimPO. [NeurIPS 2024] SimPO: Simple Preference Optimization with a Reference-Free Reward

RLHF-Reward-Modeling. Recipes to train reward model for RLHF.

lightplane. Lightplane implements a highly memory-efficient differentiable radiance field renderer, and a module for unprojecting features from images to 3D grids.

llama3-from-scratch. llama3 implementation one matrix multiplication at a time

TinyLLaVA_Factory. A Framework of Small-scale Large Multimodal Models

CogVLM2. GPT4V-level open-source multi-modal model based on Llama3-8B

IC-Light. More relighting!

llama3. The official Meta Llama 3 GitHub site

gdown. Google Drive public file downloader when curl/wget fails.

StoryDiffusion. Accepted as [NeurIPS 2024] Spotlight Presentation Paper

MAP-NEO. Python

ldm3d-inpainting. Joint image and Depth inpainting, ldm3d

Self-Rewarding-Language-Models. Python

outlines. Structured Outputs

LLaVA-NeXT. Python

DeepSeek-V2. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

VILA. VILA is a family of state-of-the-art vision language models (VLMs) for diverse multimodal AI tasks across the edge, data center, and cloud.

gpt-fast. Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

U-ViT. A PyTorch implementation of the paper "All are Worth Words: A ViT Backbone for Diffusion Models".

Discffusion. Official repo for the TMLR paper "Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learners"

llama-cookbook. Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We also show you how to solve end to end problems using Llama model family and using them on various provider services

reka-vibe-eval. Multimodal language model benchmark, featuring challenging examples

calculate-flops.pytorch. The calflops is designed to calculate FLOPs、MACs and Parameters in all various neural networks, such as Linear、 CNN、 RNN、 GCN、Transformer(Bert、LlaMA etc Large Language Model)

rerun. Visualize, query, and stream to train on multimodal robotics data.

aim. Aim 💫 — An easy-to-use & supercharged open-source experiment tracker.

einx. Universal Notation for Tensor Operations in Python

EasyContext. Memory optimization and training recipes to extrapolate language models' context length to 1 million tokens, with minimal hardware.

self-rewarding-lm-pytorch. Implementation of the training framework proposed in Self-Rewarding Language Model, from MetaAI

grok-1. Grok open release

Depth-Anything. [CVPR 2024] Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data. Foundation Model for Monocular Depth Estimation

Qwen-VL. The official repo of Qwen-VL (通义千问-VL) chat & pretrained large vision language model proposed by Alibaba Cloud.

Yi. A series of large language models trained from scratch by developers @01-ai

search_with_lepton. Building a quick conversation-based search demo with Lepton AI.

LLMDataHub. A quick guide (especially) for trending instruction finetuning datasets

llm-course. Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks.

TinyLlama. The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.

MOSS-RLHF. Secrets of RLHF in Large Language Models Part I: PPO

openinterpreter. A lightweight coding agent for open models like Deepseek, Kimi, and Qwen

CodeGen2. CodeGen2 models for program synthesis

shap-e. Generate 3D objects conditioned on text or images

Multimodal-GPT. Multimodal-GPT

LLaMA-Adapter. [ICLR 2024] Fine-tuning LLaMA to follow Instructions within 1 Hour and 1.2M Parameters

Segment-Everything-Everywhere-All-At-Once. [NeurIPS 2023] Official implementation of the paper "Segment Everything Everywhere All at Once"

autoresearcher. ⚡ Automating scientific workflows with AI ⚡

NextChat. ✨ Light and Fast AI Assistant. Support: Web | iOS | MacOS | Android | Linux | Windows

AutoGPT. AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.

segment-anything. The repository provides code for running inference with the SegmentAnything Model (SAM), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.

whisper_streaming. Whisper realtime streaming for long speech-to-text transcription and translation

ChatGPT_JCM. OpenAI管理界面，聚合了OpenAI的所有接口进行界面操作(所有模型、图片、音频、微调、文件)等，支持Markdown格式(公式、图表，表格)等，后期会一点一点的将OpenAI接口进行接入大家支持一下，右上角点个Star。

PaLM-rlhf-pytorch. Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture. Basically ChatGPT but with PaLM

nanoGPT. The simplest, fastest repository for training/finetuning medium-sized GPTs.

lit-llama. Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.

chatgpt-ui. A ChatGPT web client that supports multiple users, multiple languages, and multiple database connections for persistent data storage. Provides Docker images and quick deployment scripts.

GenerativeImage2Text. GIT: A Generative Image-to-text Transformer for Vision and Language

gpt_academic. 为GPT/GLM等LLM大语言模型提供实用化交互接口，特别优化论文阅读/润色/写作体验，模块化设计，支持自定义快捷按钮&函数插件，支持Python和C++等项目剖析&自译解功能，PDF/LaTex论文翻译&总结功能，支持并行问询多种LLM模型，支持chatglm3等本地模型。接入通义千问, deepseekcoder, 讯飞星火, 文心一言, llama2, rwkv, claude2, moss等。

chatgpt-for-educators. Slides for a talk about the use and abuse of ChatGPT in the classroom

ai-pdf-chatbot-langchain. AI PDF chatbot agent built with LangChain & LangGraph

ChatReviewer. ChatReviewer: 使用ChatGPT分析论文优缺点，提出改进建议

alpaca-lora. Instruct-tune LLaMA on consumer hardware

ChatGPT-Shortcut. 🚀💪Maximize your efficiency and productivity. The ultimate hub to manage, customize, and share prompts. (English/中文/Español/العربية). 让生产力加倍的 AI 快捷指令。更高效地管理提示词，在分享社区中发现适用于不同场景的灵感。

PromptCraft-Robotics. Community for applying LLMs to robotics and a robot simulator with ChatGPT integration

dalai. The simplest way to run LLaMA on your local machine

stanford_alpaca. Code and documentation to train Stanford's Alpaca models, and generate the data.

ChatPaper. Use ChatGPT to summarize the arXiv papers. 全流程加速科研，利用chatgpt进行论文全文总结+专业翻译+润色+审稿+审稿回复

TaskMatrix. Python

evals. Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

xmanager. A platform for managing machine learning experiments

FLEX. Code for our CVPR'23 paper - "FLEX: Full-Body Grasping Without Full-Body Grasps"

prismer. The implementation of "Prismer: A Vision-Language Model with Multi-Task Experts".

openai-cookbook. Examples and guides for using the OpenAI API

EX-chatGPT. Let ChatGPT truly learn how to go online and call APIs! 'EX-ChatGPT' can rival and even surpass NewBing

stable-diffusion-webui. Stable Diffusion web UI

ControlNet. Let us control diffusion models!

ColossalAI. Making large AI models cheaper, faster and more accessible

lora. Using Low-rank adaptation to quickly fine-tune diffusion models.

whisper. Robust Speech Recognition via Large-Scale Weak Supervision

prompts.chat. f.k.a. Awesome ChatGPT Prompts. Share, discover, and collect prompts from the community. Free and open source — self-host for your organization with complete privacy.

Diffusion-SDF. [CVPR 2023] Diffusion-SDF: Text-to-Shape via Voxelized Diffusion

Open-Assistant. OpenAssistant is a chat-based assistant that understands tasks, can interact with third-party systems, and retrieve information dynamically to do so.

robotics_transformer. Python

PiPPy. Pipeline Parallelism for PyTorch

sjc. Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation (CVPR 2023)

kernl. Kernl lets you run PyTorch transformer models several times faster on GPU with a single line of code, and is designed to be easily hackable.

taichi-ngp-renderer. An Instants-NGP renderer that has been implemented using Taichi

alphatensor. Python

stable-dreamfusion. Text-to-3D & Image-to-3D & Mesh Exportation with NeRF + Diffusion.

VectorAdam. VectorAdam for Rotation Equivariant Geometry Optimization

edgevit. [ECCV 2022] EdgeViT: Competing Light-weight CNNs on Mobile Devices with Vision Transformers

CodeT. Python

botorch. Bayesian optimization in PyTorch

multimodal. TorchMultimodal is a PyTorch library for training state-of-the-art multimodal multi-task models at scale.

pytorch-lightning. Pretrain, finetune ANY AI model of ANY size on 1 or 10,000+ GPUs with zero code changes.

detrex. detrex is a research platform for DETR-based object detection, segmentation, pose estimation and other visual recognition tasks.

netron. Visualizer for neural network, deep learning and machine learning models

denoising-diffusion-pytorch. Implementation of Denoising Diffusion Probabilistic Model in Pytorch

DeCLIP. Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm

Local-Disco-Diffusion-v5.2.jupyterNote. A custom Disco Diffusion v5.2 that runs on local GPUS.

FiD. Fusion-in-Decoder

big_vision. Official codebase used to develop Vision Transformer, SigLIP, MLP-Mixer, LiT and more.

GSAM. PyTorch repository for ICLR 2022 paper (GSAM) which improves generalization (e.g. +3.8% top-1 accuracy on ImageNet with ViT-B/32)

tuixue.online-visa. https://tuixue.online/visa/ A Real-time Display of U.S. Visa Appointment Status Website 预约美帝签证各个签证处最早时间的爬虫

PaLM-colossalai. Scalable PaLM implementation of PyTorch

adapting-CLIP. Python

DiffusionCLIP. [CVPR 2022] Official PyTorch Implementation for DiffusionCLIP: Text-guided Image Manipulation Using Diffusion Models

efficientnet-jax. EfficientNet, MobileNetV3, MobileNetV2, MixNet, etc in JAX w/ Flax Linen and Objax

DINO. [ICLR 2023] Official implementation of the paper "DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection"

ivy. Convert Machine Learning Code Between Frameworks

vidt. Python

SLIP. Code release for SLIP Self-supervision meets Language-Image Pre-training

sktime. A unified framework for machine learning with time series

visual-comet. VisualCOMET: Reasoning about the Dynamic Context of a Still Image

transformer-exercises-jax. Python

PointContrast. Code for paper <PointContrast: Unsupervised Pretraining for 3D Point Cloud Understanding>

moolib. A library for distributed ML training with PyTorch

gpt-code-clippy. Full description can be found here: https://discuss.huggingface.co/t/pretrain-gpt-neo-for-open-source-github-copilot-model/7678?u=ncoop57

emacs-secondmate. An open-source, mini imitation of GitHub Copilot for Emacs.

open_clip. An open source implementation of CLIP.

OpenLT. An open-source project for long-tail classification

glide-text2im. GLIDE: a diffusion-based text-conditional image synthesis model

ml-research-containers. A list of community maintained docker images for deep reinforcement learning research

vmoe. Jupyter Notebook

instant-ngp. Instant neural graphics primitives: lightning fast NeRF and more

ConvNeXt. Code release for ConvNeXt model

Detic. Code release for "Detecting Twenty-thousand Classes using Image-level Supervision".

ibot. iBOT :robot:: Image BERT Pre-Training with Online Tokenizer (ICLR 2022)

zsseg.baseline. Open-vocabulary Semantic Segmentation