Zhe Chen

detect-secrets. An enterprise friendly way of detecting and preventing secrets in code.

CLIProxyAPI. Wrap Antigravity, ChatGPT Codex, Claude Code, Grok Build as an OpenAI/Gemini/Claude/Codex compatible API service, allowing you to enjoy the free Gemini 3.1 Pro, GPT 5.5, Grok 4.3, Claude model through API

oh-my-openagent. omo/lazycodex: The coding agent for tokenmaxxers;the one and only agent harness for complex codebases. For your Codex, for your OpenCode

Maxs-Awesome-Datasets. Max的有趣数据集 / Max's awesome datasets

litellm. Python SDK, Proxy Server (AI Gateway) to call 100+ LLM APIs in OpenAI (or native) format, with cost tracking, guardrails, loadbalancing and logging. [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, VLLM, NVIDIA NIM]

harbor. Framework for evaluating and improving agents

claw-eval. Claw-Eval is an evaluation harness for evaluating LLM as agents. All tasks verified by humans.

autoresearch. AI agents running research on single-GPU nanochat training automatically

skill. PinchBench is a benchmarking system for evaluating LLM models as OpenClaw coding agents. Made with 🦀 by the humans at https://kilo.ai

flashinfer. FlashInfer: Kernel Library for LLM Serving

open-swe. An Open-Source Asynchronous Coding Agent

MiroFlow. 🏆 Top-1 on 5+ benchmarks | Web UI | Supports MiroThinker, Claude, Kimi, OpenAI

open-cowork. Open-source AI agent desktop app for Windows & macOS. One-click install Claude Code, MCP tools, and Skills — with sandbox isolation, multi-model support, and Feishu/Slack integration.

InternVL-U. InternVL-U is a 4B-parameter unified multimodal model (UMM) that brings multimodal understanding, reasoning, image generation, image editing into a single framework.

awesome-openclaw-skills. The awesome collection of OpenClaw skills. 5,400+ skills filtered and categorized from the official OpenClaw Skills Registry.🦞

Toolathlon. [ICLR 2026] The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution

evalscope. A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.

dataclaw. Agent harness to publish your agent chat history as Huggingface datasets.

MiMo-V2-Flash. MiMo-V2-Flash: Efficient Reasoning, Coding, and Agentic Foundation Model

openclaw. Your own personal AI assistant. Any OS. Any Platform. The lobster way. 🦞

mcpmark. MCPMark is a comprehensive, stress-testing MCP benchmark designed to evaluate model and agent capabilities in real-world MCP use.

MCP-Universe. MCP-Universe is a comprehensive framework designed for RL training, benchmarking, and developing AI agents for general tool-use.

MiroThinker. MiroThinker is a deep research agent optimized for complex research and prediction tasks. Our latest models, MiroThinker-1.7, achieves 74.0 and 75.3 on the BrowseComp and BrowseComp Zh, respectively.

InternVL-doc.

MiMo-VL. MiMo-VL

ZeroGUI. ZeroGUI: Automating Online GUI Learning at Zero Human Cost

Bagel. Open-source unified multimodal model

SAIL. Implementation for "The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer"

CVPR-2024-Highlight-Oral. Collection of Highlight papers

VisuLogic-Eval. Python

VisuLogic-Train. Python

SimChart9K. The proposed simulated dataset consisting of 9,536 charts and associated data annotations in CSV format.

Open-Reasoner-Zero. Official Repo for Open-Reasoner-Zero

Modules-Learn. 学习笔记 - 码云：https://gitee.com/wanzheng_96/Modules-Learn）

rllm. Democratizing Reinforcement Learning for LLMs

MathRuler. A light-weight tool for evaluating LLMs in rule-based ways.

Math-Verify. Python

verl. verl/HybridFlow: A Flexible and Efficient RL Post-Training Framework

awesome-multi-modal-reinforcement-learning. A curated list of Multi-Modal Reinforcement Learning resources (continually updated)

sglang. SGLang is a high-performance serving framework for large language models and multimodal models.

MM-EUREKA. MM-EUREKA: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning

Awesome-MLLM-Reasoning-Benchmarks. A Comprehensive Survey on Evaluating Reasoning Capabilities in Multimodal Large Language Models.

OpenRLHF-M. An Easy-to-use, Scalable and High-performance RLHF Framework designed for Multimodal Models.

VAR. [NeurIPS 2024 Best Paper Award][GPT beats diffusion🔥] [scaling laws in visual generation📈] Official impl. of "Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction". An *ultra-simple, user-friendly yet state-of-the-art* codebase for autoregressive image generation!

LLaDA. Official PyTorch implementation for "Large Language Diffusion Models"

lmm-r1. Extend OpenRLHF to support LMM RL training for reproduction of DeepSeek-R1 on multimodal tasks.

OpenRLHF. An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray (PPO & DAPO & REINFORCE++ & VLM & TIS & vLLM & Ray & Async RL)

Kimi-k1.5.

ChemVlm. [🏆AAAI'25] Official Repo for ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area.

ChuanhuChatGPT. GUI for ChatGPT API and many LLMs. Supports agents, file-based QA, GPT finetuning and query with web search. All with a neat UI.

PaddleMIX. Paddle Multimodal Integration and eXploration, supporting mainstream multi-modal tasks, including end-to-end large-scale multi-modal pretrain models and diffusion model toolbox. Equipped with high performance and flexibility.

neptune.

Tutorial. LLM&VLM Tutorial

LightCompress. [EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.

OlympiadBench. [ACL 2024]Official GitHub repo for OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems.

MIRB. Benchmarking Multi-Image Understanding in Vision and Language Models

Mantis. Official code for Paper "Mantis: Multi-Image Instruction Tuning" [TMLR 2024 Best Paper]

core-mm. Python

VisRAG. Parsing-free RAG supported by VLMs

AesBench. An expert benchmark aiming to comprehensively evaluate the aesthetic perception capacities of MLLMs.

A-Bench. [ICLR 2025] What do we expect from LMMs as AIGI evaluators and how do they perform?

MM-Math. Python

Liger-Kernel. Efficient Triton Kernels for LLM Training

every-chatgpt-gui. Every front-end GUI client for ChatGPT, Claude, and other LLMs

NaturalBench. 🚀 [NeurIPS24] Make Vision Matter in Visual-Question-Answering (VQA)! Introducing NaturalBench, a vision-centric VQA benchmark (NeurIPS'24) that challenges vision-language models with simple questions about natural imagery.

MMIU. [ICLR2025] MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models

CountGD. Includes the code for training and testing the CountGD model from the paper CountGD: Multi-Modal Open-World Counting.

distilabel. Distilabel is a framework for synthetic data and AI feedback for engineers who need fast, reliable and scalable pipelines based on verified research papers.

EvolKit. EvolKit is an innovative framework designed to automatically enhance the complexity of instructions used for fine-tuning Large Language Models (LLMs).

VCR. Official Repo for the paper: VCR: Visual Caption Restoration. Check arxiv.org/pdf/2406.06462 for details.

Eagle. Eagle: Frontier Vision-Language Models with Data-Centric Strategies

MME-RealWorld. ✨✨ [ICLR 2025] MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?

LMM-Engines. Python

scepter. SCEPTER is an open-source framework used for training, fine-tuning, and inference with generative models.

OmniBal. [ICML 2025] This is the official PyTorch implementation of "OmniBal: Towards Fast Instruction-Tuning for Vision-Language Models via Omniverse Computation Balance".

MTVQA. MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering. A comprehensive evaluation of multimodal large model multilingual text perception and comprehension capabilities across nine widely-used yet low-resource languages.

BLINK_Benchmark. This repo contains evaluation code for the paper "BLINK: Multimodal Large Language Models Can See but Not Perceive". https://arxiv.org/abs/2404.12390 [ECCV 2024]

ChartMimic. [ICLR 2025] ChartMimic: Evaluating LMM’s Cross-Modal Reasoning Capability via Chart-to-Code Generation

MLVU. 🔥🔥MLVU: Multi-task Long Video Understanding Benchmark

CharXiv. [NeurIPS 2024] CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs

Ovis. A novel Multimodal Large Language Model (MLLM) architecture, designed to structurally align visual and textual embeddings.

EVE. EVE Series: Encoder-Free Vision-Language Models from BAAI

MMInstruct. [SCIS 2024] The official implementation of the paper "MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity". The MMInstruct dataset includes 973K instructions from 24 domains and four instruction types.

dclm. DataComp for Language Models

Awesome-MLLM-LLM-Colab. Happy experimenting with MLLM and LLM models!

vllm. A high-throughput and memory-efficient inference and serving engine for LLMs

AutoAWQ. AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:

streamlit. Streamlit — A faster way to build and share data apps.

gpt-assistants-api-ui. 💬 OpenAI Assistants API chat UI 🛠️ It works easily by setting the ASSISTANT ID 📁 Supports file upload and file download 🏃 Supports Streaming API 🪟 Support to Azure OpenAI

st-chat. Streamlit Component, for a Chatbot UI

GeminiProChat. Minimal web UI for GeminiPro.

mergekit. Tools for merging pretrained large language models.

De-focus-Attention-Networks. Learning 1D Causal Visual Representation with De-focus Attention Networks

chatgpt-web-midjourney-proxy. One UI is all done with chatgpt web, midjourney, gpts,suno,luma,runway,viggle,flux,ideogram,realtime,pika,udio; Simultaneous support Web / PWA / Linux / Win / MacOS platform

LCL. [NeurIPS 2024] Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning

OmniCorpus. [ICLR 2025 Spotlight] OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text

MM-NIAH. [NeurIPS 2024] Needle In A Multimodal Haystack (MM-NIAH): A comprehensive benchmark designed to systematically evaluate the capability of existing MLLMs to comprehend long multimodal documents.

LabelLLM. The Open-Source Data Annotation Platform

PIIP. [NeurIPS 2024 Spotlight ⭐️ & TPAMI 2025] Parameter-Inverted Image Pyramid Networks (PIIP)

Video-MME. ✨✨[CVPR 2025] Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis

xtuner. A Next-Generation Training Engine Built for Ultra-Large MoE Models

MMFuser. The official implementation of the paper "MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understanding". MMFuser addresses the limitations of current MLLMs in capturing complex image details by simply yet efficiently integrating multi-layer features from ViTs.

conv-llava. Python

InternVL.github.io. HTML

VILA. VILA is a family of state-of-the-art vision language models (VLMs) for diverse multimodal AI tasks across the edge, data center, and cloud.

PsyDI. PsyDI: Towards a Personalized and Progressively In-depth Chatbot for Psychological Measurements. (e.g. MBTI Measurement Agent)

Multi-Modality-Arena. Chatbot Arena meets multi-modality! Multi-Modality Arena allows you to benchmark vision-language models side-by-side while providing images as inputs. Supports MiniGPT-4, LLaMA-Adapter V2, LLaVA, BLIP-2, and many more!

FreeBind. Python

MuLan. MuLan: Adapting Multilingual Diffusion Models for 110+ Languages (无需额外训练为任意扩散模型支持多语言能力)

VisCPM. [ICLR'24 spotlight] Chinese and English Multimodal Large Model Series (Chat and Paint) | 基于CPM基础模型的中英双语多模态大模型系列

opencompass. OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.

ms-swift. Use PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.6, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM4.5v, Gemma4, Llava, Phi4, ...) (AAAI 2025).

openedai-vision. An OpenAI API compatible API for chat with image input and questions about the images. aka Multimodal.

llama.cpp. LLM inference in C/C++

Open-Sora-Plan. This project aim to reproduce Sora (Open AI T2V model), we wish the open source community contribute to this project.

MMT-Bench. [ICML 2024] | MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI

InternVL. [CVPR 2024 Oral] InternVL Family: A Pioneering Open-Source Alternative to GPT-4o. 接近GPT-4o表现的开源多模态对话模型

lmms-eval. One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

BoS. [ICLR 2024 Spotlight] Bounding Box Stability against Feature Dropout Reflects Detector Generalization across Environments

EgoExoLearn. [CVPR 2024] Data and benchmark code for the EgoExoLearn dataset

OpenDMD. Open source implementation and models of One-step Diffusion with Distribution Matching Distillation

NEFTune. Official repository of NEFTune: Noisy Embeddings Improves Instruction Finetuning

video-mamba-suite. The suite of modeling video with Mamba

VideoChat-Flash. [ICLR2026] VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling

MM-Vet. MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (ICML 2024)

DecryptPrompt. 总结Prompt&LLM论文，开源数据&模型，AIGC应用

Bunny. A family of lightweight multimodal models.

transformers. 🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

DiffAgent. [CVPR 2024] DiffAgent: Fast and Accurate Text-to-Image API Selection with Large Language Model

MixtralKit. A toolkit for inference and evaluation of 'mixtral-8x7b-32kseqlen' from Mistral AI

MMBench. Official Repo of "MMBench: Is Your Multi-modal Model an All-around Player?"

ALLaVA. Harnessing 1.4M GPT4V-synthesized Data for A Lite Vision-Language Model

ViT-CoMer. Official implementation of the CVPR 2024 paper ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictions.

VideoMamba. [ECCV2024] VideoMamba: State Space Model for Efficient Video Understanding

ADDP. The official implementation of ADDP (ICLR 2024)

AAAI-2024-Papers. AAAI 2024 Papers: Explore a comprehensive collection of innovative research papers presented at one of the premier artificial intelligence conferences. Seamlessly integrate code implementations for better understanding. ⭐ experience the forefront of progress in artificial intelligence with this repository!

CRA-PCN. CRA-PCN: Point Cloud Completion with Intra- and Inter-level Cross-Resolution Transformers

BEV-Planner. Python

Vision-RWKV. [ICLR 2025 Spotlight] Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures

DeepSeek-VL. DeepSeek-VL: Towards Real-World Vision-Language Understanding

LLaVA-HR. [ICLR2025] LLaVA-HR: High-Resolution Large Language-Vision Assistant

doctr. docTR (Document Text Recognition) - a seamless, high-performing & accessible library for OCR-related tasks powered by Deep Learning.

InternEvo. InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

PixArt-sigma. PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation

VideoSys. VideoSys: An easy and efficient system for video generation

GeoDiffusion. Official PyTorch implementation of GeoDiffusion in ICLR 2024 (https://arxiv.org/abs/2306.04607)

MathVista. MathVista: data, code, and evaluation for Mathematical Reasoning in Visual Contexts

MMVP. Python

Depth-Anything. [CVPR 2024] Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data. Foundation Model for Monocular Depth Estimation

MultimodalOCR. On the Hidden Mystery of OCR in Large Multimodal Models (OCRBench)

CMMMU. Python

MMMU. This repo contains evaluation code for the paper "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI"

MM-Interleaved. MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer

InternVL-MMDetSeg. Train InternViT-6B in MMSegmentation and MMDetection with DeepSpeed

PixArt-alpha. PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis

CapsFusion. [CVPR 2024] CapsFusion: Rethinking Image-Text Data at Scale

vissl. VISSL is FAIR's library of extensible, modular and scalable components for SOTA Self-Supervised Learning with images.

prismer. The implementation of "Prismer: A Vision-Language Model with Multi-Task Experts".

AlphaCLIP. [CVPR 2024] Alpha-CLIP: A CLIP Model Focusing on Wherever You Want

Academic-project-page-template. A project page template for academic papers. Demo at https://eliahuhorwitz.github.io/Academic-project-page-template/

DCNv4. [CVPR 2024] Deformable Convolution v4

VL-GPT. VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation

VLMEvalKit. Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

QLLM. A general 2-8 bits quantization toolbox with GPTQ/AWQ/HQQ/VPTQ, and export to onnx/onnx-runtime easily.

DriveMLM.

AVSegFormer. [AAAI 2024] AVSegFormer: Audio-Visual Segmentation with Transformer

Qwen-VL. The official repo of Qwen-VL (通义千问-VL) chat & pretrained large vision language model proposed by Alibaba Cloud.

UniRepLKNet. [CVPR 2024 & TPAMI 2025] UniRepLKNet

Otter. 🦦 Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability.

MAT-Memory-and-Anticipation-Transformer. [ICCV 2023] Official implementation of Memory-and-Anticipation Transformer for Online Action Understanding

maws. Code and models for the paper "The effectiveness of MAE pre-pretraining for billion-scale pretraining" https://arxiv.org/abs/2303.13496

LVIS-INSTRUCT4V.

Yi. A series of large language models trained from scratch by developers @01-ai

Siamese-Image-Modeling. [CVPR 2023]Implementation of Siamese Image Modeling for Self-Supervised Vision Representation Learning

ControlLLM. ControlLLM: Augment Language Models with Tools by Searching on Graphs

FastChat. An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

FreeMask. [NeurIPS 2023] FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Models

PlainSeg. simple and efficient baselines for practical semantic segmentation with plain ViTs

StableLLAVA. Official repo for StableLLAVA

nougat. Implementation of Nougat Neural Optical Understanding for Academic Documents

Mini-DALLE3. Mini-DALLE3: Interactive Text to Image by Prompting Large Language Models

Chinese-LLaMA-Alpaca-2. 中文LLaMA-2 & Alpaca-2大模型二期项目 + 64K超长上下文模型 (Chinese LLaMA-2 & Alpaca-2 LLMs with 64K long context models)

DreamLLM. [ICLR 2024 Spotlight] DreamLLM: Synergistic Multimodal Comprehension and Creation

Emu. Emu Series: Generative Multimodal Models from BAAI

Awesome-Multimodal-Large-Language-Models. :sparkles::sparkles:Latest Advances on Multimodal Large Language Models

Babel-ImageNet. Jupyter Notebook

Multilingual-CLIP. OpenAI CLIP text encoders for multiple languages!

DVIS. DVIS: Decoupled Video Instance Segmentation Framework

SAM-Med2D. Official implementation of SAM-Med2D

pycocoevalcap. Python 3 support for the MS COCO caption evaluation tools

cider. python codes for CIDEr - Consensus-based Image Caption Evaluation

LLaVA. [NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.

awesome-diffusion-categorized. collection of diffusion model papers categorized by their subareas

LandMark. Python

VQA. Python

ICCV2023-Diffusion-Papers. ICCV2023-Diffusion-Papers

all-seeing. [ICLR 2024 & ECCV 2024] The All-Seeing Projects: Towards Panoptic Visual Recognition&Understanding and General Relation Comprehension of the Open World"

HQTrack. Tracking Anything in High Quality

ETRIS. [ICCV-2023] The official code of Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentation

wikiart. Full retriever for art and metadata in http://wikiart.org/