Ji Lin

VILA. VILA is a family of state-of-the-art vision language models (VLMs) for diverse multimodal AI tasks across the edge, data center, and cloud.

llama_cu_awq. llama INT4 cuda inference with AWQ

TensorRT-LLM. TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

Qwen-VL. The official repo of Qwen-VL (通义千问-VL) chat & pretrained large vision language model proposed by Alibaba Cloud.

streaming-llm. [ICLR 2024] Efficient Streaming Language Models with Attention Sinks

lmdeploy. LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

TinyLlama. The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.

AutoAWQ. AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:

generative_agents. Generative Agents: Interactive Simulacra of Human Behavior

LightLLM. LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

Visual-Instruction-Tuning. SVIT: Scaling up Visual Instruction Tuning

RedPajama-Data. The RedPajama-Data repository contains code for preparing large datasets for training large language models.

vllm. A high-throughput and memory-efficient inference and serving engine for LLMs

mmc4. MultimodalC4 is a multimodal extension of c4 that interleaves millions of images with text.

llm-awq. [MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

qlora. QLoRA: Efficient Finetuning of Quantized LLMs

fastcomposer. [IJCV] FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention

IF. Python

StableLM. StableLM: Stability AI Language Models

segment-anything. The repository provides code for running inference with the SegmentAnything Model (SAM), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.

open_flamingo. An open-source framework for training large multimodal models.

TaskMatrix. Python

Prompt-Engineering-Guide. 🐙 Guides, papers, lessons, notebooks and resources for prompt engineering, context engineering, RAG, and AI Agents.

llama_index. LlamaIndex is the leading document agent and OCR platform

stanford_alpaca. Code and documentation to train Stanford's Alpaca models, and generate the data.

prismer. The implementation of "Prismer: A Vision-Language Model with Multi-Task Experts".

MiniChain. A tiny library for coding with large language models.

llama. Inference code for Llama models

FlexLLMGen. Running large language models on a single GPU for throughput-oriented scenarios.

peft. 🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

autocut. 用文本编辑器剪视频

Painter. Painter & SegGPT Series: Vision Foundation Models from BAAI

tiny-training. On-Device Training Under 256KB Memory [NeurIPS'22]

stable-diffusion-2-gui. Lightweight Stable Diffusion v 2.1 web UI: txt2img, img2img, depth2img, inpaint and upscale4x.

smoothquant. [ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

sige. [NeurIPS 2022, T-PAMI 2023] Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models

transformers. 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

cleanlab. Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.

lm-evaluation-harness. A framework for few-shot evaluation of language models.

tinyengine. [NeurIPS 2020] MCUNet: Tiny Deep Learning on IoT Devices; [NeurIPS 2021] MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning; [NeurIPS 2022] MCUNetV3: On-Device Training Under 256KB Memory

stable-diffusion. A latent text-to-image diffusion model

skypilot. Run, manage, and scale AI workloads on any AI infrastructure. Use one system to access & manage all AI compute (Kubernetes, Slurm, 20+ clouds, on-prem).

UHDM. (ECCV2022) This is the official PyTorch implementation of ECCV2022 paper: Towards Efficient and Scale-Robust Ultra-High-Definition Image Demoireing

parti.

mcunet. [NeurIPS 2020] MCUNet: Tiny Deep Learning on IoT Devices; [NeurIPS 2021] MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning

DepthShrinker. [ICML 2022] "DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks", by Yonggan Fu, Haichuan Yang, Jiayi Yuan, Meng Li, Cheng Wan, Raghuraman Krishnamoorthi, Vikas Chandra, and Yingyan (Celine) Lin.

ml-ane-transformers. Reference implementation of the Transformer architecture optimized for Apple Neural Engine (ANE)

CogVideo. text and image to video generation: CogVideoX (2024) and CogVideo (ICLR 2023)

mlc-en. Python

BEVFusion. Offical PyTorch implementation of "BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework"

bevfusion. [ICRA'23] BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation

DALLE2-pytorch. Implementation of DALL-E 2, OpenAI's updated text-to-image synthesis neural network, in Pytorch

open_clip. An open source implementation of CLIP.

DeCLIP. Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm

HowToLiveLonger. 程序员延寿指南 | A programmer's guide to live longer

Splice. Official Pytorch Implementation for "Splicing ViT Features for Semantic Appearance Transfer" presenting "Splice" (CVPR 2022 Oral)

alpa. Training and serving large-scale neural networks with auto parallelization.

RepLKNet-pytorch. Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs (CVPR 2022)

RepLKNet. Official MegEngine implementation of RepLKNet

FastFold. Optimizing AlphaFold Training and Inference on GPU Clusters

cosFormer. [ICLR 2022] Official implementation of cosformer-attention in cosFormer: Rethinking Softmax in Attention

temporal-shift-module. [ICCV 2019] TSM: Temporal Shift Module for Efficient Video Understanding

dietgpu. GPU implementation of a fast generalized ANS (asymmetric numeral system) entropy encoder and decoder, with extensions for lossless compression of numerical and other data types in HPC/ML applications.

SWAG. Official repository for "Revisiting Weakly Supervised Pre-Training of Visual Perception Models". https://arxiv.org/abs/2201.08371.

ffcv. FFCV: Fast Forward Computer Vision (and other ML workloads!)

spconv. Spatial Sparse Convolution Library

instant-ngp. Instant neural graphics primitives: lightning fast NeRF and more

ConvNeXt. Code release for ConvNeXt model

glide-text2im. GLIDE: a diffusion-based text-conditional image synthesis model

gangealing. Official PyTorch Implementation of "GAN-Supervised Dense Visual Alignment" (CVPR 2022 Oral, Best Paper Finalist)

MAE-pytorch. Unofficial PyTorch implementation of Masked Autoencoders Are Scalable Vision Learners

xcit. Official code Cross-Covariance Image Transformer (XCiT)

ColossalAI. Making large AI models cheaper, faster and more accessible

micronet. Python

NonDeepNetworks. Official Code for "Non-deep Networks"

Merlion. Merlion: A Machine Learning Framework for Time Series Intelligence

esm. Evolutionary Scale Modeling (esm): Pretrained language models for proteins

PASS. The PASS dataset: pretrained models and how to get the data

ImageNet21K. Official Pytorch Implementation of: "ImageNet-21K Pretraining for the Masses"(NeurIPS, 2021) paper

CLIP. CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image

RobustVideoMatting. Robust Video Matting in PyTorch, TensorFlow, TensorFlow.js, ONNX, CoreML!

AppleNeuralHash2ONNX. Convert Apple NeuralHash model for CSAM Detection to ONNX.

dynamic-video-depth. Code for the SIGGRAPH 2021 paper "Consistent Depth of Moving Objects in Video".

Neural-Video-Portrait-Relighting-in-Real-time-via-Consistency-Modeling. Neural Video Portrait Relighting in Real-time via Consistency Modeling

triton. Development repository for the Triton language and compiler

alias-free-gan. Alias-Free GAN project website and code

YOLOX. YOLOX is a high-performance anchor-free YOLO, exceeding yolov3~v5 with MegEngine, ONNX, TensorRT, ncnn, and OpenVINO supported. Documentation: https://yolox.readthedocs.io/

actnn. ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training

Video-Swin-Transformer. This is an official implementation for "Video Swin Transformers".

DeDLOC. Official code for "Distributed Deep Learning in Open Collaborations" (NeurIPS 2021)

ml-hypersim. Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding

UniDet. Object detection on multiple datasets with an automatically learned unified label space.

d2l-tvm. Dive into Deep Learning Compiler

decision-transformer. Official codebase for Decision Transformer: Reinforcement Learning via Sequence Modeling.

clean-fid. PyTorch - FID calculation with proper image resizing and quantization steps [CVPR 2022]

pytorchvideo. A deep learning library for video understanding research.

albumentations. Fast and flexible image augmentation library. Paper about the library: https://www.mdpi.com/2078-2489/11/2/125

depthstillation. Demo code for paper "Learning optical flow from still images", CVPR 2021.

swapping-autoencoder-pytorch. Official Implementation of Swapping Autoencoder for Deep Image Manipulation (NeurIPS 2020)

StyleCLIP. Official Implementation for "StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery" (ICCV 2021 Oral)

NN-CUDA-Example. Several simple examples for popular neural network toolkits calling custom CUDA operators.

OpenPCDet. OpenPCDet Toolbox for LiDAR-based 3D Object Detection.

Swin-Transformer. This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows".

YOLOF. You Only Look One-level Feature (YOLOF), CVPR2021, Detectron2

YOLOF. Python

co-mod-gan. [ICLR 2021, Spotlight] Large Scale Image Completion via Co-Modulated Generative Adversarial Networks

onnx-tensorflow. Tensorflow Backend for ONNX

onnxsim. Simplify your onnx model

sefa. [CVPR 2021] Closed-Form Factorization of Latent Semantics in GANs

dm-haiku. JAX-based neural network library

vissl. VISSL is FAIR's library of extensible, modular and scalable components for SOTA Self-Supervised Learning with images.

anycost-gan. [CVPR 2021] Anycost GANs for Interactive Image Synthesis and Editing

DALL-E. PyTorch package for the discrete VAE used for DALL·E.

metfaces-dataset. Python

FuSta. Hybrid Neural Fusion for Full-frame Video Stabilization

RepVGG.

natural-language-image-search. Search photos on Unsplash using natural language

RepVGG. RepVGG: Making VGG-style ConvNets Great Again

awesome-NeRF. A curated list of awesome neural radiance fields papers

tinyml. Python

mdeq. [NeurIPS'20] Multiscale Deep Equilibrium Models

PyTorch-LBFGS. A PyTorch implementation of L-BFGS.

pixel2style2pixel. Official Implementation for "Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation" (CVPR 2021) presenting the pixel2style2pixel (pSp) framework

stylegan2-ada. StyleGAN2 with adaptive discriminator augmentation (ADA) - Official TensorFlow implementation

aw_nas. aw_nas: A Modularized and Extensible NAS Framework

gen-efficientnet-pytorch. Pretrained EfficientNet, EfficientNet-Lite, MixNet, MobileNetV3 / V2, MNASNet A1 and B1, FBNet, Single-Path NAS

contrastive-unpaired-translation. Contrastive unpaired image-to-image translation, faster and lighter training than cyclegan (ECCV 2020, in PyTorch)

mmaction2. OpenMMLab's Next Generation Video Understanding Toolbox and Benchmark

SciencePlots. Matplotlib styles for scientific plotting

deep-gradient-compression. [ICLR 2018] Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

fourier-feature-networks. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

data-efficient-gans. [NeurIPS 2020] Differentiable Augmentation for Data-Efficient GAN Training

decord. An efficient video loader for deep learning with smart shuffling that's super easy to digest

mmselfsup. OpenMMLab Self-Supervised Learning Toolbox and Benchmark

hardware-aware-transformers. [ACL'20] HAT: Hardware-Aware Transformers for Efficient Natural Language Processing

tvm. Open Machine Learning Compiler Framework

gpt-3. GPT-3: Language Models are Few-Shot Learners

detr. End-to-End Object Detection with Transformers

Yet-Another-EfficientDet-Pytorch. The pytorch re-implement of the official efficientdet with SOTA performance in real time and pretrained weights.

moco. PyTorch implementation of MoCo: https://arxiv.org/abs/1911.05722

gan-compression. [CVPR 2020] GAN Compression: Efficient Architectures for Interactive Conditional GANs

simclr. SimCLRv2 - Big Self-Supervised Models are Strong Semi-Supervised Learners

once-for-all. [ICLR 2020] Once for All: Train One Network and Specialize it for Efficient Deployment

ClassyVision. An end-to-end PyTorch framework for image and video classification

SinGAN. Official pytorch implementation of the paper: "SinGAN: Learning a Generative Model from a Single Natural Image"

MinkowskiEngine. Minkowski Engine is an auto-diff neural network library for high-dimensional sparse tensors

SlowFast. PySlowFast: video understanding codebase from FAIR for reproducing state-of-the-art video models.

detectron2. Detectron2 is a platform for object detection, segmentation and other visual recognition tasks.

MobilePose. Light-weight Single Person Pose Estimator

torchprofile. Count the MACs / FLOPs of PyTorch models

amdim-public. Public repo for Augmented Multiscale Deep InfoMax representation learning

CMC. [arXiv 2019] "Contrastive Multiview Coding", also contains implementations for MoCo and InstDis

mmdetection. OpenMMLab Detection Toolbox and Benchmark

amc. [ECCV 2018] AMC: AutoML for Model Compression and Acceleration on Mobile Devices

nasbench. NASBench: A Neural Architecture Search Dataset and Benchmark

Stylized-ImageNet. Code to create Stylized-ImageNet, a stylized version of standard ImageNet (ICLR 2019 Oral)

texture-vs-shape. Pre-trained models, data, code & materials from the paper "ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness" (ICLR 2019 Oral)

pytorch-mobilenet-v3. MobileNetV3 in pytorch and ImageNet pretrained models

amc-models. [ECCV 2018] AMC: AutoML for Model Compression and Acceleration on Mobile Devices

TorchSeg. Fast, modular reference implementation and easy training of Semantic Segmentation algorithms in PyTorch.

PathLevel-EAS. Path-Level Network Transformation for Efficient Architecture Search, in ICML 2018.

knowledge-distillation-pytorch. A PyTorch implementation for exploring deep and shallow knowledge distillation (KD) experiments with flexibility

torch-gel. Group elastic net implementation in PyTorch.

N2N. Python

pytorch-mobilenet-v2. A PyTorch implementation of MobileNet V2 architecture and pretrained model.

blocksparse. Efficient GPU kernels for block-sparse matrix multiplication and convolution

impl-pruning-TF. Implementation of "Iterative pruning" on TensorFlow

caffe2. Caffe2 is a lightweight, modular, and scalable deep learning framework.

gym. A toolkit for developing and comparing reinforcement learning algorithms.

pytorch-tutorial. PyTorch Tutorial for Deep Learning Researchers

caffe. Caffe: a fast open framework for deep learning.

DenseNet. Densely Connected Convolutional Networks, In CVPR 2017 (Best Paper Award).

deep-residual-networks. Deep Residual Learning for Image Recognition

pytorch. Tensors and Dynamic neural networks in Python with strong GPU acceleration

iina. The modern video player for macOS.

EmotiW2016. Jupyter Notebook

caffe-tensorflow. Caffe models in TensorFlow

dgd_person_reid. Domain Guided Dropout for Person Re-identification

tensorflow. An Open Source Machine Learning Framework for Everyone

torch7. http://torch.ch

NCR_Code. This package contains the Matlab code implementing the NCR algorithm described in ECCV 2014 paper "Consistent Re-identification in a Camera Network".