Dong Zhang

MiMo-Code. MiMo Code: Where Models and Agents Co-Evolve

MiMo-V2.5-ASR. Robust Speech Recognition Across Languages, Dialects, and Complex Acoustic Scenarios

MiMo-V2-Flash. MiMo-V2-Flash: Efficient Reasoning, Coding, and Agentic Foundation Model

MiMo-Audio-Tokenizer. A unified tokenizer that is capable of both extracting semantic information and enabling high-fidelity audio reconstruction.

MiMo-Audio-Eval. Python

MiMo-Audio. MiMo-Audio: Audio Language Models are Few-Shot Learners

MiMo-VL. MiMo-VL

MiMo. MiMo: Unlocking the Reasoning Potential of Language Model – From Pretraining to Posttraining

RULER. This repo contains the source code for RULER: What’s the Real Context Size of Your Long-Context Language Models?

elevenlabs-mcp. The official ElevenLabs MCP server

Seed-Thinking-v1.5.

OSUM. OSUM & OSUM-EChat, open speech understanding model and empathetic spoken chatbot based on it, open-sourced by ASLP@NPU.

UnifiedReward. Official implementation of UnifiedReward & [NeurIPS 2025] UnifiedReward-Think & UnifiedReward-Flex

paraspeechcaps. Codebase for 'Scaling Rich Style-Prompted Text-to-Speech Datasets'

Megatron-LM. Ongoing research training transformer models at scale

FlowDec. An neural full-band audio codec for general audio sampled at 48 kHz with 7.5 kps or 4.5 kbps.

verl. verl/HybridFlow: A Flexible and Efficient RL Post-Training Framework

Spark-TTS. Spark-TTS Inference Code

csm. A Conversational Speech Generation Model

slamkit. SlamKit is an open source tool kit for efficient training of SpeechLMs. It was used for "Slamming: Training a Speech Language Model on One GPU in a Day"

Awesome-RL-based-Reasoning-MLLMs. This repository provides valuable reference for researchers in the field of multimodality, please start your exploratory travel in RL-based Reasoning MLLMs!

LightZero. [NeurIPS 2023 Spotlight] LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios (awesome MCTS)

VideoWorld. [CVPR 2025] VideoWorld is a simple generative model that learns purely from unlabeled videos—much like how babies learn by observing their environment.

Awesome-LLM-Strawberry. A collection of LLM papers, blogs, and projects, with a focus on OpenAI o1 🍓 and reasoning techniques.

Zonos. Zonos-v0.1 is a leading open-weight text-to-speech model trained on more than 200k hours of varied multilingual speech, delivering expressiveness and quality on par with—or even surpassing—top TTS providers.

focalcodec. A low-bitrate single-codebook 16 / 24 kHz speech codec based on focal modulation

VideoRAG. [KDD'2026] "VideoRAG: Chat with Your Videos"

preference-flow-matching. Official code implementation for the work Preference Alignment with Flow Matching (NeurIPS 2024)

hibiki. Hibiki is a model for streaming speech translation (also known as simultaneous translation). Unlike offline translation—where one waits for the end of the source utterance to start translating--- Hibiki adapts its flow to accumulate just enough context to produce a correct translation in real-time, chunk by chunk.

audiobox-aesthetics. Unified automatic quality assessment for speech, music, and sound.

audio_understanding. Python

FireRedASR. Open-source industrial-grade ASR models supporting Mandarin, Chinese dialects and English, achieving a new SOTA on public Mandarin ASR benchmarks, while also offering outstanding singing lyrics recognition capability.

SpeechGPT-2.0-preview. GPT-4o-level, real-time spoken dialogue system.

openai-realtime-agents. This is a simple demonstration of more advanced, agentic patterns built on top of the Realtime API.

llama.cpp. LLM inference in C/C++

audio-foundation-model-dataset.

VITA. ✨✨[NeurIPS 2025] VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction

ebook2audiobook. Generate audiobooks from e-books, voice cloning & 1158+ languages!

emphases. Crowdsourced and Automatic Speech Prominence Estimation

CMMLU. CMMLU: Measuring massive multitask language understanding in Chinese

3D-Speaker. A Repository for Single- and Multi-modal Speaker Verification, Speaker Recognition and Speaker Diarization

fastrtc. The python library for real-time communication

blt. Code for BLT research paper

flow_matching. A PyTorch library for implementing flow matching algorithms, featuring continuous and discrete flow matching implementations. It includes practical examples for both text and image modalities.

LLM-RL-Papers. Monitoring recent cross-research on LLM & RL on arXiv for control. If there are good papers, PRs are welcome.

open-notebook. An Open Source implementation of Notebook LM with more flexibility and features

stable-codec. A family of state-of-the-art Transformer-based audio codecs for low-bitrate high-quality audio coding.

Freeze-Omni. ✨✨Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM

AudioSetCaps. A 6-million Audio-Caption Paired Dataset Built with a LLMs and ALMs-based Automatic Pipeline

usdm. Official PyTorch implementation of "Paralinguistics-Aware Speech-Empowered LLMs for Natural Conversation" (NeurIPS 2024)

ClearerVoice-Studio. An AI-Powered Speech Processing Toolkit and Open Source SOTA Pretrained Models, Supporting Speech Enhancement, Separation, and Target Speaker Extraction, etc.

podcastfy. An Open Source Python alternative to NotebookLM's podcast feature: Transforming Multimodal Content into Captivating Multilingual Audio Conversations with GenAI

CrisperWhisper. Verbatim Automatic Speech Recognition with improved word-level timestamps and filler detection

UltraChat. Large-scale, Informative, and Diverse Multi-round Chat Data (and Models)

magpie. [ICLR 2025] Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing. Your efficient and high-quality synthetic data generation pipeline!

PixIT. Companion repo for the paper "PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings" published at Odyssey 2024

DiariZen. A toolkit for speaker diarization.

svoice. We provide a PyTorch implementation of the paper Voice Separation with an Unknown Number of Multiple Speakers In which, we present a new method for separating a mixed audio sequence, in which multiple voices speak simultaneously. The new method employs gated neural networks that are trained to separate the voices at multiple processing steps, while maintaining the speaker in each output channel fixed. A different model is trained for every number of possible speakers, and the model with the largest number of speakers is employed to select the actual number of speakers in a given sample. Our method greatly outperforms the current state of the art, which, as we show, is not competitive for more than two speakers.

denoiser. Real Time Speech Enhancement in the Waveform Domain (Interspeech 2020)We provide a PyTorch implementation of the paper Real Time Speech Enhancement in the Waveform Domain. In which, we present a causal speech enhancement model working on the raw waveform that runs in real-time on a laptop CPU. The proposed model is based on an encoder-decoder architecture with skip-connections. It is optimized on both time and frequency domains, using multiple loss functions. Empirical evidence shows that it is capable of removing various kinds of background noise including stationary and non-stationary noises, as well as room reverb. Additionally, we suggest a set of data augmentation techniques applied directly on the raw waveform which further improve model performance and its generalization abilities.

StreamingBench. StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding

sglang. SGLang is a high-performance serving framework for large language models and multimodal models.

VoxInstruct. VoxInstruct: Expressive Human Instruction-to-Speech Generation with Unified Multilingual Codec Language Modelling

reverb. Open source inference code for Rev's model

sentence-transformers. State-of-the-Art Embeddings, Retrieval, and Reranking

CLAP. Contrastive Language-Audio Pretraining

Whisper-Finetune. Fine-tune the Whisper speech recognition model to support training without timestamp data, training with timestamp data, and training without speech data. Accelerate inference and support Web deployment, Windows desktop deployment, and Android deployment

awesome-speech-enhancement. speech enhancement\speech seperation\sound source localization

rnnoise. Recurrent neural network for audio noise reduction

EmpatheticDialogues. Dialogue model that produces empathetic responses when trained on the EmpatheticDialogues dataset.

BitStack. [ICLR 2025] BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments

hertz-dev. first base model for full-duplex conversational audio

NeurIPS2024-CoVoMix. Official repo for CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations

lm-evaluation-harness. A framework for few-shot evaluation of language models.

simple-evals. Python

NCSSD. Generative Expressive Conversational Speech Synthesis (Accepted by MM'2024)

awesome-diarization. A curated list of awesome Speaker Diarization papers, libraries, datasets, and other resources.

CleanS2S. High-quality and streaming Speech-to-Speech interactive agent in a single file. 只用一个文件实现的流式全双工语音交互原型智能体！

MMAU. Python

FireRedTTS. An Open-Sourced LLM-empowered Foundation TTS System

moonshine. Very low latency speech to text, intent recognition, and text to speech, for building voice agents and interfaces

lingua. Meta Lingua: a lean, efficient, and easy-to-hack codebase to research LLMs.

Westlake-Omni. Python

spiritlm. Inference code for the paper "Spirit-LM Interleaved Spoken and Written Language Model".

MossFormer2. This is the audio sample repository for speech separation model "MossFormer2".

emotional-speech-annotations. This repository contains prompts & best practices to annotate audio clips with a very high degree of details using Audio-Language-Models

ichigo. Local realtime voice AI

audioset_tagging_cnn. Python

test. Measuring Massive Multitask Language Understanding | ICLR 2021

AudioClassification-Pytorch. The Pytorch implementation of sound classification supports EcapaTdnn, PANNS, TDNN, Res2Net, ResNetSE and other models, as well as a variety of preprocessing methods.

Pyramid-Flow. [ICLR 2025] Pyramidal Flow Matching for Efficient Video Generative Modeling

F5-TTS. Official code for "F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching"

SpeechCraft. The official repository of SpeechCraft dataset, a large-scale expressive bilingual speech dataset with natural language descriptions.

EmoLLM. 心理健康大模型 (LLM x Mental Health), Pre & Post-training & Dataset & Evaluation & Depoly & RAG, with InternLM / Qwen / Baichuan / DeepSeek / Mixtral / LLama / GLM series models

ultimatevocalremovergui. GUI for a Vocal Remover that uses Deep Neural Networks.

Chinese-Podcasts. 播客 🎧 编程、设计、Vlog、音乐、访谈、博客...

wesep. Target Speaker Extraction Toolkit

firecrawl. The API to search, scrape, and interact with the web at scale. 🔥

peoples-speech. The People’s Speech Dataset

mlx. MLX: An array framework for Apple silicon

voxpopuli. A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation

openai-realtime-console. React app for inspecting, building and debugging with the Realtime API

voicechat2. Local SRT/LLM/TTS Voicechat

voicefixer. General Speech Restoration

Emu3. Next-Token Prediction is All You Need

Speech-Separation-Paper-Tutorial. A must-read paper for speech separation based on neural networks

Liger-Kernel. Efficient Triton Kernels for LLM Training

OmniSenseVoice. Omni SenseVoice: High-Speed Speech Recognition with words timestamps 🗣️🎯

pyannote-audio. Neural building blocks for speaker diarization: speech activity detection, speaker change detection, overlapped speech detection, speaker embedding

DNS-Challenge. This repo contains the scripts, models, and required files for the Deep Noise Suppression (DNS) Challenge.

whisper-timestamped. Multilingual Automatic Speech Recognition with word-level timestamps and confidence

DeepFilterNet. Noise supression using deep filtering

SoCodec. Ultra-low-bitrate Speech Codec for Speech Language Modeling Applications

videollm-online. VideoLLM-online: Online Video Large Language Model for Streaming Video (CVPR 2024)

audioset-processing. Toolkit for downloading and processing Google's AudioSet dataset.

gazelle-inference. proof of concept conversation orchestrator with a speech-language model

RSTnet. Real-time Speech-Text Foundation Model Toolkit (wip)

moshi. Moshi is a speech-text foundation model and full-duplex spoken dialogue framework. It uses Mimi, a state-of-the-art streaming neural audio codec.

ParallelWaveGAN. Unofficial Parallel WaveGAN (+ MelGAN & Multi-band MelGAN & HiFi-GAN & StyleMelGAN) with Pytorch

libri-light. dataset for lightly supervised training using the librivox audio book recordings. https://librivox.org/.

vocalsound. Dataset and baseline code for the VocalSound dataset (ICASSP2022).

CNVVE. Dataset and Benchmark for Classifying Non-verbal Voice Expressions

insanely-fast-whisper. Jupyter Notebook

salmon. The official code for the SALMon🍣 benchmark (ICASSP 2025 - Oral)

versa. Versatile Evaluation of Speech and Audio

LLaMA-Omni. LLaMA-Omni is a low-latency and high-quality end-to-end speech interaction model built upon Llama-3.1-8B-Instruct, aiming to achieve speech capabilities at the GPT-4o level.

wespeaker. Research and Production Oriented Speaker Verification, Recognition and Diarization Toolkit

GigaSpeech2. An evolving, large-scale and multi-domain ASR corpus for low-resource languages with automated crawling, transcription and refinement

OpenRLHF. An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray (PPO & DAPO & REINFORCE++ & VLM & TIS & vLLM & Ray & Async RL)

BigCodec. Official implementation of the paper "BigCodec: Pushing the Limits of Low-Bitrate Neural Speech Codec"

whisper_streaming. Whisper realtime streaming for long speech-to-text transcription and translation

codec-bpe. Implementation of Acoustic BPE (Shen et al., 2024), extended for RVQ-based Neural Audio Codecs

speech-dataset-generator. 🔊 Create labeled datasets, enhance audio quality, identify speakers, support diverse dataset types. 🎧👥📊 Advanced audio processing.

ten-framework. Open-source framework for conversational voice AI agents

xcodec. AAAI 2025: Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model

mini-omni. open-source multimodal large language model that can hear, talk while thinking. Featuring real-time end-to-end speech input and streaming audio output conversational capabilities.

kmeans_pytorch. kmeans using PyTorch

DynamicVectorQuantization. Official Pytorch Implementation of Our CVPR2023 Paper: "Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantization"

WavTokenizer. [ICLR 2025] SOTA discrete acoustic codec models with 40/75 tokens per second for audio language modeling

kmcuda. Large scale K-means and K-nn implementation on NVIDIA GPU / CUDA

whisperX. WhisperX: Automatic Speech Recognition with Word-level Timestamps (& Diarization)

StreamDiffusion. StreamDiffusion: A Pipeline-Level Solution for Real-Time Interactive Generation

audiocraft. Audiocraft is a library for audio processing and generation with deep learning. It features the state-of-the-art EnCodec audio compressor / tokenizer, along with MusicGen, a simple and controllable music generation LM with textual and melodic conditioning.

align-anything. Align Anything: Training All-modality Model with Feedback

faster-whisper. Faster Whisper transcription with CTranslate2

transfusion-pytorch. Pytorch implementation of Transfusion, "Predict the Next Token and Diffuse Images with One Multi-Modal Model", from MetaAI

laughter-synthesis. Official implementation of the paper "Laughter Synthesis using Pseudo Phonetic Tokens with a Large-scale In-the-wild Laughter Corpus" accepted by INTERSPEECH 2023.

laughter-detection. Python

LanguageAgentTreeSearch. [ICML 2024] Official repository for "Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models"

silero-models. Silero Models: pre-trained text-to-speech models made embarrassingly simple

silero-vad. Silero VAD: pre-trained enterprise-grade Voice Activity Detector

youtube-dl. Command-line program to download videos from YouTube.com and other video sites

Show-o. [ICLR & NeurIPS 2025] Repository for Show-o series, One Single Transformer to Unify Multimodal Understanding and Generation.

AAR. [Official Implementation] Acoustic Autoregressive Modeling 🔥

speech-to-speech. Build local voice agents with open-source models

evaluate-zero-shot-tts. Evaluation Protocol for Large-Scale Zero-Shot TTS Literature

SimpleSpeech. The open source code for SimpleSpeech series

rtvi-web-demo. Example UI implementing the RTVI web client

efficient-speech-codec. [EMNLP 2024] ESC: Efficient Speech Coding with Cross-Scale Residual Vector Quantized Transformers

whisper-medusa. Whisper with Medusa heads

AffectGPT. EMER, OV-MER (ICML25), AffectGPT (ICML25, Oral), EmoPrefer (ICLR26)

UnifiedMLLM. UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With Large Language Model

AudioBench. AudioBench: A Universal Benchmark for Audio Large Language Models

MInference. [NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

llm-mcts. [NeurIPS 2023] We use large language models as commonsense world model and heuristic policy within Monte-Carlo Tree Search, enabling better-reasoned decision-making for daily task planning problems.

persona-hub. Official repo for the paper "Scaling Synthetic Data Creation with 1,000,000,000 Personas"

reward-bench. RewardBench: the first evaluation tool for reward models.

anole. [Extended verision ICLR 2025 Blog Track] Anole: An Open, Autoregressive and Native Multimodal Models for Interleaved Image-Text Generation

CosyVoice. Multi-lingual large voice generation model, providing inference, training and deployment full-stack ability.

SenseVoice. Multilingual speech understanding: ASR + emotion recognition + audio event detection. 50+ languages, 15x faster than Whisper, non-autoregressive.

open-tts-tracker.

edge-tts. Use Microsoft Edge's online text-to-speech service from Python WITHOUT needing Microsoft Edge or Windows or an API key

examples. Examples for Cerebrium Serverless GPUs

Awesome-ChatTTS. 官方推荐的 ChatTTS 资源汇总项目，整理了全网相关资源和常见问题 || Officially recommended ChatTTS resource collection project

ReST-MCTS. ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search (NeurIPS 2024)

SD-Eval. [NeurIPS 2024] SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words

benchmarks. This repository contains the SpeechBrain Benchmarks

NPC. Non-Autoregressive Predictive Coding

NAST. Official repository for NAST: Noise Aware Speech Tokenization for Speech Language Models (Interspeech 2024) https://arxiv.org/abs/2406.11037

chameleon. Repository for Meta Chameleon, a mixed-modal early-fusion foundation model from FAIR.

SEED-Voken. SEED-Voken: A Series of Powerful Visual Tokenizers

VQGAN-LC. Python

open-muse. Open reproduction of MUSE for fast text2image generation.

TTS-arxiv-daily. Automatically Update Text-to-speech (TTS) Papers Daily using Github Actions (Update Every 12th hours)

blsp-emo. BLSP-Emo: Towards Empathetic Large Speech-Language Models

OmniTokenizer. [NeurIPS 2024]OmniTokenizer: one model and one weight for image-video joint tokenization.

VideoLLaMA2. VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs

LibriTTS-P. LibriTTS-P: A Corpus with Speaking Style and Speaker Identity Prompts for Text-to-Speech and Style Captioning

ChatCaptioner. Official Repository of ChatCaptioner

MARS5-TTS. MARS5 speech model (TTS) from CAMB.AI

EmoBox. [INTERSPEECH 2024] EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmark

LlamaGen. Autoregressive Model Beats Diffusion: 🦙 Llama for Scalable Image Generation

SemantiCodec-inference. Ultra-low bitrate neural audio codec (0.31~1.40 kbps) with a better semantic in the latent space.

vits_chinese. Best practice TTS based on BERT and VITS with some Natural Speech Features Of Microsoft; Support ONNX streaming out!

StreamSpeech. StreamSpeech is an “All in One” seamless model for offline and simultaneous speech recognition, speech translation and speech synthesis.

vllm. A high-throughput and memory-efficient inference and serving engine for LLMs

Python-Wrapper-for-World-Vocoder. A Python wrapper for the high-quality vocoder "World"

GLM-4. GLM-4 series: Open Multilingual Multimodal Chat LMs | 开源多语言多模态对话模型

AudioMAE. This repo hosts the code and models of "Masked Autoencoders that Listen".

FunASR. Industrial-grade speech recognition toolkit: 170x realtime, 50+ languages, speaker diarization, emotion detection, streaming, and OpenAI-compatible API.

CV-VAE. [NeurIPS 2024] CV-VAE: A Compatible Video VAE for Latent Generative Video Models