Rare find

Tiktoken. High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.

github.com/tryAGI/Tiktoken
by @tryAGI

Problem

Audience

Keep exploring

Updates

No founder updates yet. Check back soon.