tiktokenizer. Converts BPE tokenizers from HuggingFace Transformers format to OpenAI Tiktoken format

github.com/shakedzy/tiktokenizer
by @shakedzy

Problem

Audience

Keep exploring

Updates

No founder updates yet. Check back soon.