tokenizers-converter. tokenizers-converter is a command-line tool for converting slow tokenizers to their fast tokenizer counterparts.

github.com/johnmai-dev/tokenizers-converter
by @johnmai-dev

Problem

Audience

Keep exploring

Updates

No founder updates yet. Check back soon.