Exact-Byte-Level-Probabilities-from-Tokenized-LMs. Example implementation of "Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles" by Buu Phan, Brandon Amos, Itai Gat, Marton Havasi, Matthew Muckley, and Karen UllrichWork conducted as part of a MetaAI internship.

github.com/facebookresearch/Exact-Byte-Level-Probabilities-from-Tokenized-LMs

Problem

Audience

Updates

No founder updates yet. Check back soon.