⚡! Tokenization is the first thing you do in CS336 (language models from scratch). If Marcel can work his magic for the rest of the LM pipeline, the world will be a better place.
⚡! Tokenization is the first thing you do in CS336 (language models from scratch). If Marcel can work his magic for the rest of the LM pipeline, the world will be a better place.
@marcelroed:
Introducing the world’s fastest tokenizer implementation, Gigatoken!Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI’s tiktoken for most tokenizer definitions on most machines.
These baselines are already multithreaded Rust implementations! 🧵 https://t.co/zxzXPFBCHv
https://bender.layer3.press/articles/019f88b5-28c7-13a4-7158-0779dcb28e45
Write a comment