I run an AI platform and we need to tokenize fast and early to make a lot of decisions on the subsequent steps (things like routing, rate limiting and such). Its really important to do this efficiently even though its not a large % of total end to end time for the request.

1/1000 of inference compute is a non-trivial workload at scale. Gartner estimates ~$28B in inference spend for 2026 making this a $28 million dollar per year workload (edit: based on the assumption above)

Source: https://www.gartner.com/en/newsroom/press-releases/2026-07-2...

Always good to make it 0.001%

Time to first token, especially for smaller models, can be sharply reduced.

Latency can be just as important as overall throughput, especially for inference providers like Groq and Cerebras.

[dead]

I initially had a Rust-based word cloud generator that generates word clouds in high resolution in ~100 milliseconds whereas it would take other generators a couple seconds to do the same thing. Does the world need a super-fast word cloud generator? No. Do I want a super-fast word cloud generator? Yes.

I later found enough optimizations to reduce the generation speed all the way down to ~16ms. Do I need a word cloud generator that fast? No. But if I have a word cloud generator it's going to be as fast as possible dammit.

"The pursuit of excellence does not need justification."

https://x.com/mitchellh/status/2074225453217505494

This depends on what your workflow is. There are use cases for tokenization that don't always involve immediately feeding the text into a model.

1000x faster on 0.1% of runtime = 0.1% saved. Amdahl remains undefeated.

Globally that’s ~50 GWh/yr, or ~5.7 MW continuous:

• 4,700 American homes

• a week of British tea

It's pretty funny but then again, why not if it's as trivial to simplify as it appears

The problem with the rest of inference is that changes are not trivially correct or incorrect, as they are with the tokenization layer.

I'm sure there's been a lot more effort put into the other, more consequential, portions of inference time.

the answer is many! This would take hours to write. Full teams and research on nearly every part. So many 'unlocks' coming.

Author here! In my case it's mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at a token-level instead of a text level. In this case we usually run for days on a huge number of CPUs to finish tokenizing something like DCLM.

From what I can tell it's also useful for inference when considering time-to-first-token (TTFT) as reported by fastokens.[0]

I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache. If you have a long prefix that's been seen before (say a system prompt), the time for tokenizing that will be a large part of your TTFT. The tokenizer cache should be warmed up in this case, so the throughput for Gigatoken would be significantly higher than reported in the repo.

[0] https://github.com/crusoecloud/fastokens

I worked on a system a couple years ago with a BERT-based model (64M parameters) used for classification. The rest of the system could process data at gigabytes per second, and so here tokenization at a measly few megabytes per second really slowed things down. The model inference was more expensive than tokenization, but tokenization was still >10% of total runtime.

Wait, since when does it matter whether something being hyper-optimized is useful? The computer going brrrr on an interesting problem is in itself the goal!

If you are training an LLM, you need to tokenize the text before it’s trained on. A lot of time this can be done in parallel with the GPU though.

I have spent way too much time waiting 10-15 minutes tokenizing my training dataset only for the run to crash over some minor bug after that. (If I was smarter, I’d test on a smaller batch first.)

It can be useful for checking input token usage before sending it to the model, e.g. preventing calls above a given token bound or grouping requests into batches.

It can also be used by the LLMs to provide the input and output token counts on the different APIs, though I'm not sure if this is how llama.cpp or other OpenAI-like APIs calculate the input/output tokens of a request.

I've data where i cannot store metadata that i need to search semantically so i embed it on the fly at every search with static embedding and tokenizing was more than 99% of the cpu time. Granted that was due the naive implementation of the default tokenizer which was o^2 with document length and just switching to a proper scanner solved most of it without going to simd and whatnot, but still.

Pre-training data is pre-tokenized ahead of time before being used to not waste any GPU compute.

A massive speedup like this is a nice efficiency savings on some of these data pipelines for sure.

Numbers are for the Gigatoken API, but compatibility mode just means eating a bunch of Python overhead (creating lists, reading strings to bytes). You can expect a modest ~200-300x speedup with compatibility mode depending on how you use it.

I can add some benchmarks for compatibility mode in the future. I have a little more juice to squeeze out of the Python interop though, so not quite ready for it yet.

Both the example libraries compared (tokenizers and tiktoken) are Rust-based with Python bindings. There's just a few levers in Rust that can speed it up even more particularly with LLM assistance as the AI Use Discloure here notes:

> Final profiling stages and the last ~4x worth of performance from eliminating branching and improving the pretoken cache hierarchy

1 year ago everyone would have called you insane for suggesting this. Now we all shrug and say yeah maybe we can do this and it’s actually a good idea?

We should rewrite all Rust code in Python. Not for any technical reason. I'm just sick of the Rust cult at this point.