Tool 01
Compare teaching BPE with real OpenAI tiktoken encodings. Inspect token IDs, merge steps, vocabulary entries, and rough API cost estimates.
Analyzing...
| Token | ID | Match |
|---|---|---|
| No matches. | ||
Tokenization is the first step in every LLM pipeline. Before a model can reason about text, it must convert raw characters into token IDs — fixed vocabulary entries the network was trained on. Understanding tokenization explains API costs, context limits, and why the same sentence can look different across models.
LLMs do not read letters or words directly. They read integers — token IDs — that index into a vocabulary of subword pieces.
A single English word might be one token ("hello") or several ("un" + "believ" + "able"). Emoji, code, and non-English text can consume surprisingly many tokens. API pricing and context windows are counted in tokens, not characters — so tokenization directly affects cost and how much you can fit in one request.
Early approaches split text by characters or bytes. Modern models mostly use subword tokenization — especially Byte Pair Encoding (BPE).
BPE starts with individual characters and repeatedly merges the most frequent adjacent pairs until it builds a vocabulary of useful chunks. Common words become single tokens; rare words get split into smaller pieces. This balances vocabulary size with the ability to represent any text.
Each merge in BPE has a rank — lower rank means the merge happened earlier during training and is more "important." When encoding new text, the tokenizer applies merges in rank order.
In this tool's teaching BPE mode, you can watch merge steps unfold. Real production tokenizers like tiktoken ship with a pre-built merge table trained on huge corpora — you get the end result without re-running training.
Different models use different vocabularies. GPT-4 family models use cl100k_base (~100k tokens). GPT-4o and newer models use o200k_base (~200k tokens). The same sentence can produce different token counts and IDs across encodings.
Compare mode in the playground above shows these differences side by side. A few extra tokens per request adds up at scale.
Switch to Compare tokenizers and paste the same paragraph under different encodings — note the token count delta.
Try code snippets, emoji, and mixed languages to see where tokenizers struggle. Use vocabulary lookup to find how a specific piece maps to an ID. Check the cost estimate to connect token count to real API pricing.