zhudong.si中文EN
TOPIC EXPLAINED

What Is a Token? The Smallest Language Unit of Large Models

Keyword TokenUpdated 2026-10-13
Quick Answer:

A token is the smallest unit large language models use to process text: the model does not read "characters" or "words" directly, but first slices the input into a sequence of tokens, then predicts the next token one by one. A sentence like "你好,世界" ("hello, world") may be split into 4–6 tokens; English averages about 4 characters per token, Chinese about 1.5–2 characters per token. Context windows (e.g., 128K), training-data scale (e.g., over 10 trillion tokens), and API billing are all counted in tokens. But the ByteDance Seed In-Depth Research Report warns: assuming a model processes more tokens per day does not mean there is more new information — total token count does not equal intelligence, and cross-company token comparisons are especially misleading.

From Text to Tokens: How Tokenization Works

What a large model sees is not Chinese characters and letters, but a string of integer IDs. Tokenization is the process of slicing text into tokens and mapping them to IDs. The mainstream method is BPE (byte-pair encoding): starting from individual characters, it repeatedly merges high-frequency adjacent pairs into new tokens, eventually forming a "vocabulary" of tens to hundreds of thousands.

An example makes it clear:

Text Possible Segmentation Notes
"artificial" "art" + "ific" + "ial" High-frequency roots kept intact
"人工智能" "人工" + "智能" Common Chinese words may form whole tokens
"supercalifragilistic" Split into 5–6 fragments Rare words get shattered
"123456" "123" + "456" Numbers often split into groups of 3 digits

This segmentation has two direct consequences. First, the same sentence consumes different numbers of tokens in Chinese versus English: English is about 4 characters per token, Chinese about 1.5–2 characters per token — Chinese users can actually fit more "meaning" into the same-length context window. Second, rare words, misspellings, and invented words get shattered into fragments, and models struggle more with them — which is why large models occasionally "act dumb" on obscure characters and complex numbers. Third, tokenization is irreversible lossy compression: once the segmentation is fixed, the model can never see the original character boundaries, which is part of why certain word games (like "count how many r's") stump large models.

The Three Most Common Token Numbers

Once you understand tokens, three numbers in the news become easy to grasp:

Context window: how long an input the model can "remember" at once, measured in tokens. A 128K context is roughly 190,000 Chinese characters or a 300,000-word English book. But note: a long context doesn't mean effective use — attention computation grows quadratically with sequence length, and "needle in a haystack" tests show the model's utilization of information in the middle of ultra-long inputs declines.

Training-data scale: Seed2.0 Pro's official training-data summary disclosed over 10 trillion tokens of pre-training text and over 1 billion images. This is among the largest training-data disclosures made public. But the report stresses that quality training data cannot be generated automatically: traffic contains massive redundancy, weak feedback, and commercial objectives; what needs study is effective feedback output, not total token count.

API billing: almost all large-model APIs bill by token, with different unit prices for input and output. Being "frugal with words" when writing prompts saves real money — which is also why prompt engineering has economic value.

Why Token Counts Don't Equal Intelligence

This is the report's most worth-underlining point. The executive summary states plainly: ByteDance's applications and infrastructure can reduce friction between research and deployment, but cannot automatically produce quality training data. The report also lists three "water" sources in token growth: free subsidies (vendors burning money to give away API calls), long outputs (the more verbose the model, the more tokens), and repeated requests (retries, polling).

The deeper reason is measurement basis. Long contexts, video encoding, internal applications, repeated inference, price subsidies, and model routing all change call volumes — without a unified measurement basis, cross-company token comparisons are especially misleading. Company A announcing "10 trillion tokens processed daily" versus Company B announcing "1 trillion" may differ only in that one side also counts internal retries. The report accordingly refused to write the huge daily call volumes circulating online into its monitoring baseline.

The report also proposes a cost perspective: once agents scale up, each agent not only generates tokens but also runs tests, maintains files, operates browsers, and waits for external feedback — token unit price should not be used as a substitute for "cost per accepted task." The real cost is the sum of inference, tools, environment, validation, and operations.

Tokens and Scaling: Is More Always Better?

In Scaling Law discussions, tokens are the unit of measurement for the "data" dimension. The empirical rule proposed by the Chinchilla paper: given a compute budget, model parameter count and training token count should grow proportionally, with an optimal ratio of about 20 tokens per parameter. But that describes "how to allocate given a budget," not "more tokens is always better."

The report's judgment: foundational capability and environmental learning may be complementary, but that doesn't justify unlimited expansion of every resource. Large models may increase the return per interaction while making each interaction more expensive. The choice requires comparing quality, cost, latency, and intervention rate — not looking at a single leaderboard. In other words, tokens are fuel, but engine efficiency, road conditions (data quality), and driving skill (algorithms) equally determine how far you go.

Tokens and Chinese: Why Tokenizer Quality Matters

For Chinese users, tokenizer quality directly affects experience and cost. A good Chinese tokenizer includes high-frequency words like "的", "是", and "人工智能" in its vocabulary, each taking 1 token; a poor tokenizer may split common words into individual characters, consuming 30% more tokens for the same sentence — the context window effectively shrinks, and the API bill effectively rises.

This is also why every large-model vendor trains its own tokenizer: the vocabulary is optimized for the language distribution of the training data. Models centered on Chinese typically have more efficient Chinese tokenization. To judge whether a model is Chinese-friendly, besides leaderboard scores, you can look at its token consumption on the same Chinese text — a very practical metric. Incidentally, the open-source RL frameworks like verl initiated by ByteDance's Seed team, and the disclosure of Seed2.0 Pro's 10-trillion-plus-token training data, both show that leading labs are optimizing for "effective output per token" rather than "total token count."

FAQ

How many Chinese characters does 1 token equal?

There is no fixed answer; it depends on the tokenizer. Rough estimate: Chinese is about 1.5–2 characters per token, English about 4 characters per token. Common Chinese words (such as "的", "是", "人工智能") often each take 1 token, while rare characters may take 2–3 tokens per character. For precise calculation, use each model's official tokenizer tool.

What does a 128K context window mean?

It means the model can process up to 128,000 tokens in a single input-plus-output, roughly 190,000 Chinese characters. But note three things: information in the middle of ultra-long inputs is easily "forgotten"; attention computation grows quadratically with length, making it very expensive; and many tasks never fill such a long window anyway.

Why is Chinese more "cost-effective" for large models?

Because Chinese is information-dense: expressing the same meaning, Chinese typically uses 30%–50% fewer tokens than English. Token-billed APIs are naturally friendlier to Chinese users. But this also means Chinese tokenizer quality has a bigger impact on the experience.

Will tokens disappear? Are there alternatives?

Not in the short term. Tokens are the basic input unit of the Transformer architecture, and the entire ecosystem (training, inference, billing) is built around them. Research directions include byte-level models and perceptron-style tokenization, aiming to reduce the information loss from segmentation — but all are still in early stages.

Want to go deeper?

To deeply understand the Scaling and data questions behind tokens, we recommend the ByteDance Seed In-Depth Research Report. In sixteen chapters, the report systematically discusses AI for AI, environment learning, and system intelligence: analysis of Seed2.0 Pro's ten-trillion-token training-data disclosure, the methodology of "effective feedback output rather than total token count," the ten types of Scaling divergence, and why cross-company token comparisons mislead — helping you build a complete judgment framework for the "fuel" question of large models.

Read the full ByteDance Seed In-Depth Research Report report