Tokenizers Analysis

This page counts the tokens in each tokenizer's vocabulary by writing system, with a focus on Southeast Asian (SEA) tokens. Each table row shows one tokenizer, named after the model family it comes from. If other models use the same tokenizer, the row lists them too.

22,434

All SEA tokens

SEA-LION v1

8,875

Thai

SEA-LION v1

4,001

Vietnamese

SEA-LION v1

4,862

Tamil

SEA-LION v1

2,361

Khmer

SEA-LION v1

1,247

Lao

Tiny Aya +3

1,675

Burmese

SEA-LION v1

SEA Token Counts

Vocabulary tokens in each Southeast Asian script

Thai

Vietnamese

Tamil

Khmer

Lao

Burmese

Other SEA tokens

Token Counts

Southeast Asian tokens shown by default

Tokenizer
All SEA tokens
Thai
Vietnamese
Tamil
Khmer
Lao
Burmese
Other SEA tokens
22,434
8,875
4,001
4,862
2,361
660
1,675
–
CohereLabs logo
CohereLabs logo
Tiny Aya

CohereLabs

Tiny Aya Fire, Tiny Aya Global, Tiny Aya Water

12,198
3,551
2,362
2,923
1,370
1,247
745
–
Alibaba logo
Alibaba logo
Qwen 3.5

Alibaba

Qwen 3.5, Qwen 3.6, Qwen 3.8, …

8,919
5,741
2,647
268
79
37
147
–
Google logo
Google logo
Gemma 3

Google

Gemma 3, Gemma 4, Gemma 4 (E2B), …

8,903
2,177
1,455
3,850
599
302
511
9
Thinking Machines logo
Thinking Machines logo
Inkling 2

Thinking Machines

5,483
746
1,799
2,283
216
130
309
–
4,824
2,302
1,954
558
8
1
1
–
4,719
513
851
408
881
846
1,220
–
Meta logo
Meta logo
Llama 4

Meta

Llama 4 Maverick, Llama 4 Scout, Muse Glimmer, …

4,641
1,994
1,871
80
344
180
172
–
4,626
572
924
1,566
582
401
581
–
OpenAI logo
OpenAI logo
GPT OSS

OpenAI

GPT OSS, Inkling, Inkling Small

4,300
1,561
1,120
974
348
2
295
–
Alibaba logo
Alibaba logo
Qwen 1.5

Alibaba

Babel, Baichuan M2, Baichuan M3, …

4,056
2,571
1,270
30
38
33
37
77
Huawei logo
Huawei logo
openPangu 2.0

Huawei

openPangu 2.0 Pro

3,967
3,246
477
110
62
35
37
–
Liquid AI logo
Liquid AI logo
LFM2.5

Liquid AI

LFM3

3,875
2,244
999
233
185
–
214
–
3,510
2,089
1,359
22
16
12
10
2
Google logo
Google logo
Gemma 2

Google

Gemma 2, MERaLiON 2, MERaLiON 2 ASR, …

3,371
1,287
1,413
325
88
56
124
78
LG AI Research logo
LG AI Research logo
EXAONE 4.5

LG AI Research

K-EXAONE 2.0

2,694
157
2,439
73
3
–
22
–
Meta logo
Meta logo
Llama 3

Meta

Distill Llama (R1), Llama 3, Llama 3.1, …

2,573
1,394
1,171
5
3
–
–
–
Upstage logo
Upstage logo
Solar Open

Upstage

Solar Open2

2,380
1,663
662
13
–
1
32
9
Mistral AI logo
Mistral AI logo
Mistral Nemo

Mistral AI

Apertus, Apertus 1.5, Ministral 2410, …

2,188
567
864
538
–
–
219
–
CohereLabs logo
CohereLabs logo
Aya 23

CohereLabs

Aya 23, Aya Expanse, Command A 03-2025, …

2,104
49
2,029
15
3
–
8
–
2,077
504
751
373
152
88
209
–
2,057
404
1,313
125
41
41
133
–
inclusionAI logo
inclusionAI logo
Ling 2.6

inclusionAI

Ling 2.6, Ling 3.0 Flash, Ling 3.0 Tiny, …

1,946
1,148
790
8
–
–
–
–
DeepSeek logo
DeepSeek logo
DeepSeek V3

DeepSeek

DeepSeek R1, DeepSeek R1 0528, DeepSeek V3, …

1,702
1,267
188
167
18
3
59
–
Z.ai logo
Z.ai logo
GLM 4.7 Flash

Z.ai

GLM 5.1, GLM 5.2, GLM 5.3, …

1,275
145
835
5
130
–
160
–
China Telecom logo
China Telecom logo
Xing4.0

China Telecom

1,192
529
432
81
38
43
69
–
1,101
608
318
93
30
2
50
–
Baidu logo
Baidu logo
ERNIE 4.5

Baidu

ERNIE 4.5

1,062
238
377
156
68
77
146
–
898
58
832
5
3
–
–
–
BSC-LT logo
BSC-LT logo
Salamandra

BSC-LT

ALIA, Salamandra

827
–
827
–
–
–
–
–
Tencent logo
Tencent logo
Hy3

Tencent

799
388
320
74
3
–
14
–
Recursive logo
Recursive logo
Flint

Recursive

666
199
454
8
3
–
2
–
IFM logo
IFM logo
K2 Horizon

IFM

Jais 3, K2 Horizon

636
165
250
98
3
15
105
–
Arcee AI logo
Arcee AI logo
Trinity

Arcee AI

Trinity Large Thinking

508
118
234
65
19
22
50
–
Moonshot AI logo
Moonshot AI logo
Kimi K2

Moonshot AI

Kimi K2.5, Kimi K2.6, Kimi K3, …

480
114
65
73
20
13
195
–
353
250
78
25
–
–
–
–
204
156
47
1
–
–
–
–
AI2 logo
AI2 logo
Olmo 2

AI2

Granite 4.2, Olmo 2 0325, Olmo 2 1124, …

181
58
115
5
3
–
–
–
181
58
115
5
3
–
–
–
Mistral AI logo
Mistral AI logo
Mistral v0.3

Mistral AI

Mistral Large 2407, Mistral Large 2411, Mistral Small 2409

179
57
49
22
32
2
17
–
Minimax logo
Minimax logo
MiniMax M2

Minimax

MiniMax M2.1, MiniMax M2.5, MiniMax M2.7, …

178
42
65
71
–
–
–
–
153
54
54
19
9
2
15
–
SK Telecom logo
SK Telecom logo
A.X K2

SK Telecom

150
59
45
4
–
–
42
–
Poolside logo
Poolside logo
Laguna M.1

Poolside

Laguna S 2.1, Laguna XS 2.1

145
48
71
20
6
–
–
–
LG AI Research logo
LG AI Research logo
EXAONE Deep

LG AI Research

EXAONE 4.0

127
53
49
17
–
–
8
–
105
41
51
11
–
–
2
–
No records

  • The Vietnamese column counts tokens that contain Vietnamese-only letters, such as Æ¡, ư, and ạ.
  • Malay, Indonesian, and Filipino use the Latin script, so their tokens cannot be differentiated from other Latin-script tokens.