SEA Performance
Overall SEA Average
Average of 30 bootstraps. 95% CI are shown.
Model Size: ≤200B
Open instruct models only
 31B 72.02±0.03 |
 27B 69.97±0.05 |
 122B MoE 68.40±0.06 |
27B 68.34±0.07 |
26B MoE 68.18±0.03 |
27B 66.08±0.06 |
32B 65.96±0.03 |
32B 64.53±0.05 |
27B 63.58±0.06 |
27B 63.43±0.05 |
70B 62.81±0.08 |
 109B MoE 62.77±0.04 |
35B MoE 61.65±0.08 |
8B 61.23±0.06 |
 128B 60.83±0.08 |
12B 60.75±0.02 |
35B MoE 59.68±0.09 |
70B 59.44±0.05 |
8B 56.97±0.05 |
8B 56.03±0.06 |
9B 54.73±0.10 |
9B 51.61±0.11 |
4B 51.53±0.07 |
119B MoE 50.96±0.09 |
5B 50.93±0.05 |
5B 50.10±0.06 |
4B 49.75±0.07 |
4B 49.11±0.06 |
 120B MoE 48.50±0.09 |
8B 47.94±0.09 |
4B 45.84±0.07 |
 10B 44.69±0.07 |
4B 41.91±0.08 |
 32B 40.49±0.07 |
8B 40.38±0.06 |
 30B MoE 37.03±0.12 |
8B 36.96±0.06 |
 8B 31.22±0.09 |
30B MoE 28.65±0.09 |
3B 27.80±0.05 |
3B 21.70±0.10 |
 3B 21.49±0.11 |
3B 20.84±0.07 |
Language Performance by Model
Average of 30 bootstraps. 95% CI are shown.
Model Size: ≤200B
Open instruct models only
Model | SEA | MY | TL | ID | MS | TA | TH | VI |
|---|---|---|---|---|---|---|---|---|
Gemma 4 31B | 72.02 ± 0.03 | 67.29 ± 0.08 | 76.32 ± 0.08 | 77.36 ± 0.07 | 70.89 ± 0.09 | 75.41 ± 0.16 | 62.35 ± 0.07 | 74.51 ± 0.08 |
SEA-LION v4.5 (Qwen) 27B AISG | 69.97 ± 0.05 | 61.11 ± 0.16 | 71.99 ± 0.15 | 74.51 ± 0.13 | 69.69 ± 0.10 | 75.00 ± 0.14 | 66.42 ± 0.11 | 71.08 ± 0.13 |
Qwen 3.5 122B MoE Alibaba | 68.40 ± 0.06 | 59.23 ± 0.20 | 71.13 ± 0.18 | 74.18 ± 0.12 | 69.77 ± 0.13 | 72.23 ± 0.14 | 61.21 ± 0.15 | 71.05 ± 0.19 |
Qwen 3.5 27B Alibaba | 68.34 ± 0.07 | 59.35 ± 0.19 | 69.62 ± 0.20 | 74.38 ± 0.14 | 69.61 ± 0.20 | 72.77 ± 0.18 | 63.08 ± 0.13 | 69.55 ± 0.15 |
Gemma 4 26B MoE | 68.18 ± 0.03 | 61.39 ± 0.12 | 72.00 ± 0.09 | 72.95 ± 0.08 | 68.76 ± 0.08 | 72.20 ± 0.16 | 60.14 ± 0.10 | 69.80 ± 0.11 |
Qwen 3.6 27B Alibaba | 66.08 ± 0.06 | 53.70 ± 0.25 | 66.43 ± 0.23 | 73.00 ± 0.15 | 68.45 ± 0.21 | 69.04 ± 0.16 | 63.24 ± 0.19 | 68.70 ± 0.21 |
SEA-LION v4 (Qwen) 32B AISG | 65.96 ± 0.03 | 56.13 ± 0.11 | 67.77 ± 0.12 | 70.91 ± 0.09 | 67.13 ± 0.12 | 71.47 ± 0.13 | 61.24 ± 0.08 | 67.09 ± 0.11 |
Qwen 3 VL 32B Alibaba | 64.53 ± 0.05 | 47.65 ± 0.14 | 66.55 ± 0.14 | 71.79 ± 0.08 | 68.06 ± 0.10 | 68.99 ± 0.16 | 61.74 ± 0.10 | 66.93 ± 0.09 |
SEA-LION v4 (Gemma) 27B AISG | 63.58 ± 0.06 | 50.14 ± 0.19 | 70.53 ± 0.16 | 68.27 ± 0.10 | 66.72 ± 0.17 | 69.66 ± 0.20 | 54.67 ± 0.13 | 65.05 ± 0.16 |
Gemma 3 27B | 63.43 ± 0.05 | 50.53 ± 0.20 | 70.17 ± 0.13 | 68.35 ± 0.09 | 66.47 ± 0.13 | 69.47 ± 0.19 | 54.36 ± 0.13 | 64.66 ± 0.14 |
SEA-LION v3 (Llama) 70B AISG | 62.81 ± 0.08 | 38.82 ± 0.42 | 69.38 ± 0.18 | 70.23 ± 0.14 | 67.68 ± 0.20 | 65.46 ± 0.30 | 59.78 ± 0.17 | 68.31 ± 0.23 |
Llama 4 Scout 109B MoE Meta | 62.77 ± 0.04 | 53.85 ± 0.15 | 64.82 ± 0.12 | 68.00 ± 0.10 | 65.61 ± 0.14 | 67.81 ± 0.14 | 54.85 ± 0.08 | 64.44 ± 0.12 |
Qwen 3.6 35B MoE Alibaba | 61.65 ± 0.08 | 40.64 ± 0.25 | 63.64 ± 0.21 | 68.88 ± 0.13 | 66.72 ± 0.22 | 68.62 ± 0.19 | 56.26 ± 0.18 | 66.79 ± 0.23 |
Gemma 4 (E4B) 8B | 61.23 ± 0.06 | 50.93 ± 0.19 | 64.22 ± 0.17 | 66.08 ± 0.14 | 63.54 ± 0.12 | 65.96 ± 0.18 | 56.16 ± 0.20 | 61.70 ± 0.12 |
Mistral Medium 3.5 128B Mistral AI | 60.83 ± 0.08 | 35.98 ± 0.34 | 65.65 ± 0.18 | 69.30 ± 0.14 | 66.21 ± 0.13 | 67.37 ± 0.19 | 54.67 ± 0.22 | 66.64 ± 0.17 |
Gemma 3 12B | 60.75 ± 0.02 | 45.78 ± 0.00 | 66.76 ± 0.08 | 66.22 ± 0.06 | 63.86 ± 0.07 | 66.51 ± 0.08 | 52.60 ± 0.00 | 63.53 ± 0.06 |
Qwen 3.5 35B MoE Alibaba | 59.68 ± 0.09 | 40.84 ± 0.29 | 62.42 ± 0.23 | 64.52 ± 0.17 | 65.52 ± 0.24 | 65.00 ± 0.22 | 55.33 ± 0.20 | 64.13 ± 0.27 |
Llama 3.3 70B Meta | 59.44 ± 0.05 | 26.16 ± 0.20 | 66.84 ± 0.13 | 69.68 ± 0.08 | 67.05 ± 0.12 | 62.71 ± 0.17 | 56.90 ± 0.07 | 66.76 ± 0.12 |
SEA-LION v4 (Qwen VL) 8B AISG | 56.97 ± 0.05 | 35.79 ± 0.19 | 56.21 ± 0.15 | 67.39 ± 0.10 | 64.21 ± 0.10 | 53.53 ± 0.17 | 57.38 ± 0.08 | 64.26 ± 0.11 |
Qwen 3 VL 8B Alibaba | 56.03 ± 0.06 | 34.40 ± 0.22 | 55.63 ± 0.16 | 66.92 ± 0.09 | 64.61 ± 0.10 | 49.60 ± 0.17 | 57.16 ± 0.12 | 63.86 ± 0.13 |
SEA-LION v3 (Gemma 2) 9B AISG | 54.73 ± 0.10 | 16.14 ± 0.19 | 62.79 ± 0.15 | 64.52 ± 0.20 | 61.58 ± 0.23 | 61.93 ± 0.22 | 54.10 ± 0.22 | 62.03 ± 0.16 |
Qwen 3.5 9B Alibaba | 51.61 ± 0.11 | 30.82 ± 0.39 | 43.61 ± 0.23 | 61.69 ± 0.22 | 59.22 ± 0.23 | 54.40 ± 0.29 | 53.14 ± 0.21 | 58.38 ± 0.37 |
SEA-LION v4 (Qwen VL) 4B AISG | 51.53 ± 0.07 | 27.99 ± 0.17 | 51.63 ± 0.15 | 61.32 ± 0.10 | 60.58 ± 0.15 | 43.28 ± 0.19 | 56.21 ± 0.12 | 59.69 ± 0.11 |
Mistral Small 4 119B MoE Mistral AI | 50.96 ± 0.09 | 19.88 ± 0.31 | 55.42 ± 0.26 | 61.61 ± 0.19 | 58.87 ± 0.18 | 53.83 ± 0.30 | 49.10 ± 0.21 | 58.01 ± 0.19 |
SEA-LION v4.5 (Gemma E2B) 5B AISG | 50.93 ± 0.05 | 34.78 ± 0.17 | 53.01 ± 0.17 | 57.41 ± 0.10 | 57.35 ± 0.15 | 53.85 ± 0.20 | 45.32 ± 0.10 | 54.80 ± 0.17 |
Gemma 4 (E2B) 5B | 50.10 ± 0.06 | 33.40 ± 0.18 | 52.42 ± 0.18 | 56.55 ± 0.11 | 57.63 ± 0.18 | 52.92 ± 0.19 | 44.30 ± 0.18 | 53.51 ± 0.13 |
SEA-LION v4 (Gemma VL) 4B AISG | 49.75 ± 0.07 | 27.07 ± 0.16 | 56.63 ± 0.15 | 56.39 ± 0.11 | 57.10 ± 0.13 | 55.10 ± 0.19 | 43.61 ± 0.15 | 52.38 ± 0.14 |
Qwen 3 VL 4B Alibaba | 49.11 ± 0.06 | 24.36 ± 0.21 | 46.72 ± 0.17 | 60.83 ± 0.12 | 59.90 ± 0.16 | 38.44 ± 0.22 | 54.07 ± 0.12 | 59.45 ± 0.10 |
NVIDIA Nemotron 3 Super 120B MoE NVIDIA | 48.50 ± 0.09 | 7.21 ± 0.17 | 58.90 ± 0.19 | 63.14 ± 0.18 | 62.91 ± 0.20 | 33.59 ± 0.42 | 53.42 ± 0.18 | 60.31 ± 0.19 |
SEA-LION v3 (Llama) 8B AISG | 47.94 ± 0.09 | 19.12 ± 0.27 | 52.17 ± 0.25 | 58.03 ± 0.16 | 58.08 ± 0.20 | 46.95 ± 0.25 | 46.52 ± 0.18 | 54.70 ± 0.21 |
Gemma 3 4B | 45.84 ± 0.07 | 22.17 ± 0.16 | 54.19 ± 0.14 | 54.52 ± 0.10 | 54.74 ± 0.15 | 44.25 ± 0.19 | 40.61 ± 0.17 | 50.40 ± 0.15 |
MERaLiON 2 10B A*STAR | 44.69 ± 0.07 | 9.91 ± 0.20 | 52.26 ± 0.16 | 58.36 ± 0.14 | 55.68 ± 0.22 | 43.79 ± 0.24 | 42.98 ± 0.23 | 49.83 ± 0.21 |
Qwen 3.5 4B Alibaba | 41.91 ± 0.08 | 16.24 ± 0.22 | 36.22 ± 0.33 | 54.43 ± 0.22 | 52.38 ± 0.25 | 36.74 ± 0.23 | 44.29 ± 0.19 | 53.07 ± 0.28 |
Olmo 3.1 32B AI2 | 40.49 ± 0.07 | 7.33 ± 0.12 | 51.45 ± 0.14 | 53.36 ± 0.14 | 55.07 ± 0.16 | 30.81 ± 0.23 | 34.13 ± 0.16 | 51.32 ± 0.19 |
SEA-LION v4 (Apertus) 8B AISG | 40.38 ± 0.06 | 18.09 ± 0.21 | 40.85 ± 0.20 | 47.36 ± 0.13 | 50.92 ± 0.15 | 38.82 ± 0.27 | 42.94 ± 0.16 | 43.71 ± 0.21 |
GLM 4.7 Flash 30B MoE Z.ai | 37.03 ± 0.12 | 12.01 ± 0.23 | 41.86 ± 0.28 | 50.35 ± 0.28 | 51.42 ± 0.34 | 24.67 ± 0.28 | 32.10 ± 0.25 | 46.83 ± 0.30 |
Llama 3.1 8B Meta | 36.96 ± 0.06 | 9.81 ± 0.22 | 41.94 ± 0.12 | 53.03 ± 0.18 | 51.69 ± 0.15 | 21.22 ± 0.18 | 37.18 ± 0.15 | 43.88 ± 0.19 |
Apertus 8B Swiss AI | 31.22 ± 0.09 | 10.09 ± 0.22 | 27.54 ± 0.26 | 38.87 ± 0.21 | 45.08 ± 0.32 | 22.37 ± 0.31 | 34.48 ± 0.30 | 40.13 ± 0.24 |
NVIDIA Nemotron 3 Nano 30B MoE NVIDIA | 28.65 ± 0.09 | 1.75 ± 0.14 | 27.53 ± 0.30 | 37.29 ± 0.25 | 38.61 ± 0.24 | 16.26 ± 0.20 | 38.99 ± 0.29 | 40.13 ± 0.27 |
Llama 3.2 3B Meta | 27.80 ± 0.05 | 4.15 ± 0.19 | 26.28 ± 0.23 | 35.82 ± 0.23 | 43.00 ± 0.24 | 16.86 ± 0.20 | 36.15 ± 0.24 | 32.31 ± 0.21 |
MERaLiON 2 3B A*STAR | 21.70 ± 0.10 | 5.43 ± 0.15 | 23.52 ± 0.20 | 37.24 ± 0.16 | 31.94 ± 0.17 | 12.51 ± 0.24 | 22.94 ± 0.34 | 18.30 ± 0.25 |
Tiny Aya Water 3B CohereLabs | 21.49 ± 0.11 | 5.69 ± 0.17 | 23.62 ± 0.32 | 36.47 ± 0.26 | 35.94 ± 0.28 | 4.42 ± 0.16 | 9.62 ± 0.19 | 34.69 ± 0.32 |
Tiny Aya Global 3B CohereLabs | 20.84 ± 0.07 | 5.37 ± 0.18 | 23.67 ± 0.31 | 34.82 ± 0.26 | 35.33 ± 0.30 | 5.62 ± 0.17 | 8.04 ± 0.20 | 33.02 ± 0.30 |