SEA Performance
Overall SEA Average
Average of 2000 bootstraps. 95% CI are shown.
Gemini 3.1 Pro Preview
GPT 6 AstraGemma 4
Qwen 3.5
Opus 4.8Qwen 3.6
GLM 5.2
SEA-LION v4.5 (Qwen)
DeepSeek V4 Pro
MiniMax M3
Muse Glimmer
Inkling SmallGPT OSS
Mistral Medium 3.5
Nemotron 3 UltraSEA-LION v4.8 (Nemotron)SEA-LION v4.5 (Gemma E2B)Gemma 4 (E2B)
Command A+
Apertus 1.5SEA-LION v4.8 (Nemotron)Nemotron 3 SuperNemotron 3.5 Lightning
MERaLiON 2
Olmo 3.1Legend:
Reasoning model
No open weights
·
Model Size: All
·
Open/closed models
Language Performance by Model
Average of 2000 bootstraps. 95% CI are shown.
Model | SEA | MY | TL | ID | MS | TA | TH | VI |
|---|---|---|---|---|---|---|---|---|
Gemini 3.1 Pro Preview | 77.42 [-0.60, +0.59] | 75.23 [-2.26, +2.00] | 79.24 [-1.26, +1.24] | 81.21 [-1.24, +1.19] | 76.34 [-1.78, +1.74] | 77.26 [-1.84, +1.64] | 73.98 [-1.46, +1.33] | 78.66 [-1.67, +1.49] |
 GPT 6 Astra OpenAI | 76.90 [-0.64, +0.61] | 73.54 [-2.32, +2.20] | 79.97 [-1.25, +1.22] | 81.40 [-1.24, +1.22] | 74.31 [-1.69, +1.80] | 76.44 [-2.06, +2.03] | 74.78 [-1.34, +1.29] | 77.86 [-1.67, +1.58] |
Gemma 4 31B | 76.17 [-0.60, +0.59] | 70.97 [-2.06, +1.97] | 78.44 [-1.30, +1.27] | 79.75 [-1.25, +1.20] | 76.27 [-1.70, +1.77] | 76.40 [-1.78, +1.63] | 74.30 [-1.23, +1.22] | 77.09 [-1.73, +1.64] |
Qwen 3.5 397B MoE Alibaba | 76.17 [-0.61, +0.60] | 70.68 [-2.08, +2.03] | 77.26 [-1.30, +1.21] | 79.85 [-1.21, +1.24] | 76.85 [-1.70, +1.69] | 76.66 [-1.90, +1.65] | 73.76 [-1.35, +1.30] | 78.15 [-1.56, +1.50] |
 Opus 4.8 Anthropic | 75.62 [-0.66, +0.65] | 68.29 [-2.65, +2.36] | 77.67 [-1.41, +1.30] | 81.23 [-1.27, +1.20] | 74.98 [-1.95, +1.93] | 74.75 [-2.07, +1.95] | 73.72 [-1.41, +1.34] | 78.70 [-1.47, +1.39] |
Qwen 3.6 27B Alibaba | 74.64 [-0.65, +0.60] | 68.37 [-2.18, +2.12] | 75.23 [-1.25, +1.28] | 77.96 [-1.22, +1.20] | 76.26 [-1.76, +1.72] | 75.32 [-1.90, +1.80] | 73.22 [-1.30, +1.30] | 76.12 [-1.46, +1.39] |
GLM 5.2 754B MoE FP8 Z.ai | 74.04 [-0.57, +0.55] | 68.18 [-2.01, +1.87] | 75.93 [-1.30, +1.21] | 77.79 [-1.16, +1.24] | 74.40 [-1.68, +1.63] | 73.40 [-1.98, +1.81] | 71.67 [-1.29, +1.26] | 76.93 [-1.45, +1.35] |
SEA-LION v4.5 (Qwen) 27B AISG | 73.24 [-0.56, +0.57] | 67.57 [-1.90, +1.88] | 75.16 [-1.36, +1.36] | 76.51 [-1.23, +1.23] | 75.23 [-1.73, +1.67] | 71.38 [-1.78, +1.58] | 73.00 [-1.37, +1.28] | 73.85 [-1.40, +1.40] |
DeepSeek V4 Pro 1600B MoE DeepSeek | 72.58 [-0.58, +0.54] | 66.26 [-1.96, +1.93] | 76.44 [-1.21, +1.17] | 76.59 [-1.13, +1.14] | 73.75 [-1.75, +1.79] | 72.35 [-1.78, +1.62] | 69.24 [-1.47, +1.36] | 73.40 [-1.56, +1.54] |
MiniMax M3 428B MoE Minimax | 72.31 [-0.58, +0.59] | 64.23 [-2.06, +2.01] | 73.76 [-1.35, +1.38] | 76.81 [-1.23, +1.21] | 73.83 [-1.61, +1.70] | 73.71 [-1.48, +1.56] | 70.45 [-1.36, +1.31] | 73.40 [-1.72, +1.71] |
Muse Glimmer 30B Meta | 72.13 [-0.59, +0.56] | 65.95 [-2.02, +2.01] | 74.28 [-1.32, +1.30] | 75.61 [-1.13, +1.10] | 74.14 [-1.83, +1.79] | 72.03 [-1.72, +1.63] | 69.38 [-1.31, +1.33] | 73.52 [-1.47, +1.38] |
Inkling Small 276B MoE Thinking Machines | 71.53 [-0.53, +0.52] | 63.76 [-1.89, +1.95] | 71.08 [-1.22, +1.18] | 75.13 [-1.07, +1.16] | 73.87 [-1.62, +1.53] | 73.48 [-1.49, +1.37] | 70.85 [-1.27, +1.28] | 72.52 [-1.40, +1.43] |
GPT OSS 120B MoE mxfp4 OpenAI | 66.14 [-0.64, +0.66] | 46.90 [-1.92, +2.03] | 71.38 [-1.43, +1.46] | 72.28 [-1.35, +1.33] | 69.68 [-1.82, +1.98] | 66.05 [-1.98, +1.97] | 66.88 [-1.46, +1.39] | 69.81 [-1.69, +1.59] |
Mistral Medium 3.5 128B Mistral AI | 64.71 [-0.59, +0.59] | 41.83 [-1.83, +1.81] | 70.12 [-1.41, +1.48] | 70.09 [-1.18, +1.13] | 68.93 [-1.70, +1.66] | 68.80 [-1.66, +1.61] | 64.35 [-1.46, +1.42] | 68.82 [-1.47, +1.43] |
Nemotron 3 Ultra 550B MoE NVIDIA | 64.09 [-0.66, +0.65] | 35.61 [-2.13, +2.10] | 69.85 [-1.22, +1.22] | 74.86 [-1.39, +1.33] | 71.91 [-1.65, +1.80] | 60.79 [-1.99, +1.83] | 67.02 [-1.51, +1.55] | 68.57 [-1.74, +1.79] |
SEA-LION v4.8 (Nemotron) 120B MoE AISG | 63.44 [-0.63, +0.61] | 31.35 [-1.80, +1.80] | 71.01 [-1.47, +1.39] | 73.10 [-1.16, +1.21] | 73.25 [-1.71, +1.74] | 56.35 [-1.88, +1.87] | 68.66 [-1.37, +1.39] | 70.36 [-1.65, +1.67] |
SEA-LION v4.5 (Gemma E2B) 5B MoE AISG | 59.23 [-0.66, +0.67] | 47.68 [-2.22, +2.09] | 60.73 [-1.60, +1.60] | 63.96 [-1.38, +1.37] | 62.28 [-1.83, +1.81] | 59.06 [-1.72, +1.73] | 58.94 [-1.62, +1.58] | 61.95 [-1.75, +1.78] |
Gemma 4 (E2B) 5B | 58.82 [-0.64, +0.66] | 46.95 [-2.07, +2.06] | 60.68 [-1.60, +1.60] | 64.13 [-1.39, +1.41] | 61.46 [-1.88, +1.80] | 59.03 [-1.83, +1.83] | 58.32 [-1.55, +1.58] | 61.19 [-1.66, +1.69] |
Command A+ 218B MoE CohereLabs | 56.73 [-0.54, +0.55] | 16.60 [-1.37, +1.43] | 60.08 [-1.36, +1.34] | 68.66 [-1.30, +1.15] | 64.48 [-1.67, +1.70] | 58.98 [-1.68, +1.61] | 61.85 [-1.37, +1.35] | 66.44 [-1.51, +1.56] |
Apertus 1.5 70B Swiss AI | 55.20 [-0.66, +0.61] | 19.21 [-1.22, +1.25] | 59.00 [-1.52, +1.52] | 64.69 [-1.37, +1.36] | 65.54 [-1.90, +1.72] | 54.49 [-1.96, +2.09] | 58.81 [-1.59, +1.57] | 64.66 [-1.62, +1.63] |
SEA-LION v4.8 (Nemotron) 30B MoE AISG | 51.57 [-0.52, +0.52] | 10.61 [-1.02, +0.98] | 61.82 [-1.52, +1.35] | 65.87 [-1.29, +1.33] | 62.09 [-1.70, +1.63] | 33.14 [-1.16, +1.17] | 62.60 [-1.40, +1.37] | 64.86 [-1.67, +1.63] |
Nemotron 3 Super 120B MoE NVIDIA | 49.30 [-0.51, +0.53] | 4.98 [-0.67, +0.78] | 66.92 [-1.39, +1.35] | 70.78 [-1.22, +1.20] | 68.66 [-1.71, +1.80] | 21.83 [-1.25, +1.21] | 56.61 [-1.29, +1.44] | 55.31 [-1.48, +1.55] |
Nemotron 3.5 Lightning 30B MoE NVIDIA | 46.06 [-0.54, +0.54] | 3.79 [-0.52, +0.57] | 58.69 [-1.57, +1.58] | 64.34 [-1.36, +1.37] | 58.76 [-1.68, +1.75] | 17.23 [-1.13, +1.21] | 58.66 [-1.51, +1.60] | 60.97 [-1.55, +1.52] |
MERaLiON 2 10B A*STAR | 44.14 [-0.63, +0.68] | 10.64 [-1.37, +1.50] | 54.24 [-1.69, +1.66] | 58.01 [-1.66, +1.65] | 55.29 [-1.90, +1.95] | 34.10 [-1.54, +1.46] | 44.83 [-1.72, +1.68] | 51.89 [-1.81, +1.85] |
Olmo 3.1 32B AI2 | 40.88 [-0.69, +0.64] | 6.06 [-1.06, +1.13] | 48.91 [-1.92, +1.87] | 55.89 [-1.56, +1.62] | 58.14 [-2.02, +2.01] | 24.19 [-1.62, +1.61] | 41.86 [-1.92, +1.80] | 51.07 [-1.89, +2.01] |
Legend:
Reasoning model
No open weights
·
Model Size: All
·
Open/closed models