SEA Performance
Overall SEA Average
Average of 2000 bootstraps. 95% CI are shown.
Model Size: ≤200B
Open instruct models only
Language Performance by Model
Average of 2000 bootstraps. 95% CI are shown.
Model Size: ≤200B
Open instruct models only
Model | SEA | MY | TL | ID | MS | TA | TH | VI |
|---|---|---|---|---|---|---|---|---|
 Gemma 4 31B | 75.19 [-0.64, +0.61] | 69.13 [-2.16, +2.13] | 77.12 [-1.48, +1.53] | 78.92 [-1.34, +1.26] | 76.73 [-1.77, +1.77] | 76.23 [-1.93, +1.86] | 71.09 [-1.40, +1.30] | 77.09 [-1.71, +1.76] |
 SEA-LION v4.5 (Qwen) 27B AISG | 72.63 [-0.67, +0.63] | 63.22 [-2.39, +2.20] | 73.28 [-1.53, +1.52] | 75.60 [-1.38, +1.46] | 73.82 [-1.84, +1.76] | 75.62 [-1.91, +1.78] | 72.32 [-1.26, +1.28] | 74.52 [-1.53, +1.63] |
 Qwen 3.5 122B MoE Alibaba | 70.92 [-0.64, +0.66] | 61.41 [-2.15, +2.06] | 72.33 [-1.40, +1.41] | 75.57 [-1.25, +1.31] | 74.54 [-1.74, +1.75] | 70.79 [-2.06, +2.16] | 67.47 [-1.50, +1.54] | 74.33 [-1.54, +1.54] |
Qwen 3.6 27B Alibaba | 67.23 [-0.61, +0.61] | 47.92 [-1.92, +1.75] | 67.43 [-1.43, +1.41] | 74.05 [-1.25, +1.26] | 72.60 [-1.71, +1.75] | 67.74 [-1.87, +1.91] | 68.85 [-1.38, +1.34] | 72.00 [-1.51, +1.55] |
 Llama 4 Scout 109B MoE Meta | 63.43 [-0.78, +0.79] | 52.70 [-2.58, +2.49] | 65.28 [-1.67, +1.68] | 67.86 [-1.58, +1.56] | 68.81 [-2.11, +2.05] | 63.63 [-2.30, +2.34] | 61.81 [-1.66, +1.69] | 63.92 [-1.95, +2.02] |
 Mistral Medium 3.5 128B Mistral AI | 60.40 [-0.66, +0.67] | 32.78 [-1.76, +1.94] | 66.74 [-1.57, +1.63] | 67.83 [-1.37, +1.29] | 66.62 [-1.94, +1.82] | 61.37 [-1.91, +1.83] | 61.13 [-1.60, +1.58] | 66.33 [-1.86, +1.78] |
SEA-LION v4.5 (Gemma E2B) 5B MoE AISG | 49.29 [-0.77, +0.75] | 33.58 [-2.70, +2.60] | 53.04 [-1.87, +1.87] | 57.13 [-1.59, +1.66] | 55.34 [-2.03, +2.01] | 47.69 [-1.74, +1.72] | 44.94 [-1.64, +1.63] | 53.31 [-2.04, +2.01] |
 Nemotron 3 Super 120B MoE NVIDIA | 47.97 [-0.55, +0.55] | 6.17 [-0.96, +0.94] | 55.51 [-1.51, +1.57] | 63.84 [-1.41, +1.39] | 62.25 [-1.81, +1.76] | 26.83 [-1.29, +1.27] | 59.28 [-1.44, +1.46] | 61.93 [-1.80, +1.65] |
Gemma 4 (E2B) 5B | 47.25 [-0.75, +0.75] | 32.48 [-2.77, +2.82] | 51.63 [-1.96, +2.00] | 54.22 [-1.51, +1.58] | 52.97 [-1.87, +1.76] | 46.89 [-1.85, +1.73] | 42.68 [-1.58, +1.55] | 49.85 [-1.85, +1.86] |
 MERaLiON 2 10B A*STAR | 44.14 [-0.63, +0.68] | 10.64 [-1.37, +1.50] | 54.24 [-1.69, +1.66] | 58.01 [-1.66, +1.65] | 55.29 [-1.90, +1.95] | 34.10 [-1.54, +1.46] | 44.83 [-1.72, +1.68] | 51.89 [-1.81, +1.85] |
 Olmo 3.1 32B AI2 | 40.88 [-0.69, +0.64] | 6.06 [-1.06, +1.13] | 48.91 [-1.92, +1.87] | 55.89 [-1.56, +1.62] | 58.14 [-2.02, +2.01] | 24.19 [-1.62, +1.61] | 41.86 [-1.92, +1.80] | 51.07 [-1.89, +2.01] |
 GLM 4.7 Flash 30B MoE Z.ai | 34.80 [-0.52, +0.53] | 10.22 [-1.15, +1.12] | 39.92 [-1.50, +1.55] | 46.89 [-1.29, +1.21] | 48.84 [-1.55, +1.53] | 20.58 [-1.51, +1.59] | 33.06 [-1.33, +1.36] | 44.11 [-1.38, +1.44] |
 Apertus 8B Swiss AI | 27.54 [-0.49, +0.49] | 8.16 [-0.83, +0.88] | 24.15 [-1.35, +1.36] | 35.67 [-1.20, +1.23] | 40.59 [-1.56, +1.63] | 16.67 [-1.12, +1.09] | 33.05 [-1.32, +1.31] | 34.46 [-1.37, +1.43] |
 Tiny Aya Water 3B CohereLabs | 19.75 [-0.40, +0.43] | 4.84 [-0.84, +0.88] | 24.05 [-1.30, +1.19] | 33.23 [-1.13, +1.13] | 32.71 [-1.47, +1.54] | 3.24 [-0.60, +0.65] | 8.04 [-0.74, +0.75] | 32.15 [-1.36, +1.29] |