Indonesian Performance
Indonesian Scores by Model
Average of 2000 bootstraps. 95% CI are shown.
Gemma 4
Qwen 3.6Qwen 3.5
SEA-LION v4.5 (Qwen)
Muse GlimmerSEA-LION v4.8 (Nemotron)
GPT OSS
Nemotron 3 Super
Mistral Medium 3.5
GLM 4.7 FlashSEA-LION v4.8 (Nemotron)
Apertus 1.5Nemotron 3.5 LightningGemma 4 (E2B)SEA-LION v4.5 (Gemma E2B)
MERaLiON 2
Olmo 3.1
Tiny Aya WaterLegend:
Reasoning model
·
Model Size: ≤200B
·
Open models only
Indonesian Competency & Task Scores
Average of 2000 bootstraps. 95% CI are shown.
Model | ID | Cultural | Inst. Following | Knowledge | Linguistic Diagnostics | Multi-Turn Chat | NLG | NLR | NLU | Safety |
|---|---|---|---|---|---|---|---|---|---|---|
Gemma 4 31B | 79.75 [-1.25, +1.20] | 76.45 [-5.15, +5.24] | 97.75 [-2.87, +1.90] | 84.70 [-4.12, +3.76] | 78.66 [-3.54, +3.41] | 86.59 [-3.47, +3.39] | 56.53 [-0.72, +0.71] | 90.35 [-1.76, +1.57] | 83.18 [-3.10, +2.85] | 63.57 [-6.23, +5.79] |
Qwen 3.6 27B Alibaba | 77.96 [-1.22, +1.20] | 77.18 [-5.51, +5.41] | 96.69 [-3.12, +2.48] | 84.63 [-3.75, +3.54] | 68.60 [-3.73, +3.44] | 86.05 [-3.54, +3.42] | 55.18 [-0.59, +0.62] | 89.96 [-1.74, +1.65] | 82.66 [-2.92, +2.85] | 60.74 [-5.83, +5.44] |
Qwen 3.5 122B MoE Alibaba | 77.93 [-1.29, +1.25] | 73.23 [-5.58, +5.58] | 94.69 [-3.26, +2.58] | 86.01 [-3.68, +3.57] | 72.64 [-3.64, +3.48] | 88.03 [-3.14, +2.98] | 55.11 [-0.59, +0.60] | 90.07 [-1.78, +1.62] | 80.66 [-3.19, +2.97] | 60.89 [-6.57, +6.23] |
SEA-LION v4.5 (Qwen) 27B AISG | 76.51 [-1.23, +1.23] | 76.34 [-5.89, +5.62] | 95.94 [-3.09, +2.39] | 83.27 [-3.85, +3.69] | 63.20 [-3.98, +3.83] | 83.50 [-3.59, +3.46] | 54.77 [-0.57, +0.59] | 89.50 [-1.85, +1.82] | 82.30 [-3.13, +3.03] | 59.74 [-5.90, +5.54] |
Muse Glimmer 30B Meta | 75.61 [-1.13, +1.10] | 61.95 [-5.93, +5.89] | 94.16 [-4.16, +3.22] | 83.91 [-3.95, +3.76] | 66.77 [-3.67, +3.63] | 87.98 [-2.96, +2.80] | 53.85 [-0.51, +0.50] | 83.89 [-1.93, +1.93] | 83.99 [-2.85, +2.73] | 64.01 [-3.53, +3.32] |
SEA-LION v4.8 (Nemotron) 120B MoE AISG | 73.10 [-1.16, +1.21] | 70.02 [-5.50, +5.30] | 86.76 [-5.10, +4.55] | 83.85 [-3.69, +3.61] | 56.20 [-3.92, +3.81] | 86.07 [-3.45, +3.43] | 53.95 [-0.52, +0.59] | 87.25 [-1.84, +1.84] | 78.77 [-3.13, +3.10] | 54.98 [-3.48, +3.36] |
 GPT OSS 120B MoE mxfp4 OpenAI | 72.28 [-1.35, +1.33] | 69.72 [-4.95, +5.01] | 88.90 [-5.21, +4.67] | 75.92 [-4.55, +4.50] | 55.90 [-4.01, +3.65] | 85.28 [-4.53, +3.89] | 53.81 [-0.49, +0.50] | 85.70 [-2.09, +1.95] | 78.74 [-3.25, +3.20] | 56.58 [-5.55, +5.19] |
Nemotron 3 Super 120B MoE NVIDIA | 70.78 [-1.22, +1.20] | 67.85 [-4.71, +5.02] | 84.89 [-4.78, +4.63] | 82.57 [-3.61, +3.64] | 54.77 [-3.79, +3.61] | 82.70 [-3.77, +3.50] | 52.87 [-0.49, +0.53] | 84.12 [-2.08, +1.93] | 77.92 [-3.09, +3.01] | 49.32 [-4.31, +3.76] |
Mistral Medium 3.5 128B Mistral AI | 70.09 [-1.18, +1.13] | 39.25 [-4.87, +4.76] | 91.18 [-4.40, +3.94] | 79.64 [-4.10, +4.02] | 62.20 [-3.61, +3.49] | 83.49 [-4.32, +4.07] | 55.78 [-0.71, +0.74] | 84.15 [-2.01, +1.84] | 79.56 [-3.20, +3.28] | 55.53 [-3.52, +3.18] |
GLM 4.7 Flash 30B MoE Z.ai | 67.89 [-1.32, +1.27] | 61.66 [-4.50, +4.81] | 85.53 [-4.46, +4.24] | 73.97 [-4.09, +3.95] | 48.16 [-3.42, +3.53] | 73.18 [-5.34, +5.29] | 53.88 [-0.57, +0.57] | 83.41 [-1.92, +1.83] | 73.57 [-3.32, +3.26] | 57.62 [-5.19, +4.75] |
SEA-LION v4.8 (Nemotron) 30B MoE AISG | 65.87 [-1.29, +1.33] | 58.29 [-4.64, +4.87] | 80.27 [-5.27, +4.97] | 76.11 [-4.32, +4.35] | 46.04 [-3.82, +3.78] | 84.54 [-3.84, +3.41] | 52.49 [-0.63, +0.68] | 82.58 [-2.22, +2.11] | 66.45 [-3.38, +3.12] | 46.07 [-5.69, +4.93] |
Apertus 1.5 70B Swiss AI | 64.69 [-1.37, +1.36] | 67.32 [-4.63, +4.59] | 88.82 [-4.41, +3.80] | 56.15 [-5.48, +5.35] | 42.67 [-3.71, +3.94] | 70.23 [-4.48, +4.24] | 53.99 [-0.53, +0.57] | 76.28 [-2.98, +2.88] | 69.35 [-3.82, +3.80] | 57.36 [-5.41, +4.78] |
Nemotron 3.5 Lightning 30B MoE NVIDIA | 64.34 [-1.36, +1.37] | 48.82 [-4.68, +4.74] | 75.70 [-6.05, +5.73] | 75.70 [-4.33, +4.30] | 46.64 [-3.97, +3.79] | 82.32 [-4.19, +3.88] | 50.89 [-0.58, +0.61] | 80.43 [-2.33, +2.18] | 63.35 [-3.34, +3.48] | 55.19 [-5.68, +5.20] |
Gemma 4 (E2B) 5B | 64.13 [-1.39, +1.41] | 54.52 [-4.77, +4.44] | 90.23 [-4.64, +4.29] | 53.28 [-5.28, +4.93] | 43.55 [-3.76, +3.66] | 77.45 [-5.11, +4.96] | 53.28 [-0.47, +0.49] | 77.17 [-2.50, +2.46] | 72.87 [-3.78, +3.49] | 54.84 [-5.45, +5.68] |
SEA-LION v4.5 (Gemma E2B) 5B MoE AISG | 63.96 [-1.38, +1.37] | 55.34 [-4.73, +4.92] | 89.17 [-4.76, +4.17] | 52.95 [-4.99, +4.80] | 44.51 [-3.57, +3.65] | 75.96 [-4.77, +4.83] | 53.70 [-0.51, +0.52] | 76.06 [-2.44, +2.37] | 72.97 [-3.62, +3.40] | 55.02 [-5.58, +5.62] |
MERaLiON 2 10B A*STAR | 58.01 [-1.66, +1.65] | 44.10 [-5.45, +5.64] | 72.44 [-7.33, +6.97] | 50.25 [-5.80, +5.70] | 44.88 [-4.58, +4.72] | 54.76 [-5.15, +5.14] | 55.33 [-0.69, +0.67] | 72.98 [-2.86, +2.74] | 71.10 [-3.81, +3.60] | 56.21 [-5.72, +5.10] |
Olmo 3.1 32B AI2 | 55.89 [-1.56, +1.62] | 54.90 [-4.61, +4.64] | 82.76 [-6.10, +5.57] | 45.45 [-5.70, +5.76] | 27.66 [-3.78, +4.05] | 71.90 [-4.92, +4.61] | 51.62 [-0.55, +0.54] | 66.97 [-3.53, +3.59] | 62.84 [-4.24, +3.97] | 38.91 [-6.21, +7.03] |
Tiny Aya Water 3B CohereLabs | 33.23 [-1.13, +1.13] | 0.00 [-0.00, +0.00] | 70.29 [-6.00, +5.19] | 34.32 [-4.99, +5.06] | 13.43 [-2.02, +2.41] | 30.62 [-4.69, +4.81] | 40.71 [-0.53, +0.50] | 30.60 [-2.48, +2.35] | 46.21 [-3.03, +3.09] | 32.90 [-2.50, +2.51] |
Legend:
Reasoning model
·
Model Size: ≤200B
·
Open models only