Indonesian Performance
Indonesian Scores by Model
Average of 2000 bootstraps. 95% CI are shown.
GPT 6 Astra
Opus 4.8
Gemini 3.1 Pro Preview
Qwen 3.5
Gemma 4
Qwen 3.6
GLM 5.2
MiniMax M3
DeepSeek V4 Pro
SEA-LION v4.5 (Qwen)
Muse Glimmer
Inkling Small
Nemotron 3 Ultra
SEA-LION v4.8 (Nemotron)
GPT OSS
Nemotron 3 Super
Mistral Medium 3.5
Command A+
SEA-LION v4.8 (Nemotron)
Apertus 1.5
Nemotron 3.5 Lightning
Gemma 4 (E2B)
SEA-LION v4.5 (Gemma E2B)
MERaLiON 2
Olmo 3.1Legend:
Reasoning model
No open weights
·
Model Size: All
·
Open/closed models
Indonesian Competency & Task Scores
Average of 2000 bootstraps. 95% CI are shown.
Model | ID | Cultural | Inst. Following | Knowledge | Linguistic Diagnostics | Multi-Turn Chat | NLG | NLR | NLU | Safety |
|---|---|---|---|---|---|---|---|---|---|---|
GPT 6 Astra OpenAI | 81.40 [-1.24, +1.22] | 76.60 [-5.14, +4.80] | 96.72 [-3.39, +2.56] | 92.26 [-3.05, +2.66] | 84.00 [-3.55, +3.33] | 85.64 [-4.04, +3.48] | 55.55 [-0.60, +0.61] | 91.91 [-1.61, +1.47] | 86.40 [-3.00, +2.60] | 63.50 [-6.79, +5.90] |
Opus 4.8 Anthropic | 81.23 [-1.27, +1.20] | 76.13 [-6.14, +6.12] | 95.83 [-3.45, +2.63] | 90.55 [-3.18, +2.99] | 79.42 [-3.37, +3.36] | 87.66 [-3.25, +3.17] | 55.67 [-0.57, +0.58] | 90.62 [-1.72, +1.58] | 85.71 [-2.80, +2.65] | 69.44 [-6.01, +5.75] |
Gemini 3.1 Pro Preview | 81.21 [-1.24, +1.19] | 76.33 [-5.94, +5.54] | 95.25 [-3.70, +3.09] | 91.35 [-3.06, +2.90] | 83.38 [-2.93, +3.01] | 84.55 [-3.70, +3.51] | 55.88 [-0.64, +0.65] | 91.07 [-1.55, +1.54] | 85.46 [-2.81, +2.63] | 67.59 [-6.06, +5.61] |
Qwen 3.5 397B MoE Alibaba | 79.85 [-1.21, +1.24] | 74.71 [-6.23, +5.97] | 97.04 [-3.11, +2.25] | 87.96 [-3.55, +3.54] | 78.48 [-3.47, +3.42] | 88.15 [-3.07, +2.97] | 55.03 [-0.59, +0.59] | 90.44 [-1.70, +1.59] | 81.61 [-3.25, +3.12] | 65.21 [-6.09, +5.69] |
Gemma 4 31B | 79.75 [-1.25, +1.20] | 76.45 [-5.15, +5.24] | 97.75 [-2.87, +1.90] | 84.70 [-4.12, +3.76] | 78.66 [-3.54, +3.41] | 86.59 [-3.47, +3.39] | 56.53 [-0.72, +0.71] | 90.35 [-1.76, +1.57] | 83.18 [-3.10, +2.85] | 63.57 [-6.23, +5.79] |
Qwen 3.6 27B Alibaba | 77.96 [-1.22, +1.20] | 77.18 [-5.51, +5.41] | 96.69 [-3.12, +2.48] | 84.63 [-3.75, +3.54] | 68.60 [-3.73, +3.44] | 86.05 [-3.54, +3.42] | 55.18 [-0.59, +0.62] | 89.96 [-1.74, +1.65] | 82.66 [-2.92, +2.85] | 60.74 [-5.83, +5.44] |
GLM 5.2 754B MoE FP8 Z.ai | 77.79 [-1.16, +1.24] | 72.58 [-5.85, +5.54] | 96.08 [-3.10, +2.49] | 84.51 [-3.56, +3.44] | 73.58 [-3.40, +3.26] | 86.70 [-3.56, +3.26] | 53.70 [-0.59, +0.60] | 88.43 [-1.79, +1.63] | 83.79 [-2.90, +2.79] | 60.70 [-5.74, +5.60] |
MiniMax M3 428B MoE Minimax | 76.81 [-1.23, +1.21] | 75.00 [-5.27, +4.98] | 90.18 [-4.71, +4.22] | 84.59 [-3.80, +3.74] | 67.85 [-3.02, +3.08] | 85.40 [-3.52, +3.48] | 55.02 [-0.54, +0.55] | 88.58 [-1.75, +1.67] | 80.62 [-3.00, +2.84] | 64.06 [-4.71, +4.27] |
DeepSeek V4 Pro 1600B MoE DeepSeek | 76.59 [-1.13, +1.14] | 60.27 [-5.27, +5.14] | 93.44 [-3.80, +3.34] | 86.54 [-3.58, +3.63] | 74.13 [-3.55, +3.37] | 86.50 [-3.11, +2.89] | 55.77 [-0.60, +0.61] | 85.87 [-1.82, +1.82] | 83.40 [-3.00, +2.99] | 63.37 [-4.92, +4.53] |
SEA-LION v4.5 (Qwen) 27B AISG | 76.51 [-1.23, +1.23] | 76.34 [-5.89, +5.62] | 95.94 [-3.09, +2.39] | 83.27 [-3.85, +3.69] | 63.20 [-3.98, +3.83] | 83.50 [-3.59, +3.46] | 54.77 [-0.57, +0.59] | 89.50 [-1.85, +1.82] | 82.30 [-3.13, +3.03] | 59.74 [-5.90, +5.54] |
Muse Glimmer 30B Meta | 75.61 [-1.13, +1.10] | 61.95 [-5.93, +5.89] | 94.16 [-4.16, +3.22] | 83.91 [-3.95, +3.76] | 66.77 [-3.67, +3.63] | 87.98 [-2.96, +2.80] | 53.85 [-0.51, +0.50] | 83.89 [-1.93, +1.93] | 83.99 [-2.85, +2.73] | 64.01 [-3.53, +3.32] |
Inkling Small 276B MoE Thinking Machines | 75.13 [-1.07, +1.16] | 61.62 [-5.57, +5.48] | 95.96 [-2.63, +2.25] | 84.87 [-3.87, +3.83] | 65.78 [-4.09, +3.77] | 89.63 [-2.44, +2.41] | 54.10 [-0.49, +0.50] | 85.21 [-1.90, +1.74] | 80.17 [-3.17, +3.06] | 58.80 [-4.41, +3.75] |
Nemotron 3 Ultra 550B MoE NVIDIA | 74.86 [-1.39, +1.33] | 73.29 [-5.33, +5.15] | 82.65 [-6.46, +6.04] | 86.81 [-3.44, +3.23] | 68.34 [-3.59, +3.42] | 89.02 [-2.59, +2.42] | 54.19 [-0.58, +0.54] | 87.82 [-1.75, +1.64] | 80.69 [-3.01, +2.84] | 50.88 [-6.75, +6.73] |
SEA-LION v4.8 (Nemotron) 120B MoE AISG | 73.10 [-1.16, +1.21] | 70.02 [-5.50, +5.30] | 86.76 [-5.10, +4.55] | 83.85 [-3.69, +3.61] | 56.20 [-3.92, +3.81] | 86.07 [-3.45, +3.43] | 53.95 [-0.52, +0.59] | 87.25 [-1.84, +1.84] | 78.77 [-3.13, +3.10] | 54.98 [-3.48, +3.36] |
GPT OSS 120B MoE mxfp4 OpenAI | 72.28 [-1.35, +1.33] | 69.72 [-4.95, +5.01] | 88.90 [-5.21, +4.67] | 75.92 [-4.55, +4.50] | 55.90 [-4.01, +3.65] | 85.28 [-4.53, +3.89] | 53.81 [-0.49, +0.50] | 85.70 [-2.09, +1.95] | 78.74 [-3.25, +3.20] | 56.58 [-5.55, +5.19] |
Nemotron 3 Super 120B MoE NVIDIA | 70.78 [-1.22, +1.20] | 67.85 [-4.71, +5.02] | 84.89 [-4.78, +4.63] | 82.57 [-3.61, +3.64] | 54.77 [-3.79, +3.61] | 82.70 [-3.77, +3.50] | 52.87 [-0.49, +0.53] | 84.12 [-2.08, +1.93] | 77.92 [-3.09, +3.01] | 49.32 [-4.31, +3.76] |
Mistral Medium 3.5 128B Mistral AI | 70.09 [-1.18, +1.13] | 39.25 [-4.87, +4.76] | 91.18 [-4.40, +3.94] | 79.64 [-4.10, +4.02] | 62.20 [-3.61, +3.49] | 83.49 [-4.32, +4.07] | 55.78 [-0.71, +0.74] | 84.15 [-2.01, +1.84] | 79.56 [-3.20, +3.28] | 55.53 [-3.52, +3.18] |
Command A+ 218B MoE CohereLabs | 68.66 [-1.30, +1.15] | 59.90 [-4.52, +4.79] | 88.71 [-4.18, +3.79] | 77.91 [-4.11, +3.89] | 52.63 [-3.24, +3.46] | 82.75 [-3.65, +3.58] | 42.42 [-0.63, +0.64] | 83.40 [-1.85, +1.78] | 74.20 [-3.29, +3.21] | 56.06 [-5.27, +4.63] |
SEA-LION v4.8 (Nemotron) 30B MoE AISG | 65.87 [-1.29, +1.33] | 58.29 [-4.64, +4.87] | 80.27 [-5.27, +4.97] | 76.11 [-4.32, +4.35] | 46.04 [-3.82, +3.78] | 84.54 [-3.84, +3.41] | 52.49 [-0.63, +0.68] | 82.58 [-2.22, +2.11] | 66.45 [-3.38, +3.12] | 46.07 [-5.69, +4.93] |
Apertus 1.5 70B Swiss AI | 64.69 [-1.37, +1.36] | 67.32 [-4.63, +4.59] | 88.82 [-4.41, +3.80] | 56.15 [-5.48, +5.35] | 42.67 [-3.71, +3.94] | 70.23 [-4.48, +4.24] | 53.99 [-0.53, +0.57] | 76.28 [-2.98, +2.88] | 69.35 [-3.82, +3.80] | 57.36 [-5.41, +4.78] |
Nemotron 3.5 Lightning 30B MoE NVIDIA | 64.34 [-1.36, +1.37] | 48.82 [-4.68, +4.74] | 75.70 [-6.05, +5.73] | 75.70 [-4.33, +4.30] | 46.64 [-3.97, +3.79] | 82.32 [-4.19, +3.88] | 50.89 [-0.58, +0.61] | 80.43 [-2.33, +2.18] | 63.35 [-3.34, +3.48] | 55.19 [-5.68, +5.20] |
Gemma 4 (E2B) 5B | 64.13 [-1.39, +1.41] | 54.52 [-4.77, +4.44] | 90.23 [-4.64, +4.29] | 53.28 [-5.28, +4.93] | 43.55 [-3.76, +3.66] | 77.45 [-5.11, +4.96] | 53.28 [-0.47, +0.49] | 77.17 [-2.50, +2.46] | 72.87 [-3.78, +3.49] | 54.84 [-5.45, +5.68] |
SEA-LION v4.5 (Gemma E2B) 5B MoE AISG | 63.96 [-1.38, +1.37] | 55.34 [-4.73, +4.92] | 89.17 [-4.76, +4.17] | 52.95 [-4.99, +4.80] | 44.51 [-3.57, +3.65] | 75.96 [-4.77, +4.83] | 53.70 [-0.51, +0.52] | 76.06 [-2.44, +2.37] | 72.97 [-3.62, +3.40] | 55.02 [-5.58, +5.62] |
MERaLiON 2 10B A*STAR | 58.01 [-1.66, +1.65] | 44.10 [-5.45, +5.64] | 72.44 [-7.33, +6.97] | 50.25 [-5.80, +5.70] | 44.88 [-4.58, +4.72] | 54.76 [-5.15, +5.14] | 55.33 [-0.69, +0.67] | 72.98 [-2.86, +2.74] | 71.10 [-3.81, +3.60] | 56.21 [-5.72, +5.10] |
Olmo 3.1 32B AI2 | 55.89 [-1.56, +1.62] | 54.90 [-4.61, +4.64] | 82.76 [-6.10, +5.57] | 45.45 [-5.70, +5.76] | 27.66 [-3.78, +4.05] | 71.90 [-4.92, +4.61] | 51.62 [-0.55, +0.54] | 66.97 [-3.53, +3.59] | 62.84 [-4.24, +3.97] | 38.91 [-6.21, +7.03] |
Legend:
Reasoning model
No open weights
·
Model Size: All
·
Open/closed models