Malay Performance

Malay Scores by Model

Average of 30 bootstraps. 95% CI are shown.

Model Size: ≤200B

Open instruct models only

Alibaba logo
Alibaba logo
Qwen 3 Next
80B
MoE
62.80±0.12
AISG logo
AISG logo
SEA-LION v4 (Qwen)
32B
61.36±0.14
Alibaba logo
Alibaba logo
Qwen 3
30B
MoE
61.13±0.14
AISG logo
AISG logo
SEA-LION v4 (Gemma)
27B
61.10±0.16
Google logo
Google logo
Gemma 3
27B
60.92±0.17
AISG logo
AISG logo
SEA-LION v3 (Llama)
70B
59.94±0.17
Alibaba logo
Alibaba logo
Qwen 3
32B
59.67±0.17
Alibaba logo
Alibaba logo
Qwen 2.5
72B
59.48±0.14
Meta logo
Meta logo
Llama 3.3
70B
58.80±0.15
Google logo
Google logo
Gemma 3
12B
57.96±0.12
AI2 logo
AI2 logo
Tulu 3
70B
57.39±0.19
Meta logo
Meta logo
Llama 4 Scout
109B
MoE
57.38±0.11
Meta logo
Meta logo
Llama 3.1
70B
55.67±0.18
Alibaba logo
Alibaba logo
Qwen 3
14B
55.04±0.11
CohereLabs logo
CohereLabs logo
Command A 03-2025
111B
54.76±0.17
AISG logo
AISG logo
SEA-LION v3 (Gemma 2)
9B
54.70±0.18
Google logo
Google logo
Gemma 2
27B
54.06±0.22
Alibaba logo
Alibaba logo
Qwen 3
8B
53.83±0.15
Alibaba logo
Alibaba logo
Qwen 2.5
32B
53.71±0.13
Mistral AI logo
Mistral AI logo
Mistral Large 2411
123B
52.73±0.21
Alibaba logo
Alibaba logo
Qwen 2.5
14B
51.49±0.13
AISG logo
AISG logo
SEA-LION v3 (Llama)
8B
51.32±0.28
Google logo
Google logo
Gemma 2
9B
49.72±0.21
A*STAR logo
A*STAR logo
MERaLiON 2
10B
49.12±0.22
CohereLabs logo
CohereLabs logo
Aya Expanse
32B
48.35±0.20
AI2 logo
AI2 logo
Olmo 2 0325
32B
48.28±0.23
Alibaba logo
Alibaba logo
Qwen 2.5
7B
46.94±0.13
Baidu logo
Baidu logo
ERNIE 4.5
21B
MoE
45.83±0.18
Meta logo
Meta logo
Llama 3.1
8B
45.49±0.22
SAIL logo
SAIL logo
Sailor2
20B
45.19±0.17
Mistral AI logo
Mistral AI logo
Mistral Small 3.1 2503
24B
44.39±0.22
CohereLabs logo
CohereLabs logo
Command R+ 08-2024
104B
44.12±0.26
CohereLabs logo
CohereLabs logo
Aya Expanse
8B
44.12±0.18
SAIL logo
SAIL logo
Sailor2
8B
43.90±0.19
Meta logo
Meta logo
Llama 3
70B
43.42±0.10
AI2 logo
AI2 logo
Tulu 3
8B
42.86±0.14
Swiss AI logo
Swiss AI logo
Apertus
70B
42.69±0.24
Microsoft logo
Microsoft logo
phi-4
14B
39.71±0.24
CohereLabs logo
CohereLabs logo
Command R 08-2024
32B
39.00±0.21
Swiss AI logo
Swiss AI logo
Apertus
8B
38.91±0.31
AI2 logo
AI2 logo
Olmo 2 1124
13B
38.54±0.19
Alibaba-DAMO logo
Alibaba-DAMO logo
SeaLLMs V3
7B
36.82±0.28
Meta logo
Meta logo
Llama 3
8B
35.77±0.16
Alibaba-DAMO logo
Alibaba-DAMO logo
Babel
9B
31.66±0.22
AI2 logo
AI2 logo
Olmo 2 1124
7B
29.81±0.22
Alibaba-DAMO logo
Alibaba-DAMO logo
Babel
83B
28.79±0.27
CohereLabs logo
CohereLabs logo
Command R7B 12-2024
7B
28.08±0.26
Mistral AI logo
Mistral AI logo
Ministral 2410
8B
23.78±0.30

Malay Competencies

Average of 30 bootstraps. 95% CI are shown.

Model Size: ≤200B

Open instruct models only

Model
MS
Instruction Following
Multi-Turn Chat
NLG
NLU
Safety
Knowledge
Alibaba logo
Alibaba logo
Qwen 3 Next
80B
MoE

Alibaba

62.80

± 0.12

83.33

± 0.32

57.47

± 0.62

90.14

± 0.01

67.90

± 0.13

7.58

± 0.17

70.36

± 0.19

AISG logo
AISG logo
SEA-LION v4 (Qwen)
32B

AISG

61.36

± 0.14

81.21

± 0.50

42.30

± 0.70

90.27

± 0.02

68.42

± 0.09

16.31

± 0.13

69.65

± 0.18

Alibaba logo
Alibaba logo
Qwen 3
30B
MoE

Alibaba

61.13

± 0.14

81.46

± 0.47

51.48

± 0.57

90.07

± 0.02

67.60

± 0.05

13.06

± 0.11

63.12

± 0.26

AISG logo
AISG logo
SEA-LION v4 (Gemma)
27B

AISG

61.10

± 0.16

83.37

± 0.65

44.78

± 0.80

91.71

± 0.02

69.31

± 0.10

14.57

± 0.21

62.86

± 0.28

Google logo
Google logo
Gemma 3
27B

Google

60.92

± 0.17

81.65

± 0.59

45.89

± 0.56

91.83

± 0.01

69.16

± 0.08

13.86

± 0.25

63.13

± 0.34

AISG logo
AISG logo
SEA-LION v3 (Llama)
70B

AISG

59.94

± 0.17

85.62

± 0.51

25.55

± 0.75

91.01

± 0.02

69.71

± 0.14

13.62

± 0.36

74.12

± 0.44

Alibaba logo
Alibaba logo
Qwen 3
32B

Alibaba

59.67

± 0.17

80.38

± 0.62

41.64

± 0.74

88.97

± 0.03

66.47

± 0.13

13.18

± 0.29

67.38

± 0.27

Alibaba logo
Alibaba logo
Qwen 2.5
72B

Alibaba

59.48

± 0.14

79.24

± 0.58

32.76

± 0.55

89.60

± 0.03

69.11

± 0.10

15.39

± 0.22

70.78

± 0.32

Meta logo
Meta logo
Llama 3.3
70B

Meta

58.80

± 0.15

86.51

± 0.40

17.07

± 0.57

90.10

± 0.02

69.75

± 0.09

17.48

± 0.19

71.86

± 0.19

Google logo
Google logo
Gemma 3
12B

Google

57.96

± 0.12

81.65

± 0.67

38.02

± 0.60

91.02

± 0.02

67.67

± 0.07

11.57

± 0.18

57.83

± 0.25

AI2 logo
AI2 logo
Tulu 3
70B

AI2

57.39

± 0.19

80.00

± 0.65

27.60

± 0.58

91.06

± 0.02

67.60

± 0.20

11.11

± 0.27

66.97

± 0.30

Meta logo
Meta logo
Llama 4 Scout
109B
MoE

Meta

57.38

± 0.11

84.38

± 0.46

21.34

± 0.52

91.03

± 0.02

68.72

± 0.07

14.05

± 0.13

64.79

± 0.14

Meta logo
Meta logo
Llama 3.1
70B

Meta

55.67

± 0.18

75.05

± 0.86

13.97

± 0.48

90.34

± 0.02

69.16

± 0.17

15.90

± 0.24

69.58

± 0.35

Alibaba logo
Alibaba logo
Qwen 3
14B

Alibaba

55.04

± 0.11

78.35

± 0.42

31.61

± 0.52

87.44

± 0.02

66.47

± 0.11

12.54

± 0.22

53.83

± 0.26

CohereLabs logo
CohereLabs logo
Command A 03-2025
111B

CohereLabs

54.76

± 0.17

84.79

± 0.63

40.92

± 0.72

87.16

± 0.07

49.65

± 0.38

0.00

± 0.00

66.04

± 0.35

AISG logo
AISG logo
SEA-LION v3 (Gemma 2)
9B

AISG

54.70

± 0.18

81.05

± 0.85

25.46

± 0.63

90.38

± 0.03

65.12

± 0.15

12.19

± 0.41

53.98

± 0.44

Google logo
Google logo
Gemma 2
27B

Google

54.06

± 0.22

74.00

± 0.80

16.03

± 0.55

90.48

± 0.03

68.19

± 0.12

14.11

± 0.53

61.53

± 0.38

Alibaba logo
Alibaba logo
Qwen 3
8B

Alibaba

53.83

± 0.15

79.05

± 0.63

26.29

± 0.66

87.19

± 0.04

63.43

± 0.15

15.30

± 0.17

51.70

± 0.33

Alibaba logo
Alibaba logo
Qwen 2.5
32B

Alibaba

53.71

± 0.13

79.24

± 0.64

19.25

± 0.44

86.01

± 0.04

67.36

± 0.09

9.77

± 0.22

60.61

± 0.25

Mistral AI logo
Mistral AI logo
Mistral Large 2411
123B

Mistral AI

52.73

± 0.21

79.56

± 0.87

21.49

± 0.49

89.19

± 0.04

53.06

± 0.30

14.72

± 0.54

58.38

± 0.45

Alibaba logo
Alibaba logo
Qwen 2.5
14B

Alibaba

51.49

± 0.13

74.54

± 0.61

15.00

± 0.41

85.33

± 0.05

65.20

± 0.09

15.65

± 0.17

53.19

± 0.19

AISG logo
AISG logo
SEA-LION v3 (Llama)
8B

AISG

51.32

± 0.28

78.13

± 0.74

20.11

± 0.82

89.46

± 0.03

60.30

± 0.20

15.08

± 0.29

44.86

± 0.53

Google logo
Google logo
Gemma 2
9B

Google

49.72

± 0.21

73.14

± 0.88

12.08

± 0.44

89.08

± 0.04

61.98

± 0.15

10.75

± 0.28

51.31

± 0.36

A*STAR logo
A*STAR logo
MERaLiON 2
10B

A*STAR

49.12

± 0.22

69.11

± 0.95

11.61

± 0.31

88.16

± 0.04

62.51

± 0.20

14.04

± 0.34

49.27

± 0.52

CohereLabs logo
CohereLabs logo
Aya Expanse
32B

CohereLabs

48.35

± 0.20

73.87

± 0.79

18.74

± 0.74

86.71

± 0.04

48.22

± 0.19

13.13

± 0.25

49.44

± 0.26

AI2 logo
AI2 logo
Olmo 2 0325
32B

AI2

48.28

± 0.23

75.08

± 1.16

11.29

± 0.48

82.23

± 0.10

61.19

± 0.27

12.31

± 0.18

47.58

± 0.64

Alibaba logo
Alibaba logo
Qwen 2.5
7B

Alibaba

46.94

± 0.13

69.37

± 0.68

14.51

± 0.38

81.40

± 0.05

56.37

± 0.08

13.91

± 0.12

46.07

± 0.27

Baidu logo
Baidu logo
ERNIE 4.5
21B
MoE

Baidu

45.83

± 0.18

74.57

± 0.85

14.38

± 0.52

88.08

± 0.04

50.07

± 0.27

14.74

± 0.16

33.11

± 0.64

Meta logo
Meta logo
Llama 3.1
8B

Meta

45.49

± 0.22

64.63

± 1.08

10.07

± 0.37

87.30

± 0.03

59.11

± 0.17

15.03

± 0.24

36.82

± 0.54

SAIL logo
SAIL logo
Sailor2
20B

SAIL

45.19

± 0.17

41.02

± 0.68

25.88

± 0.68

76.16

± 0.18

65.47

± 0.14

12.73

± 0.12

49.87

± 0.31

Mistral AI logo
Mistral AI logo
Mistral Small 3.1 2503
24B

Mistral AI

44.39

± 0.22

64.22

± 0.99

9.71

± 0.44

82.03

± 0.07

55.72

± 0.26

0.00

± 0.00

54.68

± 0.63

CohereLabs logo
CohereLabs logo
Command R+ 08-2024
104B

CohereLabs

44.12

± 0.26

67.81

± 1.06

7.39

± 0.46

84.03

± 0.07

62.48

± 0.29

4.18

± 0.36

38.85

± 0.76

CohereLabs logo
CohereLabs logo
Aya Expanse
8B

CohereLabs

44.12

± 0.18

63.78

± 0.78

13.30

± 0.65

81.71

± 0.06

56.00

± 0.18

12.41

± 0.25

37.53

± 0.20

SAIL logo
SAIL logo
Sailor2
8B

SAIL

43.90

± 0.19

38.92

± 0.89

25.56

± 0.43

85.24

± 0.07

51.53

± 0.16

19.51

± 0.31

42.67

± 0.39

Meta logo
Meta logo
Llama 3
70B

Meta

43.42

± 0.10

24.10

± 0.47

9.07

± 0.44

89.75

± 0.02

66.39

± 0.10

9.59

± 0.16

61.64

± 0.29

AI2 logo
AI2 logo
Tulu 3
8B

AI2

42.86

± 0.14

73.08

± 0.59

12.67

± 0.48

87.26

± 0.05

53.32

± 0.18

0.00

± 0.00

30.81

± 0.35

Swiss AI logo
Swiss AI logo
Apertus
70B

Swiss AI

42.69

± 0.24

62.73

± 1.01

14.80

± 0.46

82.01

± 0.09

48.52

± 0.28

17.96

± 0.45

30.15

± 0.77

Microsoft logo
Microsoft logo
phi-4
14B

Microsoft

39.71

± 0.24

61.52

± 1.09

15.11

± 0.51

72.87

± 0.11

39.61

± 0.40

0.00

± 0.00

49.11

± 0.40

CohereLabs logo
CohereLabs logo
Command R 08-2024
32B

CohereLabs

39.00

± 0.21

64.63

± 0.79

5.01

± 0.39

81.29

± 0.08

33.72

± 0.16

10.82

± 0.58

38.56

± 0.63

Swiss AI logo
Swiss AI logo
Apertus
8B

Swiss AI

38.91

± 0.31

65.59

± 0.97

5.79

± 0.42

86.17

± 0.06

35.86

± 0.29

9.26

± 0.76

30.81

± 0.85

AI2 logo
AI2 logo
Olmo 2 1124
13B

AI2

38.54

± 0.19

64.92

± 0.89

6.70

± 0.30

71.47

± 0.14

46.86

± 0.21

14.19

± 0.25

27.11

± 0.58

Alibaba-DAMO logo
Alibaba-DAMO logo
SeaLLMs V3
7B

Alibaba-DAMO

36.82

± 0.28

52.35

± 1.06

8.95

± 0.40

82.02

± 0.09

39.73

± 0.35

9.01

± 0.68

28.88

± 0.84

Meta logo
Meta logo
Llama 3
8B

Meta

35.77

± 0.16

26.89

± 0.46

5.47

± 0.45

84.96

± 0.06

53.66

± 0.16

8.43

± 0.16

35.19

± 0.49

Alibaba-DAMO logo
Alibaba-DAMO logo
Babel
9B

Alibaba-DAMO

31.66

± 0.22

40.35

± 0.95

3.79

± 0.33

72.51

± 0.13

35.27

± 0.31

9.34

± 0.63

28.69

± 0.82

AI2 logo
AI2 logo
Olmo 2 1124
7B

AI2

29.81

± 0.22

55.02

± 1.26

4.24

± 0.33

59.43

± 0.13

30.84

± 0.30

6.78

± 0.45

22.57

± 0.50

Alibaba-DAMO logo
Alibaba-DAMO logo
Babel
83B

Alibaba-DAMO

28.79

± 0.27

40.41

± 1.24

7.13

± 0.58

66.77

± 0.18

22.52

± 0.51

0.00

± 0.00

35.89

± 1.06

CohereLabs logo
CohereLabs logo
Command R7B 12-2024
7B

CohereLabs

28.08

± 0.26

48.83

± 1.43

2.08

± 0.34

46.82

± 0.15

45.54

± 0.26

0.00

± 0.00

25.23

± 0.70

Mistral AI logo
Mistral AI logo
Ministral 2410
8B

Mistral AI

23.78

± 0.30

28.03

± 1.12

3.74

± 0.38

57.48

± 0.17

25.65

± 0.23

7.46

± 0.59

20.34

± 0.93

No records

Malay Tasks

Average of 30 bootstraps. 95% CI are shown.

Model Size: ≤200B

Open instruct models only

Model
MS
Instruction Following
SEA-IFEval
Alibaba logo
Alibaba logo
Qwen 3 Next
80B
MoE

Alibaba

62.80

± 0.12

83.33

± 0.32

83.33

± 0.32

AISG logo
AISG logo
SEA-LION v4 (Qwen)
32B

AISG

61.36

± 0.14

81.21

± 0.50

81.21

± 0.50

Alibaba logo
Alibaba logo
Qwen 3
30B
MoE

Alibaba

61.13

± 0.14

81.46

± 0.47

81.46

± 0.47

AISG logo
AISG logo
SEA-LION v4 (Gemma)
27B

AISG

61.10

± 0.16

83.37

± 0.65

83.37

± 0.65

Google logo
Google logo
Gemma 3
27B

Google

60.92

± 0.17

81.65

± 0.59

81.65

± 0.59

AISG logo
AISG logo
SEA-LION v3 (Llama)
70B

AISG

59.94

± 0.17

85.62

± 0.51

85.62

± 0.51

Alibaba logo
Alibaba logo
Qwen 3
32B

Alibaba

59.67

± 0.17

80.38

± 0.62

80.38

± 0.62

Alibaba logo
Alibaba logo
Qwen 2.5
72B

Alibaba

59.48

± 0.14

79.24

± 0.58

79.24

± 0.58

Meta logo
Meta logo
Llama 3.3
70B

Meta

58.80

± 0.15

86.51

± 0.40

86.51

± 0.40

Google logo
Google logo
Gemma 3
12B

Google

57.96

± 0.12

81.65

± 0.67

81.65

± 0.67

AI2 logo
AI2 logo
Tulu 3
70B

AI2

57.39

± 0.19

80.00

± 0.65

80.00

± 0.65

Meta logo
Meta logo
Llama 4 Scout
109B
MoE

Meta

57.38

± 0.11

84.38

± 0.46

84.38

± 0.46

Meta logo
Meta logo
Llama 3.1
70B

Meta

55.67

± 0.18

75.05

± 0.86

75.05

± 0.86

Alibaba logo
Alibaba logo
Qwen 3
14B

Alibaba

55.04

± 0.11

78.35

± 0.42

78.35

± 0.42

CohereLabs logo
CohereLabs logo
Command A 03-2025
111B

CohereLabs

54.76

± 0.17

84.79

± 0.63

84.79

± 0.63

AISG logo
AISG logo
SEA-LION v3 (Gemma 2)
9B

AISG

54.70

± 0.18

81.05

± 0.85

81.05

± 0.85

Google logo
Google logo
Gemma 2
27B

Google

54.06

± 0.22

74.00

± 0.80

74.00

± 0.80

Alibaba logo
Alibaba logo
Qwen 3
8B

Alibaba

53.83

± 0.15

79.05

± 0.63

79.05

± 0.63

Alibaba logo
Alibaba logo
Qwen 2.5
32B

Alibaba

53.71

± 0.13

79.24

± 0.64

79.24

± 0.64

Mistral AI logo
Mistral AI logo
Mistral Large 2411
123B

Mistral AI

52.73

± 0.21

79.56

± 0.87

79.56

± 0.87

Alibaba logo
Alibaba logo
Qwen 2.5
14B

Alibaba

51.49

± 0.13

74.54

± 0.61

74.54

± 0.61

AISG logo
AISG logo
SEA-LION v3 (Llama)
8B

AISG

51.32

± 0.28

78.13

± 0.74

78.13

± 0.74

Google logo
Google logo
Gemma 2
9B

Google

49.72

± 0.21

73.14

± 0.88

73.14

± 0.88

A*STAR logo
A*STAR logo
MERaLiON 2
10B

A*STAR

49.12

± 0.22

69.11

± 0.95

69.11

± 0.95

CohereLabs logo
CohereLabs logo
Aya Expanse
32B

CohereLabs

48.35

± 0.20

73.87

± 0.79

73.87

± 0.79

AI2 logo
AI2 logo
Olmo 2 0325
32B

AI2

48.28

± 0.23

75.08

± 1.16

75.08

± 1.16

Alibaba logo
Alibaba logo
Qwen 2.5
7B

Alibaba

46.94

± 0.13

69.37

± 0.68

69.37

± 0.68

Baidu logo
Baidu logo
ERNIE 4.5
21B
MoE

Baidu

45.83

± 0.18

74.57

± 0.85

74.57

± 0.85

Meta logo
Meta logo
Llama 3.1
8B

Meta

45.49

± 0.22

64.63

± 1.08

64.63

± 1.08

SAIL logo
SAIL logo
Sailor2
20B

SAIL

45.19

± 0.17

41.02

± 0.68

41.02

± 0.68

Mistral AI logo
Mistral AI logo
Mistral Small 3.1 2503
24B

Mistral AI

44.39

± 0.22

64.22

± 0.99

64.22

± 0.99

CohereLabs logo
CohereLabs logo
Command R+ 08-2024
104B

CohereLabs

44.12

± 0.26

67.81

± 1.06

67.81

± 1.06

CohereLabs logo
CohereLabs logo
Aya Expanse
8B

CohereLabs

44.12

± 0.18

63.78

± 0.78

63.78

± 0.78

SAIL logo
SAIL logo
Sailor2
8B

SAIL

43.90

± 0.19

38.92

± 0.89

38.92

± 0.89

Meta logo
Meta logo
Llama 3
70B

Meta

43.42

± 0.10

24.10

± 0.47

24.10

± 0.47

AI2 logo
AI2 logo
Tulu 3
8B

AI2

42.86

± 0.14

73.08

± 0.59

73.08

± 0.59

Swiss AI logo
Swiss AI logo
Apertus
70B

Swiss AI

42.69

± 0.24

62.73

± 1.01

62.73

± 1.01

Microsoft logo
Microsoft logo
phi-4
14B

Microsoft

39.71

± 0.24

61.52

± 1.09

61.52

± 1.09

CohereLabs logo
CohereLabs logo
Command R 08-2024
32B

CohereLabs

39.00

± 0.21

64.63

± 0.79

64.63

± 0.79

Swiss AI logo
Swiss AI logo
Apertus
8B

Swiss AI

38.91

± 0.31

65.59

± 0.97

65.59

± 0.97

AI2 logo
AI2 logo
Olmo 2 1124
13B

AI2

38.54

± 0.19

64.92

± 0.89

64.92

± 0.89

Alibaba-DAMO logo
Alibaba-DAMO logo
SeaLLMs V3
7B

Alibaba-DAMO

36.82

± 0.28

52.35

± 1.06

52.35

± 1.06

Meta logo
Meta logo
Llama 3
8B

Meta

35.77

± 0.16

26.89

± 0.46

26.89

± 0.46

Alibaba-DAMO logo
Alibaba-DAMO logo
Babel
9B

Alibaba-DAMO

31.66

± 0.22

40.35

± 0.95

40.35

± 0.95

AI2 logo
AI2 logo
Olmo 2 1124
7B

AI2

29.81

± 0.22

55.02

± 1.26

55.02

± 1.26

Alibaba-DAMO logo
Alibaba-DAMO logo
Babel
83B

Alibaba-DAMO

28.79

± 0.27

40.41

± 1.24

40.41

± 1.24

CohereLabs logo
CohereLabs logo
Command R7B 12-2024
7B

CohereLabs

28.08

± 0.26

48.83

± 1.43

48.83

± 1.43

Mistral AI logo
Mistral AI logo
Ministral 2410
8B

Mistral AI

23.78

± 0.30

28.03

± 1.12

28.03

± 1.12

No records