Tamil Performance

Tamil Scores by Model

Average of 30 bootstraps. 95% CI are shown.

Model Size: ≤200B

Open instruct models only

AISG logo
AISG logo
SEA-LION v4 (Gemma)
27B
64.43±0.16
Google logo
Google logo
Gemma 3
27B
64.36±0.22
AISG logo
AISG logo
SEA-LION v4 (Qwen)
32B
62.30±0.15
Alibaba logo
Alibaba logo
Qwen 3 Next
80B
MoE
60.05±0.13
Google logo
Google logo
Gemma 3
12B
59.86±0.21
Alibaba logo
Alibaba logo
Qwen 3
32B
58.88±0.24
Meta logo
Meta logo
Llama 4 Scout
109B
MoE
58.69±0.15
AISG logo
AISG logo
SEA-LION v3 (Llama)
70B
57.99±0.21
Alibaba logo
Alibaba logo
Qwen 3
30B
MoE
56.06±0.16
Meta logo
Meta logo
Llama 3.3
70B
54.74±0.15
AISG logo
AISG logo
SEA-LION v3 (Gemma 2)
9B
53.89±0.21
Google logo
Google logo
Gemma 2
27B
53.44±0.23
CohereLabs logo
CohereLabs logo
Command A 03-2025
111B
53.29±0.20
Alibaba logo
Alibaba logo
Qwen 3
14B
52.30±0.17
AI2 logo
AI2 logo
Tulu 3
70B
50.61±0.23
Mistral AI logo
Mistral AI logo
Mistral Large 2411
123B
49.44±0.29
Google logo
Google logo
Gemma 2
9B
48.17±0.23
Meta logo
Meta logo
Llama 3.1
70B
46.08±0.21
A*STAR logo
A*STAR logo
MERaLiON 2
10B
45.29±0.25
Alibaba logo
Alibaba logo
Qwen 2.5
32B
44.29±0.22
Alibaba logo
Alibaba logo
Qwen 2.5
72B
42.51±0.15
AISG logo
AISG logo
SEA-LION v3 (Llama)
8B
42.13±0.32
Baidu logo
Baidu logo
ERNIE 4.5
21B
MoE
40.75±0.18
CohereLabs logo
CohereLabs logo
Aya Expanse
32B
40.58±0.15
CohereLabs logo
CohereLabs logo
Command R 08-2024
32B
36.43±0.27
SAIL logo
SAIL logo
Sailor2
20B
35.23±0.20
Alibaba logo
Alibaba logo
Qwen 3
8B
33.23±0.16
Alibaba logo
Alibaba logo
Qwen 2.5
14B
32.49±0.22
Meta logo
Meta logo
Llama 3
70B
31.13±0.16
CohereLabs logo
CohereLabs logo
Command R+ 08-2024
104B
29.43±0.39
Swiss AI logo
Swiss AI logo
Apertus
70B
28.48±0.30
Alibaba-DAMO logo
Alibaba-DAMO logo
Babel
83B
25.82±0.33
AI2 logo
AI2 logo
Tulu 3
8B
24.58±0.25
Microsoft logo
Microsoft logo
phi-4
14B
22.74±0.23
Alibaba logo
Alibaba logo
Qwen 2.5
7B
21.63±0.13
SAIL logo
SAIL logo
Sailor2
8B
20.59±0.16
AI2 logo
AI2 logo
Olmo 2 0325
32B
19.42±0.31
Meta logo
Meta logo
Llama 3.1
8B
17.53±0.26
Swiss AI logo
Swiss AI logo
Apertus
8B
17.21±0.27
CohereLabs logo
CohereLabs logo
Aya Expanse
8B
17.05±0.23
Alibaba-DAMO logo
Alibaba-DAMO logo
Babel
9B
16.70±0.21
CohereLabs logo
CohereLabs logo
Command R7B 12-2024
7B
13.32±0.18
Alibaba-DAMO logo
Alibaba-DAMO logo
SeaLLMs V3
7B
11.92±0.18
Mistral AI logo
Mistral AI logo
Ministral 2410
8B
11.11±0.19
Mistral AI logo
Mistral AI logo
Mistral Small 3.1 2503
24B
10.39±0.28
Meta logo
Meta logo
Llama 3
8B
8.95±0.16
AI2 logo
AI2 logo
Olmo 2 1124
13B
8.48±0.18
AI2 logo
AI2 logo
Olmo 2 1124
7B
6.98±0.14

Tamil Competencies

Average of 30 bootstraps. 95% CI are shown.

Model Size: ≤200B

Open instruct models only

Model
TA
Instruction Following
Linguistic Diagnostics
Multi-Turn Chat
NLG
NLR
NLU
AISG logo
AISG logo
SEA-LION v4 (Gemma)
27B

AISG

64.43

± 0.16

71.05

± 0.64

71.55

± 0.48

40.52

± 0.67

50.37

± 0.05

71.26

± 0.16

81.86

± 0.22

Google logo
Google logo
Gemma 3
27B

Google

64.36

± 0.22

71.33

± 0.91

71.32

± 0.33

40.19

± 0.82

50.50

± 0.06

71.71

± 0.12

81.08

± 0.24

AISG logo
AISG logo
SEA-LION v4 (Qwen)
32B

AISG

62.30

± 0.15

78.38

± 0.63

71.39

± 0.26

20.96

± 0.57

49.73

± 0.06

69.41

± 0.11

83.94

± 0.16

Alibaba logo
Alibaba logo
Qwen 3 Next
80B
MoE

Alibaba

60.05

± 0.13

73.71

± 0.50

71.51

± 0.29

23.41

± 0.57

47.63

± 0.04

62.50

± 0.16

81.52

± 0.08

Google logo
Google logo
Gemma 3
12B

Google

59.86

± 0.21

70.67

± 0.87

58.39

± 0.50

36.03

± 0.56

49.59

± 0.06

61.95

± 0.18

82.53

± 0.22

Alibaba logo
Alibaba logo
Qwen 3
32B

Alibaba

58.88

± 0.24

76.06

± 0.71

69.40

± 0.84

17.14

± 0.53

44.71

± 0.08

62.93

± 0.18

83.05

± 0.21

Meta logo
Meta logo
Llama 4 Scout
109B
MoE

Meta

58.69

± 0.15

80.70

± 0.62

62.58

± 0.29

12.10

± 0.49

50.18

± 0.07

63.61

± 0.05

82.97

± 0.10

AISG logo
AISG logo
SEA-LION v3 (Llama)
70B

AISG

57.99

± 0.21

75.97

± 0.94

60.28

± 1.06

15.69

± 0.44

48.62

± 0.08

64.72

± 0.31

82.68

± 0.26

Alibaba logo
Alibaba logo
Qwen 3
30B
MoE

Alibaba

56.06

± 0.16

66.38

± 0.79

63.44

± 0.38

19.20

± 0.44

42.10

± 0.05

63.60

± 0.11

81.67

± 0.09

Meta logo
Meta logo
Llama 3.3
70B

Meta

54.74

± 0.15

70.22

± 0.73

55.64

± 0.42

8.43

± 0.52

46.86

± 0.11

65.14

± 0.18

82.12

± 0.19

AISG logo
AISG logo
SEA-LION v3 (Gemma 2)
9B

AISG

53.89

± 0.21

66.83

± 0.79

62.07

± 0.42

17.53

± 0.54

42.47

± 0.09

52.68

± 0.21

81.79

± 0.23

Google logo
Google logo
Gemma 2
27B

Google

53.44

± 0.23

63.24

± 1.09

62.72

± 0.36

7.47

± 0.30

44.45

± 0.13

61.24

± 0.21

81.52

± 0.29

CohereLabs logo
CohereLabs logo
Command A 03-2025
111B

CohereLabs

53.29

± 0.20

68.60

± 0.98

63.64

± 0.58

17.46

± 0.36

47.94

± 0.06

48.83

± 0.33

73.26

± 0.44

Alibaba logo
Alibaba logo
Qwen 3
14B

Alibaba

52.30

± 0.17

68.44

± 0.68

60.92

± 0.48

12.80

± 0.40

41.64

± 0.10

45.50

± 0.19

84.48

± 0.20

AI2 logo
AI2 logo
Tulu 3
70B

AI2

50.61

± 0.23

59.56

± 0.94

47.47

± 0.68

10.19

± 0.46

44.37

± 0.13

62.63

± 0.30

79.47

± 0.39

Mistral AI logo
Mistral AI logo
Mistral Large 2411
123B

Mistral AI

49.44

± 0.29

62.41

± 1.10

49.21

± 1.18

10.70

± 0.56

47.26

± 0.08

45.49

± 0.45

81.59

± 0.46

Google logo
Google logo
Gemma 2
9B

Google

48.17

± 0.23

58.57

± 1.08

63.86

± 0.52

6.62

± 0.39

35.18

± 0.10

42.72

± 0.39

82.08

± 0.26

Meta logo
Meta logo
Llama 3.1
70B

Meta

46.08

± 0.21

50.70

± 1.18

49.12

± 0.73

8.15

± 0.39

27.39

± 0.16

59.55

± 0.55

81.58

± 0.25

A*STAR logo
A*STAR logo
MERaLiON 2
10B

A*STAR

45.29

± 0.25

58.89

± 0.90

58.11

± 0.49

5.86

± 0.40

26.24

± 0.14

42.79

± 0.42

79.83

± 0.36

Alibaba logo
Alibaba logo
Qwen 2.5
32B

Alibaba

44.29

± 0.22

66.79

± 0.81

43.65

± 0.55

7.51

± 0.44

35.04

± 0.08

39.56

± 0.14

73.20

± 0.20

Alibaba logo
Alibaba logo
Qwen 2.5
72B

Alibaba

42.51

± 0.15

63.49

± 1.02

40.11

± 0.35

10.93

± 0.32

37.34

± 0.09

31.67

± 0.23

71.51

± 0.24

AISG logo
AISG logo
SEA-LION v3 (Llama)
8B

AISG

42.13

± 0.32

63.71

± 1.09

29.91

± 1.13

9.24

± 0.50

42.81

± 0.09

31.62

± 0.76

75.47

± 0.31

Baidu logo
Baidu logo
ERNIE 4.5
21B
MoE

Baidu

40.75

± 0.18

55.27

± 0.79

16.30

± 0.48

12.20

± 0.46

40.44

± 0.09

45.63

± 0.40

74.69

± 0.32

CohereLabs logo
CohereLabs logo
Aya Expanse
32B

CohereLabs

40.58

± 0.15

54.67

± 0.74

35.43

± 0.44

6.81

± 0.32

36.62

± 0.04

35.90

± 0.20

74.05

± 0.22

CohereLabs logo
CohereLabs logo
Command R 08-2024
32B

CohereLabs

36.43

± 0.27

38.98

± 1.14

24.42

± 1.11

1.91

± 0.31

37.81

± 0.13

37.14

± 0.53

78.31

± 0.32

SAIL logo
SAIL logo
Sailor2
20B

SAIL

35.23

± 0.20

29.24

± 0.85

42.96

± 0.26

10.96

± 0.45

22.97

± 0.07

61.84

± 0.13

43.42

± 0.13

Alibaba logo
Alibaba logo
Qwen 3
8B

Alibaba

33.23

± 0.16

62.67

± 0.59

28.57

± 0.31

12.30

± 0.40

20.38

± 0.04

24.69

± 0.16

50.78

± 0.42

Alibaba logo
Alibaba logo
Qwen 2.5
14B

Alibaba

32.49

± 0.22

43.87

± 0.91

38.80

± 0.60

6.38

± 0.26

25.85

± 0.07

23.27

± 0.19

56.78

± 0.23

Meta logo
Meta logo
Llama 3
70B

Meta

31.13

± 0.16

15.40

± 0.72

38.92

± 0.44

4.80

± 0.34

31.35

± 0.05

50.44

± 0.29

45.85

± 0.04

CohereLabs logo
CohereLabs logo
Command R+ 08-2024
104B

CohereLabs

29.43

± 0.39

35.68

± 1.10

20.33

± 1.22

2.77

± 0.24

40.17

± 0.10

15.34

± 0.68

62.28

± 0.55

Swiss AI logo
Swiss AI logo
Apertus
70B

Swiss AI

28.48

± 0.30

51.02

± 1.39

0.00

± 0.00

8.55

± 0.56

39.26

± 0.10

1.45

± 0.31

70.60

± 0.44

Alibaba-DAMO logo
Alibaba-DAMO logo
Babel
83B

Alibaba-DAMO

25.82

± 0.33

33.40

± 0.84

21.72

± 1.31

2.08

± 0.34

24.48

± 0.13

26.08

± 0.63

47.16

± 0.55

AI2 logo
AI2 logo
Tulu 3
8B

AI2

24.58

± 0.25

48.22

± 1.14

9.82

± 0.51

2.56

± 0.27

36.04

± 0.11

27.27

± 0.52

23.58

± 0.34

Microsoft logo
Microsoft logo
phi-4
14B

Microsoft

22.74

± 0.23

32.19

± 0.98

0.00

± 0.00

8.13

± 0.40

31.25

± 0.10

14.79

± 0.71

50.06

± 0.55

Alibaba logo
Alibaba logo
Qwen 2.5
7B

Alibaba

21.63

± 0.13

44.98

± 0.65

3.59

± 0.28

2.64

± 0.33

17.57

± 0.08

10.35

± 0.13

50.67

± 0.27

SAIL logo
SAIL logo
Sailor2
8B

SAIL

20.59

± 0.16

25.78

± 0.90

0.00

± 0.00

7.24

± 0.39

23.60

± 0.08

22.00

± 0.18

44.95

± 0.12

AI2 logo
AI2 logo
Olmo 2 0325
32B

AI2

19.42

± 0.31

51.56

± 0.95

12.81

± 1.20

2.40

± 0.28

25.23

± 0.06

20.06

± 0.58

4.44

± 0.46

Meta logo
Meta logo
Llama 3.1
8B

Meta

17.53

± 0.26

36.98

± 1.21

5.15

± 0.92

3.92

± 0.35

29.43

± 0.10

2.99

± 0.39

26.72

± 0.51

Swiss AI logo
Swiss AI logo
Apertus
8B

Swiss AI

17.21

± 0.27

42.54

± 1.16

0.00

± 0.00

2.30

± 0.34

30.78

± 0.13

0.00

± 0.00

27.67

± 0.91

CohereLabs logo
CohereLabs logo
Aya Expanse
8B

CohereLabs

17.05

± 0.23

32.60

± 1.08

12.17

± 0.61

2.24

± 0.22

19.69

± 0.05

17.19

± 0.32

18.40

± 0.41

Alibaba-DAMO logo
Alibaba-DAMO logo
Babel
9B

Alibaba-DAMO

16.70

± 0.21

38.03

± 0.90

0.00

± 0.00

2.54

± 0.22

28.21

± 0.13

1.89

± 0.57

29.55

± 0.32

CohereLabs logo
CohereLabs logo
Command R7B 12-2024
7B

CohereLabs

13.32

± 0.18

38.57

± 0.98

0.00

± 0.00

1.51

± 0.25

24.67

± 0.09

0.00

± 0.00

15.19

± 0.50

Alibaba-DAMO logo
Alibaba-DAMO logo
SeaLLMs V3
7B

Alibaba-DAMO

11.92

± 0.18

32.79

± 0.83

0.00

± 0.00

2.13

± 0.30

20.07

± 0.08

0.00

± 0.00

16.54

± 0.38

Mistral AI logo
Mistral AI logo
Ministral 2410
8B

Mistral AI

11.11

± 0.19

22.03

± 1.00

0.00

± 0.00

0.99

± 0.19

18.72

± 0.10

1.27

± 0.30

23.62

± 0.52

Mistral AI logo
Mistral AI logo
Mistral Small 3.1 2503
24B

Mistral AI

10.39

± 0.28

31.97

± 1.43

0.00

± 0.00

1.93

± 0.30

11.22

± 0.08

0.00

± 0.00

17.25

± 0.44

Meta logo
Meta logo
Llama 3
8B

Meta

8.95

± 0.16

20.16

± 0.84

0.00

± 0.00

2.46

± 0.24

27.01

± 0.07

4.06

± 0.46

0.00

± 0.00

AI2 logo
AI2 logo
Olmo 2 1124
13B

AI2

8.48

± 0.18

32.22

± 1.12

0.00

± 0.00

1.18

± 0.23

17.41

± 0.06

0.08

± 0.06

0.00

± 0.00

AI2 logo
AI2 logo
Olmo 2 1124
7B

AI2

6.98

± 0.14

26.51

± 0.83

0.00

± 0.00

0.29

± 0.12

15.09

± 0.06

0.00

± 0.00

0.00

± 0.00

No records

Tamil Tasks

Average of 30 bootstraps. 95% CI are shown.

Model Size: ≤200B

Open instruct models only

Model
TA
Instruction Following
SEA-IFEval
AISG logo
AISG logo
SEA-LION v4 (Gemma)
27B

AISG

64.43

± 0.16

71.05

± 0.64

71.05

± 0.64

Google logo
Google logo
Gemma 3
27B

Google

64.36

± 0.22

71.33

± 0.91

71.33

± 0.91

AISG logo
AISG logo
SEA-LION v4 (Qwen)
32B

AISG

62.30

± 0.15

78.38

± 0.63

78.38

± 0.63

Alibaba logo
Alibaba logo
Qwen 3 Next
80B
MoE

Alibaba

60.05

± 0.13

73.71

± 0.50

73.71

± 0.50

Google logo
Google logo
Gemma 3
12B

Google

59.86

± 0.21

70.67

± 0.87

70.67

± 0.87

Alibaba logo
Alibaba logo
Qwen 3
32B

Alibaba

58.88

± 0.24

76.06

± 0.71

76.06

± 0.71

Meta logo
Meta logo
Llama 4 Scout
109B
MoE

Meta

58.69

± 0.15

80.70

± 0.62

80.70

± 0.62

AISG logo
AISG logo
SEA-LION v3 (Llama)
70B

AISG

57.99

± 0.21

75.97

± 0.94

75.97

± 0.94

Alibaba logo
Alibaba logo
Qwen 3
30B
MoE

Alibaba

56.06

± 0.16

66.38

± 0.79

66.38

± 0.79

Meta logo
Meta logo
Llama 3.3
70B

Meta

54.74

± 0.15

70.22

± 0.73

70.22

± 0.73

AISG logo
AISG logo
SEA-LION v3 (Gemma 2)
9B

AISG

53.89

± 0.21

66.83

± 0.79

66.83

± 0.79

Google logo
Google logo
Gemma 2
27B

Google

53.44

± 0.23

63.24

± 1.09

63.24

± 1.09

CohereLabs logo
CohereLabs logo
Command A 03-2025
111B

CohereLabs

53.29

± 0.20

68.60

± 0.98

68.60

± 0.98

Alibaba logo
Alibaba logo
Qwen 3
14B

Alibaba

52.30

± 0.17

68.44

± 0.68

68.44

± 0.68

AI2 logo
AI2 logo
Tulu 3
70B

AI2

50.61

± 0.23

59.56

± 0.94

59.56

± 0.94

Mistral AI logo
Mistral AI logo
Mistral Large 2411
123B

Mistral AI

49.44

± 0.29

62.41

± 1.10

62.41

± 1.10

Google logo
Google logo
Gemma 2
9B

Google

48.17

± 0.23

58.57

± 1.08

58.57

± 1.08

Meta logo
Meta logo
Llama 3.1
70B

Meta

46.08

± 0.21

50.70

± 1.18

50.70

± 1.18

A*STAR logo
A*STAR logo
MERaLiON 2
10B

A*STAR

45.29

± 0.25

58.89

± 0.90

58.89

± 0.90

Alibaba logo
Alibaba logo
Qwen 2.5
32B

Alibaba

44.29

± 0.22

66.79

± 0.81

66.79

± 0.81

Alibaba logo
Alibaba logo
Qwen 2.5
72B

Alibaba

42.51

± 0.15

63.49

± 1.02

63.49

± 1.02

AISG logo
AISG logo
SEA-LION v3 (Llama)
8B

AISG

42.13

± 0.32

63.71

± 1.09

63.71

± 1.09

Baidu logo
Baidu logo
ERNIE 4.5
21B
MoE

Baidu

40.75

± 0.18

55.27

± 0.79

55.27

± 0.79

CohereLabs logo
CohereLabs logo
Aya Expanse
32B

CohereLabs

40.58

± 0.15

54.67

± 0.74

54.67

± 0.74

CohereLabs logo
CohereLabs logo
Command R 08-2024
32B

CohereLabs

36.43

± 0.27

38.98

± 1.14

38.98

± 1.14

SAIL logo
SAIL logo
Sailor2
20B

SAIL

35.23

± 0.20

29.24

± 0.85

29.24

± 0.85

Alibaba logo
Alibaba logo
Qwen 3
8B

Alibaba

33.23

± 0.16

62.67

± 0.59

62.67

± 0.59

Alibaba logo
Alibaba logo
Qwen 2.5
14B

Alibaba

32.49

± 0.22

43.87

± 0.91

43.87

± 0.91

Meta logo
Meta logo
Llama 3
70B

Meta

31.13

± 0.16

15.40

± 0.72

15.40

± 0.72

CohereLabs logo
CohereLabs logo
Command R+ 08-2024
104B

CohereLabs

29.43

± 0.39

35.68

± 1.10

35.68

± 1.10

Swiss AI logo
Swiss AI logo
Apertus
70B

Swiss AI

28.48

± 0.30

51.02

± 1.39

51.02

± 1.39

Alibaba-DAMO logo
Alibaba-DAMO logo
Babel
83B

Alibaba-DAMO

25.82

± 0.33

33.40

± 0.84

33.40

± 0.84

AI2 logo
AI2 logo
Tulu 3
8B

AI2

24.58

± 0.25

48.22

± 1.14

48.22

± 1.14

Microsoft logo
Microsoft logo
phi-4
14B

Microsoft

22.74

± 0.23

32.19

± 0.98

32.19

± 0.98

Alibaba logo
Alibaba logo
Qwen 2.5
7B

Alibaba

21.63

± 0.13

44.98

± 0.65

44.98

± 0.65

SAIL logo
SAIL logo
Sailor2
8B

SAIL

20.59

± 0.16

25.78

± 0.90

25.78

± 0.90

AI2 logo
AI2 logo
Olmo 2 0325
32B

AI2

19.42

± 0.31

51.56

± 0.95

51.56

± 0.95

Meta logo
Meta logo
Llama 3.1
8B

Meta

17.53

± 0.26

36.98

± 1.21

36.98

± 1.21

Swiss AI logo
Swiss AI logo
Apertus
8B

Swiss AI

17.21

± 0.27

42.54

± 1.16

42.54

± 1.16

CohereLabs logo
CohereLabs logo
Aya Expanse
8B

CohereLabs

17.05

± 0.23

32.60

± 1.08

32.60

± 1.08

Alibaba-DAMO logo
Alibaba-DAMO logo
Babel
9B

Alibaba-DAMO

16.70

± 0.21

38.03

± 0.90

38.03

± 0.90

CohereLabs logo
CohereLabs logo
Command R7B 12-2024
7B

CohereLabs

13.32

± 0.18

38.57

± 0.98

38.57

± 0.98

Alibaba-DAMO logo
Alibaba-DAMO logo
SeaLLMs V3
7B

Alibaba-DAMO

11.92

± 0.18

32.79

± 0.83

32.79

± 0.83

Mistral AI logo
Mistral AI logo
Ministral 2410
8B

Mistral AI

11.11

± 0.19

22.03

± 1.00

22.03

± 1.00

Mistral AI logo
Mistral AI logo
Mistral Small 3.1 2503
24B

Mistral AI

10.39

± 0.28

31.97

± 1.43

31.97

± 1.43

Meta logo
Meta logo
Llama 3
8B

Meta

8.95

± 0.16

20.16

± 0.84

20.16

± 0.84

AI2 logo
AI2 logo
Olmo 2 1124
13B

AI2

8.48

± 0.18

32.22

± 1.12

32.22

± 1.12

AI2 logo
AI2 logo
Olmo 2 1124
7B

AI2

6.98

± 0.14

26.51

± 0.83

26.51

± 0.83

No records