Light/Dark
Click on each task to view more details.
Translations
Natural Language Generation (NLG)
Translations of a given text from one of the Southeast Asian languages to English and vice versa.
Metric: MetricX-24 score
Summarization
Summarization of a given text.
Metric: Rouge-L
Natural Language Inference (NLI)
Natural Language Reasoning (NLR)
Identification of entailment, contradiction or neutral relationships between X and Y.
Metric: Accuracy
Causal Reasoning
Identification of cause and effect relationships between X and Y.
Sentiment Analysis
Natural Language Understanding (NLU)
Identification of sentiment (positive, negative, neutral) from a given text.
Question Answering
Answering questions based on a given context or document.
Metric: F1 or Accuracy
Metaphor understanding
Identification of metaphorical language in a given text.
Syntactic Understanding
Linguistic Diagnostics
Identification of which of the two sentences is more syntactically acceptable
Pragmatic Understanding
Identification of context-dependent meaning in a given text.
SEA-IFEval
Instruction Following
Ability to follow specific natural language constraints
SEA-MTBench
Multi-turn Chat
Ability to engage in multi-turn conversations
Metric: Win Rate against a reference
Toxicity detection
Safety
Identification of the presence of toxic language in a given text.
Cultural Alignment
Cultural Knowledge
Identification of the culturally appropriate response given a situation
General Knowledge
Knowledge
General knowledge of the world