Evaluation Details

Methodology

Evaluation Parameters

SEA-HELM evaluations are conducted using the following parameters:

  • Number of evaluation runs: 8 independent runs per model to account for variability in model outputs.

  • Number of bootstrap runs: 2000 bootstrapped runs which sample different sets of questions with replacement to estimate the confidence intervals for each task. Stratified bootstrapping was applied for multiple choice tasks to ensure that each option is represented in the resamples.

  • Generation parameters: We use the model-specific defaults when available in the model configurations. For any unspecified parameters, we apply the vLLM default settings.

Note: All prompts in SEA-HELM are presented in their native languages using zero-shot prompting for instruct/reasoning models and five-shot prompting for base models.

Scoring

Score Aggregation Methodology

Our scoring system follows a hierarchical approach, aggregating results from individual tasks up to the overall SEA score:

  • 📋 Task Level

    Each question is first scored as the average of its 8 independent runs. We then draw 2000 bootstrap resamples of the task's questions (with replacement) and take the mean of each resample. The reported task score is the mean of those 2000 bootstrap means, and its 95% confidence interval is the 2.5th to 97.5th percentile of the same distribution.

  • 🎯 Competency Level

    For each bootstrap resample, we calculate a competency score by averaging all task scores within that competency area. The final competency score is the mean of these per-resample scores, and its 95% confidence interval is calculated from their 2.5th and 97.5th percentiles.

  • 🌏 Language Level

    Language scores aggregate all competency scores available for that specific language, calculated using the same approach as the competency-level aggregation.

  • 🏆 SEA Level (Overall Score)

    The SEA score represents the performance across all Southeast Asian languages and is calculated as the aggregate of the individual language scores and uses the same approach as the language/competency-level aggregation.