Aditya Venkatesh's Personal Website

Benchmarks

Benchmarks provide standardized datasets and tasks to compare model performance. Popular benchmarks for LLMs include GLUE and SuperGLUE for language understanding, SQuAD for question answering, as well as more specialized domain tests focused on coding ability or multilingual competence. These benchmarks help track progress and identify gaps across diverse challenges.

Core Automatic Metrics

Common quantitative metrics include perplexity, which measures how well a model predicts text; accuracy and F1 score for classification-type tasks; and BLEU and ROUGE for evaluating text similarity in translation or summarization tasks. These metrics offer objective, reproducible ways to gauge model capability on discrete aspects of language.

Limitations of Metrics

While useful, automatic metrics have blind spots. They often miss subtleties like contextual appropriateness, creativity, or ethical risks. Some metrics can be gamed by models optimizing for score rather than quality, leading to misleading conclusions. Therefore, relying solely on metrics without complementary evaluation methods, such as human evaluation, can obscure a model’s true performance capabilities.