Large Language Models for Machine Translation Quality Annotation: Humans

Apple Machine Learning Research published the October 2026 EMNLP paper "Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged," by Hala Almaghout, Christian Federmann, and Qin Gao. Apple Machine Learning Research's paper tests large language models as judges of machine translation quality, comparing LLM agreement with human annotators on Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA).

This piece covers how the comparison was run, what a long-context set of 70 language pairs and the public WMT23 and WMT25 data showed, and where human and LLM annotators are challenged. It is for readers who score translations or who are weighing an LLM in place of a human annotator across language pairs, domains, and levels of annotation granularity.

Large Language Models for Machine Translation: what actually changed

Large language models are considered a more efficient and cost-effective alternative to human judgment for machine translation evaluation. This Speech and Natural Language Processing paper tests that view on the coverage the authors spell out. Evaluation spans "a large number of language pairs, domains and levels of annotation granularity," and LLMs "must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation." The test uses two prominent schemes, MQM and ESA, and compares model annotations with human annotators on both scores and error spans.

LLM agreement with human annotators exceeds agreement between human annotators for some evaluation tasks. Both comparisons "vary substantially across annotation schemes, language pairs and domains and remain unreliable for most settings." Humans are particularly challenged by fine-grained MQM annotations and by low-resource language pairs. LLMs struggle with minor errors, wrong language variants, and error span annotation. Almaghout, Federmann, and Gao say this leaves room to improve human annotation and LLM annotation, possibly through collaborative pipelines that address the reliability issues identified in the work.

Large Language Models for Machine Translation: how it works

Agreement with human annotators is the measure for MQM and for ESA. Multidimensional Quality Metrics is the scheme tied here to fine-grained quality annotation. Error Span Annotation is the scheme centered on error spans. The authors investigate both score agreement and error span annotation agreement across a variety of language pairs and domains. They also state a model weakness outright: LLMs struggle with error span annotation, with minor errors, and with wrong language variants, the three LLM challenges named in the paper.

The study reports results on a long-context test set of 70 language pairs and on the publicly available WMT23 and WMT25 data. On that material, LLM annotators are compared with human annotators for MQM and ESA, for scores and for error spans, across a variety of language pairs and domains. Agreement varies substantially by scheme, pair, and domain. The 70-pair set is the long-context sample, WMT23 and WMT25 are the public sets, and low-resource pairs are a particular challenge for human annotators.

Large Language Models for Machine Translation: why it matters now

LLMs are considered a more efficient and cost-effective alternative to human judgment for machine translation evaluation. The authors pair that view with a condition. Evaluation spans a large number of language pairs, domains, and levels of annotation granularity, and models "must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation." Hala Almaghout, Christian Federmann, and Qin Gao publish the MQM and ESA comparison at EMNLP in October 2026, using a 70-pair long-context set together with WMT23 and WMT25.

For some evaluation tasks, LLM agreement with a human annotator is higher than agreement between human annotators. Across annotation schemes, language pairs, and domains, both comparisons vary substantially and remain unreliable for most settings. Humans are particularly challenged by fine-grained MQM annotations and by low-resource language pairs. LLMs are challenged by minor errors, wrong language variants, and error span annotation. Using an LLM as the alternative to a human judge relies, in most settings, on agreement the authors call unreliable.

Large Language Models for Machine Translation: who is affected

Human annotators are challenged on the two fronts the paper names. They are particularly challenged by fine-grained MQM annotations and by low-resource language pairs. MQM, or Multidimensional Quality Metrics, is one of the two prominent schemes in the study, alongside Error Span Annotation. Agreement between human annotators also varies substantially across schemes, language pairs, and domains, and it remains unreliable for most settings. For fine-grained MQM and for low-resource pairs, those annotators are the group the paper calls particularly challenged.

LLM annotators are the other group under test. They are the option already considered a more efficient and cost-effective alternative to human judgment, and the paper measures where that option struggles. The models struggle with minor errors, wrong language variants, and error span annotation. ESA is the error-span scheme, so the span weakness falls on that method. The comparison uses a long-context set of 70 language pairs and the public WMT23 and WMT25 data, across a variety of pairs and domains. The authors investigate both score agreement and error span annotation agreement.

Large Language Models for Machine Translation: what to watch

LLM agreement with human annotators exceeds agreement between human annotators for some evaluation tasks. Both agreements vary substantially across annotation schemes, language pairs, and domains, and both remain unreliable for most settings. The authors investigate score agreement and error span annotation agreement for both MQM and ESA. They report both on a long-context set of 70 language pairs and on the publicly available WMT23 and WMT25 data. A stronger match on some tasks leaves the broader finding unchanged: most settings remain unreliable.

The authors name a path that uses human annotators and LLM annotators together. They write that the results "highlight the potential for improvement for both human and LLM annotation performance, possibly through human-LLM collaborative annotation pipelines that address the reliability issues identified in this work." The human challenges they list are fine-grained MQM annotations and low-resource language pairs. The model challenges are minor errors, wrong language variants, and error span annotation. Those pipelines are the authors' proposal for the reliability issues that remain in most settings.