Is your LLM really clever? Can it mark its own homework?
ELOQUENT Lab 2026
3d edition of the lab for evaluation of generative language model quality at CLEF, the Conference and Labs of the Evaluation Forum
2026 Schedule

Workshop in Jena

The third ELOQUENT Workshop will be held at the CLEF conference in Jena, September 21-24 2026.

a spektrometer built in Jena

Workshop schedule

  • Tuesday 10:00-18:00 Posters: all participants are welcome to hang a poster for their experiment
  • Tuesday 11:15-12:05 CLEF Lab overview session, including the ELOQUENT overview of all tasks
  • Tuesday 14-15:30 ELOQUENT Session 1: Topical Quiz and Sensemaking tasks
    • Task Context topical quiz generation, quiz scoring, and the PISA reading profiency tests
    • Markarit Vartamperian: Quiz generation task
    • Pavel Šindelář: Sensemaking task
    • Participants: Poster boasters
    • All:Plans for next year
  • Tuesday 16-17:30 ELOQUENT Session 2: Cultural Robustness and Diversity task
    • Jussi Karlgren: Task overview and evolution over editions
    • Bruno Nadalić-Šotić and Bram Koorstra: Influence of Conversational Context, Personalization, and Prompt Framing on Cultural Signal Purity
    • Josiane Mothe: Cultural Robustness in Multilingual LLMs
    • All: Plans for next year
  • Thursday 14-15:30 ELOQUENT and PAN joint session: Voight Kampff task
    • Builder task - intro: PAN presentation on classifiers
    • Builder task - Team Suspiciously Coherent
    • Builder task - Team DACTYL
    • Breaker task - intro: ELOQUENT presentation on text generation
    • Breaker task - Team DArgk
      • l>
    • All:Plans for next year
  • Thursday 16-17:30 CLEF closing session, including ELOQUENT announcement for the 2027 edition

Task Voight-Kampff

Can your LLM fool a classifier to believe it is human?

This task explores whether automatically-generated text can be distinguished from human-authored text, and is organised in collaboration with the PAN lab at CLEF.

  • More information about the task.

    • part human, part machine

    Robustness Task Cultural Robustness and Diversity

    Will your machine respond with the same content to all of us?

    This task has run in two variants in previous editions, and this year will tests how well a model achieves consistency across several languages and how well the model adapts to the local culture of a linguistic area.

  • More information about the task

    • janus, a two-faced deity, depicted on a roman coin

    Topical PISA Quiz Task Generating and Scoring Exams

    Can your language model prep, sit, or rate an exam for you?

    In an evolved version of the first year's Topical Quiz Task and the second year's Sensemaking task, this year the Exam task is developed together with the OECD for the purposes of supporting future PISA tests. It will have two subtasks: creating test items from a given text and scoring student responses to test items.

    study session

    Fourth Edition Coming up in 2027!

    There will be a fourth edition of ELOQUENT in 2027 with some tasks continued and some new exciting tasks launched! First announcements will be made at CLEF 2026, with details to follow!

    Indicative Timeline

    • Fall 2026: task formulation -- get in touch with us if you want to be involved!
    • January 2027: tasks formally announced
    • March 2027: ELOQUENT presentation at ECIR
    • May 2027: submission deadline for experiments
    • June 2027: report submission deadline
    • September 2027: 4th ELOQUENT workshop

    Second Edition, 2025

    The second edition of ELOQUENT ran in 2025 and involved four tasks: Voight-Kampff, Robustness and Consistency, Preference Prediction, and Sensemaking (a development of the Topical Quiz task) with organisers Ekaterina Artemova, Ondřej Bojar, Marie Isabel Engels, Vladislav Mikhailov, Jussi Karlgren, Pavel Šindelář, Erik Velldal, Lilja Øvrelid

    First Edition, 2024

    The first edition of ELOQUENT ran in 2024 and involved four tasks: Voight-Kampff, Hallucigen, Robustness and Consistency, and Topical Quiz with organisers Luise Dürlich, Evangelia Gogoulou, Liane Guillou, Jussi Karlgren, Joakim Nivre, Magnus Sahlgren, Aarne Talman

    Organising Committee
    some people at a work table
    • AMD Silo AI: Maria Barrett, Jussi Karlgren, Georgios Stampoulidis
    • Charles University: Ondřej Bojar and Pavel Šindelář
    • Fraunhofer IAIS: Marie Isabel Engels
    • OECD: Mario Piacentini, Luis Francisco Vargas Madriz, Katherina Thomas
    • Université Grenoble Alpes: Diandra Fabre, Lorraine Goeuriot, Philippe Mulhem, Didier Schwab, Markarit Vartampetian
    • Université de Toulouse, IRIT: Josiane Mothe
    • University of Tennessee, Knoxville: Rohit Gunti

    Contact us at eloquent-clef2026-organizers AT googlegroups.com

    Thank you

    The ELOQUENT lab is partially supported by the OpenEuroLLM and the DeployAI projects through their activities on building, evaluating, and disseminating generative language models.

    Page layout from Codepen.