Is your LLM really clever? Can it mark its own homework? The ELOQUENT lab provides shared tasks for evaluation of generative language model quality.
A lab at CLEF, the Conference and Labs of the Evaluation Forum
Task 1:
Topical Competence

Does your LLM know what it is talking about?

a lecturer holding forth about something complex

This task will test and verify that a system based on a generative language model is able to handle material from some given topical domain of interest, by having systems automatically generate tests of domain knowledge.

More information on the task page!

Task 2:

Can it be trusted? Or does it make stuff up?

This task will test whether your model is able to detect hallucinations in both human-authored and machine-generated contexts.

a graffiti from Born with a grinning psychedelic face

More information on the task page!

Task 3:

Will it respond with the same content to all of us?

This task will test the capability of a model to handle input variation -- e.g. dialectal, sociolectal, and cross-cultural -- as represented by human-generated varieties of input prompts. The results will be assessed by how variation in output is conditioned on variation of equivalent but non-identical input prompts.

More information on the task page!

janus, a two-faced deity, depicted on a roman coin

Task 4:

Has a machine written this? Or has a human author put together these words?

This task will explore whether automatically-generated text can be distinguished from human-authored text. This task will be organised in collaboration with the PAN lab at CLEF.

part human, part machine

More information on the task page!

2024 Workshop in Grenoble

The first ELOQUENT Workshop will be in Grenoble, September 9-12 2024.

CLEF 2024 Grenoble Logo

The workshop program will hold overview presentations, an invited keynote, and some selected participant presentations.

Sign up for the tasks!

Here is how to participate in the discussion about task details: sign up to join the conversation through the CLEF registration form.



  • Fall 2023: discussion and task formulation
  • February 2024: tasks open and public announcement of tasks on mailing lists
  • Last week of March: ECIR presentation of ELOQUENT
  • 22 April 2024: registration for participation closes
  • May 2024: submission deadline of experimental runs from participants
  • June 2024: participant report submission deadline
  • July 2024: camera ready report submission deadline
  • September 2024: workshop at CLEF

Organising committee

Contact us at eloquent-clef2024-organizers AT

Thank you

The ELOQUENT lab is supported by the DeployAI project.

Page layout from Codepen.