What is measured
The complete list of evals, with status and implementation, is on the evals page. This page fixes the protocol for the ones the leaderboard shows. Task family, first release. Next-item prediction over behavioral event data. A person’s history, in time order, is given to the system. The system ranks the entire catalog. The target is the person’s actual next item, which the system never saw. This is scored on logged actions, not on survey answers, human ratings, or a judge model’s opinion. Cell. A result is a cell in a design matrix, and a claim that does not name its cell is not comparable. The axes: individual or population; how much of the person’s history the system was given; how much prompting it received; the task family. Every row on the leaderboard states its cell in the snapshot header.
Two numbers ride along with every row: n, with intervals clustered by person, and
cost, as fit time now and dollars per thousand predictions once external systems are run.
Diversity
A simulator that ranks well can still produce a narrow, uniform population. The diversity columns measure the distributional gap between real and simulated behavior. The method is adapted from Mehri et al., Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors (2026): represent each behavior, quantize the representations into discrete distributions by clustering, and compute divergences between the real and simulated distributions. That work scored 24 language-model user simulators on conversation data with forward KL, backward KL, and Jensen-Shannon divergence over k-means clusters of embedded behavior descriptions. On event data the same idea is simpler. Each person in the held-out set has a real next item. Each simulator samples a next item for the same person from the same history. Both sets of items are mapped onto a fixed clustering of the catalog (item embeddings, k-means, k fixed per snapshot and named in the results file), giving one real distribution and one simulated distribution over clusters.- JS divergence between the two. Lower is better. The floor is a collapsed-mode simulator that emits the most popular item for everyone, so the number has a bottom.
- Spread ratio: entropy of the simulated distribution over entropy of the real one. One is the target. Below one is mode collapse. Above one is noise dressed as diversity.
- Forward and backward KL are recorded in the results file for the reasons the source paper gives: forward KL falls when the simulator misses real behaviors, backward KL when it produces behaviors real people do not.
Calibration
Log loss of the held-out next item under the simulator’s own distribution over the catalog, averaged over persons, against popularity. A system that emits a ranked list without probabilities is scored on ranking only and its calibration cell is left empty rather than filled with a proxy.The controls
- Popularity. The most frequent training items, same order for everyone. Every ranking row for a non-baseline system states this beside its number.
- Stranger. The same system scored with another person’s history, permuted within the target domain so the domain prior is held constant. This separates knowing the person from knowing the population. Required in every transfer cell.
- Collapsed mode. For diversity: always emit the most popular item. Gives the divergence a floor.
Snapshots
Each snapshot is a frozen, versioned dataset with a published cutoff date and a stated protocol. The snapshot id appears on every row. Only data whose terms allow this use is scored, and Jean does not redistribute any snapshot; the source is linked and the build is deterministic from it. The scored holdout is never sold, licensed, or trained on by Jean. Data Jean supplies to customers comes from a different corpus. Snapshots are fixed, not rolling. A new snapshot gets a new id and old rows stay under the old one.What verified means
Rows are never edited. A rerun adds a row with a new run id. Each release of the results
file names its date; each row names the commit of the evaluation code that produced it.
Conflict of interest
Jean has entries on its own leaderboard, and sells behavioral data and models. Refereeing while selling contestants the data to win is how a benchmark loses authority, so:- Jean’s rows are labeled with the same verification levels as anyone else’s and start as internal runs, not verified.
- The rerun scripts for Jean’s rows are published first, so they are the easiest rows to check.
- Protocol and scoring code are public. The scored holdout is not sold or trained on.
- Every run is recorded, including the ones that do not flatter Jean. Featuring is a choice about emphasis, never about existence.
What is not measured
- Dialogue user simulation. Systems like UserLM-8b simulate the user side of a conversation. That is a different task family with different ground truth, and it is not scored here. If the audience for this leaderboard turns out to be there, a second track will be designed rather than this one stretched.
- Causal claims. Nothing on the board says what happens if a price, message, or policy changes. That needs randomized or natural-experiment evidence, which no snapshot here carries. See the simulation product page for the same caveat on the API.
- Survey answers, human ratings, judge scores. The target is always something a person did.
Submit
Open a discussion on the submissions dataset, which is public and keeps the exchange on the record, or email jonathan@jeantechnologies.com with the subject “Leaderboard submission”. Either way, send the following.- System name, organization, and a link to the paper or release.
- Kind: trained, prompted, or baseline. For trained systems, the training data and whether any snapshot’s source is in it.
- An artifact or an API Jean can rerun against the frozen snapshot: weights, a container, or an endpoint that accepts an ordered item history and returns scores over a candidate set. Without one, the entry is listed as reported.
- Your own numbers, if you have them, with the snapshot id, metric, n, and controls.
- Cost: seconds and dollars per thousand persons scored, if known.