Every Eval Ever (EEE) and Hugging Face Community Eval are mutually compatible. Enable cross-posting and interpretation of assessment results while linking to open models, leaderboards, and a unified, standardized metadata store.
EEE was launched in February 2026 as a project of the EvalEval Coalition. This is the first cross-institutional effort to improve how AI assessment results are reported by both first- and third-party evaluators. Hugging Face launched Community Evals in February 2026 to decentralize how benchmark scores are reported in the hub. Together, they help close the gap in how users, researchers, and policy makers trust, understand, and select assessments and models.
Evaluation results are a way to measure model functionality, compare models to each other, and make inferences about safety and governance, but they are scattered and difficult to compare. These exist in their own formats: papers, leaderboards, blog posts, harness logs, and more. The same model for the same benchmark often returns different scores depending on who runs it and how. For example, LLaMA 65B is reported at both 63.7 and 48.8 in MMLU. These gaps may arise from assessment settings that are not commonly reported.
EEE is a correction on the reporting side. This is a JSON schema of one of the evaluation results that records:
Companion JSONL files for who is running it, which model, how to access it, generation settings, what the metrics actually mean (recommended), and output for each sample.
The schema was built based on feedback from researchers and policy researchers, and incorporates results from all sources, so harness logs, leaderboard scrapings, and paper numbers all have the same shape. The GitHub repository has converters, samples, and contributor guides. Since its inception, Hugging Face’s datastore has grown to approximately 229,000 evaluation results across over 22,000 models and 2,200 benchmarks, extracted from 31 different report formats. Reproducing just these runs from scratch would cost hundreds of thousands of dollars, but this is a reasonable argument for not littering the data after someone has paid to generate it.
Learn more about the schema and how to contribute.
Integration and attribution are now improved. Contributors can now submit their EEE results to Hugging Face Community Evals. We built a converter that takes EEE records and writes the small YAML files that Hugging Face expects, so you don’t have to manually maintain the same results in two formats.
This is a new feature for existing EEE contributors as well as anyone reporting or reading ratings. First-party evaluators reporting on their own models and third-party evaluators reporting on others’ models can submit to both community evaluations and EEE, and anyone viewing the hub can retrieve results retroactively to the full record. When you submit data through your organization’s official Hugging Face account, your results will be displayed with a verified checkmark in EvalEval, letting readers know the numbers come directly from the source. The rest of this article explains what community ratings are and how the converter works.
How Hugging Face Community Evals and EvalEval work together
There are two sides to Hugging Face Community Evals.
The benchmark resides in a dataset repository that registers itself by adding eval.yaml. Once registered, that dataset page collects and displays a leaderboard of all scores reported across the hub. The list of official benchmarks will grow over time.
Model scores reside in .eval_results/*.yaml in the model repository. These are displayed on the model card and reflected on the corresponding benchmark leaderboard. Both your own results and those submitted by others through pull requests are aggregated, and each score is given a badge to indicate whether it was author-submitted, community-submitted, or independently verified. Anyone can add scores to the model by opening the PR in the appropriate YAML file. Model authors can also close PRs and hide results in their repositories.
One of these leaderboards looks like this:
Community rates Humanity’s Last Test leaderboard on the hub
This is where EEE and community evaluation fit in. When you send results to both, two things happen. First, your scores will be displayed on the Hugging Face model page and populated into benchmark leaderboards. And second, it has a source badge that links directly to the full EEE record. There you’ll find generation configurations, harness versions, reproducibility notes, and instance-level data.
Evaluation (MMLU-Pro) of the EEE datastore (a) cross-linked to the Hugging Face model card (b) at the file level. Source Eval The Eval badge links to the complete JSON record.
The two destinations perform different tasks toward the same goal. Hugging Face displays results where people see the model, along with a link to the source. EEE maintains a complete structured record that makes the results interpretable and on top of that powers the assessment cards. Sending data to both will show the same ratings at once, making them easier to read. This is the important point of reporting either.
You can check their mutual compatibility below. The same GPQA scores that appear on the model card above are also rendered on the Eval card, composing the EEE execution data, including benchmark and model metadata, into one interpretable record. Same rating, different surfaces:
structure
Hugging Face stores evaluation scores in the model repository as YAML under .eval_results/. The only required fields are benchmark dataset, task, and value. The source block is the part that creates backlinks to EEE.
– Dataset: ID: openai/gsm8k task_id: gsm8k Value: 96.8 Date: ‘2024-07-16’ Notes: ‘8-shot CoT’ Source: URL: https://huggingface.co/datasets/evalal/EEE_datastore/blob/main/ flat/objects///.json Name: EvalEval
The converter fills this in from an existing record. This maps source_data.hf_repo to dataset.id, evaluation_name to task_id, score_details.score to value, evaluation_timestamp to date, and drops the datastore object URL as the source link to the per-record EEE JSON. We are currently processing four official benchmarks: MMLU-Pro, GPQA, HLE, and GSM8K.
Converters do more than just reshape fields. If you specify one EEE datastore collection, that collection is downloaded along with the records it references, the object hash is checked, and the scores mapped to supported benchmarks. Audit existing content before writing anything live. Read all .eval_results YAML in the model’s main branch and open PRs and compare by dataset and task instead of file name. If the score already exists, it is marked ready_present, if another score exists, it is flagged as score_conflict, and if the model repository is not resolved on the hub, it is marked missing_hf_model. All others are marked as ready.
Nothing will be pushed without your signoff. The tool writes a review file that you can inspect with a local YAML preview, displays a report on what’s ready and what needs attention, and opens the PR only after you enter a commit message by typing “OPEN PRS”. Unless –force is specified, replays reuse the collection’s cached results.
Converter review steps. Excluded entries (here, models without a matching hub repository) are listed with the EEE source URL, and the ready PR awaits an explicit OPEN PRS confirmation.
start here
Send the complete record to the EEE datastore.
Taking advantage of EEE requires only one additional step and is largely automated by the converter. The Community Ratings Converter Tool is located in our GitHub repository. To process the collection, run the following command:
uv run tools/hf-community-evals/community_evals_converter.py MMLU-Pro \ –datastore evalval/EEE_datastore@main
Review the preview and report that will be generated, and when you are ready to send, type OPEN PRS. Complete schema, CLI, and converter documentation can be found at evalvalai.com/every_eval_ever/hf-community-evals.
