Illustration: an AI team keeps its test prompts, reviewed reference answers and graders inside the OpenAI Evals dashboard. The dashboard is already read-only, and on November 30, 2026 it shuts down.
A golden dataset is a curated set of test inputs with reviewed reference outputs or explicit pass criteria that a team uses to decide whether an AI system is good enough to release. A golden dataset belongs to the team, not to the evaluation tool, so the same comparison can continue when the tool changes.
By Ho Bae, Founder & CEO, CUBIG Corp. · Updated November 2026.
Key takeaways
- A golden dataset is the reviewed test set a team uses to approve or block a release, and it belongs to the team rather than to any evaluation tool.
- OpenAI lists the Evals dashboard, API and graders as read-only now and shutting down on November 30, 2026.
- OpenAI’s guide to moving to Promptfoo describes manually recreating prompts, provider, test cases and assertions, so test cases kept only in the dashboard need a plan before the shutdown.
- Keep inputs, pass criteria, grader definitions, data version, system version and past results in the team’s own files.
- On the new tool, change one thing at a time and compare item by item with the last recorded results before trusting the scores.
What is a golden dataset?
A golden dataset is the test set a team trusts enough to block or approve a release. Langfuse’s guide to golden dataset evaluation describes it as “a curated collection of inputs with reviewed reference outputs (or explicit pass criteria)” and notes that the concepts are tool-agnostic.
The value of a golden dataset comes from comparison under the same conditions. Stanford’s Holistic Evaluation of Language Models (HELM, Liang et al., 2022-2023) benchmarked 30 models “on the same core scenarios and metrics under standardized conditions” and reported that, before HELM, models were evaluated on an average of 17.9% of its core scenarios. HELM shows the same principle for public benchmarks: comparisons drift when the test set or the conditions change between runs.
What changes when OpenAI Evals shuts down on November 30, 2026?
OpenAI’s deprecations page lists the “Evals dashboard, API, and graders for eval workflows” with a shutdown date of November 30, 2026 and a current status of read-only; the notice date was June 3, 2026. The same page lists Agent Builder for shutdown on the same day and gpt-5 (including gpt-5-mini, gpt-5-nano and gpt-5-pro) and o3 (including o3-pro) for December 11, 2026.
OpenAI recommends Promptfoo as the next tool. The OpenAI cookbook guide on moving from OpenAI Evals to Promptfoo describes the move as manually recreating the evaluation, meaning its prompts, provider, test cases and assertions, in a Promptfoo configuration file. The guide says this approach “does not require an OpenAI Evals export feature,” so a team that kept its test cases only inside the dashboard should plan to recreate them before November 30.
What does a golden dataset need to record to survive a tool change?
A golden dataset survives a tool change when the team can rebuild the same evaluation from its own files. Each item and each run should carry the following records.
- Inputs: the exact prompts or records the system receives, including the edge cases that failed before.
- Reference outputs or pass criteria: what counts as a pass for each item, and who reviewed it.
- Grader definitions: the rubric, the judge prompt and the model used for any LLM-as-a-judge grader.
- Data version: the version of the test set and of any data the system reads during the task.
- System version: the model, prompt and settings each result came from.
- Past results: scores per item, so the team can check the first run on a new tool against the last run on the old one.

Why can scores differ after moving to a new evaluation tool?
The OpenAI cookbook guide states that similarity-based scoring may not produce identical numbers across systems, and that any manually recreated grader, especially an LLM-as-a-judge grader, should be validated before a team relies on it for regression decisions. A score change after the move can come from the grader, not from the model.
The first run on the new tool should therefore change one thing only. The team keeps the model, prompt, test set and data version fixed, reruns the golden dataset, and compares item by item with the last recorded results. Items that flip from pass to fail point first to the grader or the recreated configuration; rerunning them shows how much is run-to-run variation, and the grader should be validated before the new tool is used for release decisions.

For teams on gpt-5 or o3, the retirement on December 11, 2026 adds a second change. CUBIG’s article on separating a model change from a data change covers how to pin the data version and record a baseline before a model retires. Moving the evaluation tool and the model in the same few weeks makes it hard to tell which change moved the score.
CUBIG Shorts
Yesterday’s Approval Was for Yesterday’s Input
A short take on carrying a past evaluation forward: an approval holds only for the inputs it was given.
How do you check the golden dataset itself?
Any labeled dataset can contain errors of its own, and a golden dataset is no exception. When CUBIG examined the public, synthetic UCI AI4I 2020 dataset in November 2026, the file had no missing values across 10,000 records, yet 27 records had a failure label that did not match the failure-mode flags. The details are in CUBIG’s predictive maintenance data checks. A wrong reference answer can grade a correct output as a failure, or a wrong output as a pass.
A tool migration is a good moment to review the set: check duplicated items, labels that disagree with the source fields, cases that no longer match current policy, and gaps where a known failure type has no test item.
How does Syntitan fit into evaluation data work?
Syntitan is CUBIG’s AI-ready data platform. Syntitan keeps the original Dataset unchanged and runs preparation on a copy inside a Project, and each successful processing run becomes a new Version, so the data version behind a result stays identifiable.
For tabular data in supported binary classification tasks, Ready for Model compares results before and after data preparation with four internal task models under the same evaluation conditions. For agent use cases, Ready for Agent generates Tasks and Cases from a team’s work documents and provides a Case CSV for review, an Agent input CSV and a run guide; the team then runs the cases in its own agent environment. Those cases can become new items in a golden dataset after review.
Syntitan does not replace an evaluation tool such as Promptfoo. Promptfoo runs and stores the evaluation itself, including test cases, assertions and graders in a configuration file; the team still owns the pass criteria and the record of which data each result used.
A tool change does not have to break a team’s evaluation history. The test set, the criteria and the data version have to live with the team before the old tool goes away, and the first run on the new tool should be checked against the old results before anyone trusts it. Try Syntitan on the data behind one AI task.
