Skip to content
For the complete documentation index, see llms.txt. Markdown versions of documentation pages are available by appending .md to the page URL.
Primary navigation

Suggested
Overview Models Agents Tools Audio & voice Production API reference

Evals

Manage and run evals in the OpenAI platform.

Create eval
evals.create(**kwargs) -> EvalCreateResponse
POST
/evals
Delete an eval
evals.delete(eval_id) -> EvalDeleteResponse
DELETE
/evals/{eval_id}
List evals
evals.list(**kwargs) -> SyncCursorPage[EvalListResponse]
GET
/evals
Get an eval
evals.retrieve(eval_id) -> EvalRetrieveResponse
GET
/evals/{eval_id}
Update an eval
evals.update(eval_id, **kwargs) -> EvalUpdateResponse
POST
/evals/{eval_id}
Models
class EvalCreateResponse: …

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

  • Improve the quality of my chatbot
  • See how well my chatbot handles customer support
  • Check if o4-mini is better at my usecase than gpt-6-astra
class EvalCustomDataSourceConfig: …

A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces. The response schema defines the shape of the data that will be:

  • Used to define your testing criteria and
  • What data is required when creating a run
class EvalDeleteResponse: …
class EvalListResponse: …

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

  • Improve the quality of my chatbot
  • See how well my chatbot handles customer support
  • Check if o4-mini is better at my usecase than gpt-6-astra
class EvalRetrieveResponse: …

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

  • Improve the quality of my chatbot
  • See how well my chatbot handles customer support
  • Check if o4-mini is better at my usecase than gpt-6-astra
class EvalStoredCompletionsDataSourceConfig: …

Deprecated in favor of LogsDataSourceConfig.

class EvalUpdateResponse: …

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

  • Improve the quality of my chatbot
  • See how well my chatbot handles customer support
  • Check if o4-mini is better at my usecase than gpt-6-astra

EvalsRuns

Manage and run evals in the OpenAI platform.

Cancel eval run
evals.runs.cancel(run_id, **kwargs) -> RunCancelResponse
POST
/evals/{eval_id}/runs/{run_id}/cancel
Create eval run
evals.runs.create(eval_id, **kwargs) -> RunCreateResponse
POST
/evals/{eval_id}/runs
Delete eval run
evals.runs.delete(run_id, **kwargs) -> RunDeleteResponse
DELETE
/evals/{eval_id}/runs/{run_id}
Get eval runs
evals.runs.list(eval_id, **kwargs) -> SyncCursorPage[RunListResponse]
GET
/evals/{eval_id}/runs
Get an eval run
evals.runs.retrieve(run_id, **kwargs) -> RunRetrieveResponse
GET
/evals/{eval_id}/runs/{run_id}
Models
class CreateEvalCompletionsRunDataSource: …

A CompletionsRunDataSource object describing a model sampling configuration.

class CreateEvalJSONLRunDataSource: …

A JsonlRunDataSource object with that specifies a JSONL file that matches the eval

class EvalAPIError: …

An object representing an error response from the Eval API.

class RunCancelResponse: …

A schema representing an evaluation run.

class RunCreateResponse: …

A schema representing an evaluation run.

class RunDeleteResponse: …
class RunListResponse: …

A schema representing an evaluation run.

class RunRetrieveResponse: …

A schema representing an evaluation run.

EvalsRunsOutput Items

Manage and run evals in the OpenAI platform.

Get eval run output items
evals.runs.output_items.list(run_id, **kwargs) -> SyncCursorPage[OutputItemListResponse]
GET
/evals/{eval_id}/runs/{run_id}/output_items
Get an output item of an eval run
evals.runs.output_items.retrieve(output_item_id, **kwargs) -> OutputItemRetrieveResponse
GET
/evals/{eval_id}/runs/{run_id}/output_items/{output_item_id}
Models
class OutputItemListResponse: …

A schema representing an evaluation run output item.

class OutputItemRetrieveResponse: …

A schema representing an evaluation run output item.