Skip to content
For the complete documentation index, see llms.txt. Markdown versions of documentation pages are available by appending .md to the page URL.
Primary navigation

Suggested
Overview Models Agents Tools Voice & Audio Production API reference
API Dashboard
Try ChatGPT
API Reference

Evals

Manage and run evals in the OpenAI platform.

Create eval
client.evals.create(EvalCreateParams { data_source_config, testing_criteria, metadata, name } body, RequestOptionsoptions?): EvalCreateResponse { id, created_at, data_source_config, 4 more }
POST/evals
Delete an eval
client.evals.delete(stringevalID, RequestOptionsoptions?): EvalDeleteResponse { deleted, eval_id, object }
DELETE/evals/{eval_id}
List evals
client.evals.list(EvalListParams { after, limit, order, order_by } query?, RequestOptionsoptions?): CursorPage<EvalListResponse { id, created_at, data_source_config, 4 more } >
GET/evals
Get an eval
client.evals.retrieve(stringevalID, RequestOptionsoptions?): EvalRetrieveResponse { id, created_at, data_source_config, 4 more }
GET/evals/{eval_id}
Update an eval
client.evals.update(stringevalID, EvalUpdateParams { metadata, name } body, RequestOptionsoptions?): EvalUpdateResponse { id, created_at, data_source_config, 4 more }
POST/evals/{eval_id}
ModelsExpand Collapse
EvalCreateResponse { id, created_at, data_source_config, 4 more }

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

  • Improve the quality of my chatbot
  • See how well my chatbot handles customer support
  • Check if o4-mini is better at my usecase than gpt-4o
EvalCustomDataSourceConfig { schema, type }

A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces. The response schema defines the shape of the data that will be:

  • Used to define your testing criteria and
  • What data is required when creating a run
EvalDeleteResponse { deleted, eval_id, object }
EvalListResponse { id, created_at, data_source_config, 4 more }

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

  • Improve the quality of my chatbot
  • See how well my chatbot handles customer support
  • Check if o4-mini is better at my usecase than gpt-4o
EvalRetrieveResponse { id, created_at, data_source_config, 4 more }

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

  • Improve the quality of my chatbot
  • See how well my chatbot handles customer support
  • Check if o4-mini is better at my usecase than gpt-4o
EvalStoredCompletionsDataSourceConfig { schema, type, metadata }

Deprecated in favor of LogsDataSourceConfig.

EvalUpdateResponse { id, created_at, data_source_config, 4 more }

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

  • Improve the quality of my chatbot
  • See how well my chatbot handles customer support
  • Check if o4-mini is better at my usecase than gpt-4o

EvalsRuns

Manage and run evals in the OpenAI platform.

Cancel eval run
client.evals.runs.cancel(stringrunID, RunCancelParams { eval_id } params, RequestOptionsoptions?): RunCancelResponse { id, created_at, data_source, 11 more }
POST/evals/{eval_id}/runs/{run_id}
Create eval run
client.evals.runs.create(stringevalID, RunCreateParams { data_source, metadata, name } body, RequestOptionsoptions?): RunCreateResponse { id, created_at, data_source, 11 more }
POST/evals/{eval_id}/runs
Delete eval run
client.evals.runs.delete(stringrunID, RunDeleteParams { eval_id } params, RequestOptionsoptions?): RunDeleteResponse { deleted, object, run_id }
DELETE/evals/{eval_id}/runs/{run_id}
Get eval runs
client.evals.runs.list(stringevalID, RunListParams { after, limit, order, status } query?, RequestOptionsoptions?): CursorPage<RunListResponse { id, created_at, data_source, 11 more } >
GET/evals/{eval_id}/runs
Get an eval run
client.evals.runs.retrieve(stringrunID, RunRetrieveParams { eval_id } params, RequestOptionsoptions?): RunRetrieveResponse { id, created_at, data_source, 11 more }
GET/evals/{eval_id}/runs/{run_id}
ModelsExpand Collapse
CreateEvalCompletionsRunDataSource { source, type, input_messages, 2 more }

A CompletionsRunDataSource object describing a model sampling configuration.

CreateEvalJSONLRunDataSource { source, type }

A JsonlRunDataSource object with that specifies a JSONL file that matches the eval

EvalAPIError { code, message }

An object representing an error response from the Eval API.

RunCancelResponse { id, created_at, data_source, 11 more }

A schema representing an evaluation run.

RunCreateResponse { id, created_at, data_source, 11 more }

A schema representing an evaluation run.

RunDeleteResponse { deleted, object, run_id }
RunListResponse { id, created_at, data_source, 11 more }

A schema representing an evaluation run.

RunRetrieveResponse { id, created_at, data_source, 11 more }

A schema representing an evaluation run.

EvalsRunsOutput Items

Manage and run evals in the OpenAI platform.

Get eval run output items
client.evals.runs.outputItems.list(stringrunID, OutputItemListParams { eval_id, after, limit, 2 more } params, RequestOptionsoptions?): CursorPage<OutputItemListResponse { id, created_at, datasource_item, 7 more } >
GET/evals/{eval_id}/runs/{run_id}/output_items
Get an output item of an eval run
client.evals.runs.outputItems.retrieve(stringoutputItemID, OutputItemRetrieveParams { eval_id, run_id } params, RequestOptionsoptions?): OutputItemRetrieveResponse { id, created_at, datasource_item, 7 more }
GET/evals/{eval_id}/runs/{run_id}/output_items/{output_item_id}
ModelsExpand Collapse
OutputItemListResponse { id, created_at, datasource_item, 7 more }

A schema representing an evaluation run output item.

OutputItemRetrieveResponse { id, created_at, datasource_item, 7 more }

A schema representing an evaluation run output item.