Evals
Manage and run evals in the OpenAI platform.
List evals
Create eval
Get an eval
Update an eval
Delete an eval
ModelsExpand Collapse
EvalCustomDataSourceConfig object { schema, type } A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces.
The response schema defines the shape of the data that will be:
- Used to define your testing criteria and
- What data is required when creating a run
A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces.
The response schema defines the shape of the data that will be:
- Used to define your testing criteria and
- What data is required when creating a run
The json schema for the run data source items. Learn how to build JSON schemas here.
EvalStoredCompletionsDataSourceConfig object { schema, type, metadata } Deprecated in favor of LogsDataSourceConfig.
Deprecated in favor of LogsDataSourceConfig.
The json schema for the run data source items. Learn how to build JSON schemas here.
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
EvalListResponse object { id, created_at, data_source_config, 4 more } An Eval object with a data source config and testing criteria.
An Eval represents a task to be done for your LLM integration.
Like:
- Improve the quality of my chatbot
- See how well my chatbot handles customer support
- Check if o4-mini is better at my usecase than gpt-4o
An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:
- Improve the quality of my chatbot
- See how well my chatbot handles customer support
- Check if o4-mini is better at my usecase than gpt-4o
data_source_config: EvalCustomDataSourceConfig { schema, type } or object { schema, type, metadata } or EvalStoredCompletionsDataSourceConfig { schema, type, metadata } Configuration of data sources used in runs of the evaluation.
Configuration of data sources used in runs of the evaluation.
EvalCustomDataSourceConfig object { schema, type } A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces.
The response schema defines the shape of the data that will be:
- Used to define your testing criteria and
- What data is required when creating a run
A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces.
The response schema defines the shape of the data that will be:
- Used to define your testing criteria and
- What data is required when creating a run
The json schema for the run data source items. Learn how to build JSON schemas here.
LogsDataSourceConfig object { schema, type, metadata } A LogsDataSourceConfig which specifies the metadata property of your logs query.
This is usually metadata like usecase=chatbot or prompt-version=v2, etc.
The schema returned by this data source config is used to defined what variables are available in your evals.
item and sample are both defined when using this data source config.
A LogsDataSourceConfig which specifies the metadata property of your logs query.
This is usually metadata like usecase=chatbot or prompt-version=v2, etc.
The schema returned by this data source config is used to defined what variables are available in your evals.
item and sample are both defined when using this data source config.
The json schema for the run data source items. Learn how to build JSON schemas here.
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
EvalStoredCompletionsDataSourceConfig object { schema, type, metadata } Deprecated in favor of LogsDataSourceConfig.
Deprecated in favor of LogsDataSourceConfig.
The json schema for the run data source items. Learn how to build JSON schemas here.
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
testing_criteria: array of LabelModelGrader { input, labels, model, 3 more } or StringCheckGrader { input, name, operation, 2 more } or TextSimilarityGrader { evaluation_metric, input, name, 2 more } or 2 moreA list of testing criteria.
A list of testing criteria.
LabelModelGrader object { input, labels, model, 3 more } A LabelModelGrader object which uses a model to assign labels to each item
in the evaluation.
A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
input: array of object { content, role, type }
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
GraderInputs = array of string or ResponseInputText { text, type } or object { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
StringCheckGrader object { input, name, operation, 2 more } A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
EvalCreateResponse object { id, created_at, data_source_config, 4 more } An Eval object with a data source config and testing criteria.
An Eval represents a task to be done for your LLM integration.
Like:
- Improve the quality of my chatbot
- See how well my chatbot handles customer support
- Check if o4-mini is better at my usecase than gpt-4o
An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:
- Improve the quality of my chatbot
- See how well my chatbot handles customer support
- Check if o4-mini is better at my usecase than gpt-4o
data_source_config: EvalCustomDataSourceConfig { schema, type } or object { schema, type, metadata } or EvalStoredCompletionsDataSourceConfig { schema, type, metadata } Configuration of data sources used in runs of the evaluation.
Configuration of data sources used in runs of the evaluation.
EvalCustomDataSourceConfig object { schema, type } A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces.
The response schema defines the shape of the data that will be:
- Used to define your testing criteria and
- What data is required when creating a run
A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces.
The response schema defines the shape of the data that will be:
- Used to define your testing criteria and
- What data is required when creating a run
The json schema for the run data source items. Learn how to build JSON schemas here.
LogsDataSourceConfig object { schema, type, metadata } A LogsDataSourceConfig which specifies the metadata property of your logs query.
This is usually metadata like usecase=chatbot or prompt-version=v2, etc.
The schema returned by this data source config is used to defined what variables are available in your evals.
item and sample are both defined when using this data source config.
A LogsDataSourceConfig which specifies the metadata property of your logs query.
This is usually metadata like usecase=chatbot or prompt-version=v2, etc.
The schema returned by this data source config is used to defined what variables are available in your evals.
item and sample are both defined when using this data source config.
The json schema for the run data source items. Learn how to build JSON schemas here.
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
EvalStoredCompletionsDataSourceConfig object { schema, type, metadata } Deprecated in favor of LogsDataSourceConfig.
Deprecated in favor of LogsDataSourceConfig.
The json schema for the run data source items. Learn how to build JSON schemas here.
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
testing_criteria: array of LabelModelGrader { input, labels, model, 3 more } or StringCheckGrader { input, name, operation, 2 more } or TextSimilarityGrader { evaluation_metric, input, name, 2 more } or 2 moreA list of testing criteria.
A list of testing criteria.
LabelModelGrader object { input, labels, model, 3 more } A LabelModelGrader object which uses a model to assign labels to each item
in the evaluation.
A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
input: array of object { content, role, type }
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
GraderInputs = array of string or ResponseInputText { text, type } or object { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
StringCheckGrader object { input, name, operation, 2 more } A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
EvalRetrieveResponse object { id, created_at, data_source_config, 4 more } An Eval object with a data source config and testing criteria.
An Eval represents a task to be done for your LLM integration.
Like:
- Improve the quality of my chatbot
- See how well my chatbot handles customer support
- Check if o4-mini is better at my usecase than gpt-4o
An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:
- Improve the quality of my chatbot
- See how well my chatbot handles customer support
- Check if o4-mini is better at my usecase than gpt-4o
data_source_config: EvalCustomDataSourceConfig { schema, type } or object { schema, type, metadata } or EvalStoredCompletionsDataSourceConfig { schema, type, metadata } Configuration of data sources used in runs of the evaluation.
Configuration of data sources used in runs of the evaluation.
EvalCustomDataSourceConfig object { schema, type } A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces.
The response schema defines the shape of the data that will be:
- Used to define your testing criteria and
- What data is required when creating a run
A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces.
The response schema defines the shape of the data that will be:
- Used to define your testing criteria and
- What data is required when creating a run
The json schema for the run data source items. Learn how to build JSON schemas here.
LogsDataSourceConfig object { schema, type, metadata } A LogsDataSourceConfig which specifies the metadata property of your logs query.
This is usually metadata like usecase=chatbot or prompt-version=v2, etc.
The schema returned by this data source config is used to defined what variables are available in your evals.
item and sample are both defined when using this data source config.
A LogsDataSourceConfig which specifies the metadata property of your logs query.
This is usually metadata like usecase=chatbot or prompt-version=v2, etc.
The schema returned by this data source config is used to defined what variables are available in your evals.
item and sample are both defined when using this data source config.
The json schema for the run data source items. Learn how to build JSON schemas here.
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
EvalStoredCompletionsDataSourceConfig object { schema, type, metadata } Deprecated in favor of LogsDataSourceConfig.
Deprecated in favor of LogsDataSourceConfig.
The json schema for the run data source items. Learn how to build JSON schemas here.
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
testing_criteria: array of LabelModelGrader { input, labels, model, 3 more } or StringCheckGrader { input, name, operation, 2 more } or TextSimilarityGrader { evaluation_metric, input, name, 2 more } or 2 moreA list of testing criteria.
A list of testing criteria.
LabelModelGrader object { input, labels, model, 3 more } A LabelModelGrader object which uses a model to assign labels to each item
in the evaluation.
A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
input: array of object { content, role, type }
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
GraderInputs = array of string or ResponseInputText { text, type } or object { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
StringCheckGrader object { input, name, operation, 2 more } A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
EvalUpdateResponse object { id, created_at, data_source_config, 4 more } An Eval object with a data source config and testing criteria.
An Eval represents a task to be done for your LLM integration.
Like:
- Improve the quality of my chatbot
- See how well my chatbot handles customer support
- Check if o4-mini is better at my usecase than gpt-4o
An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:
- Improve the quality of my chatbot
- See how well my chatbot handles customer support
- Check if o4-mini is better at my usecase than gpt-4o
data_source_config: EvalCustomDataSourceConfig { schema, type } or object { schema, type, metadata } or EvalStoredCompletionsDataSourceConfig { schema, type, metadata } Configuration of data sources used in runs of the evaluation.
Configuration of data sources used in runs of the evaluation.
EvalCustomDataSourceConfig object { schema, type } A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces.
The response schema defines the shape of the data that will be:
- Used to define your testing criteria and
- What data is required when creating a run
A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces.
The response schema defines the shape of the data that will be:
- Used to define your testing criteria and
- What data is required when creating a run
The json schema for the run data source items. Learn how to build JSON schemas here.
LogsDataSourceConfig object { schema, type, metadata } A LogsDataSourceConfig which specifies the metadata property of your logs query.
This is usually metadata like usecase=chatbot or prompt-version=v2, etc.
The schema returned by this data source config is used to defined what variables are available in your evals.
item and sample are both defined when using this data source config.
A LogsDataSourceConfig which specifies the metadata property of your logs query.
This is usually metadata like usecase=chatbot or prompt-version=v2, etc.
The schema returned by this data source config is used to defined what variables are available in your evals.
item and sample are both defined when using this data source config.
The json schema for the run data source items. Learn how to build JSON schemas here.
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
EvalStoredCompletionsDataSourceConfig object { schema, type, metadata } Deprecated in favor of LogsDataSourceConfig.
Deprecated in favor of LogsDataSourceConfig.
The json schema for the run data source items. Learn how to build JSON schemas here.
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
testing_criteria: array of LabelModelGrader { input, labels, model, 3 more } or StringCheckGrader { input, name, operation, 2 more } or TextSimilarityGrader { evaluation_metric, input, name, 2 more } or 2 moreA list of testing criteria.
A list of testing criteria.
LabelModelGrader object { input, labels, model, 3 more } A LabelModelGrader object which uses a model to assign labels to each item
in the evaluation.
A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
input: array of object { content, role, type }
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
GraderInputs = array of string or ResponseInputText { text, type } or object { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
StringCheckGrader object { input, name, operation, 2 more } A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
EvalsRuns
Manage and run evals in the OpenAI platform.
Get eval runs
Create eval run
Get an eval run
Cancel eval run
Delete eval run
ModelsExpand Collapse
CreateEvalCompletionsRunDataSource object { source, type, input_messages, 2 more } A CompletionsRunDataSource object describing a model sampling configuration.
A CompletionsRunDataSource object describing a model sampling configuration.
source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more } Determines what populates the item namespace in this run’s data source.
Determines what populates the item namespace in this run’s data source.
StoredCompletionsRunDataSource object { type, created_after, created_before, 3 more } A StoredCompletionsRunDataSource configuration describing a set of filters
A StoredCompletionsRunDataSource configuration describing a set of filters
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
input_messages: optional object { template, type } or object { item_reference, type } Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
TemplateInputMessages object { template, type }
template: array of EasyInputMessage { content, role, phase, type } or object { content, role, type } A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
EasyInputMessage object { content, role, phase, type } A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
Text, image, or audio input to the model, used to generate a response.
Can also contain previous assistant responses.
Text, image, or audio input to the model, used to generate a response. Can also contain previous assistant responses.
A list of one or many input items to the model, containing different content
types.
A list of one or many input items to the model, containing different content types.
ResponseInputImage object { detail, type, file_id, image_url } An image input to the model. Learn about image inputs.
An image input to the model. Learn about image inputs.
role: "user" or "assistant" or "system" or "developer"The role of the message input. One of user, assistant, system, or
developer.
The role of the message input. One of user, assistant, system, or
developer.
phase: optional "commentary" or "final_answer"Labels an assistant message as intermediate commentary (commentary) or the final answer (final_answer).
For models like gpt-5.3-codex and beyond, when sending follow-up requests, preserve and resend
phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
Labels an assistant message as intermediate commentary (commentary) or the final answer (final_answer).
For models like gpt-5.3-codex and beyond, when sending follow-up requests, preserve and resend
phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
EvalMessageObject object { content, role, type } A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
GraderInputs = array of string or ResponseInputText { text, type } or object { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1defaults tonone, which does not perform reasoning. The supported reasoning values forgpt-5.1arenone,low,medium, andhigh. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1default tomediumreasoning effort, and do not supportnone. - The
gpt-5-promodel defaults to (and only supports)highreasoning effort. xhighis supported for all models aftergpt-5.1-codex-max.
response_format: optional ResponseFormatText { type } or ResponseFormatJSONSchema { json_schema, type } or ResponseFormatJSONObject { type } An object specifying the format that the model must output.
Setting to { "type": "json_schema", "json_schema": {...} } enables
Structured Outputs which ensures the model will match your supplied JSON
schema. Learn more in the Structured Outputs
guide.
Setting to { "type": "json_object" } enables the older JSON mode, which
ensures the message the model generates is valid JSON. Using json_schema
is preferred for models that support it.
An object specifying the format that the model must output.
Setting to { "type": "json_schema", "json_schema": {...} } enables
Structured Outputs which ensures the model will match your supplied JSON
schema. Learn more in the Structured Outputs
guide.
Setting to { "type": "json_object" } enables the older JSON mode, which
ensures the message the model generates is valid JSON. Using json_schema
is preferred for models that support it.
ResponseFormatJSONSchema object { json_schema, type } JSON Schema response format. Used to generate structured JSON responses.
Learn more about Structured Outputs.
JSON Schema response format. Used to generate structured JSON responses. Learn more about Structured Outputs.
json_schema: object { name, description, schema, strict } Structured Outputs configuration options, including a JSON Schema.
Structured Outputs configuration options, including a JSON Schema.
The name of the response format. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
A description of what the response format is for, used by the model to determine how to respond in the format.
The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.
Whether to enable strict schema adherence when generating the output.
If set to true, the model will always follow the exact schema defined
in the schema field. Only a subset of JSON Schema is supported when
strict is true. To learn more, read the Structured Outputs
guide.
CreateEvalJSONLRunDataSource object { source, type } A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
RunListResponse object { id, created_at, data_source, 11 more } A schema representing an evaluation run.
A schema representing an evaluation run.
data_source: CreateEvalJSONLRunDataSource { source, type } or CreateEvalCompletionsRunDataSource { source, type, input_messages, 2 more } or object { source, type, input_messages, 2 more } Information about the run’s data source.
Information about the run’s data source.
CreateEvalJSONLRunDataSource object { source, type } A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
CreateEvalCompletionsRunDataSource object { source, type, input_messages, 2 more } A CompletionsRunDataSource object describing a model sampling configuration.
A CompletionsRunDataSource object describing a model sampling configuration.
source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more } Determines what populates the item namespace in this run’s data source.
Determines what populates the item namespace in this run’s data source.
StoredCompletionsRunDataSource object { type, created_after, created_before, 3 more } A StoredCompletionsRunDataSource configuration describing a set of filters
A StoredCompletionsRunDataSource configuration describing a set of filters
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
input_messages: optional object { template, type } or object { item_reference, type } Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
TemplateInputMessages object { template, type }
template: array of EasyInputMessage { content, role, phase, type } or object { content, role, type } A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
EasyInputMessage object { content, role, phase, type } A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
Text, image, or audio input to the model, used to generate a response.
Can also contain previous assistant responses.
Text, image, or audio input to the model, used to generate a response. Can also contain previous assistant responses.
A list of one or many input items to the model, containing different content
types.
A list of one or many input items to the model, containing different content types.
ResponseInputImage object { detail, type, file_id, image_url } An image input to the model. Learn about image inputs.
An image input to the model. Learn about image inputs.
role: "user" or "assistant" or "system" or "developer"The role of the message input. One of user, assistant, system, or
developer.
The role of the message input. One of user, assistant, system, or
developer.
phase: optional "commentary" or "final_answer"Labels an assistant message as intermediate commentary (commentary) or the final answer (final_answer).
For models like gpt-5.3-codex and beyond, when sending follow-up requests, preserve and resend
phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
Labels an assistant message as intermediate commentary (commentary) or the final answer (final_answer).
For models like gpt-5.3-codex and beyond, when sending follow-up requests, preserve and resend
phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
EvalMessageObject object { content, role, type } A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
GraderInputs = array of string or ResponseInputText { text, type } or object { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1defaults tonone, which does not perform reasoning. The supported reasoning values forgpt-5.1arenone,low,medium, andhigh. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1default tomediumreasoning effort, and do not supportnone. - The
gpt-5-promodel defaults to (and only supports)highreasoning effort. xhighis supported for all models aftergpt-5.1-codex-max.
response_format: optional ResponseFormatText { type } or ResponseFormatJSONSchema { json_schema, type } or ResponseFormatJSONObject { type } An object specifying the format that the model must output.
Setting to { "type": "json_schema", "json_schema": {...} } enables
Structured Outputs which ensures the model will match your supplied JSON
schema. Learn more in the Structured Outputs
guide.
Setting to { "type": "json_object" } enables the older JSON mode, which
ensures the message the model generates is valid JSON. Using json_schema
is preferred for models that support it.
An object specifying the format that the model must output.
Setting to { "type": "json_schema", "json_schema": {...} } enables
Structured Outputs which ensures the model will match your supplied JSON
schema. Learn more in the Structured Outputs
guide.
Setting to { "type": "json_object" } enables the older JSON mode, which
ensures the message the model generates is valid JSON. Using json_schema
is preferred for models that support it.
ResponseFormatJSONSchema object { json_schema, type } JSON Schema response format. Used to generate structured JSON responses.
Learn more about Structured Outputs.
JSON Schema response format. Used to generate structured JSON responses. Learn more about Structured Outputs.
json_schema: object { name, description, schema, strict } Structured Outputs configuration options, including a JSON Schema.
Structured Outputs configuration options, including a JSON Schema.
The name of the response format. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
A description of what the response format is for, used by the model to determine how to respond in the format.
The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.
Whether to enable strict schema adherence when generating the output.
If set to true, the model will always follow the exact schema defined
in the schema field. Only a subset of JSON Schema is supported when
strict is true. To learn more, read the Structured Outputs
guide.
ResponsesRunDataSource object { source, type, input_messages, 2 more } A ResponsesRunDataSource object describing a model sampling configuration.
A ResponsesRunDataSource object describing a model sampling configuration.
source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more } Determines what populates the item namespace in this run’s data source.
Determines what populates the item namespace in this run’s data source.
EvalResponsesSource object { type, created_after, created_before, 8 more } A EvalResponsesSource object describing a run data source configuration.
A EvalResponsesSource object describing a run data source configuration.
Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.
Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.
Optional string to search the ‘instructions’ field. This is a query parameter used to select responses.
Metadata filter for the responses. This is a query parameter used to select responses.
The name of the model to find responses for. This is a query parameter used to select responses.
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1defaults tonone, which does not perform reasoning. The supported reasoning values forgpt-5.1arenone,low,medium, andhigh. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1default tomediumreasoning effort, and do not supportnone. - The
gpt-5-promodel defaults to (and only supports)highreasoning effort. xhighis supported for all models aftergpt-5.1-codex-max.
Sampling temperature. This is a query parameter used to select responses.
List of tool names. This is a query parameter used to select responses.
input_messages: optional object { template, type } or object { item_reference, type } Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
InputMessagesTemplate object { template, type }
template: array of object { content, role } or object { content, role, type } A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
EvalMessageObject object { content, role, type } A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
GraderInputs = array of string or ResponseInputText { text, type } or object { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1defaults tonone, which does not perform reasoning. The supported reasoning values forgpt-5.1arenone,low,medium, andhigh. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1default tomediumreasoning effort, and do not supportnone. - The
gpt-5-promodel defaults to (and only supports)highreasoning effort. xhighis supported for all models aftergpt-5.1-codex-max.
text: optional object { format } Configuration options for a text response from the model. Can be plain
text or structured JSON data. Learn more:
Configuration options for a text response from the model. Can be plain text or structured JSON data. Learn more:
An object specifying the format that the model must output.
Configuring { "type": "json_schema" } enables Structured Outputs,
which ensures the model will match your supplied JSON schema. Learn more in the
Structured Outputs guide.
The default format is { "type": "text" } with no additional options.
Not recommended for gpt-4o and newer models:
Setting to { "type": "json_object" } enables the older JSON mode, which
ensures the message the model generates is valid JSON. Using json_schema
is preferred for models that support it.
tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 moreAn array of tools the model may call while generating a response. You
can specify which tool to use by setting the tool_choice parameter.
The two categories of tools you can provide the model are:
- Built-in tools: Tools that are provided by OpenAI that extend the
model’s capabilities, like web search
or file search. Learn more about
built-in tools.
- Function calls (custom tools): Functions that are defined by you,
enabling the model to call your own code. Learn more about
function calling.
An array of tools the model may call while generating a response. You
can specify which tool to use by setting the tool_choice parameter.
The two categories of tools you can provide the model are:
- Built-in tools: Tools that are provided by OpenAI that extend the model’s capabilities, like web search or file search. Learn more about built-in tools.
- Function calls (custom tools): Functions that are defined by you, enabling the model to call your own code. Learn more about function calling.
Function object { name, parameters, strict, 3 more } Defines a function in your own code the model can choose to call. Learn more about function calling.
Defines a function in your own code the model can choose to call. Learn more about function calling.
FileSearch object { type, vector_store_ids, filters, 2 more } A tool that searches for relevant content from uploaded files. Learn more about the file search tool.
A tool that searches for relevant content from uploaded files. Learn more about the file search tool.
A filter to apply.
A filter to apply.
ComparisonFilter object { key, type, value } A filter used to compare a specified attribute key to a given value using a defined comparison operation.
A filter used to compare a specified attribute key to a given value using a defined comparison operation.
CompoundFilter object { filters, type } Combine multiple filters using and or or.
Combine multiple filters using and or or.
Array of filters to combine. Items can be ComparisonFilter or CompoundFilter.
Array of filters to combine. Items can be ComparisonFilter or CompoundFilter.
ComparisonFilter object { key, type, value } A filter used to compare a specified attribute key to a given value using a defined comparison operation.
A filter used to compare a specified attribute key to a given value using a defined comparison operation.
The maximum number of results to return. This number should be between 1 and 50 inclusive.
ranking_options: optional object { hybrid_search, ranker, score_threshold } Ranking options for search.
Ranking options for search.
Computer object { type } A tool that controls a virtual computer. Learn more about the computer tool.
A tool that controls a virtual computer. Learn more about the computer tool.
ComputerUsePreview object { display_height, display_width, environment, type } A tool that controls a virtual computer. Learn more about the computer tool.
A tool that controls a virtual computer. Learn more about the computer tool.
WebSearch object { type, filters, search_context_size, user_location } Search the Internet for sources related to the prompt. Learn more about the
web search tool.
Search the Internet for sources related to the prompt. Learn more about the web search tool.
type: "web_search" or "web_search_2025_08_26"The type of the web search tool. One of web_search or web_search_2025_08_26.
The type of the web search tool. One of web_search or web_search_2025_08_26.
search_context_size: optional "low" or "medium" or "high"High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.
High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.
user_location: optional object { city, country, region, 2 more } The approximate location of the user.
The approximate location of the user.
The two-letter ISO country code of the user, e.g. US.
The IANA timezone of the user, e.g. America/Los_Angeles.
Mcp object { server_label, type, allowed_tools, 7 more } Give the model access to additional tools via remote Model Context Protocol
(MCP) servers. Learn more about MCP.
Give the model access to additional tools via remote Model Context Protocol (MCP) servers. Learn more about MCP.
allowed_tools: optional array of string or object { read_only, tool_names } List of allowed tool names or a filter object.
List of allowed tool names or a filter object.
McpToolFilter object { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
An OAuth access token that can be used with a remote MCP server, either with a custom MCP server URL or a service connector. Your application must handle the OAuth authorization flow and provide the token here.
connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 moreIdentifier for service connectors, like those available in ChatGPT. One of
server_url or connector_id must be provided. Learn more about service
connectors here.
Currently supported connector_id values are:
- Dropbox:
connector_dropbox
- Gmail:
connector_gmail
- Google Calendar:
connector_googlecalendar
- Google Drive:
connector_googledrive
- Microsoft Teams:
connector_microsoftteams
- Outlook Calendar:
connector_outlookcalendar
- Outlook Email:
connector_outlookemail
- SharePoint:
connector_sharepoint
Identifier for service connectors, like those available in ChatGPT. One of
server_url or connector_id must be provided. Learn more about service
connectors here.
Currently supported connector_id values are:
- Dropbox:
connector_dropbox - Gmail:
connector_gmail - Google Calendar:
connector_googlecalendar - Google Drive:
connector_googledrive - Microsoft Teams:
connector_microsoftteams - Outlook Calendar:
connector_outlookcalendar - Outlook Email:
connector_outlookemail - SharePoint:
connector_sharepoint
Optional HTTP headers to send to the MCP server. Use for authentication or other purposes.
require_approval: optional object { always, never } or "always" or "never"Specify which of the MCP server’s tools require approval.
Specify which of the MCP server’s tools require approval.
McpToolApprovalFilter object { always, never } Specify which of the MCP server’s tools require approval. Can be
always, never, or a filter object associated with tools
that require approval.
Specify which of the MCP server’s tools require approval. Can be
always, never, or a filter object associated with tools
that require approval.
always: optional object { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
never: optional object { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
CodeInterpreter object { container, type } A tool that runs Python code to help generate a response to a prompt.
A tool that runs Python code to help generate a response to a prompt.
container: string or object { type, file_ids, memory_limit, network_policy } The code interpreter container. Can be a container ID or an object that
specifies uploaded file IDs to make available to your code, along with an
optional memory_limit setting.
The code interpreter container. Can be a container ID or an object that
specifies uploaded file IDs to make available to your code, along with an
optional memory_limit setting.
CodeInterpreterToolAuto object { type, file_ids, memory_limit, network_policy } Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
An optional list of uploaded files to make available to your code.
memory_limit: optional "1g" or "4g" or "16g" or "64g"The memory limit for the code interpreter container.
The memory limit for the code interpreter container.
network_policy: optional ContainerNetworkPolicyDisabled { type } or ContainerNetworkPolicyAllowlist { allowed_domains, type, domain_secrets } Network access policy for the container.
Network access policy for the container.
ImageGeneration object { type, action, background, 9 more } A tool that generates images using the GPT image models.
A tool that generates images using the GPT image models.
action: optional "generate" or "edit" or "auto"Whether to generate a new image or edit an existing image. Default: auto.
Whether to generate a new image or edit an existing image. Default: auto.
background: optional "transparent" or "opaque" or "auto"Background type for the generated image. One of transparent,
opaque, or auto. Default: auto.
Background type for the generated image. One of transparent,
opaque, or auto. Default: auto.
input_fidelity: optional "high" or "low"Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for gpt-image-1 and gpt-image-1.5 and later models, unsupported for gpt-image-1-mini. Supports high and low. Defaults to low.
Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for gpt-image-1 and gpt-image-1.5 and later models, unsupported for gpt-image-1-mini. Supports high and low. Defaults to low.
input_image_mask: optional object { file_id, image_url } Optional mask for inpainting. Contains image_url
(string, optional) and file_id (string, optional).
Optional mask for inpainting. Contains image_url
(string, optional) and file_id (string, optional).
model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"The image generation model to use. Default: gpt-image-1.
The image generation model to use. Default: gpt-image-1.
Compression level for the output image. Default: 100.
output_format: optional "png" or "webp" or "jpeg"The output format of the generated image. One of png, webp, or
jpeg. Default: png.
The output format of the generated image. One of png, webp, or
jpeg. Default: png.
Number of partial images to generate in streaming mode, from 0 (default value) to 3.
LocalShell object { type } A tool that allows the model to execute shell commands in a local environment.
A tool that allows the model to execute shell commands in a local environment.
Shell object { type, environment } A tool that allows the model to execute shell commands.
A tool that allows the model to execute shell commands.
environment: optional ContainerAuto { type, file_ids, memory_limit, 2 more } or LocalEnvironment { type, skills } or ContainerReference { container_id, type }
ContainerAuto object { type, file_ids, memory_limit, 2 more }
An optional list of uploaded files to make available to your code.
network_policy: optional ContainerNetworkPolicyDisabled { type } or ContainerNetworkPolicyAllowlist { allowed_domains, type, domain_secrets } Network access policy for the container.
Network access policy for the container.
skills: optional array of SkillReference { skill_id, type, version } or InlineSkill { description, name, source, type } An optional list of skills referenced by id or inline data.
An optional list of skills referenced by id or inline data.
Custom object { name, type, defer_loading, 2 more } A custom tool that processes input using a specified format. Learn more about custom tools
A custom tool that processes input using a specified format. Learn more about custom tools
Namespace object { description, name, tools, type } Groups function/custom tools under a shared namespace.
Groups function/custom tools under a shared namespace.
tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more } The function/custom tools available inside this namespace.
The function/custom tools available inside this namespace.
Custom object { name, type, defer_loading, 2 more } A custom tool that processes input using a specified format. Learn more about custom tools
A custom tool that processes input using a specified format. Learn more about custom tools
ToolSearch object { type, description, execution, parameters } Hosted or BYOT tool search configuration for deferred tools.
Hosted or BYOT tool search configuration for deferred tools.
WebSearchPreview object { type, search_content_types, search_context_size, user_location } This tool searches the web for relevant results to use in a response. Learn more about the web search tool.
This tool searches the web for relevant results to use in a response. Learn more about the web search tool.
type: "web_search_preview" or "web_search_preview_2025_03_11"The type of the web search tool. One of web_search_preview or web_search_preview_2025_03_11.
The type of the web search tool. One of web_search_preview or web_search_preview_2025_03_11.
search_context_size: optional "low" or "medium" or "high"High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.
High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.
user_location: optional object { type, city, country, 2 more } The user’s location.
The user’s location.
The two-letter ISO country code of the user, e.g. US.
The IANA timezone of the user, e.g. America/Los_Angeles.
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
per_model_usage: array of object { cached_tokens, completion_tokens, invocation_count, 3 more } Usage statistics for each model during the evaluation run.
Usage statistics for each model during the evaluation run.
per_testing_criteria_results: array of object { failed, passed, testing_criteria } Results per testing criteria applied during the evaluation run.
Results per testing criteria applied during the evaluation run.
RunCreateResponse object { id, created_at, data_source, 11 more } A schema representing an evaluation run.
A schema representing an evaluation run.
data_source: CreateEvalJSONLRunDataSource { source, type } or CreateEvalCompletionsRunDataSource { source, type, input_messages, 2 more } or object { source, type, input_messages, 2 more } Information about the run’s data source.
Information about the run’s data source.
CreateEvalJSONLRunDataSource object { source, type } A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
CreateEvalCompletionsRunDataSource object { source, type, input_messages, 2 more } A CompletionsRunDataSource object describing a model sampling configuration.
A CompletionsRunDataSource object describing a model sampling configuration.
source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more } Determines what populates the item namespace in this run’s data source.
Determines what populates the item namespace in this run’s data source.
StoredCompletionsRunDataSource object { type, created_after, created_before, 3 more } A StoredCompletionsRunDataSource configuration describing a set of filters
A StoredCompletionsRunDataSource configuration describing a set of filters
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
input_messages: optional object { template, type } or object { item_reference, type } Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
TemplateInputMessages object { template, type }
template: array of EasyInputMessage { content, role, phase, type } or object { content, role, type } A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
EasyInputMessage object { content, role, phase, type } A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
Text, image, or audio input to the model, used to generate a response.
Can also contain previous assistant responses.
Text, image, or audio input to the model, used to generate a response. Can also contain previous assistant responses.
A list of one or many input items to the model, containing different content
types.
A list of one or many input items to the model, containing different content types.
ResponseInputImage object { detail, type, file_id, image_url } An image input to the model. Learn about image inputs.
An image input to the model. Learn about image inputs.
role: "user" or "assistant" or "system" or "developer"The role of the message input. One of user, assistant, system, or
developer.
The role of the message input. One of user, assistant, system, or
developer.
phase: optional "commentary" or "final_answer"Labels an assistant message as intermediate commentary (commentary) or the final answer (final_answer).
For models like gpt-5.3-codex and beyond, when sending follow-up requests, preserve and resend
phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
Labels an assistant message as intermediate commentary (commentary) or the final answer (final_answer).
For models like gpt-5.3-codex and beyond, when sending follow-up requests, preserve and resend
phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
EvalMessageObject object { content, role, type } A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
GraderInputs = array of string or ResponseInputText { text, type } or object { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1defaults tonone, which does not perform reasoning. The supported reasoning values forgpt-5.1arenone,low,medium, andhigh. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1default tomediumreasoning effort, and do not supportnone. - The
gpt-5-promodel defaults to (and only supports)highreasoning effort. xhighis supported for all models aftergpt-5.1-codex-max.
response_format: optional ResponseFormatText { type } or ResponseFormatJSONSchema { json_schema, type } or ResponseFormatJSONObject { type } An object specifying the format that the model must output.
Setting to { "type": "json_schema", "json_schema": {...} } enables
Structured Outputs which ensures the model will match your supplied JSON
schema. Learn more in the Structured Outputs
guide.
Setting to { "type": "json_object" } enables the older JSON mode, which
ensures the message the model generates is valid JSON. Using json_schema
is preferred for models that support it.
An object specifying the format that the model must output.
Setting to { "type": "json_schema", "json_schema": {...} } enables
Structured Outputs which ensures the model will match your supplied JSON
schema. Learn more in the Structured Outputs
guide.
Setting to { "type": "json_object" } enables the older JSON mode, which
ensures the message the model generates is valid JSON. Using json_schema
is preferred for models that support it.
ResponseFormatJSONSchema object { json_schema, type } JSON Schema response format. Used to generate structured JSON responses.
Learn more about Structured Outputs.
JSON Schema response format. Used to generate structured JSON responses. Learn more about Structured Outputs.
json_schema: object { name, description, schema, strict } Structured Outputs configuration options, including a JSON Schema.
Structured Outputs configuration options, including a JSON Schema.
The name of the response format. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
A description of what the response format is for, used by the model to determine how to respond in the format.
The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.
Whether to enable strict schema adherence when generating the output.
If set to true, the model will always follow the exact schema defined
in the schema field. Only a subset of JSON Schema is supported when
strict is true. To learn more, read the Structured Outputs
guide.
ResponsesRunDataSource object { source, type, input_messages, 2 more } A ResponsesRunDataSource object describing a model sampling configuration.
A ResponsesRunDataSource object describing a model sampling configuration.
source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more } Determines what populates the item namespace in this run’s data source.
Determines what populates the item namespace in this run’s data source.
EvalResponsesSource object { type, created_after, created_before, 8 more } A EvalResponsesSource object describing a run data source configuration.
A EvalResponsesSource object describing a run data source configuration.
Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.
Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.
Optional string to search the ‘instructions’ field. This is a query parameter used to select responses.
Metadata filter for the responses. This is a query parameter used to select responses.
The name of the model to find responses for. This is a query parameter used to select responses.
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1defaults tonone, which does not perform reasoning. The supported reasoning values forgpt-5.1arenone,low,medium, andhigh. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1default tomediumreasoning effort, and do not supportnone. - The
gpt-5-promodel defaults to (and only supports)highreasoning effort. xhighis supported for all models aftergpt-5.1-codex-max.
Sampling temperature. This is a query parameter used to select responses.
List of tool names. This is a query parameter used to select responses.
input_messages: optional object { template, type } or object { item_reference, type } Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
InputMessagesTemplate object { template, type }
template: array of object { content, role } or object { content, role, type } A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
EvalMessageObject object { content, role, type } A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
GraderInputs = array of string or ResponseInputText { text, type } or object { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1defaults tonone, which does not perform reasoning. The supported reasoning values forgpt-5.1arenone,low,medium, andhigh. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1default tomediumreasoning effort, and do not supportnone. - The
gpt-5-promodel defaults to (and only supports)highreasoning effort. xhighis supported for all models aftergpt-5.1-codex-max.
text: optional object { format } Configuration options for a text response from the model. Can be plain
text or structured JSON data. Learn more:
Configuration options for a text response from the model. Can be plain text or structured JSON data. Learn more:
An object specifying the format that the model must output.
Configuring { "type": "json_schema" } enables Structured Outputs,
which ensures the model will match your supplied JSON schema. Learn more in the
Structured Outputs guide.
The default format is { "type": "text" } with no additional options.
Not recommended for gpt-4o and newer models:
Setting to { "type": "json_object" } enables the older JSON mode, which
ensures the message the model generates is valid JSON. Using json_schema
is preferred for models that support it.
tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 moreAn array of tools the model may call while generating a response. You
can specify which tool to use by setting the tool_choice parameter.
The two categories of tools you can provide the model are:
- Built-in tools: Tools that are provided by OpenAI that extend the
model’s capabilities, like web search
or file search. Learn more about
built-in tools.
- Function calls (custom tools): Functions that are defined by you,
enabling the model to call your own code. Learn more about
function calling.
An array of tools the model may call while generating a response. You
can specify which tool to use by setting the tool_choice parameter.
The two categories of tools you can provide the model are:
- Built-in tools: Tools that are provided by OpenAI that extend the model’s capabilities, like web search or file search. Learn more about built-in tools.
- Function calls (custom tools): Functions that are defined by you, enabling the model to call your own code. Learn more about function calling.
Function object { name, parameters, strict, 3 more } Defines a function in your own code the model can choose to call. Learn more about function calling.
Defines a function in your own code the model can choose to call. Learn more about function calling.
FileSearch object { type, vector_store_ids, filters, 2 more } A tool that searches for relevant content from uploaded files. Learn more about the file search tool.
A tool that searches for relevant content from uploaded files. Learn more about the file search tool.
A filter to apply.
A filter to apply.
ComparisonFilter object { key, type, value } A filter used to compare a specified attribute key to a given value using a defined comparison operation.
A filter used to compare a specified attribute key to a given value using a defined comparison operation.
CompoundFilter object { filters, type } Combine multiple filters using and or or.
Combine multiple filters using and or or.
Array of filters to combine. Items can be ComparisonFilter or CompoundFilter.
Array of filters to combine. Items can be ComparisonFilter or CompoundFilter.
ComparisonFilter object { key, type, value } A filter used to compare a specified attribute key to a given value using a defined comparison operation.
A filter used to compare a specified attribute key to a given value using a defined comparison operation.
The maximum number of results to return. This number should be between 1 and 50 inclusive.
ranking_options: optional object { hybrid_search, ranker, score_threshold } Ranking options for search.
Ranking options for search.
Computer object { type } A tool that controls a virtual computer. Learn more about the computer tool.
A tool that controls a virtual computer. Learn more about the computer tool.
ComputerUsePreview object { display_height, display_width, environment, type } A tool that controls a virtual computer. Learn more about the computer tool.
A tool that controls a virtual computer. Learn more about the computer tool.
WebSearch object { type, filters, search_context_size, user_location } Search the Internet for sources related to the prompt. Learn more about the
web search tool.
Search the Internet for sources related to the prompt. Learn more about the web search tool.
type: "web_search" or "web_search_2025_08_26"The type of the web search tool. One of web_search or web_search_2025_08_26.
The type of the web search tool. One of web_search or web_search_2025_08_26.
search_context_size: optional "low" or "medium" or "high"High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.
High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.
user_location: optional object { city, country, region, 2 more } The approximate location of the user.
The approximate location of the user.
The two-letter ISO country code of the user, e.g. US.
The IANA timezone of the user, e.g. America/Los_Angeles.
Mcp object { server_label, type, allowed_tools, 7 more } Give the model access to additional tools via remote Model Context Protocol
(MCP) servers. Learn more about MCP.
Give the model access to additional tools via remote Model Context Protocol (MCP) servers. Learn more about MCP.
allowed_tools: optional array of string or object { read_only, tool_names } List of allowed tool names or a filter object.
List of allowed tool names or a filter object.
McpToolFilter object { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
An OAuth access token that can be used with a remote MCP server, either with a custom MCP server URL or a service connector. Your application must handle the OAuth authorization flow and provide the token here.
connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 moreIdentifier for service connectors, like those available in ChatGPT. One of
server_url or connector_id must be provided. Learn more about service
connectors here.
Currently supported connector_id values are:
- Dropbox:
connector_dropbox
- Gmail:
connector_gmail
- Google Calendar:
connector_googlecalendar
- Google Drive:
connector_googledrive
- Microsoft Teams:
connector_microsoftteams
- Outlook Calendar:
connector_outlookcalendar
- Outlook Email:
connector_outlookemail
- SharePoint:
connector_sharepoint
Identifier for service connectors, like those available in ChatGPT. One of
server_url or connector_id must be provided. Learn more about service
connectors here.
Currently supported connector_id values are:
- Dropbox:
connector_dropbox - Gmail:
connector_gmail - Google Calendar:
connector_googlecalendar - Google Drive:
connector_googledrive - Microsoft Teams:
connector_microsoftteams - Outlook Calendar:
connector_outlookcalendar - Outlook Email:
connector_outlookemail - SharePoint:
connector_sharepoint
Optional HTTP headers to send to the MCP server. Use for authentication or other purposes.
require_approval: optional object { always, never } or "always" or "never"Specify which of the MCP server’s tools require approval.
Specify which of the MCP server’s tools require approval.
McpToolApprovalFilter object { always, never } Specify which of the MCP server’s tools require approval. Can be
always, never, or a filter object associated with tools
that require approval.
Specify which of the MCP server’s tools require approval. Can be
always, never, or a filter object associated with tools
that require approval.
always: optional object { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
never: optional object { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
CodeInterpreter object { container, type } A tool that runs Python code to help generate a response to a prompt.
A tool that runs Python code to help generate a response to a prompt.
container: string or object { type, file_ids, memory_limit, network_policy } The code interpreter container. Can be a container ID or an object that
specifies uploaded file IDs to make available to your code, along with an
optional memory_limit setting.
The code interpreter container. Can be a container ID or an object that
specifies uploaded file IDs to make available to your code, along with an
optional memory_limit setting.
CodeInterpreterToolAuto object { type, file_ids, memory_limit, network_policy } Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
An optional list of uploaded files to make available to your code.
memory_limit: optional "1g" or "4g" or "16g" or "64g"The memory limit for the code interpreter container.
The memory limit for the code interpreter container.
network_policy: optional ContainerNetworkPolicyDisabled { type } or ContainerNetworkPolicyAllowlist { allowed_domains, type, domain_secrets } Network access policy for the container.
Network access policy for the container.
ImageGeneration object { type, action, background, 9 more } A tool that generates images using the GPT image models.
A tool that generates images using the GPT image models.
action: optional "generate" or "edit" or "auto"Whether to generate a new image or edit an existing image. Default: auto.
Whether to generate a new image or edit an existing image. Default: auto.
background: optional "transparent" or "opaque" or "auto"Background type for the generated image. One of transparent,
opaque, or auto. Default: auto.
Background type for the generated image. One of transparent,
opaque, or auto. Default: auto.
input_fidelity: optional "high" or "low"Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for gpt-image-1 and gpt-image-1.5 and later models, unsupported for gpt-image-1-mini. Supports high and low. Defaults to low.
Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for gpt-image-1 and gpt-image-1.5 and later models, unsupported for gpt-image-1-mini. Supports high and low. Defaults to low.
input_image_mask: optional object { file_id, image_url } Optional mask for inpainting. Contains image_url
(string, optional) and file_id (string, optional).
Optional mask for inpainting. Contains image_url
(string, optional) and file_id (string, optional).
model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"The image generation model to use. Default: gpt-image-1.
The image generation model to use. Default: gpt-image-1.
Compression level for the output image. Default: 100.
output_format: optional "png" or "webp" or "jpeg"The output format of the generated image. One of png, webp, or
jpeg. Default: png.
The output format of the generated image. One of png, webp, or
jpeg. Default: png.
Number of partial images to generate in streaming mode, from 0 (default value) to 3.
LocalShell object { type } A tool that allows the model to execute shell commands in a local environment.
A tool that allows the model to execute shell commands in a local environment.
Shell object { type, environment } A tool that allows the model to execute shell commands.
A tool that allows the model to execute shell commands.
environment: optional ContainerAuto { type, file_ids, memory_limit, 2 more } or LocalEnvironment { type, skills } or ContainerReference { container_id, type }
ContainerAuto object { type, file_ids, memory_limit, 2 more }
An optional list of uploaded files to make available to your code.
network_policy: optional ContainerNetworkPolicyDisabled { type } or ContainerNetworkPolicyAllowlist { allowed_domains, type, domain_secrets } Network access policy for the container.
Network access policy for the container.
skills: optional array of SkillReference { skill_id, type, version } or InlineSkill { description, name, source, type } An optional list of skills referenced by id or inline data.
An optional list of skills referenced by id or inline data.
Custom object { name, type, defer_loading, 2 more } A custom tool that processes input using a specified format. Learn more about custom tools
A custom tool that processes input using a specified format. Learn more about custom tools
Namespace object { description, name, tools, type } Groups function/custom tools under a shared namespace.
Groups function/custom tools under a shared namespace.
tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more } The function/custom tools available inside this namespace.
The function/custom tools available inside this namespace.
Custom object { name, type, defer_loading, 2 more } A custom tool that processes input using a specified format. Learn more about custom tools
A custom tool that processes input using a specified format. Learn more about custom tools
ToolSearch object { type, description, execution, parameters } Hosted or BYOT tool search configuration for deferred tools.
Hosted or BYOT tool search configuration for deferred tools.
WebSearchPreview object { type, search_content_types, search_context_size, user_location } This tool searches the web for relevant results to use in a response. Learn more about the web search tool.
This tool searches the web for relevant results to use in a response. Learn more about the web search tool.
type: "web_search_preview" or "web_search_preview_2025_03_11"The type of the web search tool. One of web_search_preview or web_search_preview_2025_03_11.
The type of the web search tool. One of web_search_preview or web_search_preview_2025_03_11.
search_context_size: optional "low" or "medium" or "high"High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.
High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.
user_location: optional object { type, city, country, 2 more } The user’s location.
The user’s location.
The two-letter ISO country code of the user, e.g. US.
The IANA timezone of the user, e.g. America/Los_Angeles.
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
per_model_usage: array of object { cached_tokens, completion_tokens, invocation_count, 3 more } Usage statistics for each model during the evaluation run.
Usage statistics for each model during the evaluation run.
per_testing_criteria_results: array of object { failed, passed, testing_criteria } Results per testing criteria applied during the evaluation run.
Results per testing criteria applied during the evaluation run.
RunRetrieveResponse object { id, created_at, data_source, 11 more } A schema representing an evaluation run.
A schema representing an evaluation run.
data_source: CreateEvalJSONLRunDataSource { source, type } or CreateEvalCompletionsRunDataSource { source, type, input_messages, 2 more } or object { source, type, input_messages, 2 more } Information about the run’s data source.
Information about the run’s data source.
CreateEvalJSONLRunDataSource object { source, type } A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
CreateEvalCompletionsRunDataSource object { source, type, input_messages, 2 more } A CompletionsRunDataSource object describing a model sampling configuration.
A CompletionsRunDataSource object describing a model sampling configuration.
source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more } Determines what populates the item namespace in this run’s data source.
Determines what populates the item namespace in this run’s data source.
StoredCompletionsRunDataSource object { type, created_after, created_before, 3 more } A StoredCompletionsRunDataSource configuration describing a set of filters
A StoredCompletionsRunDataSource configuration describing a set of filters
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
input_messages: optional object { template, type } or object { item_reference, type } Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
TemplateInputMessages object { template, type }
template: array of EasyInputMessage { content, role, phase, type } or object { content, role, type } A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
EasyInputMessage object { content, role, phase, type } A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
Text, image, or audio input to the model, used to generate a response.
Can also contain previous assistant responses.
Text, image, or audio input to the model, used to generate a response. Can also contain previous assistant responses.
A list of one or many input items to the model, containing different content
types.
A list of one or many input items to the model, containing different content types.
ResponseInputImage object { detail, type, file_id, image_url } An image input to the model. Learn about image inputs.
An image input to the model. Learn about image inputs.
role: "user" or "assistant" or "system" or "developer"The role of the message input. One of user, assistant, system, or
developer.
The role of the message input. One of user, assistant, system, or
developer.
phase: optional "commentary" or "final_answer"Labels an assistant message as intermediate commentary (commentary) or the final answer (final_answer).
For models like gpt-5.3-codex and beyond, when sending follow-up requests, preserve and resend
phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
Labels an assistant message as intermediate commentary (commentary) or the final answer (final_answer).
For models like gpt-5.3-codex and beyond, when sending follow-up requests, preserve and resend
phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
EvalMessageObject object { content, role, type } A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
GraderInputs = array of string or ResponseInputText { text, type } or object { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1defaults tonone, which does not perform reasoning. The supported reasoning values forgpt-5.1arenone,low,medium, andhigh. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1default tomediumreasoning effort, and do not supportnone. - The
gpt-5-promodel defaults to (and only supports)highreasoning effort. xhighis supported for all models aftergpt-5.1-codex-max.
response_format: optional ResponseFormatText { type } or ResponseFormatJSONSchema { json_schema, type } or ResponseFormatJSONObject { type } An object specifying the format that the model must output.
Setting to { "type": "json_schema", "json_schema": {...} } enables
Structured Outputs which ensures the model will match your supplied JSON
schema. Learn more in the Structured Outputs
guide.
Setting to { "type": "json_object" } enables the older JSON mode, which
ensures the message the model generates is valid JSON. Using json_schema
is preferred for models that support it.
An object specifying the format that the model must output.
Setting to { "type": "json_schema", "json_schema": {...} } enables
Structured Outputs which ensures the model will match your supplied JSON
schema. Learn more in the Structured Outputs
guide.
Setting to { "type": "json_object" } enables the older JSON mode, which
ensures the message the model generates is valid JSON. Using json_schema
is preferred for models that support it.
ResponseFormatJSONSchema object { json_schema, type } JSON Schema response format. Used to generate structured JSON responses.
Learn more about Structured Outputs.
JSON Schema response format. Used to generate structured JSON responses. Learn more about Structured Outputs.
json_schema: object { name, description, schema, strict } Structured Outputs configuration options, including a JSON Schema.
Structured Outputs configuration options, including a JSON Schema.
The name of the response format. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
A description of what the response format is for, used by the model to determine how to respond in the format.
The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.
Whether to enable strict schema adherence when generating the output.
If set to true, the model will always follow the exact schema defined
in the schema field. Only a subset of JSON Schema is supported when
strict is true. To learn more, read the Structured Outputs
guide.
ResponsesRunDataSource object { source, type, input_messages, 2 more } A ResponsesRunDataSource object describing a model sampling configuration.
A ResponsesRunDataSource object describing a model sampling configuration.
source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more } Determines what populates the item namespace in this run’s data source.
Determines what populates the item namespace in this run’s data source.
EvalResponsesSource object { type, created_after, created_before, 8 more } A EvalResponsesSource object describing a run data source configuration.
A EvalResponsesSource object describing a run data source configuration.
Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.
Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.
Optional string to search the ‘instructions’ field. This is a query parameter used to select responses.
Metadata filter for the responses. This is a query parameter used to select responses.
The name of the model to find responses for. This is a query parameter used to select responses.
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1defaults tonone, which does not perform reasoning. The supported reasoning values forgpt-5.1arenone,low,medium, andhigh. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1default tomediumreasoning effort, and do not supportnone. - The
gpt-5-promodel defaults to (and only supports)highreasoning effort. xhighis supported for all models aftergpt-5.1-codex-max.
Sampling temperature. This is a query parameter used to select responses.
List of tool names. This is a query parameter used to select responses.
input_messages: optional object { template, type } or object { item_reference, type } Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
InputMessagesTemplate object { template, type }
template: array of object { content, role } or object { content, role, type } A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
EvalMessageObject object { content, role, type } A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
GraderInputs = array of string or ResponseInputText { text, type } or object { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1defaults tonone, which does not perform reasoning. The supported reasoning values forgpt-5.1arenone,low,medium, andhigh. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1default tomediumreasoning effort, and do not supportnone. - The
gpt-5-promodel defaults to (and only supports)highreasoning effort. xhighis supported for all models aftergpt-5.1-codex-max.
text: optional object { format } Configuration options for a text response from the model. Can be plain
text or structured JSON data. Learn more:
Configuration options for a text response from the model. Can be plain text or structured JSON data. Learn more:
An object specifying the format that the model must output.
Configuring { "type": "json_schema" } enables Structured Outputs,
which ensures the model will match your supplied JSON schema. Learn more in the
Structured Outputs guide.
The default format is { "type": "text" } with no additional options.
Not recommended for gpt-4o and newer models:
Setting to { "type": "json_object" } enables the older JSON mode, which
ensures the message the model generates is valid JSON. Using json_schema
is preferred for models that support it.
tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 moreAn array of tools the model may call while generating a response. You
can specify which tool to use by setting the tool_choice parameter.
The two categories of tools you can provide the model are:
- Built-in tools: Tools that are provided by OpenAI that extend the
model’s capabilities, like web search
or file search. Learn more about
built-in tools.
- Function calls (custom tools): Functions that are defined by you,
enabling the model to call your own code. Learn more about
function calling.
An array of tools the model may call while generating a response. You
can specify which tool to use by setting the tool_choice parameter.
The two categories of tools you can provide the model are:
- Built-in tools: Tools that are provided by OpenAI that extend the model’s capabilities, like web search or file search. Learn more about built-in tools.
- Function calls (custom tools): Functions that are defined by you, enabling the model to call your own code. Learn more about function calling.
Function object { name, parameters, strict, 3 more } Defines a function in your own code the model can choose to call. Learn more about function calling.
Defines a function in your own code the model can choose to call. Learn more about function calling.
FileSearch object { type, vector_store_ids, filters, 2 more } A tool that searches for relevant content from uploaded files. Learn more about the file search tool.
A tool that searches for relevant content from uploaded files. Learn more about the file search tool.
A filter to apply.
A filter to apply.
ComparisonFilter object { key, type, value } A filter used to compare a specified attribute key to a given value using a defined comparison operation.
A filter used to compare a specified attribute key to a given value using a defined comparison operation.
CompoundFilter object { filters, type } Combine multiple filters using and or or.
Combine multiple filters using and or or.
Array of filters to combine. Items can be ComparisonFilter or CompoundFilter.
Array of filters to combine. Items can be ComparisonFilter or CompoundFilter.
ComparisonFilter object { key, type, value } A filter used to compare a specified attribute key to a given value using a defined comparison operation.
A filter used to compare a specified attribute key to a given value using a defined comparison operation.
The maximum number of results to return. This number should be between 1 and 50 inclusive.
ranking_options: optional object { hybrid_search, ranker, score_threshold } Ranking options for search.
Ranking options for search.
Computer object { type } A tool that controls a virtual computer. Learn more about the computer tool.
A tool that controls a virtual computer. Learn more about the computer tool.
ComputerUsePreview object { display_height, display_width, environment, type } A tool that controls a virtual computer. Learn more about the computer tool.
A tool that controls a virtual computer. Learn more about the computer tool.
WebSearch object { type, filters, search_context_size, user_location } Search the Internet for sources related to the prompt. Learn more about the
web search tool.
Search the Internet for sources related to the prompt. Learn more about the web search tool.
type: "web_search" or "web_search_2025_08_26"The type of the web search tool. One of web_search or web_search_2025_08_26.
The type of the web search tool. One of web_search or web_search_2025_08_26.
search_context_size: optional "low" or "medium" or "high"High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.
High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.
user_location: optional object { city, country, region, 2 more } The approximate location of the user.
The approximate location of the user.
The two-letter ISO country code of the user, e.g. US.
The IANA timezone of the user, e.g. America/Los_Angeles.
Mcp object { server_label, type, allowed_tools, 7 more } Give the model access to additional tools via remote Model Context Protocol
(MCP) servers. Learn more about MCP.
Give the model access to additional tools via remote Model Context Protocol (MCP) servers. Learn more about MCP.
allowed_tools: optional array of string or object { read_only, tool_names } List of allowed tool names or a filter object.
List of allowed tool names or a filter object.
McpToolFilter object { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
An OAuth access token that can be used with a remote MCP server, either with a custom MCP server URL or a service connector. Your application must handle the OAuth authorization flow and provide the token here.
connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 moreIdentifier for service connectors, like those available in ChatGPT. One of
server_url or connector_id must be provided. Learn more about service
connectors here.
Currently supported connector_id values are:
- Dropbox:
connector_dropbox
- Gmail:
connector_gmail
- Google Calendar:
connector_googlecalendar
- Google Drive:
connector_googledrive
- Microsoft Teams:
connector_microsoftteams
- Outlook Calendar:
connector_outlookcalendar
- Outlook Email:
connector_outlookemail
- SharePoint:
connector_sharepoint
Identifier for service connectors, like those available in ChatGPT. One of
server_url or connector_id must be provided. Learn more about service
connectors here.
Currently supported connector_id values are:
- Dropbox:
connector_dropbox - Gmail:
connector_gmail - Google Calendar:
connector_googlecalendar - Google Drive:
connector_googledrive - Microsoft Teams:
connector_microsoftteams - Outlook Calendar:
connector_outlookcalendar - Outlook Email:
connector_outlookemail - SharePoint:
connector_sharepoint
Optional HTTP headers to send to the MCP server. Use for authentication or other purposes.
require_approval: optional object { always, never } or "always" or "never"Specify which of the MCP server’s tools require approval.
Specify which of the MCP server’s tools require approval.
McpToolApprovalFilter object { always, never } Specify which of the MCP server’s tools require approval. Can be
always, never, or a filter object associated with tools
that require approval.
Specify which of the MCP server’s tools require approval. Can be
always, never, or a filter object associated with tools
that require approval.
always: optional object { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
never: optional object { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
CodeInterpreter object { container, type } A tool that runs Python code to help generate a response to a prompt.
A tool that runs Python code to help generate a response to a prompt.
container: string or object { type, file_ids, memory_limit, network_policy } The code interpreter container. Can be a container ID or an object that
specifies uploaded file IDs to make available to your code, along with an
optional memory_limit setting.
The code interpreter container. Can be a container ID or an object that
specifies uploaded file IDs to make available to your code, along with an
optional memory_limit setting.
CodeInterpreterToolAuto object { type, file_ids, memory_limit, network_policy } Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
An optional list of uploaded files to make available to your code.
memory_limit: optional "1g" or "4g" or "16g" or "64g"The memory limit for the code interpreter container.
The memory limit for the code interpreter container.
network_policy: optional ContainerNetworkPolicyDisabled { type } or ContainerNetworkPolicyAllowlist { allowed_domains, type, domain_secrets } Network access policy for the container.
Network access policy for the container.
ImageGeneration object { type, action, background, 9 more } A tool that generates images using the GPT image models.
A tool that generates images using the GPT image models.
action: optional "generate" or "edit" or "auto"Whether to generate a new image or edit an existing image. Default: auto.
Whether to generate a new image or edit an existing image. Default: auto.
background: optional "transparent" or "opaque" or "auto"Background type for the generated image. One of transparent,
opaque, or auto. Default: auto.
Background type for the generated image. One of transparent,
opaque, or auto. Default: auto.
input_fidelity: optional "high" or "low"Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for gpt-image-1 and gpt-image-1.5 and later models, unsupported for gpt-image-1-mini. Supports high and low. Defaults to low.
Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for gpt-image-1 and gpt-image-1.5 and later models, unsupported for gpt-image-1-mini. Supports high and low. Defaults to low.
input_image_mask: optional object { file_id, image_url } Optional mask for inpainting. Contains image_url
(string, optional) and file_id (string, optional).
Optional mask for inpainting. Contains image_url
(string, optional) and file_id (string, optional).
model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"The image generation model to use. Default: gpt-image-1.
The image generation model to use. Default: gpt-image-1.
Compression level for the output image. Default: 100.
output_format: optional "png" or "webp" or "jpeg"The output format of the generated image. One of png, webp, or
jpeg. Default: png.
The output format of the generated image. One of png, webp, or
jpeg. Default: png.
Number of partial images to generate in streaming mode, from 0 (default value) to 3.
LocalShell object { type } A tool that allows the model to execute shell commands in a local environment.
A tool that allows the model to execute shell commands in a local environment.
Shell object { type, environment } A tool that allows the model to execute shell commands.
A tool that allows the model to execute shell commands.
environment: optional ContainerAuto { type, file_ids, memory_limit, 2 more } or LocalEnvironment { type, skills } or ContainerReference { container_id, type }
ContainerAuto object { type, file_ids, memory_limit, 2 more }
An optional list of uploaded files to make available to your code.
network_policy: optional ContainerNetworkPolicyDisabled { type } or ContainerNetworkPolicyAllowlist { allowed_domains, type, domain_secrets } Network access policy for the container.
Network access policy for the container.
skills: optional array of SkillReference { skill_id, type, version } or InlineSkill { description, name, source, type } An optional list of skills referenced by id or inline data.
An optional list of skills referenced by id or inline data.
Custom object { name, type, defer_loading, 2 more } A custom tool that processes input using a specified format. Learn more about custom tools
A custom tool that processes input using a specified format. Learn more about custom tools
Namespace object { description, name, tools, type } Groups function/custom tools under a shared namespace.
Groups function/custom tools under a shared namespace.
tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more } The function/custom tools available inside this namespace.
The function/custom tools available inside this namespace.
Custom object { name, type, defer_loading, 2 more } A custom tool that processes input using a specified format. Learn more about custom tools
A custom tool that processes input using a specified format. Learn more about custom tools
ToolSearch object { type, description, execution, parameters } Hosted or BYOT tool search configuration for deferred tools.
Hosted or BYOT tool search configuration for deferred tools.
WebSearchPreview object { type, search_content_types, search_context_size, user_location } This tool searches the web for relevant results to use in a response. Learn more about the web search tool.
This tool searches the web for relevant results to use in a response. Learn more about the web search tool.
type: "web_search_preview" or "web_search_preview_2025_03_11"The type of the web search tool. One of web_search_preview or web_search_preview_2025_03_11.
The type of the web search tool. One of web_search_preview or web_search_preview_2025_03_11.
search_context_size: optional "low" or "medium" or "high"High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.
High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.
user_location: optional object { type, city, country, 2 more } The user’s location.
The user’s location.
The two-letter ISO country code of the user, e.g. US.
The IANA timezone of the user, e.g. America/Los_Angeles.
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
per_model_usage: array of object { cached_tokens, completion_tokens, invocation_count, 3 more } Usage statistics for each model during the evaluation run.
Usage statistics for each model during the evaluation run.
per_testing_criteria_results: array of object { failed, passed, testing_criteria } Results per testing criteria applied during the evaluation run.
Results per testing criteria applied during the evaluation run.
RunCancelResponse object { id, created_at, data_source, 11 more } A schema representing an evaluation run.
A schema representing an evaluation run.
data_source: CreateEvalJSONLRunDataSource { source, type } or CreateEvalCompletionsRunDataSource { source, type, input_messages, 2 more } or object { source, type, input_messages, 2 more } Information about the run’s data source.
Information about the run’s data source.
CreateEvalJSONLRunDataSource object { source, type } A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
CreateEvalCompletionsRunDataSource object { source, type, input_messages, 2 more } A CompletionsRunDataSource object describing a model sampling configuration.
A CompletionsRunDataSource object describing a model sampling configuration.
source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more } Determines what populates the item namespace in this run’s data source.
Determines what populates the item namespace in this run’s data source.
StoredCompletionsRunDataSource object { type, created_after, created_before, 3 more } A StoredCompletionsRunDataSource configuration describing a set of filters
A StoredCompletionsRunDataSource configuration describing a set of filters
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
input_messages: optional object { template, type } or object { item_reference, type } Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
TemplateInputMessages object { template, type }
template: array of EasyInputMessage { content, role, phase, type } or object { content, role, type } A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
EasyInputMessage object { content, role, phase, type } A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
Text, image, or audio input to the model, used to generate a response.
Can also contain previous assistant responses.
Text, image, or audio input to the model, used to generate a response. Can also contain previous assistant responses.
A list of one or many input items to the model, containing different content
types.
A list of one or many input items to the model, containing different content types.
ResponseInputImage object { detail, type, file_id, image_url } An image input to the model. Learn about image inputs.
An image input to the model. Learn about image inputs.
role: "user" or "assistant" or "system" or "developer"The role of the message input. One of user, assistant, system, or
developer.
The role of the message input. One of user, assistant, system, or
developer.
phase: optional "commentary" or "final_answer"Labels an assistant message as intermediate commentary (commentary) or the final answer (final_answer).
For models like gpt-5.3-codex and beyond, when sending follow-up requests, preserve and resend
phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
Labels an assistant message as intermediate commentary (commentary) or the final answer (final_answer).
For models like gpt-5.3-codex and beyond, when sending follow-up requests, preserve and resend
phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
EvalMessageObject object { content, role, type } A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
GraderInputs = array of string or ResponseInputText { text, type } or object { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1defaults tonone, which does not perform reasoning. The supported reasoning values forgpt-5.1arenone,low,medium, andhigh. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1default tomediumreasoning effort, and do not supportnone. - The
gpt-5-promodel defaults to (and only supports)highreasoning effort. xhighis supported for all models aftergpt-5.1-codex-max.
response_format: optional ResponseFormatText { type } or ResponseFormatJSONSchema { json_schema, type } or ResponseFormatJSONObject { type } An object specifying the format that the model must output.
Setting to { "type": "json_schema", "json_schema": {...} } enables
Structured Outputs which ensures the model will match your supplied JSON
schema. Learn more in the Structured Outputs
guide.
Setting to { "type": "json_object" } enables the older JSON mode, which
ensures the message the model generates is valid JSON. Using json_schema
is preferred for models that support it.
An object specifying the format that the model must output.
Setting to { "type": "json_schema", "json_schema": {...} } enables
Structured Outputs which ensures the model will match your supplied JSON
schema. Learn more in the Structured Outputs
guide.
Setting to { "type": "json_object" } enables the older JSON mode, which
ensures the message the model generates is valid JSON. Using json_schema
is preferred for models that support it.
ResponseFormatJSONSchema object { json_schema, type } JSON Schema response format. Used to generate structured JSON responses.
Learn more about Structured Outputs.
JSON Schema response format. Used to generate structured JSON responses. Learn more about Structured Outputs.
json_schema: object { name, description, schema, strict } Structured Outputs configuration options, including a JSON Schema.
Structured Outputs configuration options, including a JSON Schema.
The name of the response format. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
A description of what the response format is for, used by the model to determine how to respond in the format.
The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.
Whether to enable strict schema adherence when generating the output.
If set to true, the model will always follow the exact schema defined
in the schema field. Only a subset of JSON Schema is supported when
strict is true. To learn more, read the Structured Outputs
guide.
ResponsesRunDataSource object { source, type, input_messages, 2 more } A ResponsesRunDataSource object describing a model sampling configuration.
A ResponsesRunDataSource object describing a model sampling configuration.
source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more } Determines what populates the item namespace in this run’s data source.
Determines what populates the item namespace in this run’s data source.
EvalResponsesSource object { type, created_after, created_before, 8 more } A EvalResponsesSource object describing a run data source configuration.
A EvalResponsesSource object describing a run data source configuration.
Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.
Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.
Optional string to search the ‘instructions’ field. This is a query parameter used to select responses.
Metadata filter for the responses. This is a query parameter used to select responses.
The name of the model to find responses for. This is a query parameter used to select responses.
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1defaults tonone, which does not perform reasoning. The supported reasoning values forgpt-5.1arenone,low,medium, andhigh. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1default tomediumreasoning effort, and do not supportnone. - The
gpt-5-promodel defaults to (and only supports)highreasoning effort. xhighis supported for all models aftergpt-5.1-codex-max.
Sampling temperature. This is a query parameter used to select responses.
List of tool names. This is a query parameter used to select responses.
input_messages: optional object { template, type } or object { item_reference, type } Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
InputMessagesTemplate object { template, type }
template: array of object { content, role } or object { content, role, type } A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
EvalMessageObject object { content, role, type } A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
GraderInputs = array of string or ResponseInputText { text, type } or object { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
InputImage object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1defaults tonone, which does not perform reasoning. The supported reasoning values forgpt-5.1arenone,low,medium, andhigh. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1default tomediumreasoning effort, and do not supportnone. - The
gpt-5-promodel defaults to (and only supports)highreasoning effort. xhighis supported for all models aftergpt-5.1-codex-max.
text: optional object { format } Configuration options for a text response from the model. Can be plain
text or structured JSON data. Learn more:
Configuration options for a text response from the model. Can be plain text or structured JSON data. Learn more:
An object specifying the format that the model must output.
Configuring { "type": "json_schema" } enables Structured Outputs,
which ensures the model will match your supplied JSON schema. Learn more in the
Structured Outputs guide.
The default format is { "type": "text" } with no additional options.
Not recommended for gpt-4o and newer models:
Setting to { "type": "json_object" } enables the older JSON mode, which
ensures the message the model generates is valid JSON. Using json_schema
is preferred for models that support it.
tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 moreAn array of tools the model may call while generating a response. You
can specify which tool to use by setting the tool_choice parameter.
The two categories of tools you can provide the model are:
- Built-in tools: Tools that are provided by OpenAI that extend the
model’s capabilities, like web search
or file search. Learn more about
built-in tools.
- Function calls (custom tools): Functions that are defined by you,
enabling the model to call your own code. Learn more about
function calling.
An array of tools the model may call while generating a response. You
can specify which tool to use by setting the tool_choice parameter.
The two categories of tools you can provide the model are:
- Built-in tools: Tools that are provided by OpenAI that extend the model’s capabilities, like web search or file search. Learn more about built-in tools.
- Function calls (custom tools): Functions that are defined by you, enabling the model to call your own code. Learn more about function calling.
Function object { name, parameters, strict, 3 more } Defines a function in your own code the model can choose to call. Learn more about function calling.
Defines a function in your own code the model can choose to call. Learn more about function calling.
FileSearch object { type, vector_store_ids, filters, 2 more } A tool that searches for relevant content from uploaded files. Learn more about the file search tool.
A tool that searches for relevant content from uploaded files. Learn more about the file search tool.
A filter to apply.
A filter to apply.
ComparisonFilter object { key, type, value } A filter used to compare a specified attribute key to a given value using a defined comparison operation.
A filter used to compare a specified attribute key to a given value using a defined comparison operation.
CompoundFilter object { filters, type } Combine multiple filters using and or or.
Combine multiple filters using and or or.
Array of filters to combine. Items can be ComparisonFilter or CompoundFilter.
Array of filters to combine. Items can be ComparisonFilter or CompoundFilter.
ComparisonFilter object { key, type, value } A filter used to compare a specified attribute key to a given value using a defined comparison operation.
A filter used to compare a specified attribute key to a given value using a defined comparison operation.
The maximum number of results to return. This number should be between 1 and 50 inclusive.
ranking_options: optional object { hybrid_search, ranker, score_threshold } Ranking options for search.
Ranking options for search.
Computer object { type } A tool that controls a virtual computer. Learn more about the computer tool.
A tool that controls a virtual computer. Learn more about the computer tool.
ComputerUsePreview object { display_height, display_width, environment, type } A tool that controls a virtual computer. Learn more about the computer tool.
A tool that controls a virtual computer. Learn more about the computer tool.
WebSearch object { type, filters, search_context_size, user_location } Search the Internet for sources related to the prompt. Learn more about the
web search tool.
Search the Internet for sources related to the prompt. Learn more about the web search tool.
type: "web_search" or "web_search_2025_08_26"The type of the web search tool. One of web_search or web_search_2025_08_26.
The type of the web search tool. One of web_search or web_search_2025_08_26.
search_context_size: optional "low" or "medium" or "high"High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.
High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.
user_location: optional object { city, country, region, 2 more } The approximate location of the user.
The approximate location of the user.
The two-letter ISO country code of the user, e.g. US.
The IANA timezone of the user, e.g. America/Los_Angeles.
Mcp object { server_label, type, allowed_tools, 7 more } Give the model access to additional tools via remote Model Context Protocol
(MCP) servers. Learn more about MCP.
Give the model access to additional tools via remote Model Context Protocol (MCP) servers. Learn more about MCP.
allowed_tools: optional array of string or object { read_only, tool_names } List of allowed tool names or a filter object.
List of allowed tool names or a filter object.
McpToolFilter object { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
An OAuth access token that can be used with a remote MCP server, either with a custom MCP server URL or a service connector. Your application must handle the OAuth authorization flow and provide the token here.
connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 moreIdentifier for service connectors, like those available in ChatGPT. One of
server_url or connector_id must be provided. Learn more about service
connectors here.
Currently supported connector_id values are:
- Dropbox:
connector_dropbox
- Gmail:
connector_gmail
- Google Calendar:
connector_googlecalendar
- Google Drive:
connector_googledrive
- Microsoft Teams:
connector_microsoftteams
- Outlook Calendar:
connector_outlookcalendar
- Outlook Email:
connector_outlookemail
- SharePoint:
connector_sharepoint
Identifier for service connectors, like those available in ChatGPT. One of
server_url or connector_id must be provided. Learn more about service
connectors here.
Currently supported connector_id values are:
- Dropbox:
connector_dropbox - Gmail:
connector_gmail - Google Calendar:
connector_googlecalendar - Google Drive:
connector_googledrive - Microsoft Teams:
connector_microsoftteams - Outlook Calendar:
connector_outlookcalendar - Outlook Email:
connector_outlookemail - SharePoint:
connector_sharepoint
Optional HTTP headers to send to the MCP server. Use for authentication or other purposes.
require_approval: optional object { always, never } or "always" or "never"Specify which of the MCP server’s tools require approval.
Specify which of the MCP server’s tools require approval.
McpToolApprovalFilter object { always, never } Specify which of the MCP server’s tools require approval. Can be
always, never, or a filter object associated with tools
that require approval.
Specify which of the MCP server’s tools require approval. Can be
always, never, or a filter object associated with tools
that require approval.
always: optional object { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
never: optional object { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
CodeInterpreter object { container, type } A tool that runs Python code to help generate a response to a prompt.
A tool that runs Python code to help generate a response to a prompt.
container: string or object { type, file_ids, memory_limit, network_policy } The code interpreter container. Can be a container ID or an object that
specifies uploaded file IDs to make available to your code, along with an
optional memory_limit setting.
The code interpreter container. Can be a container ID or an object that
specifies uploaded file IDs to make available to your code, along with an
optional memory_limit setting.
CodeInterpreterToolAuto object { type, file_ids, memory_limit, network_policy } Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
An optional list of uploaded files to make available to your code.
memory_limit: optional "1g" or "4g" or "16g" or "64g"The memory limit for the code interpreter container.
The memory limit for the code interpreter container.
network_policy: optional ContainerNetworkPolicyDisabled { type } or ContainerNetworkPolicyAllowlist { allowed_domains, type, domain_secrets } Network access policy for the container.
Network access policy for the container.
ImageGeneration object { type, action, background, 9 more } A tool that generates images using the GPT image models.
A tool that generates images using the GPT image models.
action: optional "generate" or "edit" or "auto"Whether to generate a new image or edit an existing image. Default: auto.
Whether to generate a new image or edit an existing image. Default: auto.
background: optional "transparent" or "opaque" or "auto"Background type for the generated image. One of transparent,
opaque, or auto. Default: auto.
Background type for the generated image. One of transparent,
opaque, or auto. Default: auto.
input_fidelity: optional "high" or "low"Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for gpt-image-1 and gpt-image-1.5 and later models, unsupported for gpt-image-1-mini. Supports high and low. Defaults to low.
Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for gpt-image-1 and gpt-image-1.5 and later models, unsupported for gpt-image-1-mini. Supports high and low. Defaults to low.
input_image_mask: optional object { file_id, image_url } Optional mask for inpainting. Contains image_url
(string, optional) and file_id (string, optional).
Optional mask for inpainting. Contains image_url
(string, optional) and file_id (string, optional).
model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"The image generation model to use. Default: gpt-image-1.
The image generation model to use. Default: gpt-image-1.
Compression level for the output image. Default: 100.
output_format: optional "png" or "webp" or "jpeg"The output format of the generated image. One of png, webp, or
jpeg. Default: png.
The output format of the generated image. One of png, webp, or
jpeg. Default: png.
Number of partial images to generate in streaming mode, from 0 (default value) to 3.
LocalShell object { type } A tool that allows the model to execute shell commands in a local environment.
A tool that allows the model to execute shell commands in a local environment.
Shell object { type, environment } A tool that allows the model to execute shell commands.
A tool that allows the model to execute shell commands.
environment: optional ContainerAuto { type, file_ids, memory_limit, 2 more } or LocalEnvironment { type, skills } or ContainerReference { container_id, type }
ContainerAuto object { type, file_ids, memory_limit, 2 more }
An optional list of uploaded files to make available to your code.
network_policy: optional ContainerNetworkPolicyDisabled { type } or ContainerNetworkPolicyAllowlist { allowed_domains, type, domain_secrets } Network access policy for the container.
Network access policy for the container.
skills: optional array of SkillReference { skill_id, type, version } or InlineSkill { description, name, source, type } An optional list of skills referenced by id or inline data.
An optional list of skills referenced by id or inline data.
Custom object { name, type, defer_loading, 2 more } A custom tool that processes input using a specified format. Learn more about custom tools
A custom tool that processes input using a specified format. Learn more about custom tools
Namespace object { description, name, tools, type } Groups function/custom tools under a shared namespace.
Groups function/custom tools under a shared namespace.
tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more } The function/custom tools available inside this namespace.
The function/custom tools available inside this namespace.
Custom object { name, type, defer_loading, 2 more } A custom tool that processes input using a specified format. Learn more about custom tools
A custom tool that processes input using a specified format. Learn more about custom tools
ToolSearch object { type, description, execution, parameters } Hosted or BYOT tool search configuration for deferred tools.
Hosted or BYOT tool search configuration for deferred tools.
WebSearchPreview object { type, search_content_types, search_context_size, user_location } This tool searches the web for relevant results to use in a response. Learn more about the web search tool.
This tool searches the web for relevant results to use in a response. Learn more about the web search tool.
type: "web_search_preview" or "web_search_preview_2025_03_11"The type of the web search tool. One of web_search_preview or web_search_preview_2025_03_11.
The type of the web search tool. One of web_search_preview or web_search_preview_2025_03_11.
search_context_size: optional "low" or "medium" or "high"High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.
High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.
user_location: optional object { type, city, country, 2 more } The user’s location.
The user’s location.
The two-letter ISO country code of the user, e.g. US.
The IANA timezone of the user, e.g. America/Los_Angeles.
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
per_model_usage: array of object { cached_tokens, completion_tokens, invocation_count, 3 more } Usage statistics for each model during the evaluation run.
Usage statistics for each model during the evaluation run.
per_testing_criteria_results: array of object { failed, passed, testing_criteria } Results per testing criteria applied during the evaluation run.
Results per testing criteria applied during the evaluation run.
EvalsRunsOutput Items
Manage and run evals in the OpenAI platform.
Get eval run output items
Get an output item of an eval run
ModelsExpand Collapse
OutputItemListResponse object { id, created_at, datasource_item, 7 more } A schema representing an evaluation run output item.
A schema representing an evaluation run output item.
results: array of object { name, passed, score, 2 more } A list of grader results for this output item.
A list of grader results for this output item.
sample: object { error, finish_reason, input, 7 more } A sample containing the input and output of the evaluation run.
A sample containing the input and output of the evaluation run.
OutputItemRetrieveResponse object { id, created_at, datasource_item, 7 more } A schema representing an evaluation run output item.
A schema representing an evaluation run output item.
results: array of object { name, passed, score, 2 more } A list of grader results for this output item.
A list of grader results for this output item.
sample: object { error, finish_reason, input, 7 more } A sample containing the input and output of the evaluation run.
A sample containing the input and output of the evaluation run.