Create eval run

RunCreateResponse evals().runs().create(, )

POST/evals/{eval_id}/runs

Kicks off a new run for a given evaluation, specifying the data source, and what model configuration to use to test. The datasource will be validated against the schema specified in the config of the evaluation.

ParametersExpand Collapse

RunCreateParams params

Optional<String> evalId

DataSource dataSource

Details about the run's data source.

class CreateEvalJsonlRunDataSource:

A JsonlRunDataSource object with that specifies a JSONL file that matches the eval

Source source

Determines what populates the item namespace in the data source.

Accepts one of the following:

class FileContent:

List<Content> content

The content of the jsonl file.

Item item

Optional<Sample> sample

JsonValue; type

The type of jsonl source. Always file_content.

class FileId:

String id

The identifier of the file.

JsonValue; type

The type of jsonl source. Always file_id.

JsonValue; type

The type of data source. Always jsonl.

class CreateEvalCompletionsRunDataSource:

A CompletionsRunDataSource object describing a model sampling configuration.

Source source

Determines what populates the item namespace in this run's data source.

Accepts one of the following:

class FileContent:

List<Content> content

The content of the jsonl file.

Item item

Optional<Sample> sample

JsonValue; type

The type of jsonl source. Always file_content.

class FileId:

String id

The identifier of the file.

JsonValue; type

The type of jsonl source. Always file_id.

class StoredCompletions:

A StoredCompletionsRunDataSource configuration describing a set of filters

JsonValue; type

The type of source. Always stored_completions.

Optional<Long> createdAfter

An optional Unix timestamp to filter items created after this time.

Optional<Long> createdBefore

An optional Unix timestamp to filter items created before this time.

Optional<Long> limit

An optional maximum number of items to return.

Optional<Metadata> metadata

Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

Optional<String> model

An optional model to filter by (e.g., 'gpt-4o').

Type type

The type of run data source. Always completions.

Optional<InputMessages> inputMessages

Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.

Accepts one of the following:

class Template:

List<InnerTemplate> template

A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.

Accepts one of the following:

class EasyInputMessage:

A message input to the model with a role indicating instruction following hierarchy. Instructions given with the developer or system role take precedence over instructions given with the user role. Messages with the assistant role are presumed to have been generated by the model in previous interactions.

Content content

Text, image, or audio input to the model, used to generate a response. Can also contain previous assistant responses.

Accepts one of the following:

String

List<ResponseInputContent>

Accepts one of the following:

class ResponseInputText:

A text input to the model.

String text

The text input to the model.

JsonValue; type

The type of the input item. Always input_text.

class ResponseInputImage:

An image input to the model. Learn about image inputs.

Detail detail

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

Accepts one of the following:

LOW("low")

HIGH("high")

AUTO("auto")

JsonValue; type

The type of the input item. Always input_image.

Optional<String> fileId

The ID of the file to be sent to the model.

Optional<String> imageUrl

The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.

class ResponseInputFile:

A file input to the model.

JsonValue; type

The type of the input item. Always input_file.

Optional<String> fileData

The content of the file to be sent to the model.

Optional<String> fileId

The ID of the file to be sent to the model.

Optional<String> fileUrl

The URL of the file to be sent to the model.

Optional<String> filename

The name of the file to be sent to the model.

Role role

The role of the message input. One of user, assistant, system, or developer.

Accepts one of the following:

USER("user")

ASSISTANT("assistant")

SYSTEM("system")

DEVELOPER("developer")

Optional<Type> type

The type of the message input. Always message.

class EvalItem:

Content content

Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

Accepts one of the following:

String

class ResponseInputText:

A text input to the model.

String text

The text input to the model.

JsonValue; type

The type of the input item. Always input_text.

class OutputText:

A text output from the model.

String text

The text output from the model.

JsonValue; type

The type of the output text. Always output_text.

class InputImage:

An image input block used within EvalItem content arrays.

String imageUrl

The URL of the image input.

JsonValue; type

The type of the image input. Always input_image.

Optional<String> detail

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio:

An audio input to the model.

InputAudio inputAudio

String data

Base64-encoded audio data.

Format format

The format of the audio data. Currently supported formats are mp3 and wav.

Accepts one of the following:

MP3("mp3")

WAV("wav")

JsonValue; type

The type of the input item. Always input_audio.

List<EvalContentItem>

Accepts one of the following:

String

class ResponseInputText:

A text input to the model.

String text

The text input to the model.

JsonValue; type

The type of the input item. Always input_text.

OutputText

String text

The text output from the model.

JsonValue; type

The type of the output text. Always output_text.

InputImage

String imageUrl

The URL of the image input.

JsonValue; type

The type of the image input. Always input_image.

Optional<String> detail

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio:

An audio input to the model.

InputAudio inputAudio

String data

Base64-encoded audio data.

Format format

The format of the audio data. Currently supported formats are mp3 and wav.

Accepts one of the following:

MP3("mp3")

WAV("wav")

JsonValue; type

The type of the input item. Always input_audio.

Role role

The role of the message input. One of user, assistant, system, or developer.

Accepts one of the following:

USER("user")

ASSISTANT("assistant")

SYSTEM("system")

DEVELOPER("developer")

Optional<Type> type

The type of the message input. Always message.

JsonValue; type

The type of input messages. Always template.

class ItemReference:

String itemReference

A reference to a variable in the item namespace. Ie, "item.input_trajectory"

JsonValue; type

The type of input messages. Always item_reference.

Optional<String> model

The name of the model to use for generating completions (e.g. "o3-mini").

Optional<SamplingParams> samplingParams

Optional<Long> maxCompletionTokens

The maximum number of tokens in the generated output.

Optional<ReasoningEffort> reasoningEffort

Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response.

gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.
All models before gpt-5.1 default to medium reasoning effort, and do not support none.
The gpt-5-pro model defaults to (and only supports) high reasoning effort.
xhigh is supported for all models after gpt-5.1-codex-max.

Accepts one of the following:

NONE("none")

MINIMAL("minimal")

LOW("low")

MEDIUM("medium")

HIGH("high")

XHIGH("xhigh")

Optional<ResponseFormat> responseFormat

An object specifying the format that the model must output.

Setting to { "type": "json_schema", "json_schema": {...} } enables Structured Outputs which ensures the model will match your supplied JSON schema. Learn more in the Structured Outputs guide.

Setting to { "type": "json_object" } enables the older JSON mode, which ensures the message the model generates is valid JSON. Using json_schema is preferred for models that support it.

Accepts one of the following:

class ResponseFormatText:

Default response format. Used to generate text responses.

JsonValue; type

The type of response format being defined. Always text.

class ResponseFormatJsonSchema:

JSON Schema response format. Used to generate structured JSON responses. Learn more about Structured Outputs.

JsonSchema jsonSchema

Structured Outputs configuration options, including a JSON Schema.

String name

The name of the response format. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.

Optional<String> description

A description of what the response format is for, used by the model to determine how to respond in the format.

Optional<Schema> schema

The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.

Optional<Boolean> strict

Whether to enable strict schema adherence when generating the output. If set to true, the model will always follow the exact schema defined in the schema field. Only a subset of JSON Schema is supported when strict is true. To learn more, read the Structured Outputs guide.

JsonValue; type

The type of response format being defined. Always json_schema.

class ResponseFormatJsonObject:

JSON object response format. An older method of generating JSON responses. Using json_schema is recommended for models that support it. Note that the model will not generate JSON without a system or user message instructing it to do so.

JsonValue; type

The type of response format being defined. Always json_object.

Optional<Long> seed

A seed value to initialize the randomness, during sampling.

Optional<Double> temperature

A higher temperature increases randomness in the outputs.

Optional<List<ChatCompletionFunctionTool>> tools

A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.

FunctionDefinition function

String name

The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.

Optional<String> description

A description of what the function does, used by the model to choose when and how to call the function.

Optional<FunctionParameters> parameters

The parameters the functions accepts, described as a JSON Schema object. See the guide for examples, and the JSON Schema reference for documentation about the format.

Omitting parameters defines a function with an empty parameter list.

Optional<Boolean> strict

Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the parameters field. Only a subset of JSON Schema is supported when strict is true. Learn more about Structured Outputs in the function calling guide.

JsonValue; type

The type of the tool. Currently, only function is supported.

Optional<Double> topP

An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

class CreateEvalResponsesRunDataSource:

A ResponsesRunDataSource object describing a model sampling configuration.

Source source

Determines what populates the item namespace in this run's data source.

Accepts one of the following:

class FileContent:

List<Content> content

The content of the jsonl file.

Item item

Optional<Sample> sample

JsonValue; type

The type of jsonl source. Always file_content.

class FileId:

String id

The identifier of the file.

JsonValue; type

The type of jsonl source. Always file_id.

class Responses:

A EvalResponsesSource object describing a run data source configuration.

JsonValue; type

The type of run data source. Always responses.

Optional<Long> createdAfter

Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.

minimum0

Optional<Long> createdBefore

Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.

minimum0

Optional<String> instructionsSearch

Optional string to search the 'instructions' field. This is a query parameter used to select responses.

Optional<JsonValue> metadata

Metadata filter for the responses. This is a query parameter used to select responses.

Optional<String> model

The name of the model to find responses for. This is a query parameter used to select responses.

Optional<ReasoningEffort> reasoningEffort

gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.
All models before gpt-5.1 default to medium reasoning effort, and do not support none.
The gpt-5-pro model defaults to (and only supports) high reasoning effort.
xhigh is supported for all models after gpt-5.1-codex-max.

Accepts one of the following:

NONE("none")

MINIMAL("minimal")

LOW("low")

MEDIUM("medium")

HIGH("high")

XHIGH("xhigh")

Optional<Double> temperature

Sampling temperature. This is a query parameter used to select responses.

Optional<List<String>> tools

List of tool names. This is a query parameter used to select responses.

Optional<Double> topP

Nucleus sampling parameter. This is a query parameter used to select responses.

Optional<List<String>> users

List of user identifiers. This is a query parameter used to select responses.

Type type

The type of run data source. Always responses.

Optional<InputMessages> inputMessages

Accepts one of the following:

class Template:

List<InnerTemplate> template

A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.

Accepts one of the following:

class ChatMessage:

String content

The content of the message.

String role

The role of the message (e.g. "system", "assistant", "user").

class EvalItem:

Content content

Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

Accepts one of the following:

String

class ResponseInputText:

A text input to the model.

String text

The text input to the model.

JsonValue; type

The type of the input item. Always input_text.

class OutputText:

A text output from the model.

String text

The text output from the model.

JsonValue; type

The type of the output text. Always output_text.

class InputImage:

An image input block used within EvalItem content arrays.

String imageUrl

The URL of the image input.

JsonValue; type

The type of the image input. Always input_image.

Optional<String> detail

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio:

An audio input to the model.

InputAudio inputAudio

String data

Base64-encoded audio data.

Format format

The format of the audio data. Currently supported formats are mp3 and wav.

Accepts one of the following:

MP3("mp3")

WAV("wav")

JsonValue; type

The type of the input item. Always input_audio.

List<EvalContentItem>

Accepts one of the following:

String

class ResponseInputText:

A text input to the model.

String text

The text input to the model.

JsonValue; type

The type of the input item. Always input_text.

OutputText

String text

The text output from the model.

JsonValue; type

The type of the output text. Always output_text.

InputImage

String imageUrl

The URL of the image input.

JsonValue; type

The type of the image input. Always input_image.

Optional<String> detail

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio:

An audio input to the model.

InputAudio inputAudio

String data

Base64-encoded audio data.

Format format

The format of the audio data. Currently supported formats are mp3 and wav.

Accepts one of the following:

MP3("mp3")

WAV("wav")

JsonValue; type

The type of the input item. Always input_audio.

Role role

The role of the message input. One of user, assistant, system, or developer.

Accepts one of the following:

USER("user")

ASSISTANT("assistant")

SYSTEM("system")

DEVELOPER("developer")

Optional<Type> type

The type of the message input. Always message.

JsonValue; type

The type of input messages. Always template.

class ItemReference:

String itemReference

A reference to a variable in the item namespace. Ie, "item.name"

JsonValue; type

The type of input messages. Always item_reference.

Optional<String> model

The name of the model to use for generating completions (e.g. "o3-mini").

Optional<SamplingParams> samplingParams

Optional<Long> maxCompletionTokens

The maximum number of tokens in the generated output.

Optional<ReasoningEffort> reasoningEffort

gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.
All models before gpt-5.1 default to medium reasoning effort, and do not support none.
The gpt-5-pro model defaults to (and only supports) high reasoning effort.
xhigh is supported for all models after gpt-5.1-codex-max.

Accepts one of the following:

NONE("none")

MINIMAL("minimal")

LOW("low")

MEDIUM("medium")

HIGH("high")

XHIGH("xhigh")

Optional<Long> seed

A seed value to initialize the randomness, during sampling.

Optional<Double> temperature

A higher temperature increases randomness in the outputs.

Optional<Text> text

Configuration options for a text response from the model. Can be plain text or structured JSON data. Learn more:

Optional<ResponseFormatTextConfig> format

An object specifying the format that the model must output.

Configuring { "type": "json_schema" } enables Structured Outputs, which ensures the model will match your supplied JSON schema. Learn more in the Structured Outputs guide.

The default format is { "type": "text" } with no additional options.

Not recommended for gpt-4o and newer models:

Setting to { "type": "json_object" } enables the older JSON mode, which ensures the message the model generates is valid JSON. Using json_schema is preferred for models that support it.

Accepts one of the following:

class ResponseFormatText:

Default response format. Used to generate text responses.

JsonValue; type

The type of response format being defined. Always text.

class ResponseFormatTextJsonSchemaConfig:

JSON Schema response format. Used to generate structured JSON responses. Learn more about Structured Outputs.

String name

The name of the response format. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.

Schema schema

The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.

JsonValue; type

The type of response format being defined. Always json_schema.

Optional<String> description

A description of what the response format is for, used by the model to determine how to respond in the format.

Optional<Boolean> strict

class ResponseFormatJsonObject:

JsonValue; type

The type of response format being defined. Always json_object.

Optional<List<Tool>> tools

An array of tools the model may call while generating a response. You can specify which tool to use by setting the tool_choice parameter.

The two categories of tools you can provide the model are:

Built-in tools: Tools that are provided by OpenAI that extend the model's capabilities, like web search or file search. Learn more about built-in tools.
Function calls (custom tools): Functions that are defined by you, enabling the model to call your own code. Learn more about function calling.

Accepts one of the following:

class FunctionTool:

Defines a function in your own code the model can choose to call. Learn more about function calling.

String name

The name of the function to call.

Optional<Parameters> parameters

A JSON schema object describing the parameters of the function.

Optional<Boolean> strict

Whether to enforce strict parameter validation. Default true.

JsonValue; type

The type of the function tool. Always function.

Optional<String> description

A description of the function. Used by the model to determine whether or not to call the function.

class FileSearchTool:

A tool that searches for relevant content from uploaded files. Learn more about the file search tool.

JsonValue; type

The type of the file search tool. Always file_search.

List<String> vectorStoreIds

The IDs of the vector stores to search.

Optional<Filters> filters

A filter to apply.

Accepts one of the following:

class ComparisonFilter:

A filter used to compare a specified attribute key to a given value using a defined comparison operation.

String key

The key to compare against the value.

Type type

Specifies the comparison operator: eq, ne, gt, gte, lt, lte, in, nin.

eq: equals
ne: not equal
gt: greater than
gte: greater than or equal
lt: less than
lte: less than or equal
in: in
nin: not in

Accepts one of the following:

EQ("eq")

NE("ne")

GT("gt")

GTE("gte")

LT("lt")

LTE("lte")

Value value

The value to compare against the attribute key; supports string, number, or boolean types.

Accepts one of the following:

String

double

boolean

List<ComparisonFilterValueItem>

Accepts one of the following:

String

double

class CompoundFilter:

Combine multiple filters using and or or.

List<Filter> filters

Array of filters to combine. Items can be ComparisonFilter or CompoundFilter.

Accepts one of the following:

class ComparisonFilter:

A filter used to compare a specified attribute key to a given value using a defined comparison operation.

String key

The key to compare against the value.

Type type

Specifies the comparison operator: eq, ne, gt, gte, lt, lte, in, nin.

eq: equals
ne: not equal
gt: greater than
gte: greater than or equal
lt: less than
lte: less than or equal
in: in
nin: not in

Accepts one of the following:

EQ("eq")

NE("ne")

GT("gt")

GTE("gte")

LT("lt")

LTE("lte")

Value value

The value to compare against the attribute key; supports string, number, or boolean types.

Accepts one of the following:

String

double

boolean

List<ComparisonFilterValueItem>

Accepts one of the following:

String

double

JsonValue

Type type

Type of operation: and or or.

Accepts one of the following:

AND("and")

OR("or")

Optional<Long> maxNumResults

The maximum number of results to return. This number should be between 1 and 50 inclusive.

Optional<RankingOptions> rankingOptions

Ranking options for search.

Optional<HybridSearch> hybridSearch

Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled.

double embeddingWeight

The weight of the embedding in the reciprocal ranking fusion.

double textWeight

The weight of the text in the reciprocal ranking fusion.

Optional<Ranker> ranker

The ranker to use for the file search.

Accepts one of the following:

AUTO("auto")

DEFAULT_2024_11_15("default-2024-11-15")

Optional<Double> scoreThreshold

The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results.

class ComputerTool:

A tool that controls a virtual computer. Learn more about the computer tool.

long displayHeight

The height of the computer display.

long displayWidth

The width of the computer display.

Environment environment

The type of computer environment to control.

Accepts one of the following:

WINDOWS("windows")

MAC("mac")

LINUX("linux")

UBUNTU("ubuntu")

BROWSER("browser")

JsonValue; type

The type of the computer use tool. Always computer_use_preview.

class WebSearchTool:

Search the Internet for sources related to the prompt. Learn more about the web search tool.

Type type

The type of the web search tool. One of web_search or web_search_2025_08_26.

Accepts one of the following:

WEB_SEARCH("web_search")

WEB_SEARCH_2025_08_26("web_search_2025_08_26")

Optional<Filters> filters

Filters for the search.

Optional<List<String>> allowedDomains

Allowed domains for the search. If not provided, all domains are allowed. Subdomains of the provided domains are allowed as well.

Example: ["pubmed.ncbi.nlm.nih.gov"]

Optional<SearchContextSize> searchContextSize

High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.

Accepts one of the following:

LOW("low")

MEDIUM("medium")

HIGH("high")

Optional<UserLocation> userLocation

The approximate location of the user.

Optional<String> city

Free text input for the city of the user, e.g. San Francisco.

Optional<String> country

The two-letter ISO country code of the user, e.g. US.

Optional<String> region

Free text input for the region of the user, e.g. California.

Optional<String> timezone

The IANA timezone of the user, e.g. America/Los_Angeles.

Optional<Type> type

The type of location approximation. Always approximate.

Mcp

String serverLabel

A label for this MCP server, used to identify it in tool calls.

JsonValue; type

The type of the MCP tool. Always mcp.

Optional<AllowedTools> allowedTools

List of allowed tool names or a filter object.

Accepts one of the following:

List<String>

class McpToolFilter:

A filter object to specify which tools are allowed.

Optional<Boolean> readOnly

Indicates whether or not a tool modifies data or is read-only. If an MCP server is annotated with readOnlyHint, it will match this filter.

Optional<List<String>> toolNames

List of allowed tool names.

Optional<String> authorization

An OAuth access token that can be used with a remote MCP server, either with a custom MCP server URL or a service connector. Your application must handle the OAuth authorization flow and provide the token here.

Optional<ConnectorId> connectorId

Identifier for service connectors, like those available in ChatGPT. One of server_url or connector_id must be provided. Learn more about service connectors here.

Currently supported connector_id values are:

Dropbox: connector_dropbox
Gmail: connector_gmail
Google Calendar: connector_googlecalendar
Google Drive: connector_googledrive
Microsoft Teams: connector_microsoftteams
Outlook Calendar: connector_outlookcalendar
Outlook Email: connector_outlookemail
SharePoint: connector_sharepoint

Accepts one of the following:

CONNECTOR_DROPBOX("connector_dropbox")

CONNECTOR_GMAIL("connector_gmail")

CONNECTOR_GOOGLECALENDAR("connector_googlecalendar")

CONNECTOR_GOOGLEDRIVE("connector_googledrive")

CONNECTOR_MICROSOFTTEAMS("connector_microsoftteams")

CONNECTOR_OUTLOOKCALENDAR("connector_outlookcalendar")

CONNECTOR_OUTLOOKEMAIL("connector_outlookemail")

CONNECTOR_SHAREPOINT("connector_sharepoint")

Optional<Headers> headers

Optional HTTP headers to send to the MCP server. Use for authentication or other purposes.

Optional<RequireApproval> requireApproval

Specify which of the MCP server's tools require approval.

Accepts one of the following:

class McpToolApprovalFilter:

Specify which of the MCP server's tools require approval. Can be always, never, or a filter object associated with tools that require approval.

Optional<Always> always

A filter object to specify which tools are allowed.

Optional<Boolean> readOnly

Indicates whether or not a tool modifies data or is read-only. If an MCP server is annotated with readOnlyHint, it will match this filter.

Optional<List<String>> toolNames

List of allowed tool names.

Optional<Never> never

A filter object to specify which tools are allowed.

Optional<Boolean> readOnly

Indicates whether or not a tool modifies data or is read-only. If an MCP server is annotated with readOnlyHint, it will match this filter.

Optional<List<String>> toolNames

List of allowed tool names.

enum McpToolApprovalSetting:

Specify a single approval policy for all tools. One of always or never. When set to always, all tools will require approval. When set to never, all tools will not require approval.

ALWAYS("always")

NEVER("never")

Optional<String> serverDescription

Optional description of the MCP server, used to provide more context.

Optional<String> serverUrl

The URL for the MCP server. One of server_url or connector_id must be provided.

CodeInterpreter

Container container

The code interpreter container. Can be a container ID or an object that specifies uploaded file IDs to make available to your code, along with an optional memory_limit setting.

Accepts one of the following:

String

class CodeInterpreterToolAuto:

Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.

JsonValue; type

Always auto.

Optional<List<String>> fileIds

An optional list of uploaded files to make available to your code.

Optional<MemoryLimit> memoryLimit

The memory limit for the code interpreter container.

Accepts one of the following:

_1G("1g")

_4G("4g")

_16G("16g")

_64G("64g")

JsonValue; type

The type of the code interpreter tool. Always code_interpreter.

ImageGeneration

JsonValue; type

The type of the image generation tool. Always image_generation.

Optional<Action> action

Whether to generate a new image or edit an existing image. Default: auto.

Accepts one of the following:

GENERATE("generate")

EDIT("edit")

AUTO("auto")

Optional<Background> background

Background type for the generated image. One of transparent, opaque, or auto. Default: auto.

Accepts one of the following:

TRANSPARENT("transparent")

OPAQUE("opaque")

AUTO("auto")

Optional<InputFidelity> inputFidelity

Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for gpt-image-1 and gpt-image-1.5 and later models, unsupported for gpt-image-1-mini. Supports high and low. Defaults to low.

Accepts one of the following:

HIGH("high")

LOW("low")

Optional<InputImageMask> inputImageMask

Optional mask for inpainting. Contains image_url (string, optional) and file_id (string, optional).

Optional<String> fileId

File ID for the mask image.

Optional<String> imageUrl

Base64-encoded mask image.

Optional<Model> model

The image generation model to use. Default: gpt-image-1.

Accepts one of the following:

GPT_IMAGE_1("gpt-image-1")

GPT_IMAGE_1_MINI("gpt-image-1-mini")

GPT_IMAGE_1_5("gpt-image-1.5")

Optional<Moderation> moderation

Moderation level for the generated image. Default: auto.

Accepts one of the following:

AUTO("auto")

LOW("low")

Optional<Long> outputCompression

Compression level for the output image. Default: 100.

minimum0

maximum100

Optional<OutputFormat> outputFormat

The output format of the generated image. One of png, webp, or jpeg. Default: png.

Accepts one of the following:

PNG("png")

WEBP("webp")

JPEG("jpeg")

Optional<Long> partialImages

Number of partial images to generate in streaming mode, from 0 (default value) to 3.

minimum0

maximum3

Optional<Quality> quality

The quality of the generated image. One of low, medium, high, or auto. Default: auto.

Accepts one of the following:

LOW("low")

MEDIUM("medium")

HIGH("high")

AUTO("auto")

Optional<Size> size

The size of the generated image. One of 1024x1024, 1024x1536, 1536x1024, or auto. Default: auto.

Accepts one of the following:

_1024X1024("1024x1024")

_1024X1536("1024x1536")

_1536X1024("1536x1024")

AUTO("auto")

JsonValue;

JsonValue; type

The type of the local shell tool. Always local_shell.

class FunctionShellTool:

A tool that allows the model to execute shell commands.

JsonValue; type

The type of the shell tool. Always shell.

class CustomTool:

A custom tool that processes input using a specified format. Learn more about custom tools

String name

The name of the custom tool, used to identify it in tool calls.

JsonValue; type

The type of the custom tool. Always custom.

Optional<String> description

Optional description of the custom tool, used to provide more context.

Optional<CustomToolInputFormat> format

The input format for the custom tool. Default is unconstrained text.

Accepts one of the following:

JsonValue;

JsonValue; type

Unconstrained text format. Always text.

Grammar

String definition

The grammar definition.

Syntax syntax

The syntax of the grammar definition. One of lark or regex.

Accepts one of the following:

LARK("lark")

REGEX("regex")

JsonValue; type

Grammar format. Always grammar.

class WebSearchPreviewTool:

This tool searches the web for relevant results to use in a response. Learn more about the web search tool.

Type type

The type of the web search tool. One of web_search_preview or web_search_preview_2025_03_11.

Accepts one of the following:

WEB_SEARCH_PREVIEW("web_search_preview")

WEB_SEARCH_PREVIEW_2025_03_11("web_search_preview_2025_03_11")

Optional<SearchContextSize> searchContextSize

High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.

Accepts one of the following:

LOW("low")

MEDIUM("medium")

HIGH("high")

Optional<UserLocation> userLocation

The user's location.

JsonValue; type

The type of location approximation. Always approximate.

Optional<String> city

Free text input for the city of the user, e.g. San Francisco.

Optional<String> country

The two-letter ISO country code of the user, e.g. US.

Optional<String> region

Free text input for the region of the user, e.g. California.

Optional<String> timezone

The IANA timezone of the user, e.g. America/Los_Angeles.

class ApplyPatchTool:

Allows the assistant to create, delete, or update files using unified diffs.

JsonValue; type

The type of the tool. Always apply_patch.

Optional<Double> topP

An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

Optional<Metadata> metadata

Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

Optional<String> name

The name of the run.

ReturnsExpand Collapse

class RunCreateResponse:

A schema representing an evaluation run.

String id

Unique identifier for the evaluation run.

long createdAt

Unix timestamp (in seconds) when the evaluation run was created.

DataSource dataSource

Information about the run's data source.

Accepts one of the following:

class CreateEvalJsonlRunDataSource:

A JsonlRunDataSource object with that specifies a JSONL file that matches the eval

Source source

Determines what populates the item namespace in the data source.

Accepts one of the following:

class FileContent:

List<Content> content

The content of the jsonl file.

Item item

Optional<Sample> sample

JsonValue; type

The type of jsonl source. Always file_content.

class FileId:

String id

The identifier of the file.

JsonValue; type

The type of jsonl source. Always file_id.

JsonValue; type

The type of data source. Always jsonl.

class CreateEvalCompletionsRunDataSource:

A CompletionsRunDataSource object describing a model sampling configuration.

Source source

Determines what populates the item namespace in this run's data source.

Accepts one of the following:

class FileContent:

List<Content> content

The content of the jsonl file.

Item item

Optional<Sample> sample

JsonValue; type

The type of jsonl source. Always file_content.

class FileId:

String id

The identifier of the file.

JsonValue; type

The type of jsonl source. Always file_id.

class StoredCompletions:

A StoredCompletionsRunDataSource configuration describing a set of filters

JsonValue; type

The type of source. Always stored_completions.

Optional<Long> createdAfter

An optional Unix timestamp to filter items created after this time.

Optional<Long> createdBefore

An optional Unix timestamp to filter items created before this time.

Optional<Long> limit

An optional maximum number of items to return.

Optional<Metadata> metadata

Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

Optional<String> model

An optional model to filter by (e.g., 'gpt-4o').

Type type

The type of run data source. Always completions.

Optional<InputMessages> inputMessages

Accepts one of the following:

class Template:

List<InnerTemplate> template

A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.

Accepts one of the following:

class EasyInputMessage:

Content content

Text, image, or audio input to the model, used to generate a response. Can also contain previous assistant responses.

Accepts one of the following:

String

List<ResponseInputContent>

Accepts one of the following:

class ResponseInputText:

A text input to the model.

String text

The text input to the model.

JsonValue; type

The type of the input item. Always input_text.

class ResponseInputImage:

An image input to the model. Learn about image inputs.

Detail detail

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

Accepts one of the following:

LOW("low")

HIGH("high")

AUTO("auto")

JsonValue; type

The type of the input item. Always input_image.

Optional<String> fileId

The ID of the file to be sent to the model.

Optional<String> imageUrl

The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.

class ResponseInputFile:

A file input to the model.

JsonValue; type

The type of the input item. Always input_file.

Optional<String> fileData

The content of the file to be sent to the model.

Optional<String> fileId

The ID of the file to be sent to the model.

Optional<String> fileUrl

The URL of the file to be sent to the model.

Optional<String> filename

The name of the file to be sent to the model.

Role role

The role of the message input. One of user, assistant, system, or developer.

Accepts one of the following:

USER("user")

ASSISTANT("assistant")

SYSTEM("system")

DEVELOPER("developer")

Optional<Type> type

The type of the message input. Always message.

class EvalItem:

Content content

Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

Accepts one of the following:

String

class ResponseInputText:

A text input to the model.

String text

The text input to the model.

JsonValue; type

The type of the input item. Always input_text.

class OutputText:

A text output from the model.

String text

The text output from the model.

JsonValue; type

The type of the output text. Always output_text.

class InputImage:

An image input block used within EvalItem content arrays.

String imageUrl

The URL of the image input.

JsonValue; type

The type of the image input. Always input_image.

Optional<String> detail

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio:

An audio input to the model.

InputAudio inputAudio

String data

Base64-encoded audio data.

Format format

The format of the audio data. Currently supported formats are mp3 and wav.

Accepts one of the following:

MP3("mp3")

WAV("wav")

JsonValue; type

The type of the input item. Always input_audio.

List<EvalContentItem>

Accepts one of the following:

String

class ResponseInputText:

A text input to the model.

String text

The text input to the model.

JsonValue; type

The type of the input item. Always input_text.

OutputText

String text

The text output from the model.

JsonValue; type

The type of the output text. Always output_text.

InputImage

String imageUrl

The URL of the image input.

JsonValue; type

The type of the image input. Always input_image.

Optional<String> detail

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio:

An audio input to the model.

InputAudio inputAudio

String data

Base64-encoded audio data.

Format format

The format of the audio data. Currently supported formats are mp3 and wav.

Accepts one of the following:

MP3("mp3")

WAV("wav")

JsonValue; type

The type of the input item. Always input_audio.

Role role

The role of the message input. One of user, assistant, system, or developer.

Accepts one of the following:

USER("user")

ASSISTANT("assistant")

SYSTEM("system")

DEVELOPER("developer")

Optional<Type> type

The type of the message input. Always message.

JsonValue; type

The type of input messages. Always template.

class ItemReference:

String itemReference

A reference to a variable in the item namespace. Ie, "item.input_trajectory"

JsonValue; type

The type of input messages. Always item_reference.

Optional<String> model

The name of the model to use for generating completions (e.g. "o3-mini").

Optional<SamplingParams> samplingParams

Optional<Long> maxCompletionTokens

The maximum number of tokens in the generated output.

Optional<ReasoningEffort> reasoningEffort

gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.
All models before gpt-5.1 default to medium reasoning effort, and do not support none.
The gpt-5-pro model defaults to (and only supports) high reasoning effort.
xhigh is supported for all models after gpt-5.1-codex-max.

Accepts one of the following:

NONE("none")

MINIMAL("minimal")

LOW("low")

MEDIUM("medium")

HIGH("high")

XHIGH("xhigh")

Optional<ResponseFormat> responseFormat

An object specifying the format that the model must output.

Setting to { "type": "json_schema", "json_schema": {...} } enables Structured Outputs which ensures the model will match your supplied JSON schema. Learn more in the Structured Outputs guide.

Setting to { "type": "json_object" } enables the older JSON mode, which ensures the message the model generates is valid JSON. Using json_schema is preferred for models that support it.

Accepts one of the following:

class ResponseFormatText:

Default response format. Used to generate text responses.

JsonValue; type

The type of response format being defined. Always text.

class ResponseFormatJsonSchema:

JSON Schema response format. Used to generate structured JSON responses. Learn more about Structured Outputs.

JsonSchema jsonSchema

Structured Outputs configuration options, including a JSON Schema.

String name

The name of the response format. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.

Optional<String> description

A description of what the response format is for, used by the model to determine how to respond in the format.

Optional<Schema> schema

The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.

Optional<Boolean> strict

JsonValue; type

The type of response format being defined. Always json_schema.

class ResponseFormatJsonObject:

JsonValue; type

The type of response format being defined. Always json_object.

Optional<Long> seed

A seed value to initialize the randomness, during sampling.

Optional<Double> temperature

A higher temperature increases randomness in the outputs.

Optional<List<ChatCompletionFunctionTool>> tools

FunctionDefinition function

String name

The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.

Optional<String> description

A description of what the function does, used by the model to choose when and how to call the function.

Optional<FunctionParameters> parameters

The parameters the functions accepts, described as a JSON Schema object. See the guide for examples, and the JSON Schema reference for documentation about the format.

Omitting parameters defines a function with an empty parameter list.

Optional<Boolean> strict

JsonValue; type

The type of the tool. Currently, only function is supported.

Optional<Double> topP

An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

class Responses:

A ResponsesRunDataSource object describing a model sampling configuration.

Source source

Determines what populates the item namespace in this run's data source.

Accepts one of the following:

class FileContent:

List<Content> content

The content of the jsonl file.

Item item

Optional<Sample> sample

JsonValue; type

The type of jsonl source. Always file_content.

class FileId:

String id

The identifier of the file.

JsonValue; type

The type of jsonl source. Always file_id.

class InnerResponses:

A EvalResponsesSource object describing a run data source configuration.

JsonValue; type

The type of run data source. Always responses.

Optional<Long> createdAfter

Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.

minimum0

Optional<Long> createdBefore

Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.

minimum0

Optional<String> instructionsSearch

Optional string to search the 'instructions' field. This is a query parameter used to select responses.

Optional<JsonValue> metadata

Metadata filter for the responses. This is a query parameter used to select responses.

Optional<String> model

The name of the model to find responses for. This is a query parameter used to select responses.

Optional<ReasoningEffort> reasoningEffort

gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.
All models before gpt-5.1 default to medium reasoning effort, and do not support none.
The gpt-5-pro model defaults to (and only supports) high reasoning effort.
xhigh is supported for all models after gpt-5.1-codex-max.

Accepts one of the following:

NONE("none")

MINIMAL("minimal")

LOW("low")

MEDIUM("medium")

HIGH("high")

XHIGH("xhigh")

Optional<Double> temperature

Sampling temperature. This is a query parameter used to select responses.

Optional<List<String>> tools

List of tool names. This is a query parameter used to select responses.

Optional<Double> topP

Nucleus sampling parameter. This is a query parameter used to select responses.

Optional<List<String>> users

List of user identifiers. This is a query parameter used to select responses.

JsonValue; type

The type of run data source. Always responses.

Optional<InputMessages> inputMessages

Accepts one of the following:

class Template:

List<InnerTemplate> template

A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.

Accepts one of the following:

class ChatMessage:

String content

The content of the message.

String role

The role of the message (e.g. "system", "assistant", "user").

class EvalItem:

Content content

Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

Accepts one of the following:

String

class ResponseInputText:

A text input to the model.

String text

The text input to the model.

JsonValue; type

The type of the input item. Always input_text.

class OutputText:

A text output from the model.

String text

The text output from the model.

JsonValue; type

The type of the output text. Always output_text.

class InputImage:

An image input block used within EvalItem content arrays.

String imageUrl

The URL of the image input.

JsonValue; type

The type of the image input. Always input_image.

Optional<String> detail

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio:

An audio input to the model.

InputAudio inputAudio

String data

Base64-encoded audio data.

Format format

The format of the audio data. Currently supported formats are mp3 and wav.

Accepts one of the following:

MP3("mp3")

WAV("wav")

JsonValue; type

The type of the input item. Always input_audio.

List<EvalContentItem>

Accepts one of the following:

String

class ResponseInputText:

A text input to the model.

String text

The text input to the model.

JsonValue; type

The type of the input item. Always input_text.

OutputText

String text

The text output from the model.

JsonValue; type

The type of the output text. Always output_text.

InputImage

String imageUrl

The URL of the image input.

JsonValue; type

The type of the image input. Always input_image.

Optional<String> detail

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio:

An audio input to the model.

InputAudio inputAudio

String data

Base64-encoded audio data.

Format format

The format of the audio data. Currently supported formats are mp3 and wav.

Accepts one of the following:

MP3("mp3")

WAV("wav")

JsonValue; type

The type of the input item. Always input_audio.

Role role

The role of the message input. One of user, assistant, system, or developer.

Accepts one of the following:

USER("user")

ASSISTANT("assistant")

SYSTEM("system")

DEVELOPER("developer")

Optional<Type> type

The type of the message input. Always message.

JsonValue; type

The type of input messages. Always template.

class ItemReference:

String itemReference

A reference to a variable in the item namespace. Ie, "item.name"

JsonValue; type

The type of input messages. Always item_reference.

Optional<String> model

The name of the model to use for generating completions (e.g. "o3-mini").

Optional<SamplingParams> samplingParams

Optional<Long> maxCompletionTokens

The maximum number of tokens in the generated output.

Optional<ReasoningEffort> reasoningEffort

gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.
All models before gpt-5.1 default to medium reasoning effort, and do not support none.
The gpt-5-pro model defaults to (and only supports) high reasoning effort.
xhigh is supported for all models after gpt-5.1-codex-max.

Accepts one of the following:

NONE("none")

MINIMAL("minimal")

LOW("low")

MEDIUM("medium")

HIGH("high")

XHIGH("xhigh")

Optional<Long> seed

A seed value to initialize the randomness, during sampling.

Optional<Double> temperature

A higher temperature increases randomness in the outputs.

Optional<Text> text

Configuration options for a text response from the model. Can be plain text or structured JSON data. Learn more:

Optional<ResponseFormatTextConfig> format

An object specifying the format that the model must output.

Configuring { "type": "json_schema" } enables Structured Outputs, which ensures the model will match your supplied JSON schema. Learn more in the Structured Outputs guide.

The default format is { "type": "text" } with no additional options.

Not recommended for gpt-4o and newer models:

Setting to { "type": "json_object" } enables the older JSON mode, which ensures the message the model generates is valid JSON. Using json_schema is preferred for models that support it.

Accepts one of the following:

class ResponseFormatText:

Default response format. Used to generate text responses.

JsonValue; type

The type of response format being defined. Always text.

class ResponseFormatTextJsonSchemaConfig:

JSON Schema response format. Used to generate structured JSON responses. Learn more about Structured Outputs.

String name

The name of the response format. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.

Schema schema

The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.

JsonValue; type

The type of response format being defined. Always json_schema.

Optional<String> description

A description of what the response format is for, used by the model to determine how to respond in the format.

Optional<Boolean> strict

class ResponseFormatJsonObject:

JsonValue; type

The type of response format being defined. Always json_object.

Optional<List<Tool>> tools

An array of tools the model may call while generating a response. You can specify which tool to use by setting the tool_choice parameter.

The two categories of tools you can provide the model are:

Built-in tools: Tools that are provided by OpenAI that extend the model's capabilities, like web search or file search. Learn more about built-in tools.
Function calls (custom tools): Functions that are defined by you, enabling the model to call your own code. Learn more about function calling.

Accepts one of the following:

class FunctionTool:

Defines a function in your own code the model can choose to call. Learn more about function calling.

String name

The name of the function to call.

Optional<Parameters> parameters

A JSON schema object describing the parameters of the function.

Optional<Boolean> strict

Whether to enforce strict parameter validation. Default true.

JsonValue; type

The type of the function tool. Always function.

Optional<String> description

A description of the function. Used by the model to determine whether or not to call the function.

class FileSearchTool:

A tool that searches for relevant content from uploaded files. Learn more about the file search tool.

JsonValue; type

The type of the file search tool. Always file_search.

List<String> vectorStoreIds

The IDs of the vector stores to search.

Optional<Filters> filters

A filter to apply.

Accepts one of the following:

class ComparisonFilter:

A filter used to compare a specified attribute key to a given value using a defined comparison operation.

String key

The key to compare against the value.

Type type

Specifies the comparison operator: eq, ne, gt, gte, lt, lte, in, nin.

eq: equals
ne: not equal
gt: greater than
gte: greater than or equal
lt: less than
lte: less than or equal
in: in
nin: not in

Accepts one of the following:

EQ("eq")

NE("ne")

GT("gt")

GTE("gte")

LT("lt")

LTE("lte")

Value value

The value to compare against the attribute key; supports string, number, or boolean types.

Accepts one of the following:

String

double

boolean

List<ComparisonFilterValueItem>

Accepts one of the following:

String

double

class CompoundFilter:

Combine multiple filters using and or or.

List<Filter> filters

Array of filters to combine. Items can be ComparisonFilter or CompoundFilter.

Accepts one of the following:

class ComparisonFilter:

A filter used to compare a specified attribute key to a given value using a defined comparison operation.

String key

The key to compare against the value.

Type type

Specifies the comparison operator: eq, ne, gt, gte, lt, lte, in, nin.

eq: equals
ne: not equal
gt: greater than
gte: greater than or equal
lt: less than
lte: less than or equal
in: in
nin: not in

Accepts one of the following:

EQ("eq")

NE("ne")

GT("gt")

GTE("gte")

LT("lt")

LTE("lte")

Value value

The value to compare against the attribute key; supports string, number, or boolean types.

Accepts one of the following:

String

double

boolean

List<ComparisonFilterValueItem>

Accepts one of the following:

String

double

JsonValue

Type type

Type of operation: and or or.

Accepts one of the following:

AND("and")

OR("or")

Optional<Long> maxNumResults

The maximum number of results to return. This number should be between 1 and 50 inclusive.

Optional<RankingOptions> rankingOptions

Ranking options for search.

Optional<HybridSearch> hybridSearch

Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled.

double embeddingWeight

The weight of the embedding in the reciprocal ranking fusion.

double textWeight

The weight of the text in the reciprocal ranking fusion.

Optional<Ranker> ranker

The ranker to use for the file search.

Accepts one of the following:

AUTO("auto")

DEFAULT_2024_11_15("default-2024-11-15")

Optional<Double> scoreThreshold

The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results.

class ComputerTool:

A tool that controls a virtual computer. Learn more about the computer tool.

long displayHeight

The height of the computer display.

long displayWidth

The width of the computer display.

Environment environment

The type of computer environment to control.

Accepts one of the following:

WINDOWS("windows")

MAC("mac")

LINUX("linux")

UBUNTU("ubuntu")

BROWSER("browser")

JsonValue; type

The type of the computer use tool. Always computer_use_preview.

class WebSearchTool:

Search the Internet for sources related to the prompt. Learn more about the web search tool.

Type type

The type of the web search tool. One of web_search or web_search_2025_08_26.

Accepts one of the following:

WEB_SEARCH("web_search")

WEB_SEARCH_2025_08_26("web_search_2025_08_26")

Optional<Filters> filters

Filters for the search.

Optional<List<String>> allowedDomains

Allowed domains for the search. If not provided, all domains are allowed. Subdomains of the provided domains are allowed as well.

Example: ["pubmed.ncbi.nlm.nih.gov"]

Optional<SearchContextSize> searchContextSize

High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.

Accepts one of the following:

LOW("low")

MEDIUM("medium")

HIGH("high")

Optional<UserLocation> userLocation

The approximate location of the user.

Optional<String> city

Free text input for the city of the user, e.g. San Francisco.

Optional<String> country

The two-letter ISO country code of the user, e.g. US.

Optional<String> region

Free text input for the region of the user, e.g. California.

Optional<String> timezone

The IANA timezone of the user, e.g. America/Los_Angeles.

Optional<Type> type

The type of location approximation. Always approximate.

Mcp

String serverLabel

A label for this MCP server, used to identify it in tool calls.

JsonValue; type

The type of the MCP tool. Always mcp.

Optional<AllowedTools> allowedTools

List of allowed tool names or a filter object.

Accepts one of the following:

List<String>

class McpToolFilter:

A filter object to specify which tools are allowed.

Optional<Boolean> readOnly

Indicates whether or not a tool modifies data or is read-only. If an MCP server is annotated with readOnlyHint, it will match this filter.

Optional<List<String>> toolNames

List of allowed tool names.

Optional<String> authorization

Optional<ConnectorId> connectorId

Identifier for service connectors, like those available in ChatGPT. One of server_url or connector_id must be provided. Learn more about service connectors here.

Currently supported connector_id values are:

Dropbox: connector_dropbox
Gmail: connector_gmail
Google Calendar: connector_googlecalendar
Google Drive: connector_googledrive
Microsoft Teams: connector_microsoftteams
Outlook Calendar: connector_outlookcalendar
Outlook Email: connector_outlookemail
SharePoint: connector_sharepoint

Accepts one of the following:

CONNECTOR_DROPBOX("connector_dropbox")

CONNECTOR_GMAIL("connector_gmail")

CONNECTOR_GOOGLECALENDAR("connector_googlecalendar")

CONNECTOR_GOOGLEDRIVE("connector_googledrive")

CONNECTOR_MICROSOFTTEAMS("connector_microsoftteams")

CONNECTOR_OUTLOOKCALENDAR("connector_outlookcalendar")

CONNECTOR_OUTLOOKEMAIL("connector_outlookemail")

CONNECTOR_SHAREPOINT("connector_sharepoint")

Optional<Headers> headers

Optional HTTP headers to send to the MCP server. Use for authentication or other purposes.

Optional<RequireApproval> requireApproval

Specify which of the MCP server's tools require approval.

Accepts one of the following:

class McpToolApprovalFilter:

Specify which of the MCP server's tools require approval. Can be always, never, or a filter object associated with tools that require approval.

Optional<Always> always

A filter object to specify which tools are allowed.

Optional<Boolean> readOnly

Indicates whether or not a tool modifies data or is read-only. If an MCP server is annotated with readOnlyHint, it will match this filter.

Optional<List<String>> toolNames

List of allowed tool names.

Optional<Never> never

A filter object to specify which tools are allowed.

Optional<Boolean> readOnly

Indicates whether or not a tool modifies data or is read-only. If an MCP server is annotated with readOnlyHint, it will match this filter.

Optional<List<String>> toolNames

List of allowed tool names.

enum McpToolApprovalSetting:

Specify a single approval policy for all tools. One of always or never. When set to always, all tools will require approval. When set to never, all tools will not require approval.

ALWAYS("always")

NEVER("never")

Optional<String> serverDescription

Optional description of the MCP server, used to provide more context.

Optional<String> serverUrl

The URL for the MCP server. One of server_url or connector_id must be provided.

CodeInterpreter

Container container

The code interpreter container. Can be a container ID or an object that specifies uploaded file IDs to make available to your code, along with an optional memory_limit setting.

Accepts one of the following:

String

class CodeInterpreterToolAuto:

Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.

JsonValue; type

Always auto.

Optional<List<String>> fileIds

An optional list of uploaded files to make available to your code.

Optional<MemoryLimit> memoryLimit

The memory limit for the code interpreter container.

Accepts one of the following:

_1G("1g")

_4G("4g")

_16G("16g")

_64G("64g")

JsonValue; type

The type of the code interpreter tool. Always code_interpreter.

ImageGeneration

JsonValue; type

The type of the image generation tool. Always image_generation.

Optional<Action> action

Whether to generate a new image or edit an existing image. Default: auto.

Accepts one of the following:

GENERATE("generate")

EDIT("edit")

AUTO("auto")

Optional<Background> background

Background type for the generated image. One of transparent, opaque, or auto. Default: auto.

Accepts one of the following:

TRANSPARENT("transparent")

OPAQUE("opaque")

AUTO("auto")

Optional<InputFidelity> inputFidelity

Accepts one of the following:

HIGH("high")

LOW("low")

Optional<InputImageMask> inputImageMask

Optional mask for inpainting. Contains image_url (string, optional) and file_id (string, optional).

Optional<String> fileId

File ID for the mask image.

Optional<String> imageUrl

Base64-encoded mask image.

Optional<Model> model

The image generation model to use. Default: gpt-image-1.

Accepts one of the following:

GPT_IMAGE_1("gpt-image-1")

GPT_IMAGE_1_MINI("gpt-image-1-mini")

GPT_IMAGE_1_5("gpt-image-1.5")

Optional<Moderation> moderation

Moderation level for the generated image. Default: auto.

Accepts one of the following:

AUTO("auto")

LOW("low")

Optional<Long> outputCompression

Compression level for the output image. Default: 100.

minimum0

maximum100

Optional<OutputFormat> outputFormat

The output format of the generated image. One of png, webp, or jpeg. Default: png.

Accepts one of the following:

PNG("png")

WEBP("webp")

JPEG("jpeg")

Optional<Long> partialImages

Number of partial images to generate in streaming mode, from 0 (default value) to 3.

minimum0

maximum3

Optional<Quality> quality

The quality of the generated image. One of low, medium, high, or auto. Default: auto.

Accepts one of the following:

LOW("low")

MEDIUM("medium")

HIGH("high")

AUTO("auto")

Optional<Size> size

The size of the generated image. One of 1024x1024, 1024x1536, 1536x1024, or auto. Default: auto.

Accepts one of the following:

_1024X1024("1024x1024")

_1024X1536("1024x1536")

_1536X1024("1536x1024")

AUTO("auto")

JsonValue;

JsonValue; type

The type of the local shell tool. Always local_shell.

class FunctionShellTool:

A tool that allows the model to execute shell commands.

JsonValue; type

The type of the shell tool. Always shell.

class CustomTool:

A custom tool that processes input using a specified format. Learn more about custom tools

String name

The name of the custom tool, used to identify it in tool calls.

JsonValue; type

The type of the custom tool. Always custom.

Optional<String> description

Optional description of the custom tool, used to provide more context.

Optional<CustomToolInputFormat> format

The input format for the custom tool. Default is unconstrained text.

Accepts one of the following:

JsonValue;

JsonValue; type

Unconstrained text format. Always text.

Grammar

String definition

The grammar definition.

Syntax syntax

The syntax of the grammar definition. One of lark or regex.

Accepts one of the following:

LARK("lark")

REGEX("regex")

JsonValue; type

Grammar format. Always grammar.

class WebSearchPreviewTool:

This tool searches the web for relevant results to use in a response. Learn more about the web search tool.

Type type

The type of the web search tool. One of web_search_preview or web_search_preview_2025_03_11.

Accepts one of the following:

WEB_SEARCH_PREVIEW("web_search_preview")

WEB_SEARCH_PREVIEW_2025_03_11("web_search_preview_2025_03_11")

Optional<SearchContextSize> searchContextSize

High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.

Accepts one of the following:

LOW("low")

MEDIUM("medium")

HIGH("high")

Optional<UserLocation> userLocation

The user's location.

JsonValue; type

The type of location approximation. Always approximate.

Optional<String> city

Free text input for the city of the user, e.g. San Francisco.

Optional<String> country

The two-letter ISO country code of the user, e.g. US.

Optional<String> region

Free text input for the region of the user, e.g. California.

Optional<String> timezone

The IANA timezone of the user, e.g. America/Los_Angeles.

class ApplyPatchTool:

Allows the assistant to create, delete, or update files using unified diffs.

JsonValue; type

The type of the tool. Always apply_patch.

Optional<Double> topP

An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

EvalApiError error

An object representing an error response from the Eval API.

String code

The error code.

String message

The error message.

String evalId

The identifier of the associated evaluation.

Optional<Metadata> metadata

Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

String model

The model that is evaluated, if applicable.

String name

The name of the evaluation run.

JsonValue; object_

The type of the object. Always "eval.run".

List<PerModelUsage> perModelUsage

Usage statistics for each model during the evaluation run.

long cachedTokens

The number of tokens retrieved from cache.

long completionTokens

The number of completion tokens generated.

long invocationCount

The number of invocations.

String modelName

The name of the model.

long promptTokens

The number of prompt tokens used.

long totalTokens

The total number of tokens used.

List<PerTestingCriteriaResult> perTestingCriteriaResults

Results per testing criteria applied during the evaluation run.

long failed

Number of tests failed for this criteria.

long passed

Number of tests passed for this criteria.

String testingCriteria

A description of the testing criteria.

String reportUrl

The URL to the rendered evaluation run report on the UI dashboard.

ResultCounts resultCounts

Counters summarizing the outcomes of the evaluation run.

long errored

Number of output items that resulted in an error.

long failed

Number of output items that failed to pass the evaluation.

long passed

Number of output items that passed the evaluation.

long total

Total number of executed output items.

String status

The status of the evaluation run.

Create eval run

package com.openai.example;

import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.core.JsonValue;
import com.openai.models.evals.runs.CreateEvalJsonlRunDataSource;
import com.openai.models.evals.runs.RunCreateParams;
import com.openai.models.evals.runs.RunCreateResponse;
import java.util.List;

public final class Main {
    private Main() {}

    public static void main(String[] args) {
        OpenAIClient client = OpenAIOkHttpClient.fromEnv();

        RunCreateParams params = RunCreateParams.builder()
            .evalId("eval_id")
            .dataSource(CreateEvalJsonlRunDataSource.builder()
                .fileContentSource(List.of(CreateEvalJsonlRunDataSource.Source.FileContent.Content.builder()
                    .item(CreateEvalJsonlRunDataSource.Source.FileContent.Content.Item.builder()
                        .putAdditionalProperty("foo", JsonValue.from("bar"))
                        .build())
                    .build()))
                .build())
            .build();
        RunCreateResponse run = client.evals().runs().create(params);
    }
}

{
  "id": "id",
  "created_at": 0,
  "data_source": {
    "source": {
      "content": [
        {
          "item": {
            "foo": "bar"
          },
          "sample": {
            "foo": "bar"
          }
        }
      ],
      "type": "file_content"
    },
    "type": "jsonl"
  },
  "error": {
    "code": "code",
    "message": "message"
  },
  "eval_id": "eval_id",
  "metadata": {
    "foo": "string"
  },
  "model": "model",
  "name": "name",
  "object": "eval.run",
  "per_model_usage": [
    {
      "cached_tokens": 0,
      "completion_tokens": 0,
      "invocation_count": 0,
      "model_name": "model_name",
      "prompt_tokens": 0,
      "total_tokens": 0
    }
  ],
  "per_testing_criteria_results": [
    {
      "failed": 0,
      "passed": 0,
      "testing_criteria": "testing_criteria"
    }
  ],
  "report_url": "report_url",
  "result_counts": {
    "errored": 0,
    "failed": 0,
    "passed": 0,
    "total": 0
  },
  "status": "status"
}

Returns Examples

{
  "id": "id",
  "created_at": 0,
  "data_source": {
    "source": {
      "content": [
        {
          "item": {
            "foo": "bar"
          },
          "sample": {
            "foo": "bar"
          }
        }
      ],
      "type": "file_content"
    },
    "type": "jsonl"
  },
  "error": {
    "code": "code",
    "message": "message"
  },
  "eval_id": "eval_id",
  "metadata": {
    "foo": "string"
  },
  "model": "model",
  "name": "name",
  "object": "eval.run",
  "per_model_usage": [
    {
      "cached_tokens": 0,
      "completion_tokens": 0,
      "invocation_count": 0,
      "model_name": "model_name",
      "prompt_tokens": 0,
      "total_tokens": 0
    }
  ],
  "per_testing_criteria_results": [
    {
      "failed": 0,
      "passed": 0,
      "testing_criteria": "testing_criteria"
    }
  ],
  "report_url": "report_url",
  "result_counts": {
    "errored": 0,
    "failed": 0,
    "passed": 0,
    "total": 0
  },
  "status": "status"
}

Search the API docs

Getting Started

Using Codex

Configuration

Administration

Automation

Learn

Releases

Categories

Topics

Create eval run

ParametersExpand Collapse

ReturnsExpand Collapse

Create eval run

Returns Examples