Graders
GradersGrader Models
Models
GraderInputs = array of string or ResponseInputText { text, type, prompt_cache_breakpoint } or OutputText { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
GraderInputs = array of string or ResponseInputText { text, type, prompt_cache_breakpoint } or OutputText { text, type } or 2 more
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
One of the following: TextInput = string A text input to the model. ResponseInputText { text, type, prompt_cache_breakpoint } A text input to the model. text: string The text input to the model. type: "input_text" (default: "input_text") The type of the input item. Always input_text. prompt_cache_breakpoint: optional { mode, ttl } Marks the exact end of a reusable prompt prefix. When ttl is omitted, the breakpoint inherits the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block. mode: "explicit" (default: "explicit") The breakpoint mode. Always explicit. ttl: optional "30m" or "12h" The minimum lifetime of this cache breakpoint. Defaults to the request TTL. One of the following: "30m" "12h" OutputText { text, type } A text output from the model. text: string The text output from the model. type: "output_text" The type of the output text. Always output_text. InputImage { image_url, type, detail } An image input block used within EvalItem content arrays. image_url: string The URL of the image input. formaturi type: "input_image" The type of the image input. Always input_image. detail: optional string The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto. ResponseInputAudio { input_audio, type } An audio input to the model. input_audio: { data, format } data: string Base64-encoded audio data. format: "mp3" or "wav" The format of the audio data. Currently supported formats are mp3 and wav. One of the following: "mp3" "wav" type: "input_audio" The type of the input item. Always input_audio.
LabelModelGrader object { input, labels, model, 3 more } A LabelModelGrader object which uses a model to assign labels to each item
in the evaluation.
LabelModelGrader object { input, labels, model, 3 more }
A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
input: array of { content, role, type } content: string or ResponseInputText or OutputText or 3 more Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. One of the following: TextInput = string A text input to the model. ResponseInputText { text, type, prompt_cache_breakpoint } A text input to the model. text: string The text input to the model. type: "input_text" (default: "input_text") The type of the input item. Always input_text. prompt_cache_breakpoint: optional { mode, ttl } Marks the exact end of a reusable prompt prefix. When ttl is omitted, the breakpoint inherits the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block. mode: "explicit" (default: "explicit") The breakpoint mode. Always explicit. ttl: optional "30m" or "12h" The minimum lifetime of this cache breakpoint. Defaults to the request TTL. One of the following: "30m" "12h" OutputText { text, type } A text output from the model. text: string The text output from the model. type: "output_text" The type of the output text. Always output_text. InputImage { image_url, type, detail } An image input block used within EvalItem content arrays. image_url: string The URL of the image input. formaturi type: "input_image" The type of the image input. Always input_image. detail: optional string The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto. ResponseInputAudio { input_audio, type } An audio input to the model. input_audio: { data, format } data: string Base64-encoded audio data. format: "mp3" or "wav" The format of the audio data. Currently supported formats are mp3 and wav. One of the following: "mp3" "wav" type: "input_audio" The type of the input item. Always input_audio. GraderInputs = array of string or ResponseInputText or OutputText or 2 more A list of inputs, each of which may be either an input text, output text, input image, or input audio object. One of the following: TextInput = string A text input to the model. ResponseInputText { text, type, prompt_cache_breakpoint } A text input to the model. text: string The text input to the model. type: "input_text" (default: "input_text") The type of the input item. Always input_text. prompt_cache_breakpoint: optional { mode, ttl } Marks the exact end of a reusable prompt prefix. When ttl is omitted, the breakpoint inherits the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block. mode: "explicit" (default: "explicit") The breakpoint mode. Always explicit. ttl: optional "30m" or "12h" The minimum lifetime of this cache breakpoint. Defaults to the request TTL. One of the following: "30m" "12h" OutputText { text, type } A text output from the model. text: string The text output from the model. type: "output_text" The type of the output text. Always output_text. InputImage { image_url, type, detail } An image input block used within EvalItem content arrays. image_url: string The URL of the image input. formaturi type: "input_image" The type of the image input. Always input_image. detail: optional string The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto. ResponseInputAudio { input_audio, type } An audio input to the model. input_audio: { data, format } data: string Base64-encoded audio data. format: "mp3" or "wav" The format of the audio data. Currently supported formats are mp3 and wav. One of the following: "mp3" "wav" type: "input_audio" The type of the input item. Always input_audio. role: "user" or "assistant" or "system" or "developer" The role of the message input. One of user, assistant, system, or developer. One of the following: "user" "assistant" "system" "developer" type: optional "message" The type of the message input. Always message. labels: array of string The labels to assign to each item in the evaluation. model: string The model to use for the evaluation. Must support structured outputs. name: string The name of the grader. passing_labels: array of string The labels that indicate a passing result. Must be a subset of labels. type: "label_model" The object type, which is always label_model.
MultiGrader object { calculate_output, graders, name, type } A MultiGrader object combines the output of multiple graders to produce a single score.
MultiGrader object { calculate_output, graders, name, type }
A MultiGrader object combines the output of multiple graders to produce a single score.
calculate_output: string A formula to calculate the output based on grader results. graders: map[StringCheckGrader or TextSimilarityGrader or PythonGrader or 2 more] One of the following: StringCheckGrader { input, name, operation, 2 more } A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. input: string The input text. This may include template strings. name: string The name of the grader. operation: "eq" or "ne" or "like" or "ilike" The string check operation to perform. One of eq, ne, like, or ilike. One of the following: "eq" "ne" "like" "ilike" reference: string The reference text. This may include template strings. type: "string_check" The object type, which is always string_check. TextSimilarityGrader { evaluation_metric, input, name, 2 more } A TextSimilarityGrader object which grades text based on similarity metrics. evaluation_metric: "cosine" or "fuzzy_match" or "bleu" or 8 more The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l. One of the following: "cosine" "fuzzy_match" "bleu" "gleu" "meteor" "rouge_1" "rouge_2" "rouge_3" "rouge_4" "rouge_5" "rouge_l" input: string The text being graded. name: string The name of the grader. reference: string The text being graded against. type: "text_similarity" (default: "text_similarity") The type of grader. PythonGrader { name, source, type, image_tag } A PythonGrader object that runs a python script on the input. name: string The name of the grader. source: string The source code of the python script. type: "python" The object type, which is always python. image_tag: optional string The image tag to use for the python script. ScoreModelGrader { input, model, name, 3 more } A ScoreModelGrader object that uses a model to assign a score to the input. input: array of { content, role, type } The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings. content: string or ResponseInputText or OutputText or 3 more Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. One of the following: TextInput = string A text input to the model. ResponseInputText { text, type, prompt_cache_breakpoint } A text input to the model. text: string The text input to the model. type: "input_text" (default: "input_text") The type of the input item. Always input_text. prompt_cache_breakpoint: optional { mode, ttl } Marks the exact end of a reusable prompt prefix. When ttl is omitted, the breakpoint inherits the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block. mode: "explicit" (default: "explicit") The breakpoint mode. Always explicit. ttl: optional "30m" or "12h" The minimum lifetime of this cache breakpoint. Defaults to the request TTL. One of the following: "30m" "12h" OutputText { text, type } A text output from the model. text: string The text output from the model. type: "output_text" The type of the output text. Always output_text. InputImage { image_url, type, detail } An image input block used within EvalItem content arrays. image_url: string The URL of the image input. formaturi type: "input_image" The type of the image input. Always input_image. detail: optional string The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto. ResponseInputAudio { input_audio, type } An audio input to the model. input_audio: { data, format } data: string Base64-encoded audio data. format: "mp3" or "wav" The format of the audio data. Currently supported formats are mp3 and wav. One of the following: "mp3" "wav" type: "input_audio" The type of the input item. Always input_audio. GraderInputs = array of string or ResponseInputText or OutputText or 2 more A list of inputs, each of which may be either an input text, output text, input image, or input audio object. One of the following: TextInput = string A text input to the model. ResponseInputText { text, type, prompt_cache_breakpoint } A text input to the model. text: string The text input to the model. type: "input_text" (default: "input_text") The type of the input item. Always input_text. prompt_cache_breakpoint: optional { mode, ttl } Marks the exact end of a reusable prompt prefix. When ttl is omitted, the breakpoint inherits the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block. mode: "explicit" (default: "explicit") The breakpoint mode. Always explicit. ttl: optional "30m" or "12h" The minimum lifetime of this cache breakpoint. Defaults to the request TTL. One of the following: "30m" "12h" OutputText { text, type } A text output from the model. text: string The text output from the model. type: "output_text" The type of the output text. Always output_text. InputImage { image_url, type, detail } An image input block used within EvalItem content arrays. image_url: string The URL of the image input. formaturi type: "input_image" The type of the image input. Always input_image. detail: optional string The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto. ResponseInputAudio { input_audio, type } An audio input to the model. input_audio: { data, format } data: string Base64-encoded audio data. format: "mp3" or "wav" The format of the audio data. Currently supported formats are mp3 and wav. One of the following: "mp3" "wav" type: "input_audio" The type of the input item. Always input_audio. role: "user" or "assistant" or "system" or "developer" The role of the message input. One of user, assistant, system, or developer. One of the following: "user" "assistant" "system" "developer" type: optional "message" The type of the message input. Always message. model: string The model to use for the evaluation. name: string The name of the grader. type: "score_model" The object type, which is always score_model. range: optional array of number The service requires two numbers for the score range. Defaults to [0, 1]. sampling_params: optional { max_completions_tokens, reasoning_effort, seed, 2 more } The sampling parameters for the model. max_completions_tokens: optional integer or null The maximum number of tokens the grader model may generate in its response. minimum1 reasoning_effort: optional ReasoningEffort or null (default: "medium") Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support. seed: optional integer or null A seed value to initialize the randomness, during sampling. temperature: optional number or null A higher temperature increases randomness in the outputs. top_p: optional number or null (default: 1) An alternative to temperature for nucleus sampling; 1.0 includes all tokens. LabelModelGrader { input, labels, model, 3 more } A LabelModelGrader object which uses a model to assign labels to each item in the evaluation. input: array of { content, role, type } content: string or ResponseInputText or OutputText or 3 more Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. One of the following: TextInput = string A text input to the model. ResponseInputText { text, type, prompt_cache_breakpoint } A text input to the model. text: string The text input to the model. type: "input_text" (default: "input_text") The type of the input item. Always input_text. prompt_cache_breakpoint: optional { mode, ttl } Marks the exact end of a reusable prompt prefix. When ttl is omitted, the breakpoint inherits the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block. mode: "explicit" (default: "explicit") The breakpoint mode. Always explicit. ttl: optional "30m" or "12h" The minimum lifetime of this cache breakpoint. Defaults to the request TTL. One of the following: "30m" "12h" OutputText { text, type } A text output from the model. text: string The text output from the model. type: "output_text" The type of the output text. Always output_text. InputImage { image_url, type, detail } An image input block used within EvalItem content arrays. image_url: string The URL of the image input. formaturi type: "input_image" The type of the image input. Always input_image. detail: optional string The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto. ResponseInputAudio { input_audio, type } An audio input to the model. input_audio: { data, format } data: string Base64-encoded audio data. format: "mp3" or "wav" The format of the audio data. Currently supported formats are mp3 and wav. One of the following: "mp3" "wav" type: "input_audio" The type of the input item. Always input_audio. GraderInputs = array of string or ResponseInputText or OutputText or 2 more A list of inputs, each of which may be either an input text, output text, input image, or input audio object. One of the following: TextInput = string A text input to the model. ResponseInputText { text, type, prompt_cache_breakpoint } A text input to the model. text: string The text input to the model. type: "input_text" (default: "input_text") The type of the input item. Always input_text. prompt_cache_breakpoint: optional { mode, ttl } Marks the exact end of a reusable prompt prefix. When ttl is omitted, the breakpoint inherits the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block. mode: "explicit" (default: "explicit") The breakpoint mode. Always explicit. ttl: optional "30m" or "12h" The minimum lifetime of this cache breakpoint. Defaults to the request TTL. One of the following: "30m" "12h" OutputText { text, type } A text output from the model. text: string The text output from the model. type: "output_text" The type of the output text. Always output_text. InputImage { image_url, type, detail } An image input block used within EvalItem content arrays. image_url: string The URL of the image input. formaturi type: "input_image" The type of the image input. Always input_image. detail: optional string The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto. ResponseInputAudio { input_audio, type } An audio input to the model. input_audio: { data, format } data: string Base64-encoded audio data. format: "mp3" or "wav" The format of the audio data. Currently supported formats are mp3 and wav. One of the following: "mp3" "wav" type: "input_audio" The type of the input item. Always input_audio. role: "user" or "assistant" or "system" or "developer" The role of the message input. One of user, assistant, system, or developer. One of the following: "user" "assistant" "system" "developer" type: optional "message" The type of the message input. Always message. labels: array of string The labels to assign to each item in the evaluation. model: string The model to use for the evaluation. Must support structured outputs. name: string The name of the grader. passing_labels: array of string The labels that indicate a passing result. Must be a subset of labels. type: "label_model" The object type, which is always label_model. name: string The name of the grader. type: "multi" (default: "multi") The object type, which is always multi.
PythonGrader object { name, source, type, image_tag } A PythonGrader object that runs a python script on the input.
PythonGrader object { name, source, type, image_tag }
A PythonGrader object that runs a python script on the input.
name: string The name of the grader. source: string The source code of the python script. type: "python" The object type, which is always python. image_tag: optional string The image tag to use for the python script.
ScoreModelGrader object { input, model, name, 3 more } A ScoreModelGrader object that uses a model to assign a score to the input.
ScoreModelGrader object { input, model, name, 3 more }
A ScoreModelGrader object that uses a model to assign a score to the input.
input: array of { content, role, type } The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings. content: string or ResponseInputText or OutputText or 3 more Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. One of the following: TextInput = string A text input to the model. ResponseInputText { text, type, prompt_cache_breakpoint } A text input to the model. text: string The text input to the model. type: "input_text" (default: "input_text") The type of the input item. Always input_text. prompt_cache_breakpoint: optional { mode, ttl } Marks the exact end of a reusable prompt prefix. When ttl is omitted, the breakpoint inherits the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block. mode: "explicit" (default: "explicit") The breakpoint mode. Always explicit. ttl: optional "30m" or "12h" The minimum lifetime of this cache breakpoint. Defaults to the request TTL. One of the following: "30m" "12h" OutputText { text, type } A text output from the model. text: string The text output from the model. type: "output_text" The type of the output text. Always output_text. InputImage { image_url, type, detail } An image input block used within EvalItem content arrays. image_url: string The URL of the image input. formaturi type: "input_image" The type of the image input. Always input_image. detail: optional string The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto. ResponseInputAudio { input_audio, type } An audio input to the model. input_audio: { data, format } data: string Base64-encoded audio data. format: "mp3" or "wav" The format of the audio data. Currently supported formats are mp3 and wav. One of the following: "mp3" "wav" type: "input_audio" The type of the input item. Always input_audio. GraderInputs = array of string or ResponseInputText or OutputText or 2 more A list of inputs, each of which may be either an input text, output text, input image, or input audio object. One of the following: TextInput = string A text input to the model. ResponseInputText { text, type, prompt_cache_breakpoint } A text input to the model. text: string The text input to the model. type: "input_text" (default: "input_text") The type of the input item. Always input_text. prompt_cache_breakpoint: optional { mode, ttl } Marks the exact end of a reusable prompt prefix. When ttl is omitted, the breakpoint inherits the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block. mode: "explicit" (default: "explicit") The breakpoint mode. Always explicit. ttl: optional "30m" or "12h" The minimum lifetime of this cache breakpoint. Defaults to the request TTL. One of the following: "30m" "12h" OutputText { text, type } A text output from the model. text: string The text output from the model. type: "output_text" The type of the output text. Always output_text. InputImage { image_url, type, detail } An image input block used within EvalItem content arrays. image_url: string The URL of the image input. formaturi type: "input_image" The type of the image input. Always input_image. detail: optional string The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto. ResponseInputAudio { input_audio, type } An audio input to the model. input_audio: { data, format } data: string Base64-encoded audio data. format: "mp3" or "wav" The format of the audio data. Currently supported formats are mp3 and wav. One of the following: "mp3" "wav" type: "input_audio" The type of the input item. Always input_audio. role: "user" or "assistant" or "system" or "developer" The role of the message input. One of user, assistant, system, or developer. One of the following: "user" "assistant" "system" "developer" type: optional "message" The type of the message input. Always message. model: string The model to use for the evaluation. name: string The name of the grader. type: "score_model" The object type, which is always score_model. range: optional array of number The service requires two numbers for the score range. Defaults to [0, 1]. sampling_params: optional { max_completions_tokens, reasoning_effort, seed, 2 more } The sampling parameters for the model. max_completions_tokens: optional integer or null The maximum number of tokens the grader model may generate in its response. minimum1 reasoning_effort: optional ReasoningEffort or null (default: "medium") Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support. seed: optional integer or null A seed value to initialize the randomness, during sampling. temperature: optional number or null A higher temperature increases randomness in the outputs. top_p: optional number or null (default: 1) An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
StringCheckGrader object { input, name, operation, 2 more } A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
StringCheckGrader object { input, name, operation, 2 more }
A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
input: string The input text. This may include template strings. name: string The name of the grader. operation: "eq" or "ne" or "like" or "ilike" The string check operation to perform. One of eq, ne, like, or ilike. One of the following: "eq" "ne" "like" "ilike" reference: string The reference text. This may include template strings. type: "string_check" The object type, which is always string_check.
TextSimilarityGrader object { evaluation_metric, input, name, 2 more } A TextSimilarityGrader object which grades text based on similarity metrics.
TextSimilarityGrader object { evaluation_metric, input, name, 2 more }
A TextSimilarityGrader object which grades text based on similarity metrics.
evaluation_metric: "cosine" or "fuzzy_match" or "bleu" or 8 more The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l. One of the following: "cosine" "fuzzy_match" "bleu" "gleu" "meteor" "rouge_1" "rouge_2" "rouge_3" "rouge_4" "rouge_5" "rouge_l" input: string The text being graded. name: string The name of the grader. reference: string The text being graded against. type: "text_similarity" (default: "text_similarity") The type of grader.