Graders
GradersGrader Models
ModelsExpand Collapse
GraderInputItem = String | ResponseInputText { text, type, prompt_cache_breakpoint } | OutputText{ text, type} | 2 moreA single content item: input text, output text, input image, or input audio.
A single content item: input text, output text, input image, or input audio.
class InputImage { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
A list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
class InputImage { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
class LabelModelGrader { input, labels, model, 3 more } A LabelModelGrader object which uses a model to assign labels to each item
in the evaluation.
A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
input: Array[Input{ content, role, type}]
content: String | ResponseInputText { text, type, prompt_cache_breakpoint } | OutputText{ text, type} | 3 moreInputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
class InputImage { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
A list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
class InputImage { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
class MultiGrader { calculate_output, graders, name, type } A MultiGrader object combines the output of multiple graders to produce a single score.
A MultiGrader object combines the output of multiple graders to produce a single score.
graders: StringCheckGrader { input, name, operation, 2 more } | TextSimilarityGrader { evaluation_metric, input, name, 2 more } | PythonGrader { name, source, type, image_tag } | 2 moreA StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
class StringCheckGrader { input, name, operation, 2 more } A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
class TextSimilarityGrader { evaluation_metric, input, name, 2 more } A TextSimilarityGrader object which grades text based on similarity metrics.
A TextSimilarityGrader object which grades text based on similarity metrics.
class PythonGrader { name, source, type, image_tag } A PythonGrader object that runs a python script on the input.
A PythonGrader object that runs a python script on the input.
class ScoreModelGrader { input, model, name, 3 more } A ScoreModelGrader object that uses a model to assign a score to the input.
A ScoreModelGrader object that uses a model to assign a score to the input.
input: Array[Input{ content, role, type}]The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
content: String | ResponseInputText { text, type, prompt_cache_breakpoint } | OutputText{ text, type} | 3 moreInputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
class InputImage { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
A list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
class InputImage { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
sampling_params: SamplingParams{ max_completions_tokens, reasoning_effort, seed, 2 more}The sampling parameters for the model.
The sampling parameters for the model.
The maximum number of tokens the grader model may generate in its response.
Constrains effort on reasoning for reasoning models. Currently supported
values are none, minimal, low, medium, high, xhigh, and max.
Reducing reasoning effort can result in faster responses and fewer tokens
used on reasoning in a response. Not all reasoning models support every
value. See the
reasoning guide
for model-specific support.
class LabelModelGrader { input, labels, model, 3 more } A LabelModelGrader object which uses a model to assign labels to each item
in the evaluation.
A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
input: Array[Input{ content, role, type}]
content: String | ResponseInputText { text, type, prompt_cache_breakpoint } | OutputText{ text, type} | 3 moreInputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
class InputImage { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
A list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
class InputImage { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
class ScoreModelGrader { input, model, name, 3 more } A ScoreModelGrader object that uses a model to assign a score to the input.
A ScoreModelGrader object that uses a model to assign a score to the input.
input: Array[Input{ content, role, type}]The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
content: String | ResponseInputText { text, type, prompt_cache_breakpoint } | OutputText{ text, type} | 3 moreInputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
class InputImage { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
A list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
class InputImage { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
sampling_params: SamplingParams{ max_completions_tokens, reasoning_effort, seed, 2 more}The sampling parameters for the model.
The sampling parameters for the model.
The maximum number of tokens the grader model may generate in its response.
Constrains effort on reasoning for reasoning models. Currently supported
values are none, minimal, low, medium, high, xhigh, and max.
Reducing reasoning effort can result in faster responses and fewer tokens
used on reasoning in a response. Not all reasoning models support every
value. See the
reasoning guide
for model-specific support.
class TextSimilarityGrader { evaluation_metric, input, name, 2 more } A TextSimilarityGrader object which grades text based on similarity metrics.
A TextSimilarityGrader object which grades text based on similarity metrics.