# Grader Models ## Domain Types ### Eval Content Item - `GraderInputItem = String | ResponseInputText | OutputText{ text, type} | 2 more` A single content item: input text, output text, input image, or input audio. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `text: String` The text input to the model. - `type: :input_text` The type of the input item. Always `input_text`. - `:input_text` - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `input_audio: InputAudio{ data, format_}` - `data: String` Base64-encoded audio data. - `format_: :mp3 | :wav` The format of the audio data. Currently supported formats are `mp3` and `wav`. - `:mp3` - `:wav` - `type: :input_audio` The type of the input item. Always `input_audio`. - `:input_audio` ### Grader Inputs - `GraderInputs = Array[GraderInputItem]` A list of inputs, each of which may be either an input text, output text, input image, or input audio object. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `text: String` The text input to the model. - `type: :input_text` The type of the input item. Always `input_text`. - `:input_text` - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `input_audio: InputAudio{ data, format_}` - `data: String` Base64-encoded audio data. - `format_: :mp3 | :wav` The format of the audio data. Currently supported formats are `mp3` and `wav`. - `:mp3` - `:wav` - `type: :input_audio` The type of the input item. Always `input_audio`. - `:input_audio` ### Label Model Grader - `class LabelModelGrader` A LabelModelGrader object which uses a model to assign labels to each item in the evaluation. - `input: Array[Input{ content, role, type}]` - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `text: String` The text input to the model. - `type: :input_text` The type of the input item. Always `input_text`. - `:input_text` - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `input_audio: InputAudio{ data, format_}` - `data: String` Base64-encoded audio data. - `format_: :mp3 | :wav` The format of the audio data. Currently supported formats are `mp3` and `wav`. - `:mp3` - `:wav` - `type: :input_audio` The type of the input item. Always `input_audio`. - `:input_audio` - `GraderInputs = Array[GraderInputItem]` A list of inputs, each of which may be either an input text, output text, input image, or input audio object. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `role: :user | :assistant | :system | :developer` The role of the message input. One of `user`, `assistant`, `system`, or `developer`. - `:user` - `:assistant` - `:system` - `:developer` - `type: :message` The type of the message input. Always `message`. - `:message` - `labels: Array[String]` The labels to assign to each item in the evaluation. - `model: String` The model to use for the evaluation. Must support structured outputs. - `name: String` The name of the grader. - `passing_labels: Array[String]` The labels that indicate a passing result. Must be a subset of labels. - `type: :label_model` The object type, which is always `label_model`. - `:label_model` ### Multi Grader - `class MultiGrader` A MultiGrader object combines the output of multiple graders to produce a single score. - `calculate_output: String` A formula to calculate the output based on grader results. - `graders: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more` A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. - `class StringCheckGrader` A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. - `input: String` The input text. This may include template strings. - `name: String` The name of the grader. - `operation: :eq | :ne | :like | :ilike` The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`. - `:eq` - `:ne` - `:like` - `:ilike` - `reference: String` The reference text. This may include template strings. - `type: :string_check` The object type, which is always `string_check`. - `:string_check` - `class TextSimilarityGrader` A TextSimilarityGrader object which grades text based on similarity metrics. - `evaluation_metric: :cosine | :fuzzy_match | :bleu | 8 more` The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`, `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`, or `rouge_l`. - `:cosine` - `:fuzzy_match` - `:bleu` - `:gleu` - `:meteor` - `:rouge_1` - `:rouge_2` - `:rouge_3` - `:rouge_4` - `:rouge_5` - `:rouge_l` - `input: String` The text being graded. - `name: String` The name of the grader. - `reference: String` The text being graded against. - `type: :text_similarity` The type of grader. - `:text_similarity` - `class PythonGrader` A PythonGrader object that runs a python script on the input. - `name: String` The name of the grader. - `source: String` The source code of the python script. - `type: :python` The object type, which is always `python`. - `:python` - `image_tag: String` The image tag to use for the python script. - `class ScoreModelGrader` A ScoreModelGrader object that uses a model to assign a score to the input. - `input: Array[Input{ content, role, type}]` The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings. - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `text: String` The text input to the model. - `type: :input_text` The type of the input item. Always `input_text`. - `:input_text` - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `input_audio: InputAudio{ data, format_}` - `data: String` Base64-encoded audio data. - `format_: :mp3 | :wav` The format of the audio data. Currently supported formats are `mp3` and `wav`. - `:mp3` - `:wav` - `type: :input_audio` The type of the input item. Always `input_audio`. - `:input_audio` - `GraderInputs = Array[GraderInputItem]` A list of inputs, each of which may be either an input text, output text, input image, or input audio object. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `role: :user | :assistant | :system | :developer` The role of the message input. One of `user`, `assistant`, `system`, or `developer`. - `:user` - `:assistant` - `:system` - `:developer` - `type: :message` The type of the message input. Always `message`. - `:message` - `model: String` The model to use for the evaluation. - `name: String` The name of the grader. - `type: :score_model` The object type, which is always `score_model`. - `:score_model` - `range: Array[Float]` The range of the score. Defaults to `[0, 1]`. - `sampling_params: SamplingParams{ max_completions_tokens, reasoning_effort, seed, 2 more}` The sampling parameters for the model. - `max_completions_tokens: Integer` The maximum number of tokens the grader model may generate in its response. - `reasoning_effort: ReasoningEffort` Constrains effort on reasoning for [reasoning models](https://platform.openai.com/docs/guides/reasoning). Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - `:none` - `:minimal` - `:low` - `:medium` - `:high` - `:xhigh` - `seed: Integer` A seed value to initialize the randomness, during sampling. - `temperature: Float` A higher temperature increases randomness in the outputs. - `top_p: Float` An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - `class LabelModelGrader` A LabelModelGrader object which uses a model to assign labels to each item in the evaluation. - `input: Array[Input{ content, role, type}]` - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `GraderInputs = Array[GraderInputItem]` A list of inputs, each of which may be either an input text, output text, input image, or input audio object. - `role: :user | :assistant | :system | :developer` The role of the message input. One of `user`, `assistant`, `system`, or `developer`. - `:user` - `:assistant` - `:system` - `:developer` - `type: :message` The type of the message input. Always `message`. - `:message` - `labels: Array[String]` The labels to assign to each item in the evaluation. - `model: String` The model to use for the evaluation. Must support structured outputs. - `name: String` The name of the grader. - `passing_labels: Array[String]` The labels that indicate a passing result. Must be a subset of labels. - `type: :label_model` The object type, which is always `label_model`. - `:label_model` - `name: String` The name of the grader. - `type: :multi` The object type, which is always `multi`. - `:multi` ### Python Grader - `class PythonGrader` A PythonGrader object that runs a python script on the input. - `name: String` The name of the grader. - `source: String` The source code of the python script. - `type: :python` The object type, which is always `python`. - `:python` - `image_tag: String` The image tag to use for the python script. ### Score Model Grader - `class ScoreModelGrader` A ScoreModelGrader object that uses a model to assign a score to the input. - `input: Array[Input{ content, role, type}]` The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings. - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `text: String` The text input to the model. - `type: :input_text` The type of the input item. Always `input_text`. - `:input_text` - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `input_audio: InputAudio{ data, format_}` - `data: String` Base64-encoded audio data. - `format_: :mp3 | :wav` The format of the audio data. Currently supported formats are `mp3` and `wav`. - `:mp3` - `:wav` - `type: :input_audio` The type of the input item. Always `input_audio`. - `:input_audio` - `GraderInputs = Array[GraderInputItem]` A list of inputs, each of which may be either an input text, output text, input image, or input audio object. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `role: :user | :assistant | :system | :developer` The role of the message input. One of `user`, `assistant`, `system`, or `developer`. - `:user` - `:assistant` - `:system` - `:developer` - `type: :message` The type of the message input. Always `message`. - `:message` - `model: String` The model to use for the evaluation. - `name: String` The name of the grader. - `type: :score_model` The object type, which is always `score_model`. - `:score_model` - `range: Array[Float]` The range of the score. Defaults to `[0, 1]`. - `sampling_params: SamplingParams{ max_completions_tokens, reasoning_effort, seed, 2 more}` The sampling parameters for the model. - `max_completions_tokens: Integer` The maximum number of tokens the grader model may generate in its response. - `reasoning_effort: ReasoningEffort` Constrains effort on reasoning for [reasoning models](https://platform.openai.com/docs/guides/reasoning). Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - `:none` - `:minimal` - `:low` - `:medium` - `:high` - `:xhigh` - `seed: Integer` A seed value to initialize the randomness, during sampling. - `temperature: Float` A higher temperature increases randomness in the outputs. - `top_p: Float` An alternative to temperature for nucleus sampling; 1.0 includes all tokens. ### String Check Grader - `class StringCheckGrader` A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. - `input: String` The input text. This may include template strings. - `name: String` The name of the grader. - `operation: :eq | :ne | :like | :ilike` The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`. - `:eq` - `:ne` - `:like` - `:ilike` - `reference: String` The reference text. This may include template strings. - `type: :string_check` The object type, which is always `string_check`. - `:string_check` ### Text Similarity Grader - `class TextSimilarityGrader` A TextSimilarityGrader object which grades text based on similarity metrics. - `evaluation_metric: :cosine | :fuzzy_match | :bleu | 8 more` The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`, `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`, or `rouge_l`. - `:cosine` - `:fuzzy_match` - `:bleu` - `:gleu` - `:meteor` - `:rouge_1` - `:rouge_2` - `:rouge_3` - `:rouge_4` - `:rouge_5` - `:rouge_l` - `input: String` The text being graded. - `name: String` The name of the grader. - `reference: String` The text being graded against. - `type: :text_similarity` The type of grader. - `:text_similarity`