# Alpha # Graders ## Run grader `fine_tuning.alpha.graders.run(**kwargs) -> GraderRunResponse` **post** `/fine_tuning/alpha/graders/run` Run a grader. ### Parameters - `grader: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more` The grader used for the fine-tuning job. - `class StringCheckGrader` A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. - `input: String` The input text. This may include template strings. - `name: String` The name of the grader. - `operation: :eq | :ne | :like | :ilike` The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`. - `:eq` - `:ne` - `:like` - `:ilike` - `reference: String` The reference text. This may include template strings. - `type: :string_check` The object type, which is always `string_check`. - `:string_check` - `class TextSimilarityGrader` A TextSimilarityGrader object which grades text based on similarity metrics. - `evaluation_metric: :cosine | :fuzzy_match | :bleu | 8 more` The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`, `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`, or `rouge_l`. - `:cosine` - `:fuzzy_match` - `:bleu` - `:gleu` - `:meteor` - `:rouge_1` - `:rouge_2` - `:rouge_3` - `:rouge_4` - `:rouge_5` - `:rouge_l` - `input: String` The text being graded. - `name: String` The name of the grader. - `reference: String` The text being graded against. - `type: :text_similarity` The type of grader. - `:text_similarity` - `class PythonGrader` A PythonGrader object that runs a python script on the input. - `name: String` The name of the grader. - `source: String` The source code of the python script. - `type: :python` The object type, which is always `python`. - `:python` - `image_tag: String` The image tag to use for the python script. - `class ScoreModelGrader` A ScoreModelGrader object that uses a model to assign a score to the input. - `input: Array[Input{ content, role, type}]` The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings. - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `text: String` The text input to the model. - `type: :input_text` The type of the input item. Always `input_text`. - `:input_text` - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `input_audio: InputAudio{ data, format_}` - `data: String` Base64-encoded audio data. - `format_: :mp3 | :wav` The format of the audio data. Currently supported formats are `mp3` and `wav`. - `:mp3` - `:wav` - `type: :input_audio` The type of the input item. Always `input_audio`. - `:input_audio` - `GraderInputs = Array[GraderInputItem]` A list of inputs, each of which may be either an input text, output text, input image, or input audio object. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `role: :user | :assistant | :system | :developer` The role of the message input. One of `user`, `assistant`, `system`, or `developer`. - `:user` - `:assistant` - `:system` - `:developer` - `type: :message` The type of the message input. Always `message`. - `:message` - `model: String` The model to use for the evaluation. - `name: String` The name of the grader. - `type: :score_model` The object type, which is always `score_model`. - `:score_model` - `range: Array[Float]` The range of the score. Defaults to `[0, 1]`. - `sampling_params: SamplingParams{ max_completions_tokens, reasoning_effort, seed, 2 more}` The sampling parameters for the model. - `max_completions_tokens: Integer` The maximum number of tokens the grader model may generate in its response. - `reasoning_effort: ReasoningEffort` Constrains effort on reasoning for [reasoning models](https://platform.openai.com/docs/guides/reasoning). Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - `:none` - `:minimal` - `:low` - `:medium` - `:high` - `:xhigh` - `seed: Integer` A seed value to initialize the randomness, during sampling. - `temperature: Float` A higher temperature increases randomness in the outputs. - `top_p: Float` An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - `class MultiGrader` A MultiGrader object combines the output of multiple graders to produce a single score. - `calculate_output: String` A formula to calculate the output based on grader results. - `graders: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more` A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. - `class StringCheckGrader` A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. - `class TextSimilarityGrader` A TextSimilarityGrader object which grades text based on similarity metrics. - `class PythonGrader` A PythonGrader object that runs a python script on the input. - `class ScoreModelGrader` A ScoreModelGrader object that uses a model to assign a score to the input. - `class LabelModelGrader` A LabelModelGrader object which uses a model to assign labels to each item in the evaluation. - `input: Array[Input{ content, role, type}]` - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `GraderInputs = Array[GraderInputItem]` A list of inputs, each of which may be either an input text, output text, input image, or input audio object. - `role: :user | :assistant | :system | :developer` The role of the message input. One of `user`, `assistant`, `system`, or `developer`. - `:user` - `:assistant` - `:system` - `:developer` - `type: :message` The type of the message input. Always `message`. - `:message` - `labels: Array[String]` The labels to assign to each item in the evaluation. - `model: String` The model to use for the evaluation. Must support structured outputs. - `name: String` The name of the grader. - `passing_labels: Array[String]` The labels that indicate a passing result. Must be a subset of labels. - `type: :label_model` The object type, which is always `label_model`. - `:label_model` - `name: String` The name of the grader. - `type: :multi` The object type, which is always `multi`. - `:multi` - `model_sample: String` The model sample to be evaluated. This value will be used to populate the `sample` namespace. See [the guide](https://platform.openai.com/docs/guides/graders) for more details. The `output_json` variable will be populated if the model sample is a valid JSON string. - `item: untyped` The dataset item provided to the grader. This will be used to populate the `item` namespace. See [the guide](https://platform.openai.com/docs/guides/graders) for more details. ### Returns - `class GraderRunResponse` - `metadata: Metadata{ errors, execution_time, name, 4 more}` - `errors: Errors{ formula_parse_error, invalid_variable_error, model_grader_parse_error, 11 more}` - `formula_parse_error: bool` - `invalid_variable_error: bool` - `model_grader_parse_error: bool` - `model_grader_refusal_error: bool` - `model_grader_server_error: bool` - `model_grader_server_error_details: String` - `other_error: bool` - `python_grader_runtime_error: bool` - `python_grader_runtime_error_details: String` - `python_grader_server_error: bool` - `python_grader_server_error_type: String` - `sample_parse_error: bool` - `truncated_observation_error: bool` - `unresponsive_reward_error: bool` - `execution_time: Float` - `name: String` - `sampled_model_name: String` - `scores: Hash[Symbol, untyped]` - `token_usage: Integer` - `type: String` - `model_grader_token_usage_per_model: Hash[Symbol, untyped]` - `reward: Float` - `sub_rewards: Hash[Symbol, untyped]` ### Example ```ruby require "openai" openai = OpenAI::Client.new(api_key: "My API Key") response = openai.fine_tuning.alpha.graders.run( grader: {input: "input", name: "name", operation: :eq, reference: "reference", type: :string_check}, model_sample: "model_sample" ) puts(response) ``` #### Response ```json { "metadata": { "errors": { "formula_parse_error": true, "invalid_variable_error": true, "model_grader_parse_error": true, "model_grader_refusal_error": true, "model_grader_server_error": true, "model_grader_server_error_details": "model_grader_server_error_details", "other_error": true, "python_grader_runtime_error": true, "python_grader_runtime_error_details": "python_grader_runtime_error_details", "python_grader_server_error": true, "python_grader_server_error_type": "python_grader_server_error_type", "sample_parse_error": true, "truncated_observation_error": true, "unresponsive_reward_error": true }, "execution_time": 0, "name": "name", "sampled_model_name": "sampled_model_name", "scores": { "foo": "bar" }, "token_usage": 0, "type": "type" }, "model_grader_token_usage_per_model": { "foo": "bar" }, "reward": 0, "sub_rewards": { "foo": "bar" } } ``` ## Validate grader `fine_tuning.alpha.graders.validate(**kwargs) -> GraderValidateResponse` **post** `/fine_tuning/alpha/graders/validate` Validate a grader. ### Parameters - `grader: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more` The grader used for the fine-tuning job. - `class StringCheckGrader` A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. - `input: String` The input text. This may include template strings. - `name: String` The name of the grader. - `operation: :eq | :ne | :like | :ilike` The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`. - `:eq` - `:ne` - `:like` - `:ilike` - `reference: String` The reference text. This may include template strings. - `type: :string_check` The object type, which is always `string_check`. - `:string_check` - `class TextSimilarityGrader` A TextSimilarityGrader object which grades text based on similarity metrics. - `evaluation_metric: :cosine | :fuzzy_match | :bleu | 8 more` The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`, `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`, or `rouge_l`. - `:cosine` - `:fuzzy_match` - `:bleu` - `:gleu` - `:meteor` - `:rouge_1` - `:rouge_2` - `:rouge_3` - `:rouge_4` - `:rouge_5` - `:rouge_l` - `input: String` The text being graded. - `name: String` The name of the grader. - `reference: String` The text being graded against. - `type: :text_similarity` The type of grader. - `:text_similarity` - `class PythonGrader` A PythonGrader object that runs a python script on the input. - `name: String` The name of the grader. - `source: String` The source code of the python script. - `type: :python` The object type, which is always `python`. - `:python` - `image_tag: String` The image tag to use for the python script. - `class ScoreModelGrader` A ScoreModelGrader object that uses a model to assign a score to the input. - `input: Array[Input{ content, role, type}]` The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings. - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `text: String` The text input to the model. - `type: :input_text` The type of the input item. Always `input_text`. - `:input_text` - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `input_audio: InputAudio{ data, format_}` - `data: String` Base64-encoded audio data. - `format_: :mp3 | :wav` The format of the audio data. Currently supported formats are `mp3` and `wav`. - `:mp3` - `:wav` - `type: :input_audio` The type of the input item. Always `input_audio`. - `:input_audio` - `GraderInputs = Array[GraderInputItem]` A list of inputs, each of which may be either an input text, output text, input image, or input audio object. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `role: :user | :assistant | :system | :developer` The role of the message input. One of `user`, `assistant`, `system`, or `developer`. - `:user` - `:assistant` - `:system` - `:developer` - `type: :message` The type of the message input. Always `message`. - `:message` - `model: String` The model to use for the evaluation. - `name: String` The name of the grader. - `type: :score_model` The object type, which is always `score_model`. - `:score_model` - `range: Array[Float]` The range of the score. Defaults to `[0, 1]`. - `sampling_params: SamplingParams{ max_completions_tokens, reasoning_effort, seed, 2 more}` The sampling parameters for the model. - `max_completions_tokens: Integer` The maximum number of tokens the grader model may generate in its response. - `reasoning_effort: ReasoningEffort` Constrains effort on reasoning for [reasoning models](https://platform.openai.com/docs/guides/reasoning). Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - `:none` - `:minimal` - `:low` - `:medium` - `:high` - `:xhigh` - `seed: Integer` A seed value to initialize the randomness, during sampling. - `temperature: Float` A higher temperature increases randomness in the outputs. - `top_p: Float` An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - `class MultiGrader` A MultiGrader object combines the output of multiple graders to produce a single score. - `calculate_output: String` A formula to calculate the output based on grader results. - `graders: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more` A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. - `class StringCheckGrader` A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. - `class TextSimilarityGrader` A TextSimilarityGrader object which grades text based on similarity metrics. - `class PythonGrader` A PythonGrader object that runs a python script on the input. - `class ScoreModelGrader` A ScoreModelGrader object that uses a model to assign a score to the input. - `class LabelModelGrader` A LabelModelGrader object which uses a model to assign labels to each item in the evaluation. - `input: Array[Input{ content, role, type}]` - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `GraderInputs = Array[GraderInputItem]` A list of inputs, each of which may be either an input text, output text, input image, or input audio object. - `role: :user | :assistant | :system | :developer` The role of the message input. One of `user`, `assistant`, `system`, or `developer`. - `:user` - `:assistant` - `:system` - `:developer` - `type: :message` The type of the message input. Always `message`. - `:message` - `labels: Array[String]` The labels to assign to each item in the evaluation. - `model: String` The model to use for the evaluation. Must support structured outputs. - `name: String` The name of the grader. - `passing_labels: Array[String]` The labels that indicate a passing result. Must be a subset of labels. - `type: :label_model` The object type, which is always `label_model`. - `:label_model` - `name: String` The name of the grader. - `type: :multi` The object type, which is always `multi`. - `:multi` ### Returns - `class GraderValidateResponse` - `grader: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more` The grader used for the fine-tuning job. - `class StringCheckGrader` A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. - `input: String` The input text. This may include template strings. - `name: String` The name of the grader. - `operation: :eq | :ne | :like | :ilike` The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`. - `:eq` - `:ne` - `:like` - `:ilike` - `reference: String` The reference text. This may include template strings. - `type: :string_check` The object type, which is always `string_check`. - `:string_check` - `class TextSimilarityGrader` A TextSimilarityGrader object which grades text based on similarity metrics. - `evaluation_metric: :cosine | :fuzzy_match | :bleu | 8 more` The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`, `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`, or `rouge_l`. - `:cosine` - `:fuzzy_match` - `:bleu` - `:gleu` - `:meteor` - `:rouge_1` - `:rouge_2` - `:rouge_3` - `:rouge_4` - `:rouge_5` - `:rouge_l` - `input: String` The text being graded. - `name: String` The name of the grader. - `reference: String` The text being graded against. - `type: :text_similarity` The type of grader. - `:text_similarity` - `class PythonGrader` A PythonGrader object that runs a python script on the input. - `name: String` The name of the grader. - `source: String` The source code of the python script. - `type: :python` The object type, which is always `python`. - `:python` - `image_tag: String` The image tag to use for the python script. - `class ScoreModelGrader` A ScoreModelGrader object that uses a model to assign a score to the input. - `input: Array[Input{ content, role, type}]` The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings. - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `text: String` The text input to the model. - `type: :input_text` The type of the input item. Always `input_text`. - `:input_text` - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `input_audio: InputAudio{ data, format_}` - `data: String` Base64-encoded audio data. - `format_: :mp3 | :wav` The format of the audio data. Currently supported formats are `mp3` and `wav`. - `:mp3` - `:wav` - `type: :input_audio` The type of the input item. Always `input_audio`. - `:input_audio` - `GraderInputs = Array[GraderInputItem]` A list of inputs, each of which may be either an input text, output text, input image, or input audio object. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `role: :user | :assistant | :system | :developer` The role of the message input. One of `user`, `assistant`, `system`, or `developer`. - `:user` - `:assistant` - `:system` - `:developer` - `type: :message` The type of the message input. Always `message`. - `:message` - `model: String` The model to use for the evaluation. - `name: String` The name of the grader. - `type: :score_model` The object type, which is always `score_model`. - `:score_model` - `range: Array[Float]` The range of the score. Defaults to `[0, 1]`. - `sampling_params: SamplingParams{ max_completions_tokens, reasoning_effort, seed, 2 more}` The sampling parameters for the model. - `max_completions_tokens: Integer` The maximum number of tokens the grader model may generate in its response. - `reasoning_effort: ReasoningEffort` Constrains effort on reasoning for [reasoning models](https://platform.openai.com/docs/guides/reasoning). Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - `:none` - `:minimal` - `:low` - `:medium` - `:high` - `:xhigh` - `seed: Integer` A seed value to initialize the randomness, during sampling. - `temperature: Float` A higher temperature increases randomness in the outputs. - `top_p: Float` An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - `class MultiGrader` A MultiGrader object combines the output of multiple graders to produce a single score. - `calculate_output: String` A formula to calculate the output based on grader results. - `graders: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more` A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. - `class StringCheckGrader` A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. - `class TextSimilarityGrader` A TextSimilarityGrader object which grades text based on similarity metrics. - `class PythonGrader` A PythonGrader object that runs a python script on the input. - `class ScoreModelGrader` A ScoreModelGrader object that uses a model to assign a score to the input. - `class LabelModelGrader` A LabelModelGrader object which uses a model to assign labels to each item in the evaluation. - `input: Array[Input{ content, role, type}]` - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `GraderInputs = Array[GraderInputItem]` A list of inputs, each of which may be either an input text, output text, input image, or input audio object. - `role: :user | :assistant | :system | :developer` The role of the message input. One of `user`, `assistant`, `system`, or `developer`. - `:user` - `:assistant` - `:system` - `:developer` - `type: :message` The type of the message input. Always `message`. - `:message` - `labels: Array[String]` The labels to assign to each item in the evaluation. - `model: String` The model to use for the evaluation. Must support structured outputs. - `name: String` The name of the grader. - `passing_labels: Array[String]` The labels that indicate a passing result. Must be a subset of labels. - `type: :label_model` The object type, which is always `label_model`. - `:label_model` - `name: String` The name of the grader. - `type: :multi` The object type, which is always `multi`. - `:multi` ### Example ```ruby require "openai" openai = OpenAI::Client.new(api_key: "My API Key") response = openai.fine_tuning.alpha.graders.validate( grader: {input: "input", name: "name", operation: :eq, reference: "reference", type: :string_check} ) puts(response) ``` #### Response ```json { "grader": { "input": "input", "name": "name", "operation": "eq", "reference": "reference", "type": "string_check" } } ``` ## Domain Types ### Grader Run Response - `class GraderRunResponse` - `metadata: Metadata{ errors, execution_time, name, 4 more}` - `errors: Errors{ formula_parse_error, invalid_variable_error, model_grader_parse_error, 11 more}` - `formula_parse_error: bool` - `invalid_variable_error: bool` - `model_grader_parse_error: bool` - `model_grader_refusal_error: bool` - `model_grader_server_error: bool` - `model_grader_server_error_details: String` - `other_error: bool` - `python_grader_runtime_error: bool` - `python_grader_runtime_error_details: String` - `python_grader_server_error: bool` - `python_grader_server_error_type: String` - `sample_parse_error: bool` - `truncated_observation_error: bool` - `unresponsive_reward_error: bool` - `execution_time: Float` - `name: String` - `sampled_model_name: String` - `scores: Hash[Symbol, untyped]` - `token_usage: Integer` - `type: String` - `model_grader_token_usage_per_model: Hash[Symbol, untyped]` - `reward: Float` - `sub_rewards: Hash[Symbol, untyped]` ### Grader Validate Response - `class GraderValidateResponse` - `grader: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more` The grader used for the fine-tuning job. - `class StringCheckGrader` A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. - `input: String` The input text. This may include template strings. - `name: String` The name of the grader. - `operation: :eq | :ne | :like | :ilike` The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`. - `:eq` - `:ne` - `:like` - `:ilike` - `reference: String` The reference text. This may include template strings. - `type: :string_check` The object type, which is always `string_check`. - `:string_check` - `class TextSimilarityGrader` A TextSimilarityGrader object which grades text based on similarity metrics. - `evaluation_metric: :cosine | :fuzzy_match | :bleu | 8 more` The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`, `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`, or `rouge_l`. - `:cosine` - `:fuzzy_match` - `:bleu` - `:gleu` - `:meteor` - `:rouge_1` - `:rouge_2` - `:rouge_3` - `:rouge_4` - `:rouge_5` - `:rouge_l` - `input: String` The text being graded. - `name: String` The name of the grader. - `reference: String` The text being graded against. - `type: :text_similarity` The type of grader. - `:text_similarity` - `class PythonGrader` A PythonGrader object that runs a python script on the input. - `name: String` The name of the grader. - `source: String` The source code of the python script. - `type: :python` The object type, which is always `python`. - `:python` - `image_tag: String` The image tag to use for the python script. - `class ScoreModelGrader` A ScoreModelGrader object that uses a model to assign a score to the input. - `input: Array[Input{ content, role, type}]` The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings. - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `text: String` The text input to the model. - `type: :input_text` The type of the input item. Always `input_text`. - `:input_text` - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `input_audio: InputAudio{ data, format_}` - `data: String` Base64-encoded audio data. - `format_: :mp3 | :wav` The format of the audio data. Currently supported formats are `mp3` and `wav`. - `:mp3` - `:wav` - `type: :input_audio` The type of the input item. Always `input_audio`. - `:input_audio` - `GraderInputs = Array[GraderInputItem]` A list of inputs, each of which may be either an input text, output text, input image, or input audio object. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `role: :user | :assistant | :system | :developer` The role of the message input. One of `user`, `assistant`, `system`, or `developer`. - `:user` - `:assistant` - `:system` - `:developer` - `type: :message` The type of the message input. Always `message`. - `:message` - `model: String` The model to use for the evaluation. - `name: String` The name of the grader. - `type: :score_model` The object type, which is always `score_model`. - `:score_model` - `range: Array[Float]` The range of the score. Defaults to `[0, 1]`. - `sampling_params: SamplingParams{ max_completions_tokens, reasoning_effort, seed, 2 more}` The sampling parameters for the model. - `max_completions_tokens: Integer` The maximum number of tokens the grader model may generate in its response. - `reasoning_effort: ReasoningEffort` Constrains effort on reasoning for [reasoning models](https://platform.openai.com/docs/guides/reasoning). Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - `:none` - `:minimal` - `:low` - `:medium` - `:high` - `:xhigh` - `seed: Integer` A seed value to initialize the randomness, during sampling. - `temperature: Float` A higher temperature increases randomness in the outputs. - `top_p: Float` An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - `class MultiGrader` A MultiGrader object combines the output of multiple graders to produce a single score. - `calculate_output: String` A formula to calculate the output based on grader results. - `graders: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more` A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. - `class StringCheckGrader` A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. - `class TextSimilarityGrader` A TextSimilarityGrader object which grades text based on similarity metrics. - `class PythonGrader` A PythonGrader object that runs a python script on the input. - `class ScoreModelGrader` A ScoreModelGrader object that uses a model to assign a score to the input. - `class LabelModelGrader` A LabelModelGrader object which uses a model to assign labels to each item in the evaluation. - `input: Array[Input{ content, role, type}]` - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - `String = String` A text input to the model. - `class ResponseInputText` A text input to the model. - `class OutputText` A text output from the model. - `text: String` The text output from the model. - `type: :output_text` The type of the output text. Always `output_text`. - `:output_text` - `class InputImage` An image input block used within EvalItem content arrays. - `image_url: String` The URL of the image input. - `type: :input_image` The type of the image input. Always `input_image`. - `:input_image` - `detail: String` The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - `class ResponseInputAudio` An audio input to the model. - `GraderInputs = Array[GraderInputItem]` A list of inputs, each of which may be either an input text, output text, input image, or input audio object. - `role: :user | :assistant | :system | :developer` The role of the message input. One of `user`, `assistant`, `system`, or `developer`. - `:user` - `:assistant` - `:system` - `:developer` - `type: :message` The type of the message input. Always `message`. - `:message` - `labels: Array[String]` The labels to assign to each item in the evaluation. - `model: String` The model to use for the evaluation. Must support structured outputs. - `name: String` The name of the grader. - `passing_labels: Array[String]` The labels that indicate a passing result. Must be a subset of labels. - `type: :label_model` The object type, which is always `label_model`. - `:label_model` - `name: String` The name of the grader. - `type: :multi` The object type, which is always `multi`. - `:multi`