API Reference

Fine Tuning

Alpha

Graders

Copy Markdown

Open in Claude

Open in ChatGPT

Open in Cursor

---

Copy Markdown

View as Markdown

Run grader

fine_tuning.alpha.graders.run(**kwargs) -> GraderRunResponse { metadata, model_grader_token_usage_per_model, reward, sub_rewards }

POST/fine_tuning/alpha/graders/run

Run a grader.

ParametersExpand Collapse

grader: StringCheckGrader { input, name, operation, 2 more } | TextSimilarityGrader { evaluation_metric, input, name, 2 more } | PythonGrader { name, source, type, image_tag } | 2 more

The grader used for the fine-tuning job.

model_sample: String

The model sample to be evaluated. This value will be used to populate the sample namespace. See the guide for more details. The output_json variable will be populated if the model sample is a valid JSON string.

item: untyped

The dataset item provided to the grader. This will be used to populate the item namespace. See the guide for more details.

ReturnsExpand Collapse

class GraderRunResponse { metadata, model_grader_token_usage_per_model, reward, sub_rewards }

metadata: Metadata{ errors, execution_time, name, 4 more}

model_grader_token_usage_per_model: Hash[Symbol, untyped]

reward: Float

sub_rewards: Hash[Symbol, untyped]

Run grader

Ruby

HTTP

Python

TypeScript

Go

Ruby

Java

CLI Tool

require "openai"

openai = OpenAI::Client.new(api_key: "My API Key")

response = openai.fine_tuning.alpha.graders.run(
  grader: {input: "input", name: "name", operation: :eq, reference: "reference", type: :string_check},
  model_sample: "model_sample"
)

puts(response)

200 example

{
  "metadata": {
    "errors": {
      "formula_parse_error": true,
      "invalid_variable_error": true,
      "model_grader_parse_error": true,
      "model_grader_refusal_error": true,
      "model_grader_server_error": true,
      "model_grader_server_error_details": "model_grader_server_error_details",
      "other_error": true,
      "python_grader_runtime_error": true,
      "python_grader_runtime_error_details": "python_grader_runtime_error_details",
      "python_grader_server_error": true,
      "python_grader_server_error_type": "python_grader_server_error_type",
      "sample_parse_error": true,
      "truncated_observation_error": true,
      "unresponsive_reward_error": true
    },
    "execution_time": 0,
    "name": "name",
    "sampled_model_name": "sampled_model_name",
    "scores": {
      "foo": "bar"
    },
    "token_usage": 0,
    "type": "type"
  },
  "model_grader_token_usage_per_model": {
    "foo": "bar"
  },
  "reward": 0,
  "sub_rewards": {
    "foo": "bar"
  }
}

Returns Examples

200 example

{
  "metadata": {
    "errors": {
      "formula_parse_error": true,
      "invalid_variable_error": true,
      "model_grader_parse_error": true,
      "model_grader_refusal_error": true,
      "model_grader_server_error": true,
      "model_grader_server_error_details": "model_grader_server_error_details",
      "other_error": true,
      "python_grader_runtime_error": true,
      "python_grader_runtime_error_details": "python_grader_runtime_error_details",
      "python_grader_server_error": true,
      "python_grader_server_error_type": "python_grader_server_error_type",
      "sample_parse_error": true,
      "truncated_observation_error": true,
      "unresponsive_reward_error": true
    },
    "execution_time": 0,
    "name": "name",
    "sampled_model_name": "sampled_model_name",
    "scores": {
      "foo": "bar"
    },
    "token_usage": 0,
    "type": "type"
  },
  "model_grader_token_usage_per_model": {
    "foo": "bar"
  },
  "reward": 0,
  "sub_rewards": {
    "foo": "bar"
  }
}