API Reference

Fine Tuning

Alpha

Graders

Copy Markdown

Open in Claude

Open in ChatGPT

Open in Cursor

---

Copy Markdown

View as Markdown

Run grader

$ openai fine-tuning:alpha:graders run

POST/fine_tuning/alpha/graders/run

Run a grader.

ParametersExpand Collapse

--grader: StringCheckGrader { input, name, operation, 2 more } or TextSimilarityGrader { evaluation_metric, input, name, 2 more } or PythonGrader { name, source, type, image_tag } or 2 more

The grader used for the fine-tuning job.

--model-sample: string

The model sample to be evaluated. This value will be used to populate the sample namespace. See the guide for more details. The output_json variable will be populated if the model sample is a valid JSON string.

--item: optional unknown

The dataset item provided to the grader. This will be used to populate the item namespace. See the guide for more details.

ReturnsExpand Collapse

FineTuningAlphaGraderRunResponse: object { metadata, model_grader_token_usage_per_model, reward, sub_rewards }

metadata: object { errors, execution_time, name, 4 more }

errors: object { formula_parse_error, invalid_variable_error, model_grader_parse_error, 11 more }

formula_parse_error: boolean

invalid_variable_error: boolean

model_grader_parse_error: boolean

model_grader_refusal_error: boolean

model_grader_server_error: boolean

model_grader_server_error_details: string

other_error: boolean

python_grader_runtime_error: boolean

python_grader_runtime_error_details: string

python_grader_server_error: boolean

python_grader_server_error_type: string

sample_parse_error: boolean

truncated_observation_error: boolean

unresponsive_reward_error: boolean

execution_time: number

name: string

sampled_model_name: string

scores: map[unknown]

token_usage: number

type: string

model_grader_token_usage_per_model: map[unknown]

reward: number

sub_rewards: map[unknown]

Run grader

CLI Tool

HTTP

HTTP

TypeScript

Python

Java

Go

Ruby

CLI Tool

openai fine-tuning:alpha:graders run \
  --api-key 'My API Key' \
  --grader '{input: input, name: name, operation: eq, reference: reference, type: string_check}' \
  --model-sample model_sample

200 example

{
  "metadata": {
    "errors": {
      "formula_parse_error": true,
      "invalid_variable_error": true,
      "model_grader_parse_error": true,
      "model_grader_refusal_error": true,
      "model_grader_server_error": true,
      "model_grader_server_error_details": "model_grader_server_error_details",
      "other_error": true,
      "python_grader_runtime_error": true,
      "python_grader_runtime_error_details": "python_grader_runtime_error_details",
      "python_grader_server_error": true,
      "python_grader_server_error_type": "python_grader_server_error_type",
      "sample_parse_error": true,
      "truncated_observation_error": true,
      "unresponsive_reward_error": true
    },
    "execution_time": 0,
    "name": "name",
    "sampled_model_name": "sampled_model_name",
    "scores": {
      "foo": "bar"
    },
    "token_usage": 0,
    "type": "type"
  },
  "model_grader_token_usage_per_model": {
    "foo": "bar"
  },
  "reward": 0,
  "sub_rewards": {
    "foo": "bar"
  }
}

Returns Examples

200 example

{
  "metadata": {
    "errors": {
      "formula_parse_error": true,
      "invalid_variable_error": true,
      "model_grader_parse_error": true,
      "model_grader_refusal_error": true,
      "model_grader_server_error": true,
      "model_grader_server_error_details": "model_grader_server_error_details",
      "other_error": true,
      "python_grader_runtime_error": true,
      "python_grader_runtime_error_details": "python_grader_runtime_error_details",
      "python_grader_server_error": true,
      "python_grader_server_error_type": "python_grader_server_error_type",
      "sample_parse_error": true,
      "truncated_observation_error": true,
      "unresponsive_reward_error": true
    },
    "execution_time": 0,
    "name": "name",
    "sampled_model_name": "sampled_model_name",
    "scores": {
      "foo": "bar"
    },
    "token_usage": 0,
    "type": "type"
  },
  "model_grader_token_usage_per_model": {
    "foo": "bar"
  },
  "reward": 0,
  "sub_rewards": {
    "foo": "bar"
  }
}