Realtime
ModelsExpand Collapse
AudioTranscription { language, model, prompt }
The language of the input audio. Supplying the input language in
ISO-639-1 (e.g. en) format
will improve accuracy and latency.
model?: (string & {}) | "whisper-1" | "gpt-4o-mini-transcribe" | "gpt-4o-mini-transcribe-2025-12-15" | 2 moreThe model to use for transcription. Current options are whisper-1, gpt-4o-mini-transcribe, gpt-4o-mini-transcribe-2025-12-15, gpt-4o-transcribe, and gpt-4o-transcribe-diarize. Use gpt-4o-transcribe-diarize when you need diarization with speaker labels.
The model to use for transcription. Current options are whisper-1, gpt-4o-mini-transcribe, gpt-4o-mini-transcribe-2025-12-15, gpt-4o-transcribe, and gpt-4o-transcribe-diarize. Use gpt-4o-transcribe-diarize when you need diarization with speaker labels.
An optional text to guide the model's style or continue a previous audio
segment.
For whisper-1, the prompt is a list of keywords.
For gpt-4o-transcribe models (excluding gpt-4o-transcribe-diarize), the prompt is a free text string, for example "expect words related to technology".
ConversationItem = RealtimeConversationItemSystemMessage { content, role, type, 3 more } | RealtimeConversationItemUserMessage { content, role, type, 3 more } | RealtimeConversationItemAssistantMessage { content, role, type, 3 more } | 6 moreA single item within a Realtime conversation.
A single item within a Realtime conversation.
RealtimeConversationItemSystemMessage { content, role, type, 3 more } A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.
A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.
RealtimeConversationItemUserMessage { content, role, type, 3 more } A user message item in a Realtime conversation.
A user message item in a Realtime conversation.
content: Array<Content>The content of the message.
The content of the message.
Base64-encoded audio bytes (for input_audio), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.
detail?: "auto" | "low" | "high"The detail level of the image (for input_image). auto will default to high.
The detail level of the image (for input_image). auto will default to high.
Base64-encoded image bytes (for input_image) as a data URI. For example data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA.... Supported formats are PNG and JPEG.
The unique ID of the item. This may be provided by the client or generated by the server.
RealtimeConversationItemAssistantMessage { content, role, type, 3 more } An assistant message item in a Realtime conversation.
An assistant message item in a Realtime conversation.
content: Array<Content>The content of the message.
The content of the message.
Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.
The unique ID of the item. This may be provided by the client or generated by the server.
RealtimeConversationItemFunctionCall { arguments, name, type, 4 more } A function call item in a Realtime conversation.
A function call item in a Realtime conversation.
The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example {"arg1": "value1", "arg2": 42}.
The unique ID of the item. This may be provided by the client or generated by the server.
RealtimeConversationItemFunctionCallOutput { call_id, output, type, 3 more } A function call output item in a Realtime conversation.
A function call output item in a Realtime conversation.
The output of the function call, this is free text and can contain any information or simply be empty.
The unique ID of the item. This may be provided by the client or generated by the server.
RealtimeMcpApprovalResponse { id, approval_request_id, approve, 2 more } A Realtime item responding to an MCP approval request.
A Realtime item responding to an MCP approval request.
RealtimeMcpListTools { server_label, tools, type, id } A Realtime item listing tools available on an MCP server.
A Realtime item listing tools available on an MCP server.
RealtimeMcpToolCall { id, arguments, name, 5 more } A Realtime item representing an invocation of a tool on an MCP server.
A Realtime item representing an invocation of a tool on an MCP server.
error?: RealtimeMcpProtocolError { code, message, type } | RealtimeMcpToolExecutionError { message, type } | RealtimeMcphttpError { code, message, type } | nullThe error from the tool call, if any.
The error from the tool call, if any.
ConversationItemAdded { event_id, item, type, previous_item_id } Sent by the server when an Item is added to the default Conversation. This can happen in several cases:
- When the client sends a
conversation.item.create event.
- When the input audio buffer is committed. In this case the item will be a user message containing the audio from the buffer.
- When the model is generating a Response. In this case the
conversation.item.added event will be sent when the model starts generating a specific Item, and thus it will not yet have any content (and status will be in_progress).
The event will include the full content of the Item (except when model is generating a Response) except for audio data, which can be retrieved separately with a conversation.item.retrieve event if necessary.
Sent by the server when an Item is added to the default Conversation. This can happen in several cases:
- When the client sends a
conversation.item.createevent. - When the input audio buffer is committed. In this case the item will be a user message containing the audio from the buffer.
- When the model is generating a Response. In this case the
conversation.item.addedevent will be sent when the model starts generating a specific Item, and thus it will not yet have any content (andstatuswill bein_progress).
The event will include the full content of the Item (except when model is generating a Response) except for audio data, which can be retrieved separately with a conversation.item.retrieve event if necessary.
ConversationItemCreateEvent { item, type, event_id, previous_item_id } Add a new Item to the Conversation's context, including messages, function
calls, and function call responses. This event can be used both to populate a
"history" of the conversation and to add new items mid-stream, but has the
current limitation that it cannot populate assistant audio messages.
If successful, the server will respond with a conversation.item.created
event, otherwise an error event will be sent.
Add a new Item to the Conversation's context, including messages, function calls, and function call responses. This event can be used both to populate a "history" of the conversation and to add new items mid-stream, but has the current limitation that it cannot populate assistant audio messages.
If successful, the server will respond with a conversation.item.created
event, otherwise an error event will be sent.
The ID of the preceding item after which the new item will be inserted. If not set, the new item will be appended to the end of the conversation.
If set to root, the new item will be added to the beginning of the conversation.
If set to an existing ID, it allows an item to be inserted mid-conversation. If the ID cannot be found, an error will be returned and the item will not be added.
ConversationItemCreatedEvent { event_id, item, type, previous_item_id } Returned when a conversation item is created. There are several scenarios that produce this event:
- The server is generating a Response, which if successful will produce
either one or two Items, which will be of type
message
(role assistant) or type function_call.
- The input audio buffer has been committed, either by the client or the
server (in
server_vad mode). The server will take the content of the
input audio buffer and add it to a new user message Item.
- The client has sent a
conversation.item.create event to add a new Item
to the Conversation.
Returned when a conversation item is created. There are several scenarios that produce this event:
- The server is generating a Response, which if successful will produce
either one or two Items, which will be of type
message(roleassistant) or typefunction_call. - The input audio buffer has been committed, either by the client or the
server (in
server_vadmode). The server will take the content of the input audio buffer and add it to a new user message Item. - The client has sent a
conversation.item.createevent to add a new Item to the Conversation.
ConversationItemDeleteEvent { item_id, type, event_id } Send this event when you want to remove any item from the conversation
history. The server will respond with a conversation.item.deleted event,
unless the item does not exist in the conversation history, in which case the
server will respond with an error.
Send this event when you want to remove any item from the conversation
history. The server will respond with a conversation.item.deleted event,
unless the item does not exist in the conversation history, in which case the
server will respond with an error.
ConversationItemDone { event_id, item, type, previous_item_id } Returned when a conversation item is finalized.
The event will include the full content of the Item except for audio data, which can be retrieved separately with a conversation.item.retrieve event if needed.
Returned when a conversation item is finalized.
The event will include the full content of the Item except for audio data, which can be retrieved separately with a conversation.item.retrieve event if needed.
ConversationItemInputAudioTranscriptionCompletedEvent { content_index, event_id, item_id, 4 more } This event is the output of audio transcription for user audio written to the
user audio buffer. Transcription begins when the input audio buffer is
committed by the client or server (when VAD is enabled). Transcription runs
asynchronously with Response creation, so this event may come before or after
the Response events.
Realtime API models accept audio natively, and thus input transcription is a
separate process run on a separate ASR (Automatic Speech Recognition) model.
The transcript may diverge somewhat from the model's interpretation, and
should be treated as a rough guide.
This event is the output of audio transcription for user audio written to the user audio buffer. Transcription begins when the input audio buffer is committed by the client or server (when VAD is enabled). Transcription runs asynchronously with Response creation, so this event may come before or after the Response events.
Realtime API models accept audio natively, and thus input transcription is a separate process run on a separate ASR (Automatic Speech Recognition) model. The transcript may diverge somewhat from the model's interpretation, and should be treated as a rough guide.
The event type, must be
conversation.item.input_audio_transcription.completed.
usage: TranscriptTextUsageTokens { input_tokens, output_tokens, total_tokens, 2 more } | TranscriptTextUsageDuration { seconds, type } Usage statistics for the transcription, this is billed according to the ASR model's pricing rather than the realtime model's pricing.
Usage statistics for the transcription, this is billed according to the ASR model's pricing rather than the realtime model's pricing.
ConversationItemInputAudioTranscriptionDeltaEvent { event_id, item_id, type, 3 more } Returned when the text value of an input audio transcription content part is updated with incremental transcription results.
Returned when the text value of an input audio transcription content part is updated with incremental transcription results.
The event type, must be conversation.item.input_audio_transcription.delta.
The log probabilities of the transcription. These can be enabled by configurating the session with "include": ["item.input_audio_transcription.logprobs"]. Each entry in the array corresponds a log probability of which token would be selected for this chunk of transcription. This can help to identify if it was possible there were multiple valid options for a given chunk of transcription.
The log probabilities of the transcription. These can be enabled by configurating the session with "include": ["item.input_audio_transcription.logprobs"]. Each entry in the array corresponds a log probability of which token would be selected for this chunk of transcription. This can help to identify if it was possible there were multiple valid options for a given chunk of transcription.
ConversationItemInputAudioTranscriptionFailedEvent { content_index, error, event_id, 2 more } Returned when input audio transcription is configured, and a transcription
request for a user message failed. These events are separate from other
error events so that the client can identify the related Item.
Returned when input audio transcription is configured, and a transcription
request for a user message failed. These events are separate from other
error events so that the client can identify the related Item.
ConversationItemRetrieveEvent { item_id, type, event_id } Send this event when you want to retrieve the server's representation of a specific item in the conversation history. This is useful, for example, to inspect user audio after noise cancellation and VAD.
The server will respond with a conversation.item.retrieved event,
unless the item does not exist in the conversation history, in which case the
server will respond with an error.
Send this event when you want to retrieve the server's representation of a specific item in the conversation history. This is useful, for example, to inspect user audio after noise cancellation and VAD.
The server will respond with a conversation.item.retrieved event,
unless the item does not exist in the conversation history, in which case the
server will respond with an error.
ConversationItemTruncateEvent { audio_end_ms, content_index, item_id, 2 more } Send this event to truncate a previous assistant message’s audio. The server
will produce audio faster than realtime, so this event is useful when the user
interrupts to truncate audio that has already been sent to the client but not
yet played. This will synchronize the server's understanding of the audio with
the client's playback.
Truncating audio will delete the server-side text transcript to ensure there
is not text in the context that hasn't been heard by the user.
If successful, the server will respond with a conversation.item.truncated
event.
Send this event to truncate a previous assistant message’s audio. The server will produce audio faster than realtime, so this event is useful when the user interrupts to truncate audio that has already been sent to the client but not yet played. This will synchronize the server's understanding of the audio with the client's playback.
Truncating audio will delete the server-side text transcript to ensure there is not text in the context that hasn't been heard by the user.
If successful, the server will respond with a conversation.item.truncated
event.
Inclusive duration up to which audio is truncated, in milliseconds. If the audio_end_ms is greater than the actual audio duration, the server will respond with an error.
ConversationItemTruncatedEvent { audio_end_ms, content_index, event_id, 2 more } Returned when an earlier assistant audio message item is truncated by the
client with a conversation.item.truncate event. This event is used to
synchronize the server's understanding of the audio with the client's playback.
This action will truncate the audio and remove the server-side text transcript
to ensure there is no text in the context that hasn't been heard by the user.
Returned when an earlier assistant audio message item is truncated by the
client with a conversation.item.truncate event. This event is used to
synchronize the server's understanding of the audio with the client's playback.
This action will truncate the audio and remove the server-side text transcript to ensure there is no text in the context that hasn't been heard by the user.
ConversationItemWithReference { id, arguments, call_id, 7 more } The item to add to the conversation.
The item to add to the conversation.
For an item of type (message | function_call | function_call_output)
this field allows the client to assign the unique ID of the item. It is
not required because the server will generate one if not provided.
For an item of type item_reference, this field is required and is a
reference to any item that has previously existed in the conversation.
The ID of the function call (for function_call and
function_call_output items). If passed on a function_call_output
item, the server will check that a function_call item with the same
ID exists in the conversation history.
content?: Array<Content>The content of the message, applicable for message items.
- Message items of role
system support only input_text content
- Message items of role
user support input_text and input_audio
content
- Message items of role
assistant support text content.
The content of the message, applicable for message items.
- Message items of role
systemsupport onlyinput_textcontent - Message items of role
usersupportinput_textandinput_audiocontent - Message items of role
assistantsupporttextcontent.
role?: "user" | "assistant" | "system"The role of the message sender (user, assistant, system), only
applicable for message items.
The role of the message sender (user, assistant, system), only
applicable for message items.
InputAudioBufferAppendEvent { audio, type, event_id } Send this event to append audio bytes to the input audio buffer. The audio
buffer is temporary storage you can write to and later commit. A "commit" will create a new
user message item in the conversation history from the buffer content and clear the buffer.
Input audio transcription (if enabled) will be generated when the buffer is committed.
If VAD is enabled the audio buffer is used to detect speech and the server will decide
when to commit. When Server VAD is disabled, you must commit the audio buffer
manually. Input audio noise reduction operates on writes to the audio buffer.
The client may choose how much audio to place in each event up to a maximum
of 15 MiB, for example streaming smaller chunks from the client may allow the
VAD to be more responsive. Unlike most other client events, the server will
not send a confirmation response to this event.
Send this event to append audio bytes to the input audio buffer. The audio buffer is temporary storage you can write to and later commit. A "commit" will create a new user message item in the conversation history from the buffer content and clear the buffer. Input audio transcription (if enabled) will be generated when the buffer is committed.
If VAD is enabled the audio buffer is used to detect speech and the server will decide when to commit. When Server VAD is disabled, you must commit the audio buffer manually. Input audio noise reduction operates on writes to the audio buffer.
The client may choose how much audio to place in each event up to a maximum of 15 MiB, for example streaming smaller chunks from the client may allow the VAD to be more responsive. Unlike most other client events, the server will not send a confirmation response to this event.
InputAudioBufferCommitEvent { type, event_id } Send this event to commit the user input audio buffer, which will create a new user message item in the conversation. This event will produce an error if the input audio buffer is empty. When in Server VAD mode, the client does not need to send this event, the server will commit the audio buffer automatically.
Committing the input audio buffer will trigger input audio transcription (if enabled in session configuration), but it will not create a response from the model. The server will respond with an input_audio_buffer.committed event.
Send this event to commit the user input audio buffer, which will create a new user message item in the conversation. This event will produce an error if the input audio buffer is empty. When in Server VAD mode, the client does not need to send this event, the server will commit the audio buffer automatically.
Committing the input audio buffer will trigger input audio transcription (if enabled in session configuration), but it will not create a response from the model. The server will respond with an input_audio_buffer.committed event.
InputAudioBufferCommittedEvent { event_id, item_id, type, previous_item_id } Returned when an input audio buffer is committed, either by the client or
automatically in server VAD mode. The item_id property is the ID of the user
message item that will be created, thus a conversation.item.created event
will also be sent to the client.
Returned when an input audio buffer is committed, either by the client or
automatically in server VAD mode. The item_id property is the ID of the user
message item that will be created, thus a conversation.item.created event
will also be sent to the client.
InputAudioBufferDtmfEventReceivedEvent { event, received_at, type } SIP Only: Returned when an DTMF event is received. A DTMF event is a message that
represents a telephone keypad press (0–9, *, #, A–D). The event property
is the keypad that the user press. The received_at is the UTC Unix Timestamp
that the server received the event.
SIP Only: Returned when an DTMF event is received. A DTMF event is a message that
represents a telephone keypad press (0–9, *, #, A–D). The event property
is the keypad that the user press. The received_at is the UTC Unix Timestamp
that the server received the event.
InputAudioBufferSpeechStartedEvent { audio_start_ms, event_id, item_id, type } Sent by the server when in server_vad mode to indicate that speech has been
detected in the audio buffer. This can happen any time audio is added to the
buffer (unless speech is already detected). The client may want to use this
event to interrupt audio playback or provide visual feedback to the user.
The client should expect to receive a input_audio_buffer.speech_stopped event
when speech stops. The item_id property is the ID of the user message item
that will be created when speech stops and will also be included in the
input_audio_buffer.speech_stopped event (unless the client manually commits
the audio buffer during VAD activation).
Sent by the server when in server_vad mode to indicate that speech has been
detected in the audio buffer. This can happen any time audio is added to the
buffer (unless speech is already detected). The client may want to use this
event to interrupt audio playback or provide visual feedback to the user.
The client should expect to receive a input_audio_buffer.speech_stopped event
when speech stops. The item_id property is the ID of the user message item
that will be created when speech stops and will also be included in the
input_audio_buffer.speech_stopped event (unless the client manually commits
the audio buffer during VAD activation).
InputAudioBufferSpeechStoppedEvent { audio_end_ms, event_id, item_id, type } Returned in server_vad mode when the server detects the end of speech in
the audio buffer. The server will also send an conversation.item.created
event with the user message item that is created from the audio buffer.
Returned in server_vad mode when the server detects the end of speech in
the audio buffer. The server will also send an conversation.item.created
event with the user message item that is created from the audio buffer.
InputAudioBufferTimeoutTriggered { audio_end_ms, audio_start_ms, event_id, 2 more } Returned when the Server VAD timeout is triggered for the input audio buffer. This is configured
with idle_timeout_ms in the turn_detection settings of the session, and it indicates that
there hasn't been any speech detected for the configured duration.
The audio_start_ms and audio_end_ms fields indicate the segment of audio after the last
model response up to the triggering time, as an offset from the beginning of audio written
to the input audio buffer. This means it demarcates the segment of audio that was silent and
the difference between the start and end values will roughly match the configured timeout.
The empty audio will be committed to the conversation as an input_audio item (there will be a
input_audio_buffer.committed event) and a model response will be generated. There may be speech
that didn't trigger VAD but is still detected by the model, so the model may respond with
something relevant to the conversation or a prompt to continue speaking.
Returned when the Server VAD timeout is triggered for the input audio buffer. This is configured
with idle_timeout_ms in the turn_detection settings of the session, and it indicates that
there hasn't been any speech detected for the configured duration.
The audio_start_ms and audio_end_ms fields indicate the segment of audio after the last
model response up to the triggering time, as an offset from the beginning of audio written
to the input audio buffer. This means it demarcates the segment of audio that was silent and
the difference between the start and end values will roughly match the configured timeout.
The empty audio will be committed to the conversation as an input_audio item (there will be a
input_audio_buffer.committed event) and a model response will be generated. There may be speech
that didn't trigger VAD but is still detected by the model, so the model may respond with
something relevant to the conversation or a prompt to continue speaking.
OutputAudioBufferClearEvent { type, event_id } WebRTC/SIP Only: Emit to cut off the current audio response. This will trigger the server to
stop generating audio and emit a output_audio_buffer.cleared event. This
event should be preceded by a response.cancel client event to stop the
generation of the current response.
Learn more.
WebRTC/SIP Only: Emit to cut off the current audio response. This will trigger the server to
stop generating audio and emit a output_audio_buffer.cleared event. This
event should be preceded by a response.cancel client event to stop the
generation of the current response.
Learn more.
RateLimitsUpdatedEvent { event_id, rate_limits, type } Emitted at the beginning of a Response to indicate the updated rate limits.
When a Response is created some tokens will be "reserved" for the output
tokens, the rate limits shown here reflect that reservation, which is then
adjusted accordingly once the Response is completed.
Emitted at the beginning of a Response to indicate the updated rate limits. When a Response is created some tokens will be "reserved" for the output tokens, the rate limits shown here reflect that reservation, which is then adjusted accordingly once the Response is completed.
RealtimeAudioConfigInput { format, noise_reduction, transcription, turn_detection }
noise_reduction?: NoiseReduction { type } Configuration for input audio noise reduction. This can be set to null to turn off.
Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model.
Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio.
Configuration for input audio noise reduction. This can be set to null to turn off.
Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model.
Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio.
Configuration for input audio transcription, defaults to off and can be set to null to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through the /audio/transcriptions endpoint and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service.
Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to null to turn off, in which case the client must manually trigger model response.
Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.
Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency.
RealtimeAudioConfigOutput { format, speed, voice }
The speed of the model's spoken response as a multiple of the original speed. 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress.
This parameter is a post-processing adjustment to the audio after it is generated, it's also possible to prompt the model to speak faster or slower.
voice?: string | "alloy" | "ash" | "ballad" | 7 more | ID { id } The voice the model uses to respond. Supported built-in voices are
alloy, ash, ballad, coral, echo, sage, shimmer, verse,
marin, and cedar. You may also provide a custom voice object with
an id, for example { "id": "voice_1234" }. Voice cannot be changed
during the session once the model has responded with audio at least once.
We recommend marin and cedar for best quality.
The voice the model uses to respond. Supported built-in voices are
alloy, ash, ballad, coral, echo, sage, shimmer, verse,
marin, and cedar. You may also provide a custom voice object with
an id, for example { "id": "voice_1234" }. Voice cannot be changed
during the session once the model has responded with audio at least once.
We recommend marin and cedar for best quality.
RealtimeAudioInputTurnDetection = ServerVad { type, create_response, idle_timeout_ms, 4 more } | SemanticVad { type, create_response, eagerness, interrupt_response } | nullConfiguration for turn detection, ether Server VAD or Semantic VAD. This can be set to null to turn off, in which case the client must manually trigger model response.
Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.
Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency.
Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to null to turn off, in which case the client must manually trigger model response.
Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.
Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency.
ServerVad { type, create_response, idle_timeout_ms, 4 more } Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence.
Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence.
Whether or not to automatically generate a response when a VAD stop event occurs. If interrupt_response is set to false this may fail to create a response if the model is already responding.
If both create_response and interrupt_response are set to false, the model will never respond automatically but VAD events will still be emitted.
Optional timeout after which a model response will be triggered automatically. This is useful for situations in which a long pause from the user is unexpected, such as a phone call. The model will effectively prompt the user to continue the conversation based on the current context.
The timeout value will be applied after the last model response's audio has finished playing,
i.e. it's set to the response.done time plus audio playback duration.
An input_audio_buffer.timeout_triggered event (plus events
associated with the Response) will be emitted when the timeout is reached.
Idle timeout is currently only supported for server_vad mode.
Whether or not to automatically interrupt (cancel) any ongoing response with output to the default
conversation (i.e. conversation of auto) when a VAD start event occurs. If true then the response will be cancelled, otherwise it will continue until complete.
If both create_response and interrupt_response are set to false, the model will never respond automatically but VAD events will still be emitted.
Used only for server_vad mode. Amount of audio to include before the VAD detected speech (in
milliseconds). Defaults to 300ms.
SemanticVad { type, create_response, eagerness, interrupt_response } Server-side semantic turn detection which uses a model to determine when the user has finished speaking.
Server-side semantic turn detection which uses a model to determine when the user has finished speaking.
Whether or not to automatically generate a response when a VAD stop event occurs.
eagerness?: "low" | "medium" | "high" | "auto"Used only for semantic_vad mode. The eagerness of the model to respond. low will wait longer for the user to continue speaking, high will respond more quickly. auto is the default and is equivalent to medium. low, medium, and high have max timeouts of 8s, 4s, and 2s respectively.
Used only for semantic_vad mode. The eagerness of the model to respond. low will wait longer for the user to continue speaking, high will respond more quickly. auto is the default and is equivalent to medium. low, medium, and high have max timeouts of 8s, 4s, and 2s respectively.
RealtimeClientEvent = ConversationItemCreateEvent { item, type, event_id, previous_item_id } | ConversationItemDeleteEvent { item_id, type, event_id } | ConversationItemRetrieveEvent { item_id, type, event_id } | 8 moreA realtime client event.
A realtime client event.
ConversationItemCreateEvent { item, type, event_id, previous_item_id } Add a new Item to the Conversation's context, including messages, function
calls, and function call responses. This event can be used both to populate a
"history" of the conversation and to add new items mid-stream, but has the
current limitation that it cannot populate assistant audio messages.
If successful, the server will respond with a conversation.item.created
event, otherwise an error event will be sent.
Add a new Item to the Conversation's context, including messages, function calls, and function call responses. This event can be used both to populate a "history" of the conversation and to add new items mid-stream, but has the current limitation that it cannot populate assistant audio messages.
If successful, the server will respond with a conversation.item.created
event, otherwise an error event will be sent.
The ID of the preceding item after which the new item will be inserted. If not set, the new item will be appended to the end of the conversation.
If set to root, the new item will be added to the beginning of the conversation.
If set to an existing ID, it allows an item to be inserted mid-conversation. If the ID cannot be found, an error will be returned and the item will not be added.
ConversationItemDeleteEvent { item_id, type, event_id } Send this event when you want to remove any item from the conversation
history. The server will respond with a conversation.item.deleted event,
unless the item does not exist in the conversation history, in which case the
server will respond with an error.
Send this event when you want to remove any item from the conversation
history. The server will respond with a conversation.item.deleted event,
unless the item does not exist in the conversation history, in which case the
server will respond with an error.
ConversationItemRetrieveEvent { item_id, type, event_id } Send this event when you want to retrieve the server's representation of a specific item in the conversation history. This is useful, for example, to inspect user audio after noise cancellation and VAD.
The server will respond with a conversation.item.retrieved event,
unless the item does not exist in the conversation history, in which case the
server will respond with an error.
Send this event when you want to retrieve the server's representation of a specific item in the conversation history. This is useful, for example, to inspect user audio after noise cancellation and VAD.
The server will respond with a conversation.item.retrieved event,
unless the item does not exist in the conversation history, in which case the
server will respond with an error.
ConversationItemTruncateEvent { audio_end_ms, content_index, item_id, 2 more } Send this event to truncate a previous assistant message’s audio. The server
will produce audio faster than realtime, so this event is useful when the user
interrupts to truncate audio that has already been sent to the client but not
yet played. This will synchronize the server's understanding of the audio with
the client's playback.
Truncating audio will delete the server-side text transcript to ensure there
is not text in the context that hasn't been heard by the user.
If successful, the server will respond with a conversation.item.truncated
event.
Send this event to truncate a previous assistant message’s audio. The server will produce audio faster than realtime, so this event is useful when the user interrupts to truncate audio that has already been sent to the client but not yet played. This will synchronize the server's understanding of the audio with the client's playback.
Truncating audio will delete the server-side text transcript to ensure there is not text in the context that hasn't been heard by the user.
If successful, the server will respond with a conversation.item.truncated
event.
Inclusive duration up to which audio is truncated, in milliseconds. If the audio_end_ms is greater than the actual audio duration, the server will respond with an error.
InputAudioBufferAppendEvent { audio, type, event_id } Send this event to append audio bytes to the input audio buffer. The audio
buffer is temporary storage you can write to and later commit. A "commit" will create a new
user message item in the conversation history from the buffer content and clear the buffer.
Input audio transcription (if enabled) will be generated when the buffer is committed.
If VAD is enabled the audio buffer is used to detect speech and the server will decide
when to commit. When Server VAD is disabled, you must commit the audio buffer
manually. Input audio noise reduction operates on writes to the audio buffer.
The client may choose how much audio to place in each event up to a maximum
of 15 MiB, for example streaming smaller chunks from the client may allow the
VAD to be more responsive. Unlike most other client events, the server will
not send a confirmation response to this event.
Send this event to append audio bytes to the input audio buffer. The audio buffer is temporary storage you can write to and later commit. A "commit" will create a new user message item in the conversation history from the buffer content and clear the buffer. Input audio transcription (if enabled) will be generated when the buffer is committed.
If VAD is enabled the audio buffer is used to detect speech and the server will decide when to commit. When Server VAD is disabled, you must commit the audio buffer manually. Input audio noise reduction operates on writes to the audio buffer.
The client may choose how much audio to place in each event up to a maximum of 15 MiB, for example streaming smaller chunks from the client may allow the VAD to be more responsive. Unlike most other client events, the server will not send a confirmation response to this event.
InputAudioBufferClearEvent { type, event_id } Send this event to clear the audio bytes in the buffer. The server will
respond with an input_audio_buffer.cleared event.
Send this event to clear the audio bytes in the buffer. The server will
respond with an input_audio_buffer.cleared event.
OutputAudioBufferClearEvent { type, event_id } WebRTC/SIP Only: Emit to cut off the current audio response. This will trigger the server to
stop generating audio and emit a output_audio_buffer.cleared event. This
event should be preceded by a response.cancel client event to stop the
generation of the current response.
Learn more.
WebRTC/SIP Only: Emit to cut off the current audio response. This will trigger the server to
stop generating audio and emit a output_audio_buffer.cleared event. This
event should be preceded by a response.cancel client event to stop the
generation of the current response.
Learn more.
InputAudioBufferCommitEvent { type, event_id } Send this event to commit the user input audio buffer, which will create a new user message item in the conversation. This event will produce an error if the input audio buffer is empty. When in Server VAD mode, the client does not need to send this event, the server will commit the audio buffer automatically.
Committing the input audio buffer will trigger input audio transcription (if enabled in session configuration), but it will not create a response from the model. The server will respond with an input_audio_buffer.committed event.
Send this event to commit the user input audio buffer, which will create a new user message item in the conversation. This event will produce an error if the input audio buffer is empty. When in Server VAD mode, the client does not need to send this event, the server will commit the audio buffer automatically.
Committing the input audio buffer will trigger input audio transcription (if enabled in session configuration), but it will not create a response from the model. The server will respond with an input_audio_buffer.committed event.
ResponseCancelEvent { type, event_id, response_id } Send this event to cancel an in-progress response. The server will respond
with a response.done event with a status of response.status=cancelled. If
there is no response to cancel, the server will respond with an error. It's safe
to call response.cancel even if no response is in progress, an error will be
returned the session will remain unaffected.
Send this event to cancel an in-progress response. The server will respond
with a response.done event with a status of response.status=cancelled. If
there is no response to cancel, the server will respond with an error. It's safe
to call response.cancel even if no response is in progress, an error will be
returned the session will remain unaffected.
ResponseCreateEvent { type, event_id, response } This event instructs the server to create a Response, which means triggering
model inference. When in Server VAD mode, the server will create Responses
automatically.
A Response will include at least one Item, and may have two, in which case
the second will be a function call. These Items will be appended to the
conversation history by default.
The server will respond with a response.created event, events for Items
and content created, and finally a response.done event to indicate the
Response is complete.
The response.create event includes inference configuration like
instructions and tools. If these are set, they will override the Session's
configuration for this Response only.
Responses can be created out-of-band of the default Conversation, meaning that they can
have arbitrary input, and it's possible to disable writing the output to the Conversation.
Only one Response can write to the default Conversation at a time, but otherwise multiple
Responses can be created in parallel. The metadata field is a good way to disambiguate
multiple simultaneous Responses.
Clients can set conversation to none to create a Response that does not write to the default
Conversation. Arbitrary input can be provided with the input field, which is an array accepting
raw Items and references to existing Items.
This event instructs the server to create a Response, which means triggering model inference. When in Server VAD mode, the server will create Responses automatically.
A Response will include at least one Item, and may have two, in which case the second will be a function call. These Items will be appended to the conversation history by default.
The server will respond with a response.created event, events for Items
and content created, and finally a response.done event to indicate the
Response is complete.
The response.create event includes inference configuration like
instructions and tools. If these are set, they will override the Session's
configuration for this Response only.
Responses can be created out-of-band of the default Conversation, meaning that they can
have arbitrary input, and it's possible to disable writing the output to the Conversation.
Only one Response can write to the default Conversation at a time, but otherwise multiple
Responses can be created in parallel. The metadata field is a good way to disambiguate
multiple simultaneous Responses.
Clients can set conversation to none to create a Response that does not write to the default
Conversation. Arbitrary input can be provided with the input field, which is an array accepting
raw Items and references to existing Items.
SessionUpdateEvent { session, type, event_id } Send this event to update the session’s configuration.
The client may send this event at any time to update any field
except for voice and model. voice can be updated only if there have been no other audio outputs yet.
When the server receives a session.update, it will respond
with a session.updated event showing the full, effective configuration.
Only the fields that are present in the session.update are updated. To clear a field like
instructions, pass an empty string. To clear a field like tools, pass an empty array.
To clear a field like turn_detection, pass null.
Send this event to update the session’s configuration.
The client may send this event at any time to update any field
except for voice and model. voice can be updated only if there have been no other audio outputs yet.
When the server receives a session.update, it will respond
with a session.updated event showing the full, effective configuration.
Only the fields that are present in the session.update are updated. To clear a field like
instructions, pass an empty string. To clear a field like tools, pass an empty array.
To clear a field like turn_detection, pass null.
session: RealtimeSessionCreateRequest { type, audio, include, 9 more } | RealtimeTranscriptionSessionCreateRequest { type, audio, include } Update the Realtime session. Choose either a realtime
session or a transcription session.
Update the Realtime session. Choose either a realtime session or a transcription session.
RealtimeSessionCreateRequest { type, audio, include, 9 more } Realtime session object configuration.
Realtime session object configuration.
Additional fields to include in server outputs.
item.input_audio_transcription.logprobs: Include logprobs for input audio transcription.
The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior.
Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.
max_output_tokens?: number | "inf"Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
model?: (string & {}) | "gpt-realtime" | "gpt-realtime-1.5" | "gpt-realtime-2025-08-28" | 13 moreThe Realtime model used for this session.
The Realtime model used for this session.
output_modalities?: Array<"text" | "audio">The set of modalities the model can respond with. It defaults to ["audio"], indicating
that the model will respond with audio plus a transcript. ["text"] can be used to make
the model respond with text only. It is not possible to request both text and audio at the same time.
The set of modalities the model can respond with. It defaults to ["audio"], indicating
that the model will respond with audio plus a transcript. ["text"] can be used to make
the model respond with text only. It is not possible to request both text and audio at the same time.
Reference to a prompt template and its variables. Learn more.
Realtime API can write session traces to the Traces Dashboard. Set to null to disable tracing. Once tracing is enabled for a session, the configuration cannot be modified.
auto will create a trace for the session with default values for the
workflow name, group id, and metadata.
When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs.
Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost.
Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate.
Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit.
RealtimeConversationItemAssistantMessage { content, role, type, 3 more } An assistant message item in a Realtime conversation.
An assistant message item in a Realtime conversation.
content: Array<Content>The content of the message.
The content of the message.
Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.
The unique ID of the item. This may be provided by the client or generated by the server.
RealtimeConversationItemFunctionCall { arguments, name, type, 4 more } A function call item in a Realtime conversation.
A function call item in a Realtime conversation.
The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example {"arg1": "value1", "arg2": 42}.
The unique ID of the item. This may be provided by the client or generated by the server.
RealtimeConversationItemFunctionCallOutput { call_id, output, type, 3 more } A function call output item in a Realtime conversation.
A function call output item in a Realtime conversation.
The output of the function call, this is free text and can contain any information or simply be empty.
The unique ID of the item. This may be provided by the client or generated by the server.
RealtimeConversationItemSystemMessage { content, role, type, 3 more } A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.
A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.
RealtimeConversationItemUserMessage { content, role, type, 3 more } A user message item in a Realtime conversation.
A user message item in a Realtime conversation.
content: Array<Content>The content of the message.
The content of the message.
Base64-encoded audio bytes (for input_audio), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.
detail?: "auto" | "low" | "high"The detail level of the image (for input_image). auto will default to high.
The detail level of the image (for input_image). auto will default to high.
Base64-encoded image bytes (for input_image) as a data URI. For example data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA.... Supported formats are PNG and JPEG.
The unique ID of the item. This may be provided by the client or generated by the server.
RealtimeMcpToolCall { id, arguments, name, 5 more } A Realtime item representing an invocation of a tool on an MCP server.
A Realtime item representing an invocation of a tool on an MCP server.
error?: RealtimeMcpProtocolError { code, message, type } | RealtimeMcpToolExecutionError { message, type } | RealtimeMcphttpError { code, message, type } | nullThe error from the tool call, if any.
The error from the tool call, if any.
RealtimeResponse { id, audio, conversation_id, 8 more } The response resource.
The response resource.
audio?: Audio { output } Configuration for audio output.
Configuration for audio output.
output?: Output { format, voice }
voice?: (string & {}) | "alloy" | "ash" | "ballad" | 7 moreThe voice the model uses to respond. Voice cannot be changed during the
session once the model has responded with audio at least once. Current
voice options are alloy, ash, ballad, coral, echo, sage,
shimmer, verse, marin, and cedar. We recommend marin and cedar for
best quality.
The voice the model uses to respond. Voice cannot be changed during the
session once the model has responded with audio at least once. Current
voice options are alloy, ash, ballad, coral, echo, sage,
shimmer, verse, marin, and cedar. We recommend marin and cedar for
best quality.
Which conversation the response is added to, determined by the conversation
field in the response.create event. If auto, the response will be added to
the default conversation and the value of conversation_id will be an id like
conv_1234. If none, the response will not be added to any conversation and
the value of conversation_id will be null. If responses are being triggered
automatically by VAD the response will be added to the default conversation
max_output_tokens?: number | "inf"Maximum number of output tokens for a single assistant response,
inclusive of tool calls, that was used in this response.
Maximum number of output tokens for a single assistant response, inclusive of tool calls, that was used in this response.
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
The list of output items generated by the response.
The list of output items generated by the response.
RealtimeConversationItemSystemMessage { content, role, type, 3 more } A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.
A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.
RealtimeConversationItemUserMessage { content, role, type, 3 more } A user message item in a Realtime conversation.
A user message item in a Realtime conversation.
content: Array<Content>The content of the message.
The content of the message.
Base64-encoded audio bytes (for input_audio), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.
detail?: "auto" | "low" | "high"The detail level of the image (for input_image). auto will default to high.
The detail level of the image (for input_image). auto will default to high.
Base64-encoded image bytes (for input_image) as a data URI. For example data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA.... Supported formats are PNG and JPEG.
The unique ID of the item. This may be provided by the client or generated by the server.
RealtimeConversationItemAssistantMessage { content, role, type, 3 more } An assistant message item in a Realtime conversation.
An assistant message item in a Realtime conversation.
content: Array<Content>The content of the message.
The content of the message.
Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.
The unique ID of the item. This may be provided by the client or generated by the server.
RealtimeConversationItemFunctionCall { arguments, name, type, 4 more } A function call item in a Realtime conversation.
A function call item in a Realtime conversation.
The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example {"arg1": "value1", "arg2": 42}.
The unique ID of the item. This may be provided by the client or generated by the server.
RealtimeConversationItemFunctionCallOutput { call_id, output, type, 3 more } A function call output item in a Realtime conversation.
A function call output item in a Realtime conversation.
The output of the function call, this is free text and can contain any information or simply be empty.
The unique ID of the item. This may be provided by the client or generated by the server.
RealtimeMcpApprovalResponse { id, approval_request_id, approve, 2 more } A Realtime item responding to an MCP approval request.
A Realtime item responding to an MCP approval request.
RealtimeMcpListTools { server_label, tools, type, id } A Realtime item listing tools available on an MCP server.
A Realtime item listing tools available on an MCP server.
RealtimeMcpToolCall { id, arguments, name, 5 more } A Realtime item representing an invocation of a tool on an MCP server.
A Realtime item representing an invocation of a tool on an MCP server.
error?: RealtimeMcpProtocolError { code, message, type } | RealtimeMcpToolExecutionError { message, type } | RealtimeMcphttpError { code, message, type } | nullThe error from the tool call, if any.
The error from the tool call, if any.
output_modalities?: Array<"text" | "audio">The set of modalities the model used to respond, currently the only possible values are
[\"audio\"], [\"text\"]. Audio output always include a text transcript. Setting the
output to mode text will disable audio output from the model.
The set of modalities the model used to respond, currently the only possible values are
[\"audio\"], [\"text\"]. Audio output always include a text transcript. Setting the
output to mode text will disable audio output from the model.
status?: "completed" | "cancelled" | "failed" | 2 moreThe final status of the response (completed, cancelled, failed, or
incomplete, in_progress).
The final status of the response (completed, cancelled, failed, or
incomplete, in_progress).
RealtimeResponseCreateAudioOutput { output } Configuration for audio input and output.
Configuration for audio input and output.
output?: Output { format, voice }
voice?: string | "alloy" | "ash" | "ballad" | 7 more | ID { id } The voice the model uses to respond. Supported built-in voices are
alloy, ash, ballad, coral, echo, sage, shimmer, verse,
marin, and cedar. You may also provide a custom voice object with
an id, for example { "id": "voice_1234" }. Voice cannot be changed
during the session once the model has responded with audio at least once.
We recommend marin and cedar for best quality.
The voice the model uses to respond. Supported built-in voices are
alloy, ash, ballad, coral, echo, sage, shimmer, verse,
marin, and cedar. You may also provide a custom voice object with
an id, for example { "id": "voice_1234" }. Voice cannot be changed
during the session once the model has responded with audio at least once.
We recommend marin and cedar for best quality.
RealtimeResponseCreateMcpTool { server_label, type, allowed_tools, 7 more } Give the model access to additional tools via remote Model Context Protocol
(MCP) servers. Learn more about MCP.
Give the model access to additional tools via remote Model Context Protocol (MCP) servers. Learn more about MCP.
allowed_tools?: Array<string> | McpToolFilter { read_only, tool_names } | nullList of allowed tool names or a filter object.
List of allowed tool names or a filter object.
McpToolFilter { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
An OAuth access token that can be used with a remote MCP server, either with a custom MCP server URL or a service connector. Your application must handle the OAuth authorization flow and provide the token here.
connector_id?: "connector_dropbox" | "connector_gmail" | "connector_googlecalendar" | 5 moreIdentifier for service connectors, like those available in ChatGPT. One of
server_url or connector_id must be provided. Learn more about service
connectors here.
Currently supported connector_id values are:
- Dropbox:
connector_dropbox
- Gmail:
connector_gmail
- Google Calendar:
connector_googlecalendar
- Google Drive:
connector_googledrive
- Microsoft Teams:
connector_microsoftteams
- Outlook Calendar:
connector_outlookcalendar
- Outlook Email:
connector_outlookemail
- SharePoint:
connector_sharepoint
Identifier for service connectors, like those available in ChatGPT. One of
server_url or connector_id must be provided. Learn more about service
connectors here.
Currently supported connector_id values are:
- Dropbox:
connector_dropbox - Gmail:
connector_gmail - Google Calendar:
connector_googlecalendar - Google Drive:
connector_googledrive - Microsoft Teams:
connector_microsoftteams - Outlook Calendar:
connector_outlookcalendar - Outlook Email:
connector_outlookemail - SharePoint:
connector_sharepoint
Optional HTTP headers to send to the MCP server. Use for authentication or other purposes.
require_approval?: McpToolApprovalFilter { always, never } | "always" | "never" | nullSpecify which of the MCP server's tools require approval.
Specify which of the MCP server's tools require approval.
McpToolApprovalFilter { always, never } Specify which of the MCP server's tools require approval. Can be
always, never, or a filter object associated with tools
that require approval.
Specify which of the MCP server's tools require approval. Can be
always, never, or a filter object associated with tools
that require approval.
always?: Always { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
never?: Never { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
RealtimeResponseCreateParams { audio, conversation, input, 7 more } Create a new Realtime response with these parameters
Create a new Realtime response with these parameters
conversation?: (string & {}) | "auto" | "none"Controls which conversation the response is added to. Currently supports
auto and none, with auto as the default value. The auto value
means that the contents of the response will be added to the default
conversation. Set this to none to create an out-of-band response which
will not add items to default conversation.
Controls which conversation the response is added to. Currently supports
auto and none, with auto as the default value. The auto value
means that the contents of the response will be added to the default
conversation. Set this to none to create an out-of-band response which
will not add items to default conversation.
Input items to include in the prompt for the model. Using this field
creates a new context for this Response instead of using the default
conversation. An empty array [] will clear the context for this Response.
Note that this can include references to items that previously appeared in the session
using their id.
Input items to include in the prompt for the model. Using this field
creates a new context for this Response instead of using the default
conversation. An empty array [] will clear the context for this Response.
Note that this can include references to items that previously appeared in the session
using their id.
RealtimeConversationItemSystemMessage { content, role, type, 3 more } A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.
A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.
RealtimeConversationItemUserMessage { content, role, type, 3 more } A user message item in a Realtime conversation.
A user message item in a Realtime conversation.
content: Array<Content>The content of the message.
The content of the message.
Base64-encoded audio bytes (for input_audio), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.
detail?: "auto" | "low" | "high"The detail level of the image (for input_image). auto will default to high.
The detail level of the image (for input_image). auto will default to high.
Base64-encoded image bytes (for input_image) as a data URI. For example data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA.... Supported formats are PNG and JPEG.
The unique ID of the item. This may be provided by the client or generated by the server.
RealtimeConversationItemAssistantMessage { content, role, type, 3 more } An assistant message item in a Realtime conversation.
An assistant message item in a Realtime conversation.
content: Array<Content>The content of the message.
The content of the message.
Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.
The unique ID of the item. This may be provided by the client or generated by the server.
RealtimeConversationItemFunctionCall { arguments, name, type, 4 more } A function call item in a Realtime conversation.
A function call item in a Realtime conversation.
The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example {"arg1": "value1", "arg2": 42}.
The unique ID of the item. This may be provided by the client or generated by the server.
RealtimeConversationItemFunctionCallOutput { call_id, output, type, 3 more } A function call output item in a Realtime conversation.
A function call output item in a Realtime conversation.
The output of the function call, this is free text and can contain any information or simply be empty.
The unique ID of the item. This may be provided by the client or generated by the server.
RealtimeMcpApprovalResponse { id, approval_request_id, approve, 2 more } A Realtime item responding to an MCP approval request.
A Realtime item responding to an MCP approval request.
RealtimeMcpListTools { server_label, tools, type, id } A Realtime item listing tools available on an MCP server.
A Realtime item listing tools available on an MCP server.
RealtimeMcpToolCall { id, arguments, name, 5 more } A Realtime item representing an invocation of a tool on an MCP server.
A Realtime item representing an invocation of a tool on an MCP server.
error?: RealtimeMcpProtocolError { code, message, type } | RealtimeMcpToolExecutionError { message, type } | RealtimeMcphttpError { code, message, type } | nullThe error from the tool call, if any.
The error from the tool call, if any.
The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior.
Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.
max_output_tokens?: number | "inf"Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
output_modalities?: Array<"text" | "audio">The set of modalities the model used to respond, currently the only possible values are
[\"audio\"], [\"text\"]. Audio output always include a text transcript. Setting the
output to mode text will disable audio output from the model.
The set of modalities the model used to respond, currently the only possible values are
[\"audio\"], [\"text\"]. Audio output always include a text transcript. Setting the
output to mode text will disable audio output from the model.
Reference to a prompt template and its variables. Learn more.
tool_choice?: ToolChoiceOptions | ToolChoiceFunction { name, type } | ToolChoiceMcp { server_label, type, name } How the model chooses tools. Provide one of the string modes or force a specific
function/MCP tool.
How the model chooses tools. Provide one of the string modes or force a specific function/MCP tool.
ToolChoiceOptions = "none" | "auto" | "required"Controls which (if any) tool is called by the model.
none means the model will not call any tool and instead generates a message.
auto means the model can pick between generating a message or calling one or
more tools.
required means the model must call one or more tools.
Controls which (if any) tool is called by the model.
none means the model will not call any tool and instead generates a message.
auto means the model can pick between generating a message or calling one or
more tools.
required means the model must call one or more tools.
tools?: Array<RealtimeFunctionTool { description, name, parameters, type } | RealtimeResponseCreateMcpTool { server_label, type, allowed_tools, 7 more } >Tools available to the model.
Tools available to the model.
RealtimeFunctionTool { description, name, parameters, type }
RealtimeResponseCreateMcpTool { server_label, type, allowed_tools, 7 more } Give the model access to additional tools via remote Model Context Protocol
(MCP) servers. Learn more about MCP.
Give the model access to additional tools via remote Model Context Protocol (MCP) servers. Learn more about MCP.
allowed_tools?: Array<string> | McpToolFilter { read_only, tool_names } | nullList of allowed tool names or a filter object.
List of allowed tool names or a filter object.
McpToolFilter { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
An OAuth access token that can be used with a remote MCP server, either with a custom MCP server URL or a service connector. Your application must handle the OAuth authorization flow and provide the token here.
connector_id?: "connector_dropbox" | "connector_gmail" | "connector_googlecalendar" | 5 moreIdentifier for service connectors, like those available in ChatGPT. One of
server_url or connector_id must be provided. Learn more about service
connectors here.
Currently supported connector_id values are:
- Dropbox:
connector_dropbox
- Gmail:
connector_gmail
- Google Calendar:
connector_googlecalendar
- Google Drive:
connector_googledrive
- Microsoft Teams:
connector_microsoftteams
- Outlook Calendar:
connector_outlookcalendar
- Outlook Email:
connector_outlookemail
- SharePoint:
connector_sharepoint
Identifier for service connectors, like those available in ChatGPT. One of
server_url or connector_id must be provided. Learn more about service
connectors here.
Currently supported connector_id values are:
- Dropbox:
connector_dropbox - Gmail:
connector_gmail - Google Calendar:
connector_googlecalendar - Google Drive:
connector_googledrive - Microsoft Teams:
connector_microsoftteams - Outlook Calendar:
connector_outlookcalendar - Outlook Email:
connector_outlookemail - SharePoint:
connector_sharepoint
Optional HTTP headers to send to the MCP server. Use for authentication or other purposes.
require_approval?: McpToolApprovalFilter { always, never } | "always" | "never" | nullSpecify which of the MCP server's tools require approval.
Specify which of the MCP server's tools require approval.
McpToolApprovalFilter { always, never } Specify which of the MCP server's tools require approval. Can be
always, never, or a filter object associated with tools
that require approval.
Specify which of the MCP server's tools require approval. Can be
always, never, or a filter object associated with tools
that require approval.
always?: Always { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
never?: Never { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
RealtimeResponseStatus { error, reason, type } Additional details about the status.
Additional details about the status.
error?: Error { code, type } A description of the error that caused the response to fail,
populated when the status is failed.
A description of the error that caused the response to fail,
populated when the status is failed.
reason?: "turn_detected" | "client_cancelled" | "max_output_tokens" | "content_filter"The reason the Response did not complete. For a cancelled Response, one of turn_detected (the server VAD detected a new start of speech) or client_cancelled (the client sent a cancel event). For an incomplete Response, one of max_output_tokens or content_filter (the server-side safety filter activated and cut off the response).
The reason the Response did not complete. For a cancelled Response, one of turn_detected (the server VAD detected a new start of speech) or client_cancelled (the client sent a cancel event). For an incomplete Response, one of max_output_tokens or content_filter (the server-side safety filter activated and cut off the response).
RealtimeResponseUsage { input_token_details, input_tokens, output_token_details, 2 more } Usage statistics for the Response, this will correspond to billing. A
Realtime API session will maintain a conversation context and append new
Items to the Conversation, thus output from previous turns (text and
audio tokens) will become the input for later turns.
Usage statistics for the Response, this will correspond to billing. A Realtime API session will maintain a conversation context and append new Items to the Conversation, thus output from previous turns (text and audio tokens) will become the input for later turns.
Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens.
The number of input tokens used in the Response, including text and audio tokens.
RealtimeResponseUsageInputTokenDetails { audio_tokens, cached_tokens, cached_tokens_details, 2 more } Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens.
Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens.
RealtimeServerEvent = ConversationCreatedEvent { conversation, event_id, type } | ConversationItemCreatedEvent { event_id, item, type, previous_item_id } | ConversationItemDeletedEvent { event_id, item_id, type } | 43 moreA realtime server event.
A realtime server event.
ConversationCreatedEvent { conversation, event_id, type } Returned when a conversation is created. Emitted right after session creation.
Returned when a conversation is created. Emitted right after session creation.
ConversationItemCreatedEvent { event_id, item, type, previous_item_id } Returned when a conversation item is created. There are several scenarios that produce this event:
- The server is generating a Response, which if successful will produce
either one or two Items, which will be of type
message
(role assistant) or type function_call.
- The input audio buffer has been committed, either by the client or the
server (in
server_vad mode). The server will take the content of the
input audio buffer and add it to a new user message Item.
- The client has sent a
conversation.item.create event to add a new Item
to the Conversation.
Returned when a conversation item is created. There are several scenarios that produce this event:
- The server is generating a Response, which if successful will produce
either one or two Items, which will be of type
message(roleassistant) or typefunction_call. - The input audio buffer has been committed, either by the client or the
server (in
server_vadmode). The server will take the content of the input audio buffer and add it to a new user message Item. - The client has sent a
conversation.item.createevent to add a new Item to the Conversation.
ConversationItemDeletedEvent { event_id, item_id, type } Returned when an item in the conversation is deleted by the client with a
conversation.item.delete event. This event is used to synchronize the
server's understanding of the conversation history with the client's view.
Returned when an item in the conversation is deleted by the client with a
conversation.item.delete event. This event is used to synchronize the
server's understanding of the conversation history with the client's view.
ConversationItemInputAudioTranscriptionCompletedEvent { content_index, event_id, item_id, 4 more } This event is the output of audio transcription for user audio written to the
user audio buffer. Transcription begins when the input audio buffer is
committed by the client or server (when VAD is enabled). Transcription runs
asynchronously with Response creation, so this event may come before or after
the Response events.
Realtime API models accept audio natively, and thus input transcription is a
separate process run on a separate ASR (Automatic Speech Recognition) model.
The transcript may diverge somewhat from the model's interpretation, and
should be treated as a rough guide.
This event is the output of audio transcription for user audio written to the user audio buffer. Transcription begins when the input audio buffer is committed by the client or server (when VAD is enabled). Transcription runs asynchronously with Response creation, so this event may come before or after the Response events.
Realtime API models accept audio natively, and thus input transcription is a separate process run on a separate ASR (Automatic Speech Recognition) model. The transcript may diverge somewhat from the model's interpretation, and should be treated as a rough guide.
The event type, must be
conversation.item.input_audio_transcription.completed.
usage: TranscriptTextUsageTokens { input_tokens, output_tokens, total_tokens, 2 more } | TranscriptTextUsageDuration { seconds, type } Usage statistics for the transcription, this is billed according to the ASR model's pricing rather than the realtime model's pricing.
Usage statistics for the transcription, this is billed according to the ASR model's pricing rather than the realtime model's pricing.
ConversationItemInputAudioTranscriptionDeltaEvent { event_id, item_id, type, 3 more } Returned when the text value of an input audio transcription content part is updated with incremental transcription results.
Returned when the text value of an input audio transcription content part is updated with incremental transcription results.
The event type, must be conversation.item.input_audio_transcription.delta.
The log probabilities of the transcription. These can be enabled by configurating the session with "include": ["item.input_audio_transcription.logprobs"]. Each entry in the array corresponds a log probability of which token would be selected for this chunk of transcription. This can help to identify if it was possible there were multiple valid options for a given chunk of transcription.
The log probabilities of the transcription. These can be enabled by configurating the session with "include": ["item.input_audio_transcription.logprobs"]. Each entry in the array corresponds a log probability of which token would be selected for this chunk of transcription. This can help to identify if it was possible there were multiple valid options for a given chunk of transcription.
ConversationItemInputAudioTranscriptionFailedEvent { content_index, error, event_id, 2 more } Returned when input audio transcription is configured, and a transcription
request for a user message failed. These events are separate from other
error events so that the client can identify the related Item.
Returned when input audio transcription is configured, and a transcription
request for a user message failed. These events are separate from other
error events so that the client can identify the related Item.
ConversationItemRetrieved { event_id, item, type } Returned when a conversation item is retrieved with conversation.item.retrieve. This is provided as a way to fetch the server's representation of an item, for example to get access to the post-processed audio data after noise cancellation and VAD. It includes the full content of the Item, including audio data.
Returned when a conversation item is retrieved with conversation.item.retrieve. This is provided as a way to fetch the server's representation of an item, for example to get access to the post-processed audio data after noise cancellation and VAD. It includes the full content of the Item, including audio data.
ConversationItemTruncatedEvent { audio_end_ms, content_index, event_id, 2 more } Returned when an earlier assistant audio message item is truncated by the
client with a conversation.item.truncate event. This event is used to
synchronize the server's understanding of the audio with the client's playback.
This action will truncate the audio and remove the server-side text transcript
to ensure there is no text in the context that hasn't been heard by the user.
Returned when an earlier assistant audio message item is truncated by the
client with a conversation.item.truncate event. This event is used to
synchronize the server's understanding of the audio with the client's playback.
This action will truncate the audio and remove the server-side text transcript to ensure there is no text in the context that hasn't been heard by the user.
RealtimeErrorEvent { error, event_id, type } Returned when an error occurs, which could be a client problem or a server
problem. Most errors are recoverable and the session will stay open, we
recommend to implementors to monitor and log error messages by default.
Returned when an error occurs, which could be a client problem or a server problem. Most errors are recoverable and the session will stay open, we recommend to implementors to monitor and log error messages by default.
InputAudioBufferClearedEvent { event_id, type } Returned when the input audio buffer is cleared by the client with a
input_audio_buffer.clear event.
Returned when the input audio buffer is cleared by the client with a
input_audio_buffer.clear event.
InputAudioBufferCommittedEvent { event_id, item_id, type, previous_item_id } Returned when an input audio buffer is committed, either by the client or
automatically in server VAD mode. The item_id property is the ID of the user
message item that will be created, thus a conversation.item.created event
will also be sent to the client.
Returned when an input audio buffer is committed, either by the client or
automatically in server VAD mode. The item_id property is the ID of the user
message item that will be created, thus a conversation.item.created event
will also be sent to the client.
InputAudioBufferDtmfEventReceivedEvent { event, received_at, type } SIP Only: Returned when an DTMF event is received. A DTMF event is a message that
represents a telephone keypad press (0–9, *, #, A–D). The event property
is the keypad that the user press. The received_at is the UTC Unix Timestamp
that the server received the event.
SIP Only: Returned when an DTMF event is received. A DTMF event is a message that
represents a telephone keypad press (0–9, *, #, A–D). The event property
is the keypad that the user press. The received_at is the UTC Unix Timestamp
that the server received the event.
InputAudioBufferSpeechStartedEvent { audio_start_ms, event_id, item_id, type } Sent by the server when in server_vad mode to indicate that speech has been
detected in the audio buffer. This can happen any time audio is added to the
buffer (unless speech is already detected). The client may want to use this
event to interrupt audio playback or provide visual feedback to the user.
The client should expect to receive a input_audio_buffer.speech_stopped event
when speech stops. The item_id property is the ID of the user message item
that will be created when speech stops and will also be included in the
input_audio_buffer.speech_stopped event (unless the client manually commits
the audio buffer during VAD activation).
Sent by the server when in server_vad mode to indicate that speech has been
detected in the audio buffer. This can happen any time audio is added to the
buffer (unless speech is already detected). The client may want to use this
event to interrupt audio playback or provide visual feedback to the user.
The client should expect to receive a input_audio_buffer.speech_stopped event
when speech stops. The item_id property is the ID of the user message item
that will be created when speech stops and will also be included in the
input_audio_buffer.speech_stopped event (unless the client manually commits
the audio buffer during VAD activation).
InputAudioBufferSpeechStoppedEvent { audio_end_ms, event_id, item_id, type } Returned in server_vad mode when the server detects the end of speech in
the audio buffer. The server will also send an conversation.item.created
event with the user message item that is created from the audio buffer.
Returned in server_vad mode when the server detects the end of speech in
the audio buffer. The server will also send an conversation.item.created
event with the user message item that is created from the audio buffer.
RateLimitsUpdatedEvent { event_id, rate_limits, type } Emitted at the beginning of a Response to indicate the updated rate limits.
When a Response is created some tokens will be "reserved" for the output
tokens, the rate limits shown here reflect that reservation, which is then
adjusted accordingly once the Response is completed.
Emitted at the beginning of a Response to indicate the updated rate limits. When a Response is created some tokens will be "reserved" for the output tokens, the rate limits shown here reflect that reservation, which is then adjusted accordingly once the Response is completed.
ResponseAudioDeltaEvent { content_index, delta, event_id, 4 more } Returned when the model-generated audio is updated.
Returned when the model-generated audio is updated.
ResponseAudioDoneEvent { content_index, event_id, item_id, 3 more } Returned when the model-generated audio is done. Also emitted when a Response
is interrupted, incomplete, or cancelled.
Returned when the model-generated audio is done. Also emitted when a Response is interrupted, incomplete, or cancelled.
ResponseAudioTranscriptDeltaEvent { content_index, delta, event_id, 4 more } Returned when the model-generated transcription of audio output is updated.
Returned when the model-generated transcription of audio output is updated.
ResponseAudioTranscriptDoneEvent { content_index, event_id, item_id, 4 more } Returned when the model-generated transcription of audio output is done
streaming. Also emitted when a Response is interrupted, incomplete, or
cancelled.
Returned when the model-generated transcription of audio output is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.
ResponseContentPartAddedEvent { content_index, event_id, item_id, 4 more } Returned when a new content part is added to an assistant message item during
response generation.
Returned when a new content part is added to an assistant message item during response generation.
ResponseContentPartDoneEvent { content_index, event_id, item_id, 4 more } Returned when a content part is done streaming in an assistant message item.
Also emitted when a Response is interrupted, incomplete, or cancelled.
Returned when a content part is done streaming in an assistant message item. Also emitted when a Response is interrupted, incomplete, or cancelled.
ResponseCreatedEvent { event_id, response, type } Returned when a new Response is created. The first event of response creation,
where the response is in an initial state of in_progress.
Returned when a new Response is created. The first event of response creation,
where the response is in an initial state of in_progress.
ResponseDoneEvent { event_id, response, type } Returned when a Response is done streaming. Always emitted, no matter the
final state. The Response object included in the response.done event will
include all output Items in the Response but will omit the raw audio data.
Clients should check the status field of the Response to determine if it was successful
(completed) or if there was another outcome: cancelled, failed, or incomplete.
A response will contain all output items that were generated during the response, excluding
any audio content.
Returned when a Response is done streaming. Always emitted, no matter the
final state. The Response object included in the response.done event will
include all output Items in the Response but will omit the raw audio data.
Clients should check the status field of the Response to determine if it was successful
(completed) or if there was another outcome: cancelled, failed, or incomplete.
A response will contain all output items that were generated during the response, excluding any audio content.
ResponseFunctionCallArgumentsDeltaEvent { call_id, delta, event_id, 4 more } Returned when the model-generated function call arguments are updated.
Returned when the model-generated function call arguments are updated.
ResponseFunctionCallArgumentsDoneEvent { arguments, call_id, event_id, 5 more } Returned when the model-generated function call arguments are done streaming.
Also emitted when a Response is interrupted, incomplete, or cancelled.
Returned when the model-generated function call arguments are done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.
ResponseOutputItemAddedEvent { event_id, item, output_index, 2 more } Returned when a new Item is created during Response generation.
Returned when a new Item is created during Response generation.
ResponseOutputItemDoneEvent { event_id, item, output_index, 2 more } Returned when an Item is done streaming. Also emitted when a Response is
interrupted, incomplete, or cancelled.
Returned when an Item is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.
ResponseTextDeltaEvent { content_index, delta, event_id, 4 more } Returned when the text value of an "output_text" content part is updated.
Returned when the text value of an "output_text" content part is updated.
ResponseTextDoneEvent { content_index, event_id, item_id, 4 more } Returned when the text value of an "output_text" content part is done streaming. Also
emitted when a Response is interrupted, incomplete, or cancelled.
Returned when the text value of an "output_text" content part is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.
SessionCreatedEvent { event_id, session, type } Returned when a Session is created. Emitted automatically when a new
connection is established as the first server event. This event will contain
the default Session configuration.
Returned when a Session is created. Emitted automatically when a new connection is established as the first server event. This event will contain the default Session configuration.
session: RealtimeSessionCreateRequest { type, audio, include, 9 more } | RealtimeTranscriptionSessionCreateRequest { type, audio, include } The session configuration.
The session configuration.
RealtimeSessionCreateRequest { type, audio, include, 9 more } Realtime session object configuration.
Realtime session object configuration.
Additional fields to include in server outputs.
item.input_audio_transcription.logprobs: Include logprobs for input audio transcription.
The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior.
Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.
max_output_tokens?: number | "inf"Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
model?: (string & {}) | "gpt-realtime" | "gpt-realtime-1.5" | "gpt-realtime-2025-08-28" | 13 moreThe Realtime model used for this session.
The Realtime model used for this session.
output_modalities?: Array<"text" | "audio">The set of modalities the model can respond with. It defaults to ["audio"], indicating
that the model will respond with audio plus a transcript. ["text"] can be used to make
the model respond with text only. It is not possible to request both text and audio at the same time.
The set of modalities the model can respond with. It defaults to ["audio"], indicating
that the model will respond with audio plus a transcript. ["text"] can be used to make
the model respond with text only. It is not possible to request both text and audio at the same time.
Reference to a prompt template and its variables. Learn more.
Realtime API can write session traces to the Traces Dashboard. Set to null to disable tracing. Once tracing is enabled for a session, the configuration cannot be modified.
auto will create a trace for the session with default values for the
workflow name, group id, and metadata.
When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs.
Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost.
Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate.
Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit.
SessionUpdatedEvent { event_id, session, type } Returned when a session is updated with a session.update event, unless
there is an error.
Returned when a session is updated with a session.update event, unless
there is an error.
session: RealtimeSessionCreateRequest { type, audio, include, 9 more } | RealtimeTranscriptionSessionCreateRequest { type, audio, include } The session configuration.
The session configuration.
RealtimeSessionCreateRequest { type, audio, include, 9 more } Realtime session object configuration.
Realtime session object configuration.
Additional fields to include in server outputs.
item.input_audio_transcription.logprobs: Include logprobs for input audio transcription.
The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior.
Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.
max_output_tokens?: number | "inf"Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
model?: (string & {}) | "gpt-realtime" | "gpt-realtime-1.5" | "gpt-realtime-2025-08-28" | 13 moreThe Realtime model used for this session.
The Realtime model used for this session.
output_modalities?: Array<"text" | "audio">The set of modalities the model can respond with. It defaults to ["audio"], indicating
that the model will respond with audio plus a transcript. ["text"] can be used to make
the model respond with text only. It is not possible to request both text and audio at the same time.
The set of modalities the model can respond with. It defaults to ["audio"], indicating
that the model will respond with audio plus a transcript. ["text"] can be used to make
the model respond with text only. It is not possible to request both text and audio at the same time.
Reference to a prompt template and its variables. Learn more.
Realtime API can write session traces to the Traces Dashboard. Set to null to disable tracing. Once tracing is enabled for a session, the configuration cannot be modified.
auto will create a trace for the session with default values for the
workflow name, group id, and metadata.
When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs.
Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost.
Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate.
Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit.
OutputAudioBufferStarted { event_id, response_id, type } WebRTC/SIP Only: Emitted when the server begins streaming audio to the client. This event is
emitted after an audio content part has been added (response.content_part.added)
to the response.
Learn more.
WebRTC/SIP Only: Emitted when the server begins streaming audio to the client. This event is
emitted after an audio content part has been added (response.content_part.added)
to the response.
Learn more.
OutputAudioBufferStopped { event_id, response_id, type } WebRTC/SIP Only: Emitted when the output audio buffer has been completely drained on the server,
and no more audio is forthcoming. This event is emitted after the full response
data has been sent to the client (response.done).
Learn more.
WebRTC/SIP Only: Emitted when the output audio buffer has been completely drained on the server,
and no more audio is forthcoming. This event is emitted after the full response
data has been sent to the client (response.done).
Learn more.
OutputAudioBufferCleared { event_id, response_id, type } WebRTC/SIP Only: Emitted when the output audio buffer is cleared. This happens either in VAD
mode when the user has interrupted (input_audio_buffer.speech_started),
or when the client has emitted the output_audio_buffer.clear event to manually
cut off the current audio response.
Learn more.
WebRTC/SIP Only: Emitted when the output audio buffer is cleared. This happens either in VAD
mode when the user has interrupted (input_audio_buffer.speech_started),
or when the client has emitted the output_audio_buffer.clear event to manually
cut off the current audio response.
Learn more.
ConversationItemAdded { event_id, item, type, previous_item_id } Sent by the server when an Item is added to the default Conversation. This can happen in several cases:
- When the client sends a
conversation.item.create event.
- When the input audio buffer is committed. In this case the item will be a user message containing the audio from the buffer.
- When the model is generating a Response. In this case the
conversation.item.added event will be sent when the model starts generating a specific Item, and thus it will not yet have any content (and status will be in_progress).
The event will include the full content of the Item (except when model is generating a Response) except for audio data, which can be retrieved separately with a conversation.item.retrieve event if necessary.
Sent by the server when an Item is added to the default Conversation. This can happen in several cases:
- When the client sends a
conversation.item.createevent. - When the input audio buffer is committed. In this case the item will be a user message containing the audio from the buffer.
- When the model is generating a Response. In this case the
conversation.item.addedevent will be sent when the model starts generating a specific Item, and thus it will not yet have any content (andstatuswill bein_progress).
The event will include the full content of the Item (except when model is generating a Response) except for audio data, which can be retrieved separately with a conversation.item.retrieve event if necessary.
ConversationItemDone { event_id, item, type, previous_item_id } Returned when a conversation item is finalized.
The event will include the full content of the Item except for audio data, which can be retrieved separately with a conversation.item.retrieve event if needed.
Returned when a conversation item is finalized.
The event will include the full content of the Item except for audio data, which can be retrieved separately with a conversation.item.retrieve event if needed.
InputAudioBufferTimeoutTriggered { audio_end_ms, audio_start_ms, event_id, 2 more } Returned when the Server VAD timeout is triggered for the input audio buffer. This is configured
with idle_timeout_ms in the turn_detection settings of the session, and it indicates that
there hasn't been any speech detected for the configured duration.
The audio_start_ms and audio_end_ms fields indicate the segment of audio after the last
model response up to the triggering time, as an offset from the beginning of audio written
to the input audio buffer. This means it demarcates the segment of audio that was silent and
the difference between the start and end values will roughly match the configured timeout.
The empty audio will be committed to the conversation as an input_audio item (there will be a
input_audio_buffer.committed event) and a model response will be generated. There may be speech
that didn't trigger VAD but is still detected by the model, so the model may respond with
something relevant to the conversation or a prompt to continue speaking.
Returned when the Server VAD timeout is triggered for the input audio buffer. This is configured
with idle_timeout_ms in the turn_detection settings of the session, and it indicates that
there hasn't been any speech detected for the configured duration.
The audio_start_ms and audio_end_ms fields indicate the segment of audio after the last
model response up to the triggering time, as an offset from the beginning of audio written
to the input audio buffer. This means it demarcates the segment of audio that was silent and
the difference between the start and end values will roughly match the configured timeout.
The empty audio will be committed to the conversation as an input_audio item (there will be a
input_audio_buffer.committed event) and a model response will be generated. There may be speech
that didn't trigger VAD but is still detected by the model, so the model may respond with
something relevant to the conversation or a prompt to continue speaking.
ConversationItemInputAudioTranscriptionSegment { id, content_index, end, 6 more } Returned when an input audio transcription segment is identified for an item.
Returned when an input audio transcription segment is identified for an item.
McpListToolsInProgress { event_id, item_id, type } Returned when listing MCP tools is in progress for an item.
Returned when listing MCP tools is in progress for an item.
McpListToolsCompleted { event_id, item_id, type } Returned when listing MCP tools has completed for an item.
Returned when listing MCP tools has completed for an item.
McpListToolsFailed { event_id, item_id, type } Returned when listing MCP tools has failed for an item.
Returned when listing MCP tools has failed for an item.
ResponseMcpCallArgumentsDelta { delta, event_id, item_id, 4 more } Returned when MCP tool call arguments are updated during response generation.
Returned when MCP tool call arguments are updated during response generation.
ResponseMcpCallArgumentsDone { arguments, event_id, item_id, 3 more } Returned when MCP tool call arguments are finalized during response generation.
Returned when MCP tool call arguments are finalized during response generation.
ResponseMcpCallInProgress { event_id, item_id, output_index, type } Returned when an MCP tool call has started and is in progress.
Returned when an MCP tool call has started and is in progress.
ResponseMcpCallCompleted { event_id, item_id, output_index, type } Returned when an MCP tool call has completed successfully.
Returned when an MCP tool call has completed successfully.
RealtimeSession { id, expires_at, include, 17 more } Realtime session object for the beta interface.
Realtime session object for the beta interface.
Additional fields to include in server outputs.
item.input_audio_transcription.logprobs: Include logprobs for input audio transcription.
input_audio_format?: "pcm16" | "g711_ulaw" | "g711_alaw"The format of input audio. Options are pcm16, g711_ulaw, or g711_alaw.
For pcm16, input audio must be 16-bit PCM at a 24kHz sample rate,
single channel (mono), and little-endian byte order.
The format of input audio. Options are pcm16, g711_ulaw, or g711_alaw.
For pcm16, input audio must be 16-bit PCM at a 24kHz sample rate,
single channel (mono), and little-endian byte order.
input_audio_noise_reduction?: InputAudioNoiseReduction { type } Configuration for input audio noise reduction. This can be set to null to turn off.
Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model.
Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio.
Configuration for input audio noise reduction. This can be set to null to turn off.
Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model.
Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio.
Configuration for input audio transcription, defaults to off and can be set to null to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through the /audio/transcriptions endpoint and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service.
The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior.
Note that the server sets default instructions which will be used if this
field is not set and are visible in the session.created event at the
start of the session.
max_response_output_tokens?: number | "inf"Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
modalities?: Array<"text" | "audio">The set of modalities the model can respond with. To disable audio,
set this to ["text"].
The set of modalities the model can respond with. To disable audio, set this to ["text"].
model?: (string & {}) | "gpt-realtime" | "gpt-realtime-1.5" | "gpt-realtime-2025-08-28" | 13 moreThe Realtime model used for this session.
The Realtime model used for this session.
output_audio_format?: "pcm16" | "g711_ulaw" | "g711_alaw"The format of output audio. Options are pcm16, g711_ulaw, or g711_alaw.
For pcm16, output audio is sampled at a rate of 24kHz.
The format of output audio. Options are pcm16, g711_ulaw, or g711_alaw.
For pcm16, output audio is sampled at a rate of 24kHz.
Reference to a prompt template and its variables. Learn more.
The speed of the model's spoken response. 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress.
Sampling temperature for the model, limited to [0.6, 1.2]. For audio models a temperature of 0.8 is highly recommended for best performance.
How the model chooses tools. Options are auto, none, required, or
specify a function.
Tools (functions) available to the model.
Tools (functions) available to the model.
tracing?: "auto" | TracingConfiguration { group_id, metadata, workflow_name } | nullConfiguration options for tracing. Set to null to disable tracing. Once
tracing is enabled for a session, the configuration cannot be modified.
auto will create a trace for the session with default values for the
workflow name, group id, and metadata.
Configuration options for tracing. Set to null to disable tracing. Once tracing is enabled for a session, the configuration cannot be modified.
auto will create a trace for the session with default values for the
workflow name, group id, and metadata.
TracingConfiguration { group_id, metadata, workflow_name } Granular configuration for tracing.
Granular configuration for tracing.
turn_detection?: ServerVad { type, create_response, idle_timeout_ms, 4 more } | SemanticVad { type, create_response, eagerness, interrupt_response } | nullConfiguration for turn detection, ether Server VAD or Semantic VAD. This can be set to null to turn off, in which case the client must manually trigger model response.
Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.
Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency.
Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to null to turn off, in which case the client must manually trigger model response.
Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.
Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency.
ServerVad { type, create_response, idle_timeout_ms, 4 more } Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence.
Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence.
Whether or not to automatically generate a response when a VAD stop event occurs. If interrupt_response is set to false this may fail to create a response if the model is already responding.
If both create_response and interrupt_response are set to false, the model will never respond automatically but VAD events will still be emitted.
Optional timeout after which a model response will be triggered automatically. This is useful for situations in which a long pause from the user is unexpected, such as a phone call. The model will effectively prompt the user to continue the conversation based on the current context.
The timeout value will be applied after the last model response's audio has finished playing,
i.e. it's set to the response.done time plus audio playback duration.
An input_audio_buffer.timeout_triggered event (plus events
associated with the Response) will be emitted when the timeout is reached.
Idle timeout is currently only supported for server_vad mode.
Whether or not to automatically interrupt (cancel) any ongoing response with output to the default
conversation (i.e. conversation of auto) when a VAD start event occurs. If true then the response will be cancelled, otherwise it will continue until complete.
If both create_response and interrupt_response are set to false, the model will never respond automatically but VAD events will still be emitted.
Used only for server_vad mode. Amount of audio to include before the VAD detected speech (in
milliseconds). Defaults to 300ms.
SemanticVad { type, create_response, eagerness, interrupt_response } Server-side semantic turn detection which uses a model to determine when the user has finished speaking.
Server-side semantic turn detection which uses a model to determine when the user has finished speaking.
Whether or not to automatically generate a response when a VAD stop event occurs.
eagerness?: "low" | "medium" | "high" | "auto"Used only for semantic_vad mode. The eagerness of the model to respond. low will wait longer for the user to continue speaking, high will respond more quickly. auto is the default and is equivalent to medium. low, medium, and high have max timeouts of 8s, 4s, and 2s respectively.
Used only for semantic_vad mode. The eagerness of the model to respond. low will wait longer for the user to continue speaking, high will respond more quickly. auto is the default and is equivalent to medium. low, medium, and high have max timeouts of 8s, 4s, and 2s respectively.
RealtimeSessionCreateRequest { type, audio, include, 9 more } Realtime session object configuration.
Realtime session object configuration.
Additional fields to include in server outputs.
item.input_audio_transcription.logprobs: Include logprobs for input audio transcription.
The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior.
Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.
max_output_tokens?: number | "inf"Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
model?: (string & {}) | "gpt-realtime" | "gpt-realtime-1.5" | "gpt-realtime-2025-08-28" | 13 moreThe Realtime model used for this session.
The Realtime model used for this session.
output_modalities?: Array<"text" | "audio">The set of modalities the model can respond with. It defaults to ["audio"], indicating
that the model will respond with audio plus a transcript. ["text"] can be used to make
the model respond with text only. It is not possible to request both text and audio at the same time.
The set of modalities the model can respond with. It defaults to ["audio"], indicating
that the model will respond with audio plus a transcript. ["text"] can be used to make
the model respond with text only. It is not possible to request both text and audio at the same time.
Reference to a prompt template and its variables. Learn more.
Realtime API can write session traces to the Traces Dashboard. Set to null to disable tracing. Once tracing is enabled for a session, the configuration cannot be modified.
auto will create a trace for the session with default values for the
workflow name, group id, and metadata.
When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs.
Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost.
Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate.
Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit.
RealtimeToolChoiceConfig = ToolChoiceOptions | ToolChoiceFunction { name, type } | ToolChoiceMcp { server_label, type, name } How the model chooses tools. Provide one of the string modes or force a specific
function/MCP tool.
How the model chooses tools. Provide one of the string modes or force a specific function/MCP tool.
ToolChoiceOptions = "none" | "auto" | "required"Controls which (if any) tool is called by the model.
none means the model will not call any tool and instead generates a message.
auto means the model can pick between generating a message or calling one or
more tools.
required means the model must call one or more tools.
Controls which (if any) tool is called by the model.
none means the model will not call any tool and instead generates a message.
auto means the model can pick between generating a message or calling one or
more tools.
required means the model must call one or more tools.
Tools available to the model.
Tools available to the model.
RealtimeFunctionTool { description, name, parameters, type }
Mcp { server_label, type, allowed_tools, 7 more } Give the model access to additional tools via remote Model Context Protocol
(MCP) servers. Learn more about MCP.
Give the model access to additional tools via remote Model Context Protocol (MCP) servers. Learn more about MCP.
allowed_tools?: Array<string> | McpToolFilter { read_only, tool_names } | nullList of allowed tool names or a filter object.
List of allowed tool names or a filter object.
McpToolFilter { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
An OAuth access token that can be used with a remote MCP server, either with a custom MCP server URL or a service connector. Your application must handle the OAuth authorization flow and provide the token here.
connector_id?: "connector_dropbox" | "connector_gmail" | "connector_googlecalendar" | 5 moreIdentifier for service connectors, like those available in ChatGPT. One of
server_url or connector_id must be provided. Learn more about service
connectors here.
Currently supported connector_id values are:
- Dropbox:
connector_dropbox
- Gmail:
connector_gmail
- Google Calendar:
connector_googlecalendar
- Google Drive:
connector_googledrive
- Microsoft Teams:
connector_microsoftteams
- Outlook Calendar:
connector_outlookcalendar
- Outlook Email:
connector_outlookemail
- SharePoint:
connector_sharepoint
Identifier for service connectors, like those available in ChatGPT. One of
server_url or connector_id must be provided. Learn more about service
connectors here.
Currently supported connector_id values are:
- Dropbox:
connector_dropbox - Gmail:
connector_gmail - Google Calendar:
connector_googlecalendar - Google Drive:
connector_googledrive - Microsoft Teams:
connector_microsoftteams - Outlook Calendar:
connector_outlookcalendar - Outlook Email:
connector_outlookemail - SharePoint:
connector_sharepoint
Optional HTTP headers to send to the MCP server. Use for authentication or other purposes.
require_approval?: McpToolApprovalFilter { always, never } | "always" | "never" | nullSpecify which of the MCP server's tools require approval.
Specify which of the MCP server's tools require approval.
McpToolApprovalFilter { always, never } Specify which of the MCP server's tools require approval. Can be
always, never, or a filter object associated with tools
that require approval.
Specify which of the MCP server's tools require approval. Can be
always, never, or a filter object associated with tools
that require approval.
always?: Always { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
never?: Never { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
RealtimeToolsConfigUnion = RealtimeFunctionTool { description, name, parameters, type } | Mcp { server_label, type, allowed_tools, 7 more } Give the model access to additional tools via remote Model Context Protocol
(MCP) servers. Learn more about MCP.
Give the model access to additional tools via remote Model Context Protocol (MCP) servers. Learn more about MCP.
RealtimeFunctionTool { description, name, parameters, type }
Mcp { server_label, type, allowed_tools, 7 more } Give the model access to additional tools via remote Model Context Protocol
(MCP) servers. Learn more about MCP.
Give the model access to additional tools via remote Model Context Protocol (MCP) servers. Learn more about MCP.
allowed_tools?: Array<string> | McpToolFilter { read_only, tool_names } | nullList of allowed tool names or a filter object.
List of allowed tool names or a filter object.
McpToolFilter { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
An OAuth access token that can be used with a remote MCP server, either with a custom MCP server URL or a service connector. Your application must handle the OAuth authorization flow and provide the token here.
connector_id?: "connector_dropbox" | "connector_gmail" | "connector_googlecalendar" | 5 moreIdentifier for service connectors, like those available in ChatGPT. One of
server_url or connector_id must be provided. Learn more about service
connectors here.
Currently supported connector_id values are:
- Dropbox:
connector_dropbox
- Gmail:
connector_gmail
- Google Calendar:
connector_googlecalendar
- Google Drive:
connector_googledrive
- Microsoft Teams:
connector_microsoftteams
- Outlook Calendar:
connector_outlookcalendar
- Outlook Email:
connector_outlookemail
- SharePoint:
connector_sharepoint
Identifier for service connectors, like those available in ChatGPT. One of
server_url or connector_id must be provided. Learn more about service
connectors here.
Currently supported connector_id values are:
- Dropbox:
connector_dropbox - Gmail:
connector_gmail - Google Calendar:
connector_googlecalendar - Google Drive:
connector_googledrive - Microsoft Teams:
connector_microsoftteams - Outlook Calendar:
connector_outlookcalendar - Outlook Email:
connector_outlookemail - SharePoint:
connector_sharepoint
Optional HTTP headers to send to the MCP server. Use for authentication or other purposes.
require_approval?: McpToolApprovalFilter { always, never } | "always" | "never" | nullSpecify which of the MCP server's tools require approval.
Specify which of the MCP server's tools require approval.
McpToolApprovalFilter { always, never } Specify which of the MCP server's tools require approval. Can be
always, never, or a filter object associated with tools
that require approval.
Specify which of the MCP server's tools require approval. Can be
always, never, or a filter object associated with tools
that require approval.
always?: Always { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
never?: Never { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
RealtimeTracingConfig = "auto" | TracingConfiguration { group_id, metadata, workflow_name } | nullRealtime API can write session traces to the Traces Dashboard. Set to null to disable tracing. Once
tracing is enabled for a session, the configuration cannot be modified.
auto will create a trace for the session with default values for the
workflow name, group id, and metadata.
Realtime API can write session traces to the Traces Dashboard. Set to null to disable tracing. Once tracing is enabled for a session, the configuration cannot be modified.
auto will create a trace for the session with default values for the
workflow name, group id, and metadata.
TracingConfiguration { group_id, metadata, workflow_name } Granular configuration for tracing.
Granular configuration for tracing.
RealtimeTranscriptionSessionAudio { input } Configuration for input and output audio.
Configuration for input and output audio.
RealtimeTranscriptionSessionAudioInput { format, noise_reduction, transcription, turn_detection }
noise_reduction?: NoiseReduction { type } Configuration for input audio noise reduction. This can be set to null to turn off.
Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model.
Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio.
Configuration for input audio noise reduction. This can be set to null to turn off.
Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model.
Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio.
Configuration for input audio transcription, defaults to off and can be set to null to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through the /audio/transcriptions endpoint and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service.
Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to null to turn off, in which case the client must manually trigger model response.
Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.
Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency.
RealtimeTranscriptionSessionAudioInputTurnDetection = ServerVad { type, create_response, idle_timeout_ms, 4 more } | SemanticVad { type, create_response, eagerness, interrupt_response } | nullConfiguration for turn detection, ether Server VAD or Semantic VAD. This can be set to null to turn off, in which case the client must manually trigger model response.
Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.
Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency.
Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to null to turn off, in which case the client must manually trigger model response.
Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.
Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency.
ServerVad { type, create_response, idle_timeout_ms, 4 more } Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence.
Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence.
Whether or not to automatically generate a response when a VAD stop event occurs. If interrupt_response is set to false this may fail to create a response if the model is already responding.
If both create_response and interrupt_response are set to false, the model will never respond automatically but VAD events will still be emitted.
Optional timeout after which a model response will be triggered automatically. This is useful for situations in which a long pause from the user is unexpected, such as a phone call. The model will effectively prompt the user to continue the conversation based on the current context.
The timeout value will be applied after the last model response's audio has finished playing,
i.e. it's set to the response.done time plus audio playback duration.
An input_audio_buffer.timeout_triggered event (plus events
associated with the Response) will be emitted when the timeout is reached.
Idle timeout is currently only supported for server_vad mode.
Whether or not to automatically interrupt (cancel) any ongoing response with output to the default
conversation (i.e. conversation of auto) when a VAD start event occurs. If true then the response will be cancelled, otherwise it will continue until complete.
If both create_response and interrupt_response are set to false, the model will never respond automatically but VAD events will still be emitted.
Used only for server_vad mode. Amount of audio to include before the VAD detected speech (in
milliseconds). Defaults to 300ms.
SemanticVad { type, create_response, eagerness, interrupt_response } Server-side semantic turn detection which uses a model to determine when the user has finished speaking.
Server-side semantic turn detection which uses a model to determine when the user has finished speaking.
Whether or not to automatically generate a response when a VAD stop event occurs.
eagerness?: "low" | "medium" | "high" | "auto"Used only for semantic_vad mode. The eagerness of the model to respond. low will wait longer for the user to continue speaking, high will respond more quickly. auto is the default and is equivalent to medium. low, medium, and high have max timeouts of 8s, 4s, and 2s respectively.
Used only for semantic_vad mode. The eagerness of the model to respond. low will wait longer for the user to continue speaking, high will respond more quickly. auto is the default and is equivalent to medium. low, medium, and high have max timeouts of 8s, 4s, and 2s respectively.
RealtimeTruncation = "auto" | "disabled" | RealtimeTruncationRetentionRatio { retention_ratio, type, token_limits } When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs.
Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost.
Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate.
Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit.
When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs.
Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost.
Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate.
Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit.
RealtimeTruncationRetentionRatio { retention_ratio, type, token_limits } Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage.
Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage.
Fraction of post-instruction conversation tokens to retain (0.0 - 1.0) when the conversation exceeds the input token limit. Setting this to 0.8 means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates.
token_limits?: TokenLimits { post_instructions } Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used.
Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used.
Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens.
RealtimeTruncationRetentionRatio { retention_ratio, type, token_limits } Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage.
Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage.
Fraction of post-instruction conversation tokens to retain (0.0 - 1.0) when the conversation exceeds the input token limit. Setting this to 0.8 means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates.
token_limits?: TokenLimits { post_instructions } Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used.
Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used.
Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens.
ResponseCancelEvent { type, event_id, response_id } Send this event to cancel an in-progress response. The server will respond
with a response.done event with a status of response.status=cancelled. If
there is no response to cancel, the server will respond with an error. It's safe
to call response.cancel even if no response is in progress, an error will be
returned the session will remain unaffected.
Send this event to cancel an in-progress response. The server will respond
with a response.done event with a status of response.status=cancelled. If
there is no response to cancel, the server will respond with an error. It's safe
to call response.cancel even if no response is in progress, an error will be
returned the session will remain unaffected.
ResponseCreateEvent { type, event_id, response } This event instructs the server to create a Response, which means triggering
model inference. When in Server VAD mode, the server will create Responses
automatically.
A Response will include at least one Item, and may have two, in which case
the second will be a function call. These Items will be appended to the
conversation history by default.
The server will respond with a response.created event, events for Items
and content created, and finally a response.done event to indicate the
Response is complete.
The response.create event includes inference configuration like
instructions and tools. If these are set, they will override the Session's
configuration for this Response only.
Responses can be created out-of-band of the default Conversation, meaning that they can
have arbitrary input, and it's possible to disable writing the output to the Conversation.
Only one Response can write to the default Conversation at a time, but otherwise multiple
Responses can be created in parallel. The metadata field is a good way to disambiguate
multiple simultaneous Responses.
Clients can set conversation to none to create a Response that does not write to the default
Conversation. Arbitrary input can be provided with the input field, which is an array accepting
raw Items and references to existing Items.
This event instructs the server to create a Response, which means triggering model inference. When in Server VAD mode, the server will create Responses automatically.
A Response will include at least one Item, and may have two, in which case the second will be a function call. These Items will be appended to the conversation history by default.
The server will respond with a response.created event, events for Items
and content created, and finally a response.done event to indicate the
Response is complete.
The response.create event includes inference configuration like
instructions and tools. If these are set, they will override the Session's
configuration for this Response only.
Responses can be created out-of-band of the default Conversation, meaning that they can
have arbitrary input, and it's possible to disable writing the output to the Conversation.
Only one Response can write to the default Conversation at a time, but otherwise multiple
Responses can be created in parallel. The metadata field is a good way to disambiguate
multiple simultaneous Responses.
Clients can set conversation to none to create a Response that does not write to the default
Conversation. Arbitrary input can be provided with the input field, which is an array accepting
raw Items and references to existing Items.
ResponseDoneEvent { event_id, response, type } Returned when a Response is done streaming. Always emitted, no matter the
final state. The Response object included in the response.done event will
include all output Items in the Response but will omit the raw audio data.
Clients should check the status field of the Response to determine if it was successful
(completed) or if there was another outcome: cancelled, failed, or incomplete.
A response will contain all output items that were generated during the response, excluding
any audio content.
Returned when a Response is done streaming. Always emitted, no matter the
final state. The Response object included in the response.done event will
include all output Items in the Response but will omit the raw audio data.
Clients should check the status field of the Response to determine if it was successful
(completed) or if there was another outcome: cancelled, failed, or incomplete.
A response will contain all output items that were generated during the response, excluding any audio content.
SessionCreatedEvent { event_id, session, type } Returned when a Session is created. Emitted automatically when a new
connection is established as the first server event. This event will contain
the default Session configuration.
Returned when a Session is created. Emitted automatically when a new connection is established as the first server event. This event will contain the default Session configuration.
session: RealtimeSessionCreateRequest { type, audio, include, 9 more } | RealtimeTranscriptionSessionCreateRequest { type, audio, include } The session configuration.
The session configuration.
RealtimeSessionCreateRequest { type, audio, include, 9 more } Realtime session object configuration.
Realtime session object configuration.
Additional fields to include in server outputs.
item.input_audio_transcription.logprobs: Include logprobs for input audio transcription.
The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior.
Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.
max_output_tokens?: number | "inf"Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
model?: (string & {}) | "gpt-realtime" | "gpt-realtime-1.5" | "gpt-realtime-2025-08-28" | 13 moreThe Realtime model used for this session.
The Realtime model used for this session.
output_modalities?: Array<"text" | "audio">The set of modalities the model can respond with. It defaults to ["audio"], indicating
that the model will respond with audio plus a transcript. ["text"] can be used to make
the model respond with text only. It is not possible to request both text and audio at the same time.
The set of modalities the model can respond with. It defaults to ["audio"], indicating
that the model will respond with audio plus a transcript. ["text"] can be used to make
the model respond with text only. It is not possible to request both text and audio at the same time.
Reference to a prompt template and its variables. Learn more.
Realtime API can write session traces to the Traces Dashboard. Set to null to disable tracing. Once tracing is enabled for a session, the configuration cannot be modified.
auto will create a trace for the session with default values for the
workflow name, group id, and metadata.
When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs.
Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost.
Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate.
Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit.
SessionUpdateEvent { session, type, event_id } Send this event to update the session’s configuration.
The client may send this event at any time to update any field
except for voice and model. voice can be updated only if there have been no other audio outputs yet.
When the server receives a session.update, it will respond
with a session.updated event showing the full, effective configuration.
Only the fields that are present in the session.update are updated. To clear a field like
instructions, pass an empty string. To clear a field like tools, pass an empty array.
To clear a field like turn_detection, pass null.
Send this event to update the session’s configuration.
The client may send this event at any time to update any field
except for voice and model. voice can be updated only if there have been no other audio outputs yet.
When the server receives a session.update, it will respond
with a session.updated event showing the full, effective configuration.
Only the fields that are present in the session.update are updated. To clear a field like
instructions, pass an empty string. To clear a field like tools, pass an empty array.
To clear a field like turn_detection, pass null.
session: RealtimeSessionCreateRequest { type, audio, include, 9 more } | RealtimeTranscriptionSessionCreateRequest { type, audio, include } Update the Realtime session. Choose either a realtime
session or a transcription session.
Update the Realtime session. Choose either a realtime session or a transcription session.
RealtimeSessionCreateRequest { type, audio, include, 9 more } Realtime session object configuration.
Realtime session object configuration.
Additional fields to include in server outputs.
item.input_audio_transcription.logprobs: Include logprobs for input audio transcription.
The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior.
Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.
max_output_tokens?: number | "inf"Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
model?: (string & {}) | "gpt-realtime" | "gpt-realtime-1.5" | "gpt-realtime-2025-08-28" | 13 moreThe Realtime model used for this session.
The Realtime model used for this session.
output_modalities?: Array<"text" | "audio">The set of modalities the model can respond with. It defaults to ["audio"], indicating
that the model will respond with audio plus a transcript. ["text"] can be used to make
the model respond with text only. It is not possible to request both text and audio at the same time.
The set of modalities the model can respond with. It defaults to ["audio"], indicating
that the model will respond with audio plus a transcript. ["text"] can be used to make
the model respond with text only. It is not possible to request both text and audio at the same time.
Reference to a prompt template and its variables. Learn more.
Realtime API can write session traces to the Traces Dashboard. Set to null to disable tracing. Once tracing is enabled for a session, the configuration cannot be modified.
auto will create a trace for the session with default values for the
workflow name, group id, and metadata.
When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs.
Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost.
Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate.
Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit.
SessionUpdatedEvent { event_id, session, type } Returned when a session is updated with a session.update event, unless
there is an error.
Returned when a session is updated with a session.update event, unless
there is an error.
session: RealtimeSessionCreateRequest { type, audio, include, 9 more } | RealtimeTranscriptionSessionCreateRequest { type, audio, include } The session configuration.
The session configuration.
RealtimeSessionCreateRequest { type, audio, include, 9 more } Realtime session object configuration.
Realtime session object configuration.
Additional fields to include in server outputs.
item.input_audio_transcription.logprobs: Include logprobs for input audio transcription.
The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior.
Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.
max_output_tokens?: number | "inf"Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
model?: (string & {}) | "gpt-realtime" | "gpt-realtime-1.5" | "gpt-realtime-2025-08-28" | 13 moreThe Realtime model used for this session.
The Realtime model used for this session.
output_modalities?: Array<"text" | "audio">The set of modalities the model can respond with. It defaults to ["audio"], indicating
that the model will respond with audio plus a transcript. ["text"] can be used to make
the model respond with text only. It is not possible to request both text and audio at the same time.
The set of modalities the model can respond with. It defaults to ["audio"], indicating
that the model will respond with audio plus a transcript. ["text"] can be used to make
the model respond with text only. It is not possible to request both text and audio at the same time.
Reference to a prompt template and its variables. Learn more.
Realtime API can write session traces to the Traces Dashboard. Set to null to disable tracing. Once tracing is enabled for a session, the configuration cannot be modified.
auto will create a trace for the session with default values for the
workflow name, group id, and metadata.
When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs.
Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost.
Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate.
Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit.
TranscriptionSessionUpdate { session, type, event_id } Send this event to update a transcription session.
Send this event to update a transcription session.
session: Session { include, input_audio_format, input_audio_noise_reduction, 2 more } Realtime transcription session object configuration.
Realtime transcription session object configuration.
The set of items to include in the transcription. Current available items are:
item.input_audio_transcription.logprobs
input_audio_format?: "pcm16" | "g711_ulaw" | "g711_alaw"The format of input audio. Options are pcm16, g711_ulaw, or g711_alaw.
For pcm16, input audio must be 16-bit PCM at a 24kHz sample rate,
single channel (mono), and little-endian byte order.
The format of input audio. Options are pcm16, g711_ulaw, or g711_alaw.
For pcm16, input audio must be 16-bit PCM at a 24kHz sample rate,
single channel (mono), and little-endian byte order.
input_audio_noise_reduction?: InputAudioNoiseReduction { type } Configuration for input audio noise reduction. This can be set to null to turn off.
Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model.
Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio.
Configuration for input audio noise reduction. This can be set to null to turn off.
Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model.
Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio.
Configuration for input audio transcription. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service.
turn_detection?: TurnDetection { prefix_padding_ms, silence_duration_ms, threshold, type } Configuration for turn detection. Can be set to null to turn off. Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.
Configuration for turn detection. Can be set to null to turn off. Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.
Amount of audio to include before the VAD detected speech (in milliseconds). Defaults to 300ms.
Duration of silence to detect speech stop (in milliseconds). Defaults to 500ms. With shorter values the model will respond more quickly, but may jump in on short pauses from the user.
TranscriptionSessionUpdatedEvent { event_id, session, type } Returned when a transcription session is updated with a transcription_session.update event, unless
there is an error.
Returned when a transcription session is updated with a transcription_session.update event, unless
there is an error.
session: Session { client_secret, input_audio_format, input_audio_transcription, 2 more } A new Realtime transcription session configuration.
When a session is created on the server via REST API, the session object
also contains an ephemeral key. Default TTL for keys is 10 minutes. This
property is not present when a session is updated via the WebSocket API.
A new Realtime transcription session configuration.
When a session is created on the server via REST API, the session object also contains an ephemeral key. Default TTL for keys is 10 minutes. This property is not present when a session is updated via the WebSocket API.
client_secret: ClientSecret { expires_at, value } Ephemeral key returned by the API. Only present when the session is
created on the server via REST API.
Ephemeral key returned by the API. Only present when the session is created on the server via REST API.
modalities?: Array<"text" | "audio">The set of modalities the model can respond with. To disable audio,
set this to ["text"].
The set of modalities the model can respond with. To disable audio, set this to ["text"].
turn_detection?: TurnDetection { prefix_padding_ms, silence_duration_ms, threshold, type } Configuration for turn detection. Can be set to null to turn off. Server
VAD means that the model will detect the start and end of speech based on
audio volume and respond at the end of user speech.
Configuration for turn detection. Can be set to null to turn off. Server
VAD means that the model will detect the start and end of speech based on
audio volume and respond at the end of user speech.
Amount of audio to include before the VAD detected speech (in milliseconds). Defaults to 300ms.
Duration of silence to detect speech stop (in milliseconds). Defaults to 500ms. With shorter values the model will respond more quickly, but may jump in on short pauses from the user.
RealtimeClient Secrets
Create client secret
ModelsExpand Collapse
RealtimeSessionCreateResponse { client_secret, type, audio, 10 more } A new Realtime session configuration, with an ephemeral key. Default TTL
for keys is one minute.
A new Realtime session configuration, with an ephemeral key. Default TTL for keys is one minute.
audio?: Audio { input, output } Configuration for input and output audio.
Configuration for input and output audio.
input?: Input { format, noise_reduction, transcription, turn_detection }
noise_reduction?: NoiseReduction { type } Configuration for input audio noise reduction. This can be set to null to turn off.
Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model.
Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio.
Configuration for input audio noise reduction. This can be set to null to turn off.
Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model.
Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio.
Configuration for input audio transcription, defaults to off and can be set to null to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through the /audio/transcriptions endpoint and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service.
turn_detection?: ServerVad { type, create_response, idle_timeout_ms, 4 more } | SemanticVad { type, create_response, eagerness, interrupt_response } | nullConfiguration for turn detection, ether Server VAD or Semantic VAD. This can be set to null to turn off, in which case the client must manually trigger model response.
Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.
Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency.
Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to null to turn off, in which case the client must manually trigger model response.
Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.
Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency.
ServerVad { type, create_response, idle_timeout_ms, 4 more } Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence.
Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence.
Whether or not to automatically generate a response when a VAD stop event occurs. If interrupt_response is set to false this may fail to create a response if the model is already responding.
If both create_response and interrupt_response are set to false, the model will never respond automatically but VAD events will still be emitted.
Optional timeout after which a model response will be triggered automatically. This is useful for situations in which a long pause from the user is unexpected, such as a phone call. The model will effectively prompt the user to continue the conversation based on the current context.
The timeout value will be applied after the last model response's audio has finished playing,
i.e. it's set to the response.done time plus audio playback duration.
An input_audio_buffer.timeout_triggered event (plus events
associated with the Response) will be emitted when the timeout is reached.
Idle timeout is currently only supported for server_vad mode.
Whether or not to automatically interrupt (cancel) any ongoing response with output to the default
conversation (i.e. conversation of auto) when a VAD start event occurs. If true then the response will be cancelled, otherwise it will continue until complete.
If both create_response and interrupt_response are set to false, the model will never respond automatically but VAD events will still be emitted.
Used only for server_vad mode. Amount of audio to include before the VAD detected speech (in
milliseconds). Defaults to 300ms.
SemanticVad { type, create_response, eagerness, interrupt_response } Server-side semantic turn detection which uses a model to determine when the user has finished speaking.
Server-side semantic turn detection which uses a model to determine when the user has finished speaking.
Whether or not to automatically generate a response when a VAD stop event occurs.
eagerness?: "low" | "medium" | "high" | "auto"Used only for semantic_vad mode. The eagerness of the model to respond. low will wait longer for the user to continue speaking, high will respond more quickly. auto is the default and is equivalent to medium. low, medium, and high have max timeouts of 8s, 4s, and 2s respectively.
Used only for semantic_vad mode. The eagerness of the model to respond. low will wait longer for the user to continue speaking, high will respond more quickly. auto is the default and is equivalent to medium. low, medium, and high have max timeouts of 8s, 4s, and 2s respectively.
output?: Output { format, speed, voice }
The speed of the model's spoken response as a multiple of the original speed. 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress.
This parameter is a post-processing adjustment to the audio after it is generated, it's also possible to prompt the model to speak faster or slower.
voice?: (string & {}) | "alloy" | "ash" | "ballad" | 7 moreThe voice the model uses to respond. Voice cannot be changed during the
session once the model has responded with audio at least once. Current
voice options are alloy, ash, ballad, coral, echo, sage,
shimmer, verse, marin, and cedar. We recommend marin and cedar for
best quality.
The voice the model uses to respond. Voice cannot be changed during the
session once the model has responded with audio at least once. Current
voice options are alloy, ash, ballad, coral, echo, sage,
shimmer, verse, marin, and cedar. We recommend marin and cedar for
best quality.
Additional fields to include in server outputs.
item.input_audio_transcription.logprobs: Include logprobs for input audio transcription.
The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior.
Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.
max_output_tokens?: number | "inf"Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
Maximum number of output tokens for a single assistant response,
inclusive of tool calls. Provide an integer between 1 and 4096 to
limit output tokens, or inf for the maximum available tokens for a
given model. Defaults to inf.
model?: (string & {}) | "gpt-realtime" | "gpt-realtime-1.5" | "gpt-realtime-2025-08-28" | 13 moreThe Realtime model used for this session.
The Realtime model used for this session.
output_modalities?: Array<"text" | "audio">The set of modalities the model can respond with. It defaults to ["audio"], indicating
that the model will respond with audio plus a transcript. ["text"] can be used to make
the model respond with text only. It is not possible to request both text and audio at the same time.
The set of modalities the model can respond with. It defaults to ["audio"], indicating
that the model will respond with audio plus a transcript. ["text"] can be used to make
the model respond with text only. It is not possible to request both text and audio at the same time.
Reference to a prompt template and its variables. Learn more.
tool_choice?: ToolChoiceOptions | ToolChoiceFunction { name, type } | ToolChoiceMcp { server_label, type, name } How the model chooses tools. Provide one of the string modes or force a specific
function/MCP tool.
How the model chooses tools. Provide one of the string modes or force a specific function/MCP tool.
ToolChoiceOptions = "none" | "auto" | "required"Controls which (if any) tool is called by the model.
none means the model will not call any tool and instead generates a message.
auto means the model can pick between generating a message or calling one or
more tools.
required means the model must call one or more tools.
Controls which (if any) tool is called by the model.
none means the model will not call any tool and instead generates a message.
auto means the model can pick between generating a message or calling one or
more tools.
required means the model must call one or more tools.
tools?: Array<RealtimeFunctionTool { description, name, parameters, type } | McpTool { server_label, type, allowed_tools, 7 more } >Tools available to the model.
Tools available to the model.
RealtimeFunctionTool { description, name, parameters, type }
McpTool { server_label, type, allowed_tools, 7 more } Give the model access to additional tools via remote Model Context Protocol
(MCP) servers. Learn more about MCP.
Give the model access to additional tools via remote Model Context Protocol (MCP) servers. Learn more about MCP.
allowed_tools?: Array<string> | McpToolFilter { read_only, tool_names } | nullList of allowed tool names or a filter object.
List of allowed tool names or a filter object.
McpToolFilter { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
An OAuth access token that can be used with a remote MCP server, either with a custom MCP server URL or a service connector. Your application must handle the OAuth authorization flow and provide the token here.
connector_id?: "connector_dropbox" | "connector_gmail" | "connector_googlecalendar" | 5 moreIdentifier for service connectors, like those available in ChatGPT. One of
server_url or connector_id must be provided. Learn more about service
connectors here.
Currently supported connector_id values are:
- Dropbox:
connector_dropbox
- Gmail:
connector_gmail
- Google Calendar:
connector_googlecalendar
- Google Drive:
connector_googledrive
- Microsoft Teams:
connector_microsoftteams
- Outlook Calendar:
connector_outlookcalendar
- Outlook Email:
connector_outlookemail
- SharePoint:
connector_sharepoint
Identifier for service connectors, like those available in ChatGPT. One of
server_url or connector_id must be provided. Learn more about service
connectors here.
Currently supported connector_id values are:
- Dropbox:
connector_dropbox - Gmail:
connector_gmail - Google Calendar:
connector_googlecalendar - Google Drive:
connector_googledrive - Microsoft Teams:
connector_microsoftteams - Outlook Calendar:
connector_outlookcalendar - Outlook Email:
connector_outlookemail - SharePoint:
connector_sharepoint
Optional HTTP headers to send to the MCP server. Use for authentication or other purposes.
require_approval?: McpToolApprovalFilter { always, never } | "always" | "never" | nullSpecify which of the MCP server's tools require approval.
Specify which of the MCP server's tools require approval.
McpToolApprovalFilter { always, never } Specify which of the MCP server's tools require approval. Can be
always, never, or a filter object associated with tools
that require approval.
Specify which of the MCP server's tools require approval. Can be
always, never, or a filter object associated with tools
that require approval.
always?: Always { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
never?: Never { read_only, tool_names } A filter object to specify which tools are allowed.
A filter object to specify which tools are allowed.
Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
tracing?: "auto" | TracingConfiguration { group_id, metadata, workflow_name } | nullRealtime API can write session traces to the Traces Dashboard. Set to null to disable tracing. Once
tracing is enabled for a session, the configuration cannot be modified.
auto will create a trace for the session with default values for the
workflow name, group id, and metadata.
Realtime API can write session traces to the Traces Dashboard. Set to null to disable tracing. Once tracing is enabled for a session, the configuration cannot be modified.
auto will create a trace for the session with default values for the
workflow name, group id, and metadata.
TracingConfiguration { group_id, metadata, workflow_name } Granular configuration for tracing.
Granular configuration for tracing.
When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs.
Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost.
Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate.
Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit.
RealtimeTranscriptionSessionCreateResponse { id, object, type, 3 more } A Realtime transcription session configuration object.
A Realtime transcription session configuration object.
audio?: Audio { input } Configuration for input audio for the session.
Configuration for input audio for the session.
input?: Input { format, noise_reduction, transcription, turn_detection }
noise_reduction?: NoiseReduction { type } Configuration for input audio noise reduction.
Configuration for input audio noise reduction.
Configuration for turn detection. Can be set to null to turn off. Server
VAD means that the model will detect the start and end of speech based on
audio volume and respond at the end of user speech.
RealtimeTranscriptionSessionTurnDetection { prefix_padding_ms, silence_duration_ms, threshold, type } Configuration for turn detection. Can be set to null to turn off. Server
VAD means that the model will detect the start and end of speech based on
audio volume and respond at the end of user speech.
Configuration for turn detection. Can be set to null to turn off. Server
VAD means that the model will detect the start and end of speech based on
audio volume and respond at the end of user speech.
Amount of audio to include before the VAD detected speech (in milliseconds). Defaults to 300ms.
Duration of silence to detect speech stop (in milliseconds). Defaults to 500ms. With shorter values the model will respond more quickly, but may jump in on short pauses from the user.