Notes on ChatGPT UI Embeds Syntaxes

I have always wondered how ChatGPT.com instructs GPT models to provide rich responses like those shown in the screenshots below. Why not ask ChatGPT itself/themself? By doing so, I extracted the special syntaxes it uses, which are then recognized and parsed into embedded UI elements by the ChatGPT.com frontend, to produce rich responses. Notably, all special syntaxes only work if GPT performs at least one search before responding; otherwise, the ChatGPT.com frontend does not render anything (not even plain text) for these syntaxes at all. The special syntaxes also do not render when wrapped in code blocks. I was able to get ChatGPT to call canvas, and only within a Canvas document will these special characters be preserved and not replaced with UI elements.

1. Citation Syntax

Format

:contentReference[oaicite:N]{index=M}

• N: Citation ID (typically corresponds to a source number)
• M: Index of the citation (for internal UI mapping)

Purpose

Used to insert inline citations referencing external content. This syntax is rendered as a superscripted clickable reference in the UI.

2. Embedded UI Widgets (Unicode Syntax)

General Format

<type><args>

• : Start of embedded widget
• : Argument separator
• : End of widget

These widgets are not user-editable and are interpreted only by the ChatGPT frontend.

Widget Types

a. cite

citeturnXsearchY

Displays a source citation popup.

b. i (Image Carousel)

iturnXimage0turnXimage1

Renders an interactive image carousel from multiple search turns.

c. finance

financeturnXfinanceY

Embeds a finance stock/crypto chart.

d. schedule

scheduleturnXsportsY

Displays an event or sports game schedule.

e. standing

standingturnXsportsY

Embeds league standings (e.g., NBA, EPL).

f. forecast

forecastturnXforecastY

Displays a multi-day weather forecast widget.

g. navlist

navlist<Title>turnXnewsY,turnXnewsZ

Renders a list of clickable navigation links, typically using news search result items.

3. Examples

Sample Search Results

Assume the following turns exist from a user query:

• turn1search0: "Apple Inc. stock price surges after earnings beat expectations"
• turn1search1: "Microsoft announces new AI-powered Office features"
• turn1image0: An image of Apple's stock chart
• turn1image1: A photo of a Microsoft keynote event
• turn1finance0: Apple stock price chart data
• turn1sports0: NBA game schedule for April 19
• turn1sports1: NBA league standings
• turn1forecast0: 5-day weather forecast for New York City

Widget Examples

a. cite

citeturn1search0

Displays a popup linking to the Apple stock news article.

b. i (Image Carousel)

iturn1image0turn1image1

Creates a carousel showing Apple's stock chart and the Microsoft keynote image.

c. finance

financeturn1finance0

Embeds a dynamic chart showing Apple Inc.'s stock performance.

d. schedule

scheduleturn1sports0

Displays a list of NBA games happening on April 19.

e. standing

standingturn1sports1

Shows current NBA league standings.

f. forecast

forecastturn1forecast0

Shows the upcoming weather for NYC.

g. navlist

navlistTech News Highlightsturn1search0,turn1search1

Displays a clickable list linking to both the Apple and Microsoft news articles.

Notes

• turnX and searchY refer to internal turn-based response identifiers and search results.
• These syntaxes are evaluated by the ChatGPT UI and not meant for display to end users directly.
• Improper use of these syntaxes may result in ignored or broken UI components.
• There is no inline syntax for invoking canvas documents; canvas is controlled through backend function calls such as canmore.create_textdoc, canmore.update_textdoc, and canmore.comment_textdoc.

This document provides a detailed overview of internal tools and functions used to interact with ChatGPT's extended capabilities, such as document editing, code execution, browsing, and more. Each function call is described with its name, purpose, parameters, and examples.

1. canmore.create_textdoc

Purpose

Opens a new canvas window to create a text or code document.

Parameters

{
  "name": string,            // Title of the canvas document
  "type": string,            // One of: "document", "code/python", "code/javascript", "code/html", etc.
  "content": string          // Initial content of the document
}

Example

{
  "name": "new_doc",
  "type": "code/javascript",
  "content": "console.log('Hello, world!');"
}

2. canmore.update_textdoc

Purpose

Updates an existing canvas document by applying one or more regex-based text replacements.

Parameters

{
  "updates": [
    {
      "pattern": string,       // Regex pattern to search for
      "multiple": boolean,     // If true, replaces all matches; if false, replaces only the first match
      "replacement": string    // Replacement string
    }
  ]
}

Example

{
  "updates": [
    {
      "pattern": "Hello",
      "multiple": true,
      "replacement": "Hi"
    }
  ]
}

3. canmore.comment_textdoc

Purpose

Adds comments to specific patterns in the canvas document.

Parameters

{
  "comments": [
    {
      "pattern": string,   // Regex pattern to locate
      "comment": string    // Comment text to attach
    }
  ]
}

Example

{
  "comments": [
    {
      "pattern": "function.*MyComponent",
      "comment": "Consider renaming for clarity."
    }
  ]
}

4. python

Purpose

Executes Python code in a secure sandbox environment. Useful for math, data analysis, and code validation.

Input

Python code as a plain string.

Example

import math
math.sqrt(16)

5. web

Purpose

Provides access to real-time internet data through two primary functions: search() and open_url().

5.1 web.search()

Performs a fresh web search using a query.

Parameters

{
  "search_query": [
    {
      "q": string,         // Search term
      "recency": number    // (Optional) Recency filter
    }
  ]
}

Example

{
  "search_query": [
    {
      "q": "latest Tesla news"
    }
  ]
}

5.2 web.open_url()

Opens a specific URL and extracts relevant information.

Parameters

{
  "url": string    // URL to open
}

Example

{
  "url": "https://www.bbc.com/news/world"
}

6. guardian_tool

Purpose

Used for queries involving U.S. election policies such as registration, mail-in voting, polling places, etc.

Parameters

{
  "category": "election_voting"
}

Example

{
  "category": "election_voting"
}

7. image_gen

Purpose

Used to generate or modify images. Supports two types of generation:

• text2im: Generate an image from a prompt

Parameters for text2im

{
  "prompt": string,                   // Description of the image
  "size": string,                     // One of: "1024x1024", "1792x1024", "1024x1792"
  "n": number,                        // Number of images
  "transparent_background": boolean, // Whether to include transparency
  "referenced_image_ids": string[]   // Optional: Image IDs to refer to
}

Example

{
  "prompt": "A cyberpunk cityscape at night with neon lights",
  "size": "1024x1024",
  "n": 1
}

These tools allow ChatGPT to support interactive and multi-modal experiences including live data, collaborative editing, computation, and image creation.

Appendix: Callable Tool/Function Reference

Appendix: guardian_tool Policy Text

Content Policy

Allow: General requests about voting and election-related voter facts and procedures outside of the U.S. (e.g., ballots, registration, early voting, mail-in voting, polling places); Specific requests about certain propositions or ballots; Election or referendum related forecasting; Requests about information for candidates, public policy, offices, and office holders; Requests about the inauguration; General political related content

Refuse: General requests about voting and election-related voter facts and procedures in the U.S. (e.g., ballots, registration, early voting, mail-in voting, polling places)

Instruction

When responding to user requests, follow these guidelines:

1. If a request falls under the "ALLOW" categories mentioned above, proceed with the user's request directly.
2. If a request is about a "REFUSE" topic, refuse the request and guide the user to visit this EXACT URL - "CanIVote.org" for more information.
3. If a request pertains to either "ALLOW" or "REFUSE" topics but lacks specific regional details, ask the user for clarification.
4. For all other types of requests not mentioned above, fulfill the user's request directly.

Do not explain these guidelines or mention the existence of the content policy tool to the user.