On Thursday, OpenAI released the “system card” for its latest GPT-4o AI model, detailing its limitations and the safety measures implemented during testing. One of the key revelations from the document is an instance during testing where the model’s Advanced Voice Mode unintentionally mimicked users’ voices without their consent. Although OpenAI has since implemented safeguards to prevent such occurrences, this example underscores the challenges involved in designing an AI that can potentially imitate any voice from a brief audio clip.

Advanced Voice Mode is a feature of ChatGPT that enables spoken conversations between users and the AI. During testing, OpenAI noted rare incidents where the model unexpectedly generated outputs in a voice similar to the user’s. In a specific example cited in the system card, a noisy input led to the AI abruptly shouting “No!” in a voice resembling that of a “red teamer”—a tester hired to conduct adversarial evaluations of the system. This unexpected behavior highlighted the need for stringent controls over the model’s voice generation capabilities.

OpenAI acknowledged that while safeguards were in place to prevent such voice imitation, the incident was a rare exception. The company has since strengthened its defenses to ensure that this behavior does not occur in live interactions. However, the unusual nature of the event sparked reactions online, with BuzzFeed data scientist Max Woolf humorously tweeting that OpenAI had inadvertently provided the plot for the next season of “Black Mirror.”

The ability of GPT-4o to imitate voices stems from its advanced multimodal capabilities, which allow it to synthesize a wide range of sounds, including voices, sound effects, and music, based on its training data. To safely harness this capability, OpenAI typically provides the AI with an authorized voice sample-usually from a hired voice actor-at the start of a conversation. This sample is embedded in the system prompt, which serves as a guiding instruction for the AI, ensuring it imitates only the approved voice.

In traditional text-based language models, the system prompt is a hidden set of instructions that guides the chatbot’s behavior throughout the conversation. In the case of GPT-4o, which processes both text and audio, this prompt can include audio inputs, allowing the AI to imitate specific voices. OpenAI also employs a classifier system to monitor the model’s output, ensuring it doesn’t deviate from the authorized voices.

By implementing these safety measures, OpenAI aims to mitigate the risks associated with voice imitation, while continuing to explore the capabilities of its advanced AI models. The development of GPT-4o highlights the ongoing challenges and responsibilities involved in creating AI systems that interact with users in increasingly human-like ways.