Sora, ImageGen, and Codex: The Next Wave of Creative Production
Developers from OpenAI showcase "Storyboard", a custom tool built in 48 hours that transforms creative workflows. They detail how, in partnership with a film production, they used AI-assisted coding to rapidly prototype a tool that takes artists from initial sketch to high-fidelity images and even full-motion video, drastically reducing production timelines.
The Challenge: From Generic Tools to Custom Workflows
In creative fields like film and media, professionals often adapt their workflows to fit generalized, and sometimes overly complex, existing tools. The rapid emergence of new AI models presents a new opportunity: what if you could develop a tool perfectly tailored to a specific project’s creative workflow without a months-long development cycle?
This question was tested through a collaboration with Native Foreign and Vertigo Films on their animated feature, Critters. The film’s creative team wanted to partner their artists with OpenAI models in a “human-led” process, ensuring artists were driving the AI tools, not the other way around. Their initial goal was to accelerate the storyboarding process—a traditionally year-long endeavor that allows for very few iterations. The objective was to build a tool that could rapidly move from a simple sketch to a high-fidelity render, and potentially to a final cinematic frame.
“Storyboard”: A Purpose-Built Creative Application
The result of this collaboration was an internal tool dubbed Storyboard. Built using GPT models and Codex, it provides a focused workflow designed specifically for visual development.
Core Workflow and Features:
- Project-Based Organization: The application is structured around “projects.” Each project contains its own library of assets, including characters, locations, and props, ensuring consistency across a scene or the entire film.
- From Sketch to Render: An artist begins by uploading a sketch and writing a descriptive prompt. They can then select pre-configured assets (e.g., a specific character model, an underwater location) to guide the generation. These assets act as control inputs for the image generation model, ensuring the output aligns with the established visual style.
- Dual Generation Presets:
- Turbo: Quickly generates black-and-white, marker-and-ink style renderings to check composition and basic forms.
- High Fidelity: Produces full-color, highly realistic movie scenes, with specifications like “35mm photography” defined at the project level.
- Centralized Style Guide: A key feature is the project setup screen, which acts as a global style guide. Here, the creative team can define an overarching aesthetic description (e.g., “maintain the position and the composition of the initial sketch”) that is applied to every generation, ensuring a consistent look and feel.
- AI-Assisted Asset Management: The tool integrates GPT to enhance the workflow. For instance, when a new character image is uploaded, the model can automatically generate a rich text description (“a small fluffy bright yellow critter with a round chubby body”) which is then fed into the image generation prompt for more detailed outputs.
- History and Auditability: A “History” tab provides a complete audit trail, showing the initial sketch, the final generated images, the exact prompt used, and which assets were referenced. This entire history can be exported as a CSV for production tracking.
The team found that this tool was not only useful for storyboarding but also for other creative tasks like environmental and character design, allowing them to iterate on visual ideas with incredible speed.
The Development Process: Building with AI in 48 Hours
The initial prototype of Storyboard was built by a small team of five solutions engineers during a two-day internal hackathon.
- Tech Stack: The application was built with Next.js, using Supabase for the backend (managing project state, images, and authentication) and deployed on Vercel.
- AI-Powered Development: The team relied heavily on AI coding tools, including Codex and Cursor (powered by GPT models), to accelerate the development process. After building the foundational UI, they used Codex to rapidly iterate and add new features based on daily feedback from the Critters creative team.
“We would send tasks to Codeex in between meetings. We really easily reviewed and merged PRs into production, which Codex even allowed us to do from our phones.”
This AI-assisted workflow allowed them to complete nearly ten feature requests per day, turning a feedback loop that typically takes weeks into a single day. Complex tasks, such as compiling a final generation prompt based on assets from both the project and sketch level, were handled efficiently by the coding agent.
The Future: From Image to Full Motion with Sora 2
The integration of the Sora 2 API represents the next evolution for the Storyboard tool, enabling a complete workflow from sketch to image to full-motion video. The new model’s advanced understanding of motion, physics, and its ability to be guided by both text prompts and images allows for the creation of high-quality animated sequences directly from the storyboarding interface.
The key takeaway is the emergence of a new paradigm where artists and developers can collaborate more closely than ever. By leveraging AI to build custom tools, developers can transform not just the final output, but the very process of creative production, empowering artists to explore their imaginations with unprecedented speed and control.