Monster prompt, OpenAI’s business play, nano-banana and US Open experimentations

The 100-Page Prompt and the Future of Prompt Engineering

The discussion begins with KPMG’s “TaxBot,” an AI agent designed to generate 25-page advisory opinions for clients. The agent is powered by a surprisingly long 100-page prompt, sparking a debate on the role and complexity of prompt engineering.

While the prevailing narrative a year ago suggested prompt engineering would become obsolete, this example suggests the opposite. To elicit specific, high-quality agentic behavior, extensive specification is required. The panel compared this to open-source agent projects like GPT Researcher and MetaGPT, which also rely on long, complex prompts refined by community contributions.

Several reasons for such long prompts were proposed:

• Domain-Specific Knowledge: For specialized fields like tax, a long prompt can act as a substitute for knowledge not present in the base model. It’s a way to inject extensive domain knowledge directly into the context window, which can be seen as an alternative to Retrieval-Augmented Generation (RAG). One panelist argued that putting all necessary information into the context window is more reliable than “rolling the dice” with RAG to retrieve the correct chunks.
• Real-Time Data: In domains where information changes rapidly, like tax regulations or stock market data, prompts are essential for providing the model with the most current information that wouldn’t be in its training data.
• Focusing Attention: A detailed prompt acts like a “flashlight,” directing the model to focus on specific data and ignore the vastness of its internal knowledge base. This can be paired with techniques like Low-Rank Adaptation (LoRA) to further guide the model’s attention mechanism for more precise results.
• Ease of Use for Non-Experts: For users like accountants, the immediate feedback loop of editing a prompt is far more accessible and satisfying than engaging in a complex fine-tuning process.

However, the panel also viewed a 100-page prompt as a potential sign of a poorly designed system, akin to a product requiring a 100-page user manual. Fine-tuning was presented as a more robust, long-term solution, especially with newer model architectures specifically designed to be tuned efficiently. The key question is reusability: how much of the 100-page prompt is a custom solution versus a reusable template for any agent? A team building an enterprise agent would likely be more invested in simplifying and reusing prompts across different business lines rather than just seeking the quick results of iterative prompt engineering.

OpenAI’s Potential Pivot to Infrastructure

An offhand comment from OpenAI’s CFO suggested the company might eventually enter the infrastructure game, selling raw compute power in a model similar to Amazon Web Services (AWS). This would mark a significant shift from their current business of selling model access.

The primary motivation appears to be economic. The pace of hardware advancement is so rapid that state-of-the-art GPUs become obsolete for frontier model training within 24 months.

“Four years ago we had A100s, two years ago we had H100s. This year we have Blackwell. So every two years they have to refresh their whole fleet. Yet the actual lifespan of these GPUs is like five years.”

Selling access to this “last season’s” hardware would allow OpenAI to recoup its massive infrastructure investments. These older GPUs, while not sufficient for OpenAI’s cutting-edge research, are still powerful enough for many customers’ large-scale inference workloads.

This move is also seen as a strategic effort to rebalance its deep dependency on Microsoft Azure, shifting the relationship from dependency to collaboration. However, the panel raised questions about OpenAI’s ability to compete in this crowded space against established players like Google Cloud and AWS, as well as the thriving open-source ecosystem for inference optimization (e.g., vLLM, TensorRT-LLM). The success of such a venture would depend on whether OpenAI could offer a specialized infrastructure that provides superior performance to these alternatives.

Gemini’s “Nano-Banana”: A Leap in Image Generation

The launch of Google’s new image model, nicknamed “Nano-Banana,” was hailed as a significant advancement, with one panelist calling it “by far the best image generation model that I’ve seen today.” It reportedly scores 200 ELO points ahead of competitors on benchmarks.

Key capabilities that set it apart include:

• Superior Quality and Text Rendering: The model produces exceptionally high-quality images and, unlike many predecessors, handles text within images accurately.
• Multi-Turn Editing and Context Retention: Its most impressive feature is the ability to maintain the context of an image across multiple edits. A user can take a subject, place them in a new scene, change their facial expression, or add other elements, and the model preserves the subject’s identity and adapts the physics of the scene accordingly.
• Broad Applicability: These capabilities have major implications for creative industries, potentially disrupting tools like Canva and Photoshop for tasks like creating thumbnails, editing photos, and filtering images.

The panel noted that this release marks a significant comeback for Google, which now appears to be a strong contender for leadership in the AI space, particularly in image generation. While concerns about AI-generated images eroding trust in reality persist, the gradual improvement of models has also made the public more adept at spotting fakes. The discussion also highlighted positive, non-malicious use cases for editing, such as using synthetic data generation to remove cloud cover from satellite imagery to improve geospatial foundation models.

IBM Watsonx Powers New Fan Experiences at the US Open

For over 30 years, IBM has partnered with the US Open to integrate technology into the fan experience. This year, they introduced three new features powered by watsonx generative AI models, aimed at providing deeper storytelling for the millions of fans who follow the tournament digitally.

1. Match Chat: A real-time, agent-driven assistant that allows fans to ask questions about players and matches. The system is built with sophisticated NLP pipelines that can perform pronoun and player correction to provide accurate, context-aware answers through a frictionless user interface.
2. Key Points: To address that many fans don’t have time to read long articles, this feature uses summarization models to generate concise bullet points that appear at the top of match recaps and other content. The process includes a human-in-the-loop workflow with USTA editors.
3. Live Likelihood to Win: This feature combines an ensemble of predictive models to generate a pre-match win probability. As the match unfolds, proprietary equations update this probability in real-time, factoring in momentum shifts and key events like break points. This provides fans with a dynamic, data-driven narrative of the match’s progression.

• Tags:
• Prompt Engineering
• Gemini
• OpenAI
• GPU
• Cloud Infrastructure
• KPMG
• Image Generation
• Nano Banana
• IBM
• Watsonx
• US Open
• Sports Analytics