High Taste Testers: OpenAI Shipped Everything. What Actually Works?
A roundtable of AI power users reviews OpenAI DevDay and asks what actually works in daily use. Topics include personal assistants and their trust, permission, and safety trade-offs; durable agent state and remote execution; the Jev decision model and calibrated classification; model preferences across Anthropic, OpenAI, xAI, Meta, and open models; subagents and context management; and the gap between impressive capabilities and products that hold up.
Overview
This is the first High Taste Testers episode, a roundtable of AI power users recorded right after OpenAI DevDay. The conversation moves from DevDay’s release cadence, to personal assistants and the question of whether labs can ship products, to decision models (Jev), to model preferences across Anthropic, OpenAI, xAI, Meta, and open models, and finally to practical workflows for delegation and everyday use. The throughline is a recurring gap between impressive capabilities and products that actually hold up in daily use.
DevDay Impressions and Product Execution
The group’s first reaction is that DevDay was less about the event and more about the individual releases: Dots, a Sora-style 6.1 model, an ultra-fast tier, and a Notion-like shared workspace. Several participants noted that features are buried deep in the UI. Canvas-like collaboration exists, but users have to know to ask for it. The same is true of the Codex visualization feature and of a hidden Maps feature inside ChatGPT. One participant said the hiding is partly deliberate: the company protects its 8 million casual users by keeping the powerful features tucked away for power users.
The group broadly agreed that the labs are strong on research but weak on product. A quote captures the sentiment: “the labs just really suffer on product.” Codex desktop was praised at launch, but successive updates were described as regressions, including an extra sidebar, worse performance, and an input field that freezes when the connection drops. The Codex-ChatGPT merger drew criticism too, with one person noting that forking a conversation was renamed “copy” because non-developers supposedly wouldn’t understand “fork.” The group suspected that AI-written code may be part of why products change constantly and feel directionless.
A key tension is that the middle-ground products try to serve everyone and end up serving no one. Grokbot was praised as dialed in for professional personal assistance, while Muse was described as excellent for consumers. Participants argued that enterprise and consumer assistants are fundamentally different products, and that a shared Slack presence for personal bots raises unresolved questions about who is actually replying.
Remote Execution and Durable State
A significant unsolved problem is where the work runs. OpenAI’s “close your laptop, it keeps working” promise did not survive contact with reality. One participant had to leave a laptop open at the office so an agent could keep running. The core engineering challenge is externalizing agent state so it can move between local and cloud. Pi reportedly did this with a durable, append-only log, which one participant described as the actual internal representation used by Claude Code and Codex in large JSONL files. But the labs have not committed to stable APIs for those formats. Several people called for a standard agent trace format, pointing to Justin’s effort, though so far it exists mainly for research and benchmarking.
Agents Versus Assistants
The group argued that the industry’s vocabulary is shifting. 2025 was framed as the year of agents and 2026 as the year of proactive agents, but “agent” sounds scary to non-developers. “Assistant” is more familiar because wealthy people have them. The proposed pattern is something always-on that reads email, filters what matters, schedules, and reduces the user’s load rather than adding to it. A central observation was that “always-on, persistent, messages other things and kicks off work” is the right idea, though nobody has nailed the product yet.
The gap between coding agents and assistants also came up. Coding agents are trained to produce diffs, markdown, and tests. A personal assistant needs to answer conversationally and act on the user’s behalf. One participant put it bluntly: he wants his model to be a model for him, not a system prompt that asks what he wants to code today.
Building a Personal Assistant
Several participants shared working setups. One used Pi with its own Gmail, credit card, reminders, and cron jobs, calling it “better than Dots” and describing it as a Codex clone that just works. Another kept a git repo of markdown files and scripts that the agent uses as its memory and tooling, launching Codex inside it when needed. The argument for this approach is that a normal coding agent with cron jobs and a shared drive is already most of what an assistant needs. The counterargument is that a polished mobile and web experience with minimal memory configuration is what makes Grokbot compelling, and that engineers can build polished products but often don’t prioritize it.
The group also discussed operational overhead. Hosting another box means maintaining another piece of infrastructure, which defeats the goal of reducing load. One person’s Mac Mini disconnects and the assistant goes offline, and the response was “are you there?” becomes the actual failure mode.
A practical theme was trust and permissions. Participants generally agreed on read-only access with explicit approval before sending or spending. One user connected email to Dots but still wouldn’t give an agent write access, and said they don’t trust Meta with personal email, while they feel more comfortable with OpenAI’s data retention settings. Another reviews every outgoing message from Grokbot and found roughly one in five drafts contained details they didn’t want shared. Virtual card tools like Link were discussed as a way to let an agent make purchases while limiting exposure, though the speaker noted they still couldn’t set individual limits or expirations.
Prompt Injection and Safety
One participant argued that prompt injection is largely solved in newer models like Astra and Opus 5.5, which reportedly maintain a strict hierarchy between constitution, system prompt, and untrusted content. Others pushed back strongly. A friend’s OpenClaw was attacked, and a Muse incident showed the agent volunteering information to a Facebook Marketplace seller. The more worrying concern raised was not malicious injection but a helpful model that over-shares or misclassifies what is safe to reveal. Several people said they have seen real-world attempts and that the risk is not limited to adversaries.
Decision Models and Jev
The group spent considerable time on the decision model Jev. The most valued property is access to output probabilities, which the major providers had mostly stopped exposing. Log probabilities let developers see the confidence of a decision, though one participant noted that calibration in earlier structured outputs was poor: asking for a 0 to 100 score bucketed into a handful of values. Jev appears to give true per-decision probabilities, which makes it useful for classification at scale.
Use cases included ranking command-K search results, which previously required a choice between a local lookup and a streamed LLM call. A fast decision model can do both. Several participants said they had used structured outputs extensively and had even trained their own classifiers. Switching costs are low when you already have labeled data: you can use existing labels as evals and only review disagreements. One participant said they used a poor man’s Jepa-style loop, adding failing cases as evals and tuning prompts until they pass.
The group was critical of viral examples, which they felt were mostly silly. Binary classification was described as deceptively hard. A paper claiming hallucination is a binary classification problem was cited as an example of oversimplification. Benchmarks were also questioned, since overfitting to a narrow domain is easy and Jev’s appeal depends on diverse data. One participant said Jev reduced their agentic memory pipeline cost by 10 to 20x while remaining well aligned with Opus 5.5, even without reasoning. Another observed that frontier models without reasoning are poor classifiers, so few-shot examples are needed.
The group also discussed competition. Cloudflare’s small model was reported to match Jev and run faster, and its large model was reported to outperform it. A participant speculated that on-device versions will arrive, with WebGPU already able to run something close to Jev. Another noted that OpenAI will likely benefit from the brand and from its deep knowledge of how to build harnesses around such primitives.
Model Preferences and Writing Quality
Opinions on models were strong and somewhat contradictory. Astra in ultra mode was used heavily by one participant, who warned that it can burn through a weekly reset in about 20 minutes. Opus 5.5 was the preferred model for writing, with one person arguing that GPT models lack a soul by design. OpenAI’s Sam Altman reportedly acknowledged that OpenAI models are not strong at writing and is working to close that gap.
Anthropic’s model history came up as well. One participant said Opus 4.6 was the last great Opus, and that the summarization product’s customers complained when they switched to 4.8. Anthropic reportedly acknowledged that earlier Opus versions had degraded, which the group found unusual for them. Several participants thought Anthropic’s internal engineers mostly use unreleased models like Mythos and Fable rather than released ones, a gap that makes external experience less representative.
The group also noted that Opus and Sonnet were used as subagents, and that Sonnet has some writing ability but is still well behind Opus. Haiku 4.5 was discussed as nearing its one-year anniversary, and Luna was praised for speed on small tasks. Naming confusion was a common complaint: OpenAI’s Terra was never adopted, and the rollout of Soul followed by 6.1 Soul in quick succession made the lineup hard to follow.
On Grok, participants were largely negative. Grok 4.7 was called a disappointment because it is expensive, with pricing that doubles over 200k context and doubles again in fast mode. Token efficiency was described as poor. Its main advantage is real-time Twitter data and the Grokbot product itself. A bug briefly prevented Grok on X from recognizing users, although Grok.com recognized them. Grok’s X API access was praised as a free research tool that other agents can call as a subagent.
Muse Spark was praised for a million-token context, multimodal input including video and audio, and lower cost than Gemini. One participant said it is good enough for most assistant tasks but not for coding.
Open Models and Local Inference
The group was skeptical about the gap between open and frontier models. One participant ran GLM and DeepSeek-style models on DGX Sparks and found them useful for hill-climbing problems with verification but not for daily work. The frontier moved sharply with Opus 5.5 and 6.1 Soul, and open source was estimated to be a year behind. A benchmark called Slop Code Bench found that Opus 5.5 wrote about three times as much code as comparable models and that about half of it was tests, a pattern the presenter found worth publishing.
The discussion of capability generations introduced the idea of a ceiling. Models approach a plateau and cannot cross it until a breakthrough happens. The group felt that Astra, 6.1 Soul, Opus 5.5, and Fable 5.1 represent a new step function, distinguished by swarming behavior and trustworthiness rather than raw benchmark scores.
Subagents, Context, and Compaction
Subagents were a major theme. One participant noted that Codex subagents are significantly faster and cheaper than separate sessions because of cache optimizations. Setting up dozens of subagents can burn through limits quickly, so configuration matters. A participant set compaction to 400k, though the default is around 275k. Several people disliked summary-based compaction and wanted an option to strip large content like images or tool outputs instead, or to use a decision model for tool calls. Claude Code’s compaction was reported to sometimes make the agent behave as if it were a new person who doesn’t remember the work.
Workflow patterns varied. One person runs a “poor man’s Ralph loop” by queuing five to ten prompts that run overnight, and another runs about 40 background agents in parallel through a detach-mode skill. A third keeps the main thread as a pure conversation and delegates all execution to subagents with timers, pushing them until they finish. Someone else does the opposite, doing work in the main thread and discussing it on the side in an ephemeral thread. The group agreed that long-running work benefits from separating orchestration from execution, but disagreed about where the conversation should live.
One warning: overnight agents can produce more code than a human can reliably consume.
Delegation, Computer Use, and Everyday Workflows
Participants described how they divide tasks across tools. One uses Codex for coding and maintains a Hermes agent along with around 30 Grokbot instances, each specialized, communicating through Linear tickets. Another routes work across Grok for management, Muse for quick voice-based questions, and Dots for other tasks, sending the same task to all three to compare. Grokbot can kick off Cursor cloud agents with Opus 5.5 for tasks that need more capability, which one person described as significantly better than Grok 4.7.
Cursor was criticized for requiring per-project setup, environment variables, and scripts, and for loving to open PRs when the user just wants an answer. Devin was praised for behaving like a general computer rather than being tied to one repository, which makes it easy to try new ideas in a sandbox.
An important point was the attention cost of switching between assistants. One person can no longer remember which conversation happened in which assistant, and importing memories between them doesn’t work well because the clones diverge. One participant’s solution is Executor, a central MCP gateway that holds Gmail, Drive, Notion, and other connections on a home Linux box, reachable through a Cloudflare proxy. Each assistant connects to that single gateway and inherits the same tools, skills, and config. The main trade-off is that some connectors, such as Grokbot’s finance connector, are still better used natively.
In terms of everyday use, the group argued that consumer value lies in saved time rather than benchmark performance. One participant built a weekend-planning bot for family outings after arriving at a closed karate class, and said it finds places they would not discover otherwise. Google’s AI overview and quick answers were praised for speed, and some participants used Instinct instead of Google for quick travel questions such as where to catch a train in Amsterdam. The group recognized that the chat app is often a suboptimal entry point and that routing requests through a single interface loses context.
Hardware and Closing Thoughts
Hardware integration was seen as the next frontier. Grokbot already runs in Tesla vehicles through voice input. Meta’s glasses and the Tamagotchi-style Charm device were considered likely to sell out. A short-battery-life objection was raised: people already have too many devices to charge, and Apple Watch was cited as an example of the charging burden.
The closing advice was to unlock imagination and experiment with AI assistants, since many people are still in doomer mode and need concrete examples of time saved. Several participants plugged their work, including a podcast called Thursday I, open source models coming soon, and a company that makes agents more reliable.