Skill Issue: How We Used AI to Make Agents Actually Good at Supabase — Pedro Rodrigues, Supabase

The Role of Agent Skills

Agent Skills are a critical component for improving the performance of AI agents, particularly when interacting with complex, multi-product platforms like Supabase. The focus is shifting from traditional Developer Experience (DX) to Developer Agent Experience (DAX), aiming to make platforms as “agentic-friendly” as possible. Skills serve as the “secret sauce” in this endeavor, providing agents with custom information, repeatable workflows, and new capabilities through scripts.

Anatomy of a Skill: Progressive Disclosure

A common misconception is that a Skill is just a single markdown file. In reality, a Skill is a folder containing instructions and files. The core of a Skill is the skill.md file, which contains two essential front-matter fields:

• name: Identifies the Skill.
• description: Explains to the agent what the Skill does.

The key innovation that Skills introduce is the concept of progressive disclosure. Instead of loading the entire content of a Skill into the agent’s context at once, only the front-matter (the name and description) is initially loaded. This acts as an “envelope,” allowing the agent to understand what the Skill does and decide when to load the full content, thereby saving valuable context space.

A Skill can be structured like a book:

• skill.md: The main index file, containing instructions and references.
• reference/ folder: Contains other markdown files (like chapters or pages) that can be linked from the main skill.md file. You can even create a graph of references between these files.
• Scripts: Skills can also include and reference scripts (e.g., Bash, Python), enabling the agent to perform actions.

Skills vs. MCP Tools: A Complementary Relationship

There is often a debate about whether to use Skills or MCP (Machine-Credible Portability) tools. The two are not mutually exclusive but rather complementary, comparing them is like “apples to oranges.”

• MCP Tools are ideal for building integrations, especially when the agent does not have access to a local bash environment. Tools can be called by the agent and executed on a remote server, abstracting away the execution environment.
• Skills provide context and define workflows that are too detailed for an MCP tool’s description. They can orchestrate the use of MCP tools or run local scripts. Scripts included in Skills run in the user’s local environment and are tied to that specific OS (Linux, macOS, etc.), unlike server-side MCP tools.

The best practice is to use both: MCP for the integration and the action, and a Skill to describe how and when to use that action effectively.

A Framework for Testing: Eval-Driven Development

Since Skills are essentially markdown files that influence a non-deterministic LLM, testing them requires a different approach from traditional code testing. This is where evaluations (or “evals”) come in. Evals are a way to test the behavior of an agent or model.

A systematic, iterative framework for developing and testing skills, similar to test-driven development, can be followed:

1. Define Metrics: Start by defining what “good” performance means for the Skill. What specific behavior or outcome do you want to achieve?
2. Create the Skill: Write the skill.md file, along with any necessary reference files or scripts.
3. Run Evaluations: Test the Skill, either manually or through an automated set of evaluation scenarios. An eval scenario typically consists of an input, an expected output, and assertions about the agent’s behavior (e.g., steps taken, tools called).
4. Grade Results: Assess the agent’s performance against the defined metrics. This can be done deterministically (checking if a specific tool was called) or non-deterministically using an “LLM as a judge” to grade the outcome.
5. Iterate: Based on the results, refine the Skill and repeat the cycle.

Practical Demonstration: Fixing a PostgreSQL Security Flaw

The workshop demonstrated this process with a real-world scenario:

• Task: An agent is asked to create a SQL VIEW in a Supabase project to show departmental statistics (headcount, average salary).
• Without a Skill: The agent successfully creates the view. However, it introduces a critical security flaw. In PostgreSQL, when a view is created on a table with Row-Level Security (RLS) enabled, the view by default bypasses those RLS policies. It uses the permissions of the creator of the view, not the user querying it. This resulted in every employee, regardless of their role, being able to see sensitive salary data for the entire company.
• With a Skill: A supabase-security Skill was introduced. This Skill contained crucial, domain-specific knowledge about PostgreSQL security. Specifically, it instructed the agent that when creating views on tables with RLS, it must include the WITH (security_invoker = true) flag. This flag ensures that the view respects the RLS policies of the underlying tables.
• The Result: When the agent was given the same task with the Skill loaded, it correctly included the security_invoker flag in the CREATE VIEW statement, thereby fixing the security vulnerability. This powerfully illustrates how a Skill can guide an agent to follow best practices and avoid subtle but critical errors.

Automating Evaluations

The workshop also presented a simple framework for automating this testing process.

• An eval.json file defines the test cases, including the prompt, expected output, and assertions.
• A Python script runs the agent in a headless mode, once with the skill and once without it.
• The script then compares the outputs, deterministically checking if the Skill produced the desired change (e.g., the presence of the security_invoker flag in the generated SQL).

This automated pipeline is essential for ensuring that changes to a Skill don’t cause regressions and that it consistently performs as expected before being deployed to production.

• Tags:
• AI Agents
• LLM
• Evaluation
• Security
• Feature
• MCP
• PostgreSQL
• Agent Skills
• Supabase