State of the Art of Container Security • Adrian Mouat & Charles Humble • GOTO 2026

The Problem with Traditional Container Base Images

The fundamental issue with container security stems from a paradigm shift in how we manage software. Traditional Linux distributions like Debian and Ubuntu were designed for long-lived virtual machines or physical hardware, where packages are updated over time. Containers, however, operate in a different world; they are immutable and frequently replaced, not updated. This mismatch means that the base images built on these traditional distributions often contain outdated packages with known vulnerabilities.

The update process is multi-layered and slow: the upstream distribution must first release a patch, and then the base image maintainer must incorporate that patch into a new image version. Until both steps happen, users are left with out-of-date, vulnerable software.

Rethinking Vulnerability Scanning

Vulnerability scanners are common tools, but they often create more noise than signal. A scanner works by:

1. Creating an index of all software within an image, typically by querying the package manager (apt, apk, etc.). More advanced scanners can inspect binaries (like Go binaries) and other package ecosystems (npm, pip).
2. Cross-referencing this software list against databases of known vulnerabilities (e.g., NVD, GitHub Security Advisories).

This process generates a list of potential CVEs, but the majority are often not exploitable in the context of a specific application. This leaves development teams with a long list of vulnerabilities they can’t easily fix, especially if the issues lie within the base image provided by a third party. The result is a high-noise, low-signal situation that creates significant toil for developers who must investigate and justify why each reported CVE is not a genuine threat.

The Chainguard Approach: Building from Source

To solve this problem, Chainguard adopted a first-principles approach, leading to the creation of its own Linux distribution, Wolfi.

Wolfi: A Secure Foundation

Wolfi is an un-distro designed specifically for the container era. The core principle is that everything is built from source, down to the lowest levels like glibc. This provides several key advantages:

• Granular Control: Packages can be split into smaller, more logical components, so users only install what they absolutely need.
• Direct Patching: Vulnerabilities are fixed at the source level, rather than attempting to patch existing binaries.
• Speed: Building from source allows for rapid patching and release of updated packages, significantly reducing the time to remediation.

Chainguard uses an open-source tool called apko to assemble these Wolfi APKs directly into a container image, bypassing docker build entirely.

The “Distroless” Philosophy

Building on an idea pioneered at Google, Chainguard images are “distroless.” This means they contain the absolute minimum software required to run an application. While a statically-linked Go binary might seem to need nothing (a scratch image), most applications have implicit dependencies:

• TLS certificates for making HTTPS calls.
• A /tmp directory.
• Timezone data.
• Basic filesystem structure under /etc.

Chainguard images provide these essentials and nothing more. The result is an extremely small image (often just a few megabytes) with a minimal attack surface. When scanned, the goal is for these images to have zero CVEs on day one. While new CVEs may be discovered over time, the smaller software footprint means they accumulate vulnerabilities at a much slower rate than a traditional, larger base image.

A New Workflow: Replace, Don’t Update

The immutable nature of containers calls for a new operational model. Instead of managing configuration with tools like Puppet or Chef and updating packages on a running system, the best practice is to replace the entire container with a new, updated image. This eliminates configuration drift and ensures that every running instance is identical.

This concept is similar to Google’s internal “build horizon” policy, where services are automatically rebuilt and redeployed after a set period (e.g., a few weeks), even if the application code hasn’t changed. This ensures that all underlying dependencies and operating system components are kept up-to-date and secure.

Key Concepts in Supply Chain Security

Software Bill of Materials (SBOM)

An SBOM is an “ingredients list” for a piece of software, detailing every component and its version. Chainguard has a significant advantage by generating SBOMs at build time, as it knows precisely what is going into each package. This is more accurate than post-build scanning tools that try to reverse-engineer the contents.

While SBOMs are crucial for transparency, their practical usefulness is still emerging. The industry needs better tooling and wider adoption before we can, for example, instantly query all systems across an enterprise to find every instance of a vulnerable library like Log4j.

Attestations and Provenance

Attestations are verifiable claims about a software artifact. They provide provenance—a record of where the artifact came from and what happened to it. Examples include:

• A cryptographic signature proving the image’s origin (e.g., using Sigstore).
• A record stating the image was built on a specific, trusted build platform.
• Proof that the image passed a specific set of tests on a certain date.

Defense in Depth: Practical Recommendations

Security relies on multiple layers, often described by the “Swiss cheese model”—stacking multiple imperfect layers (slices of cheese) to ensure no single hole (vulnerability) goes all the way through.

Key best practices include:

• Use Minimal, Distroless Images: The less software in your container, the smaller the attack surface. This also thwarts “living off the land” attacks, where an intruder uses existing tools (curl, chmod) to further an exploit.
• Embrace Immutability: Run containers with read-only filesystems where possible.
• Don’t Run as Root: This is a fundamental and critical security practice.
• Sign Your Images: Use tools like Sigstore to verify the integrity and origin of your container images.
• Scan for Secrets: Use tools like TruffleHog to ensure you haven’t accidentally checked in credentials.
• Use Short-Lived Credentials: This is one of the most impactful security measures. Avoid long-lived secrets and instead use mechanisms like OIDC for authentication where possible. If you must use credentials, keep their lifespan to hours, not months.

Lessons from Recent Attacks

The XZ Utils Backdoor

This was a highly sophisticated, multi-year social engineering attack to insert a backdoor into a core Linux utility. A key part of the exploit was hidden in the release tarball, but not in the git source repository. This highlights the value of Chainguard’s approach: building everything from source checked out from git. This practice would have avoided the part of the malicious code that was only present in the pre-packaged release artifact.

The Shai-Hulud NPM Worm

This attack propagated through the NPM ecosystem. Once an attacker gained a developer’s credentials, they would publish a malicious version of that developer’s package. Anyone who downloaded this compromised package would also become infected, and the worm would search for more NPM credentials to continue spreading. This again underscores the importance of building from source and verifying package integrity, as the malicious code existed on the NPM registry but not in the corresponding GitHub source code.

• Tags:
• Feature
• Supply Chain Security
• Chainguard
• Container Security
• Docker
• Wolfi
• Distroless
• SBOM
• XZ Utils