State of the Art of Container Security • Adrian Mouat & Charles Humble

Adrian Mouat from Chainguard discusses the fundamental mismatch between traditional Linux distributions and the modern, immutable nature of containers. This mismatch is a primary source of security vulnerabilities, as base images built on distributions like Debian or Ubuntu were designed for long-lived systems, not for containers that are frequently replaced. This leads to outdated packages lingering in images, creating a constant stream of alerts from vulnerability scanners.

The Challenge of Vulnerability Scanning

Vulnerability scanners work by creating an inventory of all software within an image—using package managers, binary analysis, and other methods—and cross-referencing this list against databases of known vulnerabilities (CVEs). However, this often results in a high signal-to-noise ratio. Development teams receive long lists of CVEs, many of which may not be exploitable in their specific application’s context. When a vulnerability is found in a base image for which the team is already using the latest version, they are left with few options, leading to a frustrating cycle of investigation and justification with little actual security improvement.

Chainguard’s Approach: Distroless and Building from Source

Chainguard’s strategy is rooted in the “distroless” philosophy, which originated at Google. The core idea is to create minimal container images that contain only the absolute necessities for an application to run. This means stripping out shells, package managers, and other common utilities, which dramatically reduces the image size and, more importantly, the potential attack surface.

Chainguard takes this a step further with its own Linux distribution, Wolfi. Key aspects of this approach include:

• Building Everything from Source: Unlike other distributions that may patch existing binaries, Chainguard builds every package from source, down to the glibc level. This provides granular control and ensures that security fixes are applied at the source, not as an afterthought.
• Minimal Packages: Building from source allows Chainguard to create highly optimized, small packages (APKs) that include only necessary components.
• Zero-CVE Goal: The primary goal is for Chainguard images to have zero known vulnerabilities upon release. While new CVEs can be discovered over time, the minimal nature of the images means the CVE count grows much more slowly compared to traditional, larger images.

The Immutable Paradigm: Replace, Don’t Update

A central tenet of modern container management is immutability. Instead of logging into a container to run apk update, the recommended practice is to replace the entire container with a new, updated image. This approach, pioneered by companies like Google, eliminates configuration drift and ensures that every running instance is identical and verified. This is reminiscent of Google’s “build horizon” concept, where services are automatically rebuilt and redeployed on a schedule, regardless of feature changes, to ensure all underlying dependencies are kept up-to-date.

Software Supply Chain Security Tools

The conversation also explored key concepts for securing the software supply chain:

• SBOMs (Software Bill of Materials): An SBOM is a detailed inventory of all components within a piece of software. Because Chainguard builds everything from source, it can generate highly accurate SBOMs at build time. While the tooling and universal adoption are still maturing, SBOMs are a critical step toward being able to quickly identify where vulnerable components are being used across an organization.
• Attestations and Signatures: An attestation is a verifiable, signed claim about an artifact, such as its build environment, the tests it passed, or its origin (provenance). Using tools like Sigstore, images can be cryptographically signed, allowing consumers to verify that an image is authentic and has not been tampered with.
• Defense in Depth: Security should not rely on a single layer. This “Swiss cheese model” involves multiple safeguards, including:
  • Using minimal images to thwart “living off the land” attacks where attackers use existing tools on the system.
  • Running containers with read-only filesystems.
  • Never running processes as the root user.
  • Using short-lived credentials or, preferably, OIDC-based identity to eliminate the risk of long-lived secrets being stolen.

Lessons from the XZ Utils and NPM Attacks

The sophisticated XZ Utils backdoor highlighted the value of building from source. Part of the malicious payload was injected only into the release tarball, not the source code in the git repository. A build process that pulls directly from git would have been protected from that specific vector.

Similarly, attacks like the Shai-hulud worm on the NPM ecosystem, which spread by publishing malicious package versions, underscore the importance of building from a trusted source repository rather than blindly trusting packages from a public registry. These incidents also reinforce the critical need for short-lived credentials, as the attacks often propagate by stealing developer credentials.

• Tags:
• Vulnerability Scanning
• Chainguard
• Container Security
• Docker
• Wolfi
• Distroless
• SBOM
• XZ Utils
• Software Supply Chain
• Immutable Infrastructure