When you’re setting up your infrastructure to deploy that first demo service container, it’s easy to neglect the enforcement of best practices in the interest of moving fast. Before you know it, critical workloads are running, but best practices aren’t being followed. This can lead to that one resource-hungry service consuming all the memory and bringing down other services, or a third-party library exploiting privileged user access to sinister ends. 

But there’s a solution to these scenarios: continuous compliance. By investing in compliance from your infrastructure’s inception, you’ll benefit from a more secure and predictable environment from day one. Best of all, you can enforce these practices through automation, reducing developer toil. And if you already have critical workloads running in your infrastructure, it’s never too late to start leveling up.

Policy enforcement, step by step

A container image can be deployed through a service’s build and deployment pipeline, manual operation by a human, and external event triggers, such as scheduled jobs. These processes suggest two possible surfaces for your policy checkpoints: internally, inside the container infrastructure, and externally, in the build and deployment pipeline. 

For maximum coverage, aim to implement policy enforcement at both of these checkpoints. Start by integrating policy enforcement into your container infrastructure; this ensures that no matter what pathway an image takes, it’s subject to these checks. If you’re pressed for time, you can choose to address checks in the build and deployment pipelines at a later stage. (Only implementing checks in your infrastructure can be more cumbersome because the policy enforcement happens later in the deployment cycle, meaning you may have to revert a pull request or make multiple changes.)

In your container infrastructure

To enforce policies in your container infrastructure, you’ll want to set up a dedicated software component: a policy engine. Whatever software you use, it should be tightly integrated with the infrastructure so it can independently decide whether to allow a container to be deployed. 

If you use Kubernetes, Gatekeeper is an example of an off-the-shelf policy engine you can set up in your cluster. It configures admission controller webhooks in the cluster to make decisions based on the configured policies and prohibits workloads that violate these policies from running. For example, a container trying to run as a root user or using a non-tagged image won’t be allowed to start. Organizations that don’t use Kubernetes may need to invest in creating a custom solution using dedicated policy management software such as Open Policy Agent, which also works with Kubernetes.

If you’re rolling out policies to existing infrastructure, consider running your policy engine in audit mode first to identify any policy-violating workloads currently running in your cluster. Once you fix them—likely by changing how the Docker image is built or how the deployment configuration is written—you can turn off the audit mode and redeploy your workload.

Containers can have very short life spans, which poses an additional compliance challenge. If a malicious workload, like a crypto miner, only “wakes up” every few minutes or hours, it might not be around by the time you realize something’s going on and want to inspect it. I recommend adopting a solution that stores and logs container activity so you can query the logs at a later time if needed. Solutions such as Falco are able to go a step further, blocking suspicious processes at runtime. 

In your build pipeline

To ensure compliance in your build pipeline, implement static checking of Dockerfiles. Static checkers like Hadolint or Conftest will check for adherence to best practices and other custom policies, such as using a trusted base image or the presence of a specific label instruction. Since static checks run before the container image has been built, the result is a fast, automatic feedback loop.

Next, scan the container image using a vulnerability scanner such as Clair or Snyk. If the scanner finds a potential security issue, the image won’t be pushed to the registry. You’ll also want to enable container image scanning in the registry itself; most hosted and open-source registries offer this as a default. Finally, make sure you set up notifications so you’re alerted to, and can act on, any issues.

In your deployment pipeline

As with your build pipeline, static checking for Kubernetes manifests in your deployment pipeline will result in a fast feedback loop. These checks, using a tool like Conftest, should look for things like a valid service account in the deployment specification or the presence of a valid resource requirements specification. 

If you’re using a continuous deployment solution, perform these checks before you apply a change. This is often done using a hooking mechanism, in which a custom script can be executed before the application of the manifests. If you’ve adopted the GitOps model for managing your infrastructure code, you’d typically run these checks before a pull request is merged.