Container security testing is four categories: image scanning (Trivy, Grype, Snyk), cluster configuration checks (kube-bench, docker-bench), policy as code (Conftest, OPA, Kyverno), and runtime tests (Falco, chaos engineering with LitmusChaos). The team that runs the first two in CI has the foundation; the team that adds the third has enforcement; the team that adds the fourth has resilience.
Table of contents
- Layer 1: image scanning (the foundation)
- Layer 2: configuration checks (the CIS benchmark)
- Layer 3: policy as code (the enforcement)
- Layer 4: runtime tests (the resilience)
- The right test pyramid for container security
- How this fits the rest of the stack
- FAQ
Layer 1: image scanning (the foundation)
Trivy - the default pick. Scans OS packages, language deps, IaC, filesystems. Free, fast, easy. CI integration:
- name: Build image
run: docker build -t myapp:${{ github.sha }} .
- name: Scan image
run: trivy image --severity HIGH,CRITICAL --exit-code 1 myapp:${{ github.sha }}
Grype - similar coverage, Anchore’s scanner. Slightly different vulnerability database than Trivy. The team that wants two scans for cross-validation runs Trivy and Grype in parallel.
Snyk - commercial, deeper on language dependencies. The team that has 100+ repos with deep npm trees picks Snyk.
Docker Scout - built into Docker Hub, automatic on push. The team that uses Docker Hub gets this for free.
Layer 2: configuration checks (the CIS benchmark)
kube-bench - the Kubernetes CIS benchmark checker. Runs as a Kubernetes Job, checks the cluster and node configuration against the CIS Kubernetes Benchmark. The team that has a compliance requirement (SOC 2, PCI-DSS, HIPAA) runs this on every cluster.
# Run as a Job in the cluster
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
kubectl logs job.batch/kube-bench
docker-bench - the Docker CIS benchmark checker. Runs against a Docker daemon.
The team that runs kube-bench and finds 50+ failed checks has a cluster that is non-compliant by default. The team that remediates the most critical (privileged containers, host network, anonymous auth) has the foundation.
Layer 3: policy as code (the enforcement)
OPA (Open Policy Agent) - the policy engine. Rego language. The team that has any policy beyond “always allow” uses OPA.
Conftest - OPA’s test runner. Apply policies to YAML/JSON files (Kubernetes manifests, Terraform, Dockerfile):
conftest test deployment.yaml
Example policy (policy/deployment.rego):
package main
deny[msg] {
input.kind == "Deployment"
not input.spec.template.spec.containers[0].resources.limits
msg := "containers must have resource limits"
}Kyverno - Kubernetes-native policy engine. YAML policies instead of Rego. The team that wants to write policies in YAML rather than Rego uses Kyverno.
The team that runs policy as code in CI has rules enforced at deploy time. The team that runs them as a separate audit has rules that get ignored.
Layer 4: runtime tests (the resilience)
Falco - runtime detection. Watches syscalls for suspicious behavior. Covered in the container security software post.
gVisor - a user-space kernel for containers. Limits the kernel surface the container can reach. The team that runs untrusted multi-tenant workloads (e.g., a CI that builds user code) uses gVisor or Kata Containers.
Kata Containers - run containers in lightweight VMs. Stronger isolation than gVisor but more overhead. The team that needs VM-grade isolation for a specific workload uses Kata.
Chaos engineering (LitmusChaos, Chaos Mesh) - inject failures to test resilience. The team that wants to verify “what happens if this pod dies” runs chaos experiments.
The right test pyramid for container security
Bottom (most frequent, fastest):
-
Image scan in CI on every PR - 30 seconds to run.
-
Policy as code in CI on every PR - 1 second to run.
Middle (less frequent):
-
kube-bench on cluster changes - 5 minutes to run, weekly or on upgrades.
-
Grype/Trivy SBOM diff between deployments - 30 seconds to run.
Top (rare, expensive):
-
Full security audit with external pen tester - quarterly.
-
Chaos engineering day - monthly.
The team that has the bottom of the pyramid solid has the foundation. The team that tries to do chaos engineering without image scanning has the priority wrong.
FAQ
How often should I run image scans?
On every build, in CI. The team that scans on every PR has a stream of CVE-driven PRs (good - the alternative is finding out in production). The team that scans weekly has surprises.
What is the difference between Trivy and Grype?
Trivy is from Aqua Security, Grype is from Anchore. Both scan images for CVEs. The differences: vulnerability database (Grype uses Anchore’s feed, Trivy uses a combination), speed (similar), output format (Trivy is more structured). The team that picks one based on integration with their existing tools.
Do I need a service mesh for runtime security?
No - Falco (and similar) provide runtime detection without a service mesh. A service mesh adds mTLS and traffic management on top, which is separate from runtime security.
How do I write my first OPA policy?
Start with a simple deny rule: “all containers must have resource limits”. Conftest + a 5-line Rego file + a CI step is the minimum viable policy-as-code. The team that gets the first policy in CI has the foundation.
What is the difference between OPA and Kyverno?
OPA is the policy engine, Kyverno is a Kubernetes-native policy engine that uses YAML policies. OPA is more powerful (Rego is a full language); Kyverno is easier (YAML is familiar). The team that wants to enforce simple rules quickly picks Kyverno; the team that needs complex policies picks OPA.
How this fits the rest of the stack
For a sense of what the full project costs before it commits, the RunxBuild hosting calculator shows the line items together. The API, the database, the storage, the worker, the bandwidth - each one is a separate number, and the team’s mental model for the platform is the sum of those numbers.
Useful related references: