Container Security Checklist

Homayoun

10 min readMar 24, 2022

Cloud Native Concepts

Table by Aqua Cloud Native Security Platform, more details download here

Container Threat Model

Figure by Container Security by Liz Rice

Insecure Host
Misconfiguration container
Vulnerable application
Supply chain attacks
Expose secrets
Insecure networking
Integrity and confidentiality of OS images
Container escape vulnerabilities

Container Security Checklist

Checklist to build and secure the images across the following phases:

Secure the Build
Secure the Container Registry
Secure the Container Runtime
Secure the Infrastructure
Secure the Data
Secure the Workloads

Secure the Build

Secure Supply Chain

Know where images, packages came from.

Hardening Code — Secure SDLC (Software Development Life Cycle)

Do a static analysis of the code and libraries used by the code to surface any vulnerabilities in the code and its dependencies.
Improve the security and quality of their code. OWASP Open Source Application Security tools: SAST, IAST.

Secure the Image — Hardening

You can build the container images using Docker, Kaniko.

Reduce the attack surface

Package a single application per container. Small container images. Minimize the number of layers.

Use the minimal OS image:
Alpine images
Scratch images
Distroless images
Use OS optimized for running containers:
Flatcar images
CodeOS by Fedora replaced the Project Atomic.
Bottlerocket by Aws
k3os by Rancher
Container-Optimized OS — COS by Google, based on Chromium-os used by Google
Do you use Alpine, distroless or vanilla images? …
7 Google best practices for building containers
Multi-staged builds.

A well-designed multi-stage build contains only the minimal binary files and dependencies required for the final image, with no build tools or intermediate files. Optimize cache.

Use official base images.
Avoid unknown public images.
Rootless. Run as a non-root user. Least privileged user
Create a dedicated user and group on the image.

Do not use a UID below 10,000. For best security, always run your processes as a UID above 10,000. Remove setuid and setgid permissions from the images

Avoid privileged containers, which lets a container run as root on the local machine.
Use only the necessary Privileged Capabilities.
Drop kernel modules, system time, trace processes (CAP_SYS_MODULE, CAP_SYS_TIME, CAP_SYS_PTRACE ).
Enable the --read-only mode in docker, if it's possible.
Don’t leave sensitive information (secrets, tokens, keys, etc) in the image.
Not mounting Host Path.
Use Metadata Labels for Images, such as licensing information, sources, names of authors, and relation of containers to projects or components.
Used fixed image tag for inmutability.
Tagging using semantic versioning.
Not use mutable tags(latest,staging,etc). Use Inmutable tags(SHA-256, commit).
The challengue of uniquely identifying your images

Pulling images by digest
docker images --digests
docker pull alpine@sha256:b7233dafbed64e3738630b69382a8b231726aa1014ccaabc1947c5308a8910a7

Enanbled Security profiles: SELinux, AppArmor, Seccomp.
Static code analysys tool for Dockerfile like a linter. Detect misconfigurations
Hadolint
Packers (including encrypters), and downloaders are all able to evade static scanning by, for example, encrypting binary code that is only executed in memory, making the malware active only in runtime.
Trivy detect missconfigurations

Image Scanning

Check image for Common Vulnerabilities and Exposures (CVE)
Check image for secrets
Prevent attacks using the Supply Chain Attack
Scan container images for CVE (Common Vulnerabilities and Exposures).
Used dynamic analysis techniques for containers.

Container Security Scanners

Comparing the container scanners results:

Image Signing

Sign and verify images to mitigate MITM attacks. Docker offers a Content Trust mechanism that allows you to cryptographically sign images using a private key. This guarantees the image, and its tags, have not been modified.

Notary. Implementation of TUF specification.
sigstore/Cosign
Sigstore: A Solution to Software Supply Chain Security
Zero-Trust supply chains with Sigstore and SPIFFE/SPIRE

More Material about build containers

Secure the Build

Secure Supply Chain

Know where images, packages came from.

Hardening Code — Secure SDLC (Software Development Life Cycle)

Do a static analysis of the code and libraries used by the code to surface any vulnerabilities in the code and its dependencies.
Improve the security and quality of their code. OWASP Open Source Application Security tools: SAST, IAST.

Secure the Image — Hardening

You can build the container images using Docker, Kaniko.

Reduce the attack surface

Package a single application per container. Small container images. Minimize the number of layers.

Use the minimal OS image:
Alpine images
Scratch images
Distroless images
Use OS optimized for running containers:
Flatcar images
CodeOS by Fedora replaced the Project Atomic.
Bottlerocket by Aws
k3os by Rancher
Container-Optimized OS — COS by Google, based on Chromium-os used by Google
Do you use Alpine, distroless or vanilla images? …
7 Google best practices for building containers
Multi-staged builds.

A well-designed multi-stage build contains only the minimal binary files and dependencies required for the final image, with no build tools or intermediate files. Optimize cache.

Use official base images.
Avoid unknown public images.
Rootless. Run as a non-root user. Least privileged user
Create a dedicated user and group on the image.

Do not use a UID below 10,000. For best security, always run your processes as a UID above 10,000. Remove setuid and setgid permissions from the images

Avoid privileged containers, which lets a container run as root on the local machine.
Use only the necessary Privileged Capabilities.
Drop kernel modules, system time, trace processes (CAP_SYS_MODULE, CAP_SYS_TIME, CAP_SYS_PTRACE ).
Enable the --read-only mode in docker, if it's possible.
Don’t leave sensitive information (secrets, tokens, keys, etc) in the image.
Not mounting Host Path.
Use Metadata Labels for Images, such as licensing information, sources, names of authors, and relation of containers to projects or components.
Used fixed image tag for inmutability.
Tagging using semantic versioning.
Not use mutable tags(latest,staging,etc). Use Inmutable tags(SHA-256, commit).
The challengue of uniquely identifying your images

Pulling images by digest
docker images --digests
docker pull alpine@sha256:b7233dafbed64e3738630b69382a8b231726aa1014ccaabc1947c5308a8910a7

Enanbled Security profiles: SELinux, AppArmor, Seccomp.
Static code analysys tool for Dockerfile like a linter. Detect misconfigurations
Hadolint
Packers (including encrypters), and downloaders are all able to evade static scanning by, for example, encrypting binary code that is only executed in memory, making the malware active only in runtime.
Trivy detect missconfigurations

Image Scanning

Check image for Common Vulnerabilities and Exposures (CVE)
Check image for secrets
Prevent attacks using the Supply Chain Attack
Scan container images for CVE (Common Vulnerabilities and Exposures).
Used dynamic analysis techniques for containers.

Container Security Scanners

Comparing the container scanners results:

Image Signing

Notary. Implementation of TUF specification.
sigstore/Cosign
Sigstore: A Solution to Software Supply Chain Security
Zero-Trust supply chains with Sigstore and SPIFFE/SPIRE

More Material about build containers

Secure the Container Registry

Best configurations with ECR, ACR, Harbor, etc. Best practices.

Lock down access to the image registry (who can push/pull) to restrict which users can upload and download images from it. Uses Role Based Access Control (RBAC)

There is no guarantee that the image you are pulling from the registry is trusted. It may unintentionally contain security vulnerabilities, or may have intentionally been replaced with an image compromised by attackers.

Use a private registry deployed behind firewall, to reduce the risk of tampering.

Registry Resources

Secure the Container Runtime

See the following container runtimes, there are three main types of container runtimes — low-level runtimes, high-level runtimes, and virtualized runtimes or sandboxed.

Low-Level Container Runtimes:

High-Level Container Runtimes

Docker Engine
Podman
CRI-O — OCI-based implementation of Kubernetes Container Runtime Interface
Mirantes Container Runtime

Sandboxed and Virtualized Container Runtimes

Why is important Runtime Security?

Detection of IOC (Indicator Of Compromise)
Detect Zero Days attack
Compliance requirement
Recommended in highly dynamic environments

Constraints

Event context
Safety
Low overhead
Wide support of kernels

Enable detection of anomalous behaviour in applications.

Applied the secure configurations in the container runtime. By default is insecure.
Restrict access to container runtime daemon/APIs
Use if it’s possible in Rootless Mode.

Docker Security

Avoid misconfigured exposed Docker API Ports, attackers used the misconfigured port to deploy and run a malicious image that contained malware that was specifically designed to evade static scanning.
TLS encryption between the Docker client and daemon. Do not expose the Docker engine using Unix socket or remotely using http.

Never make the daemon socket available for remote connections, unless you are using Docker’s encrypted HTTPS socket, which supports authentication.

Limit the usage of mount Docker socket in a container in an untrusted environment.
Do not run Docker images with an option that exposes the socket in the container.
-v /var/run/docker.sock://var/run/docker.sock

The Docker daemon socket is a Unix network socket that facilitates communication with the Docker API. By default, this socket is owned by the root user. If anyone else obtains access to the socket, they will have permissions equivalent to root access to the host.

Run Docker in Rootless Mode. docker context use rootless
Enable the user namespaces.
Enable Docker Content Trust. Docker. DOCKER_CONTENT_TRUST=1 . Docker Content Trust implements The Update Framework (TUF) . Powered by Notary, an open-source TUF-client and server that can operate over arbitrary trusted collections of data.

More Material about Docker Security

Secure the Infrastructure

Risk:

If host is compromised, the container will be too.
Kernel exploits

Best practices:

Keep the host kernel patched to prevent a range of known vulnerabilities, many of which can result in container escape. Since the kernel is shared by the container and the host, the kernel exploits when an attacker manages to run on a container can directly affect the host.
Use CIS-Benchmark for the operating system.
Use secure computing (seccomp) to restrict host system call (syscall) access from containers.
Use Security-Enhanced Linux (SELinux) to further isolate containers.

Secure the Data

Don’t leak sensitive info in the images, avoid using environment variables for your sensitive information.

Secrets are Digital credentials:

passwords
API keys & Tokens
SSH keys
Private certificates for secure communication, transmitting and receiving data (TLS, SSL, and so on)
Private encryption keys for systems like PGP
Database names or connection strings.
Sensitive configuration settings (email address, usernames, debug flags, etc.)
Use a proper filesystem encryption technology for container storage
Use volume mounts to pass secrets to a container at runtime
Provide write/execute access only to the containers that need to modify the data in a specific host filesystem path
OPA to write controls like only allowing Read-only Root Filesystem access, listing allowed host filesystem paths to mount, and listing allowed Flex volume drivers.
Automatically scan container images for sensitive data such as credentials, tokens, SSH keys, TLS certificates, database names or connection strings and so on, before pushing them to a container registry (can be done locally and in CI).
Limit storage related syscalls and capabilities to prevent runtime privilege escalation.
Implement RBAC, or role-based access control. Every human or application only needs the minimum secrets required to operate, nothing more. Principle of Least Privilege.
Run audits regularly. Centralized audit trails are the key to knowing all the key security events.
Rotate secrets, a standard security practice.
Automatically create and store secrets

Secrets Management Tools

Open source tools:

detect-secrets by Yelp: detecting and preventing secrets in code.
git-secrets by awslabs: Prevents you from committing secrets and credentials into git repositories

Cloud Provider Key Management

Enterprise secrets vault:

Secure the Workloads … Running the containers

Avoid privileged containers
• Root access to all devices • Ability to tamper with Linux security modules like AppArmor and SELinux • Ability to install a new instance of the Docker platform, using the host’s kernel capabilities, and run Docker within Docker.

To check if the container is running in privileged mode docker inspect --format =’{{. HostConfig.Privileged}}’[container_id]

Limit container resources.

When a container is compromised, attackers may try to make use of the underlying host resources to perform malicious activity. Set memory and CPU usage limits to minimize the impact of breaches for resource-intensive containers.

docker run -d --name container-1 --cpuset-cpus 0 --cpu-shares 768 cpu-stress

Preventing a fork bomb. docker run --rm -it --pids-limit 200 debian:jessie
Segregate container networks.
The default bridge network exists on all Docker hosts — if you do not specify a different network, new containers automatically connect to it.
Use custom bridge networks to control which containers can communicate between them, and to enable automatic DNS resolution from container name to IP address.
Ensure that containers can connect to each other only if absolutely necessary, and avoid connecting sensitive containers to public-facing networks.
Docker provides network drivers that let you create your own bridge network, overlay network, or macvlan network. If you need more control, you can create a Docker network plugin.
Improve container isolation.

Protecting a container is exactly the same as protecting any process running on Linux. Ideally, the operating system on a container host should protect the host kernel from container escapes, and prevent mutual influence between containers.

Set filesystem and volumes to Read only.

This can prevent malicious activity such as deploying malware on the container or modifying configuration. docker run --read-only alpine

Complete lifecycle management restrict system calls from Within Containers
Monitor Container Activity. Analyze collected events to detect suspicious behaviourial patterns.
Create an incident response process to ensure rapid response in the case of an attack.
Apply automated patching
Confirms Inmutability. Implement drift prevention to ensure container inmutability.
Ensure you have robust auditing and forensics for quick troubleshooting and compliance reporting.