Light-Weight Container Investigation
creation-date: 2025-02-18 15:38
modification-date: 2025-02-18 15:38
Attendees
Company | Name |
---|---|
Savoir-faire Linux | Eloi Bail, Mathieu Dupré |
Utilidata | Rick Steurer, Jack Kelly, Tom |
Topics
Container technologies
Docker
proprietary solution
community solution (moby)
pros
well understood in the community
follow OCI (Open Container Initiative)
cons
docker daemon overhead
Podman
pros
Better securirty (no daemon like docker)
follow OCI
compatible with kubernetes files
cons
less well known that docker (pushed by Redhat)
LXC / LXD
First implementation of container in Linux
More oriented to package a full system
More that embedding libs & applications
LXD is a wrapper on top of LXC
pros
lightweight ?
closer to cgroup
cons
not sure if it is fully OCI compatible (to check ?)
Systemd
Running on modern Linux distribution
2 options
Run standard OCI containers
with runc or crun to run OCI Containers
Run systemd containers which are not OCI compliant (to check)
systemd-nsspawn is used as a launcher instead of runc / crun
Same isolation than container
Delivery
own
Cons
Systemd is required (not systV)
only support low level OCI files (manifest.json)
no LXC containers supported
no overlay support
Pros
Systemd is well known but not necessary for containers
Share host & containers libraires
Can run OCI containers like dockers and podman
Kubernetes
To detail
Base Image
Why ?
Align librairies
Reduce the size of image
Provide cyber security hardening
Examples (TODO @Mathieu)
Generic images
Ubuntu: for performance
alpine: for image size
Not alpine use musl libc which could break compatibility with third parties
scratch: void image if we want to reuse the host libraries
scratch image would be good if we want to have minimal image and considering host libraires mapping
Container Launcher
Purpose
Launch a single container
Technologies
RUNC
"Default" launcher
Developed in Go
Cons:
Memory usage
Pros
Well audited
CRUN
Developed in C
Developed by Redhat
Smaller in term of size
More performant than runc
Container
Container orchestrator
Purpose
Orchestrate the launch of containers
Technologies
docker compose file
Compatible with docker and podman
Not with LXC & Systemd
kubernetes (k3s)
Only with podman
Systemd
compatible with systemd
compatible with podman using quadlet
Our own ?
to detail
Container lifecycle
oneshot container vs daemon container
TODO: detail
Security advice
Operating system
It is essential that the OS is "secured" to prevent cybersecurity issues on containers
use similar cybersercurity procedure on container launcher than on regular process
SELinux / Apparmor
Secomp
Users management
Permission
Container images
minimal images or use well know images like "alpine"
Container launchers
launchers are supposed to be "secured" (e.g runc, crun)
recommendations
no privilege mode
rootless mode
Containers ressources limitation / segragation
namespaces
network
mounting point
IPC (isolate SystemdV IPC and POSIX message queues)
cgroup: isolate the cgroup tree
PID: isolate pid
Time: container time can differ
User: isolate UID and GID
UTS: hostname
cgroups
limit ressources usage (memory, CPUs ...)
Signing
Image signing is built-in in container technologies
Runtime verifying could be challenging
Ex: integration with secureboot to use the chain of trust with container layers
We would need to investigate that
ex: dm-verify shall not be compatible with docker layers
Communication between containers & between containers and hosts
Container / container
Possible to map one container to an other only using host as intermediate
this should not be a good practice
Data broker should be prefered to exchange data between containers and between host & containers
zmq
supports
Unix sockets
TCP sockets
Shared memory
mqtt
supports
TCP sockets
Others to check :)
redis
TCP sockets
Container / host
mapping directory possible on all container technologies
possible to share libraires in containers from host
Good to minimize image size
Good to keep the underlying libs up-to-date on the host
Integration step could be challenging
Ex: a container that could not run properly because of old host lib.
Should we give the ability to the container to "overwrite" a host library ?
Ex: use LD_LIBRARY_PATH to change the default lib location within the container
Shared memory & IPC
Isolation done by namespace
Could be challenging in none privilege mode
User namespace would be used and could match from one container versus an other one. We need to check it !
UID mapping can differ
Container as a VM
Kata Containers - Open Source Container Runtime Software
Some limitations seen
Hardware dependency: to check if running on ARM
Linux Kernel dependency (ex: real-time)
High memory footprint
bootc
does not work on ARM
Rootless container
HW integration
Real-time
Priority management ?
Orchestration
Out of scope
Focus on one node
No cluster management
Recommandation
OCI based container
Docker / Podman / Systemd
LXC might not be a good candidate in this case
Use very minimal base image to decrease image size
ex: Scratch which the smallest
Alpine which is smaller than ubuntu but still includes its libs & bins
Host libs / bins mapping should be used
to align the libs between containers (ex: up-to-date openssl common version)
to ensure containers size are as small as possible
Third party OSS frameworks should be used for container / container communication
Ex: zmq, mqtt, redis...etc
Actions
Savoir-faire Linux
Review the documentUtilidata
Review the document