/
Light-Weight Container Investigation

Light-Weight Container Investigation


creation-date: 2025-02-18 15:38
modification-date: 2025-02-18 15:38


Attendees

Company

Name

Company

Name

Savoir-faire Linux

Eloi Bail, Mathieu Dupré

Utilidata

Rick Steurer, Jack Kelly, Tom


Topics

Container technologies

  • Docker

    • proprietary solution

    • community solution (moby)

    • pros

      • well understood in the community

      • follow OCI (Open Container Initiative)

    • cons

      • docker daemon overhead

  • Podman

    • pros

      • Better securirty (no daemon like docker)

      • follow OCI

      • compatible with kubernetes files

    • cons

      • less well known that docker (pushed by Redhat)

  • LXC / LXD

    • First implementation of container in Linux

    • More oriented to package a full system

      • More that embedding libs & applications

    • LXD is a wrapper on top of LXC

    • pros

      • lightweight ?

      • closer to cgroup

    • cons

      • not sure if it is fully OCI compatible (to check ?)

  • Systemd

    • Running on modern Linux distribution

    • 2 options

      • Run standard OCI containers

        • with runc or crun to run OCI Containers

      • Run systemd containers which are not OCI compliant (to check)

        • systemd-nsspawn is used as a launcher instead of runc / crun

    • Same isolation than container

    • Delivery

      • own

    • Cons

      • Systemd is required (not systV)

      • only support low level OCI files (manifest.json)

        • no LXC containers supported

      • no overlay support

    • Pros

      • Systemd is well known but not necessary for containers

      • Share host & containers libraires

      • Can run OCI containers like dockers and podman

  • Kubernetes

    • To detail

Base Image

  • Why ?

    • Align librairies

    • Reduce the size of image

    • Provide cyber security hardening

  • Examples (TODO @Mathieu)

    • Generic images

      • Ubuntu: for performance

      • alpine: for image size

        • Not alpine use musl libc which could break compatibility with third parties

      • scratch: void image if we want to reuse the host libraries

        • scratch image would be good if we want to have minimal image and considering host libraires mapping

Container Launcher

  • Purpose

    • Launch a single container

  • Technologies

    • RUNC

      • "Default" launcher

      • Developed in Go

      • Cons:

        • Memory usage

      • Pros

        • Well audited

    • CRUN

      • Developed in C

      • Developed by Redhat

      • Smaller in term of size

      • More performant than runc

Container

Container orchestrator

  • Purpose

    • Orchestrate the launch of containers

  • Technologies

    • docker compose file

      • Compatible with docker and podman

      • Not with LXC & Systemd

    • kubernetes (k3s)

      • Only with podman

    • Systemd

      • compatible with systemd

      • compatible with podman using quadlet

    • Our own ?

      • to detail

Container lifecycle

  • oneshot container vs daemon container

  • TODO: detail

Security advice

  • Operating system

    • It is essential that the OS is "secured" to prevent cybersecurity issues on containers

    • use similar cybersercurity procedure on container launcher than on regular process

      • SELinux / Apparmor

      • Secomp

      • Users management

      • Permission

  • Container images

    • minimal images or use well know images like "alpine"

  • Container launchers

    • launchers are supposed to be "secured" (e.g runc, crun)

    • recommendations

      • no privilege mode

      • rootless mode

  • Containers ressources limitation / segragation

    • namespaces

      • network

      • mounting point

      • IPC (isolate SystemdV IPC and POSIX message queues)

      • cgroup: isolate the cgroup tree

      • PID: isolate pid

      • Time: container time can differ

      • User: isolate UID and GID

      • UTS: hostname

    • cgroups

      • limit ressources usage (memory, CPUs ...)

  • Signing

    • Image signing is built-in in container technologies

    • Runtime verifying could be challenging

      • Ex: integration with secureboot to use the chain of trust with container layers

      • We would need to investigate that

        • ex: dm-verify shall not be compatible with docker layers

Communication between containers & between containers and hosts

Container / container

  • Possible to map one container to an other only using host as intermediate

    • this should not be a good practice

  • Data broker should be prefered to exchange data between containers and between host & containers

    • zmq

      • supports

        • Unix sockets

        • TCP sockets

        • Shared memory

    • mqtt

      • supports

        • TCP sockets

        • Others to check :)

    • redis

      • TCP sockets

Container / host

  • mapping directory possible on all container technologies

    • possible to share libraires in containers from host

      • Good to minimize image size

      • Good to keep the underlying libs up-to-date on the host

      • Integration step could be challenging

        • Ex: a container that could not run properly because of old host lib.

        • Should we give the ability to the container to "overwrite" a host library ?

          • Ex: use LD_LIBRARY_PATH to change the default lib location within the container

  • Shared memory & IPC

    • Isolation done by namespace

    • Could be challenging in none privilege mode

      • User namespace would be used and could match from one container versus an other one. We need to check it !

      • UID mapping can differ

  • Container as a VM

  • Rootless container

    • HW integration

    • Real-time

      • Priority management ?

  • Orchestration

    • Out of scope

    • Focus on one node

    • No cluster management

Recommandation

  • OCI based container

    • Docker / Podman / Systemd

    • LXC might not be a good candidate in this case

  • Use very minimal base image to decrease image size

    • ex: Scratch which the smallest

    • Alpine which is smaller than ubuntu but still includes its libs & bins

  • Host libs / bins mapping should be used

    • to align the libs between containers (ex: up-to-date openssl common version)

    • to ensure containers size are as small as possible

  • Third party OSS frameworks should be used for container / container communication

    • Ex: zmq, mqtt, redis...etc

Actions

  • Savoir-faire Linux

    Review the document
  • Utilidata

    Review the document

Related content