Running docker inside an unprivileged LXC container on Proxmox

Published: 2021-03-25, Revised: 2024-04-19

TL;DR This is a brief description of the setup process for running docker in unprivileged LXC containers on proxmox. There are two primary sources, one is a post on Reddit¹ and a more general discussion² on linuxcontainers.org.

This guide has been verified to work with Proxmox 5.4 to 8.0.3, for both Ubuntu 22.04 and Debian 12 LXC Templates. If you are using a different setup, YMMV.

Prepare Proxmox#

On Proxmox, the overlay and aufs* Kernel modules must be enabled to support Docker-LXC-Nesting.

echo -e "overlay\naufs" >> /etc/modules-load.d/modules.conf

Reboot Proxmox and verify that the modules are active:

lsmod | grep -E 'overlay|aufs'

Create an unprivileged LXC container#

Follow the Proxmox docs to create an unprivileged LXC container, either through the web UI or using the shell.

This LXC container config will be stored at:

/etc/pve/lxc/100.conf

Open this config and add:

features: keyctl=1,nesting=1

Alternatively, use the Proxmox gui to enable these options.

Afterwards, the 100.conf will look similar to this:

arch: amd64
cores: 2
features: keyctl=1,nesting=1
hostname: docker
memory: 4096
nameserver: 192.168.40.1
net0: name=eth0,bridge=vmbr1,firewall=1,gw=192.168.40.1,ip=192.168.40.9/24,tag=40,type=veth
ostype: debian
rootfs: storagedocker:100/vm-100-disk-0.raw,size=60G
searchdomain: local.mytld.com
startup: order=1
swap: 4096
unprivileged: 1

Setup Docker in LXC#

Now, login to the newly created LXC container via ssh.

Optionally install sudo:

apt install sudo

Set time zone. In unprivileged containers, use:

dpkg-reconfigure tzdata

Update all packages:

apt-get update && apt-get upgrade && apt-get dist-upgrade && apt-get autoremove

Install Docker. This is from the docs for Debian.

sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

Change the storage driver to overlay2.

echo -e '{\n  "storage-driver": "overlay2"\n}' >> /etc/docker/daemon.json

{
    "storage-driver": "overlay2",
    "bip": "172.26.0.1/16",
    "default-address-pools":
    [
        {"base":"172.17.0.0/16","size":24}
    ]
}

Optionally install docker-compose. Follow the docs.

Test Docker#

Restart the LXC container and test Docker setup.

systemctl status docker
docker run hello-world

Hello from Docker!

Yay!

ZFS#

For ZFS, a few additional steps are required to get Docker running inside unprivileged LXCs.

Docker requires root privileges to use ZFS. Since we are running Docker inside an unprivileged LXC, file-system permissions are missing for Docker in /var/lib/docker.

There are two workarounds.

The first is to create a ZFS volume formatted as ext4 or xfs, and changing ownership to the unprivileged root user, which I am showing below. This approach was first described in a post on Reddit⁴. See also this Ansible playbook.

Prepare replacing /var/lib/docker with a ZFS volume mount

In the container:

service docker stop
cp -au /var/lib/docker /var/lib/docker.bk
rm -rf /var/lib/docker/*

Shutdown the LXC.

On Proxmox, I am assuming ZFS is already set up. Check available mounts and zfs datasets:

zfs mount
zpool status -v
zfs list

• ZFS is hierarchically organized in datasets.
• My zfs pool is called tank_ssd (a ZFS Mirror consisting of SSDs).
• In tank_ssd, I created an encrypted and compressed ZFS dataset called lxc.

The command below will create a new ZFS volume stored in tank_ssd/lxc/docker.

zfs create -s -V 8G tank_ssd/lxc/docker
zfs get volsize,referenced tank_ssd/lxc/docker

> NAME                 PROPERTY    VALUE     SOURCE
> tank_ssd/lxc/docker  volsize     8G        local
> tank_ssd/lxc/docker  referenced  88K       -

ZFS volumes are comparable to partitions and can be formatted with any file system. I prefer to use XFS over ext4, but both will work.

Format as xfs:

mkfs.xfs /dev/zvol/tank_ssd/lxc/docker

In order for Docker to be able to write to this volume, you either need to provide uid-mappings, or change the ownership to the unprivileged root user (uid 100000).

The latter can be done by mounting the volume into a temporary location (on Proxmox) and changing permissions:

mkdir /tmp/zvol_tmp
mount /dev/zvol/tank_ssd/lxc/docker /tmp/zvol_tmp
chown -R 100000:100000 /tmp/zvol_tmp
df -Th /dev/zvol/tank_ssd/lxc/docker

> Filesystem     Type  Size  Used Avail Use% Mounted on
> /dev/zd64      xfs   8.0G   90M  8.0G   2% /tmp/zvol_tmp

Unmount:

umount /tmp/zvol_tmp

Add the mount point to your unprivileged LXC config.

nano /etc/pve/lxc/100.conf
mp0: /dev/zvol/tank_ssd/lxc/docker,mp=/var/lib/docker

Start the LXC and check the Docker service inside the container:

systemctl status docker

> ● docker.service - Docker Application Container Engine
>      Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
>      Active: active (running) since Sun 2022-01-02 10:14:39 CET; 30s ago

You will notice a warning in the Docker logs:

Not using native diff for overlay2, this may cause degraded performance for building images: running in a user namespace

This is only relevant if you are building lots of Docker images (e.g. a Gitlab Runner). In this case, use a native VM if you need the performance.

You will see similar warning on the Proxmox host.

[Sun Mar  6 14:14:46 2022] overlayfs: upper fs does not support RENAME_WHITEOUT.
[Sun Mar  6 14:14:46 2022] overlayfs: upper fs missing required features.
[Sun Mar  6 14:14:47 2022] overlayfs: fs on '/var/lib/docker/check-overlayfs-support065203116/lower2' does not support file handles, falling back to xino=off.
[Sun Mar  6 14:17:12 2022] overlayfs: fs on '/var/lib/docker/overlay2/check-overlayfs-support656562844/lower2' does not support file handles, falling back to xino=off.

I haven't seen any negative consequences, even for more complex Docker images such as Gitlab or mailcow-dockerized.

In the container, check the storage driver that is used.

docker info | grep -A 7 "Storage Driver:"

> Storage Driver: overlay2
>  Backing Filesystem: xfs
>  Supports d_type: true
>  Native Overlay Diff: false
>  userxattr: true
> Logging Driver: json-file
> Cgroup Driver: systemd
> Cgroup Version: 2

Conclusions#

Now, what is neat about this setup is that it is entirely possible to have several LXC containers that run separate Docker systems.

For example, I use the above Docker LXC for hosting stable services in the local service network VLAN.

In another LXC container, I have Docker setup for experimental containers, with quick access to docker system prune --all && docker volume prune.

The performance of this Docker-LXC-nesting is negligible, since all resources are shared and running Docker containers do not consume resources, if they are not active.

Caveats#

Backups#

Having /var/lib/docker in a separate mount point (for LXC+ZFS) will exclude all docker containers from the native Proxmox backup.

Of course, Proxmox Backups can be avoided with ZFS Snapshots. However, for archiving purposes, I prefer having tar.zst files of all LXC containers.

A solution here is to move all persistent data to the LXC, so that anything in /var/lib/docker (e.g. containers, volumes) does not need to be backed up.

There is a system for setting up Docker containers I have followed since seeing it in the Funkwhale-docs.

Based on this system, the example below will go a bit beyond what it absolutely needed, but I have seen too many bad Docker container setups.

mkdir /srv/
sudo useradd -r -s /sbin/nologin -m -d /srv/miniflux -U -G docker miniflux

sudo -u miniflux -H bash
cd /srv/miniflux

mkdir -p data/miniflux-db
mkdir docker
cd docker
nano docker-compose.yml

version: '3'

services:
miniflux:
    image: miniflux/miniflux:latest
    restart: unless-stopped
    ports:
    - "127.0.0.1:18080:8080"
    depends_on:
    db:
        condition: service_healthy
    networks:
    - miniflux
    environment:
    - DATABASE_URL=postgres://${DB_USER:-miniflux}:${DB_SECRET:-eX4mP13p455w0Rd}@db/miniflux?sslmode=disable
    - CLEANUP_ARCHIVE_UNREAD_DAYS=2
    - CLEANUP_ARCHIVE_READ_DAYS=2
    - POLLING_FREQUENCY=180
    - BASE_URL=https://flux.forst.alexanderdunkel.com/
db:
    image: postgres:13
    restart: unless-stopped
    environment:
    - POSTGRES_USER=${DB_USER:-miniflux}
    - POSTGRES_PASSWORD=${DB_SECRET:-eX4mP13p455w0Rd}
    volumes:
    - /srv/miniflux/data/miniflux-db:/var/lib/postgresql/data
    networks:
    - miniflux
    healthcheck:
    test: ["CMD", "pg_isready", "-U", "${DB_USER:-miniflux}"]
    interval: 10s
    start_period: 30s

networks:
miniflux:
    name: ${NETWORK_NAME:-miniflux-network}

volumes:
  miniflux-db:

cd ~/docker
docker-compose up -d && docker-compose logs --follow --tail 100

Migrations#

Special care and possibly manual work is necessary in case of Docker+ZFS for migrations ( e.g. between nodes, clusters etc.). Since I am running a single node Proxmox Cluster, I have no information to provide here. Please add your experiences in the comments.

Special container permissions#

Except for one case (see below), I did not have any issues with this setup for over a year now, running several unprivileged LXC containers with individual docker hosts alongside.