Proxmox root migration (a report)

Published: 2022-08-04, Revised: 2024-02-13

TL;DR Migration is a tedious process because it depends on a number of custom setup parameters. Understandably, the Proxmox docs provide only basic information on how to migrate the hypervisor itself.

Background#

If there is neither a hardware nor a software change, restoring Proxmox is easy: Copy all the files back.

Most often, however, something will have changed. For me this was:

1. Migration to ZFS. This can happen in steps. I first added two ZFS pools for data_ssd (VM disks) and data_hdd (data pool). What was left was migrating the root partition of Proxmox, by creating a mirrored ZFS pool, which required a new installation.¹

2. Migrating Proxmox from 6.4 to 7.0. It is possible to do this either by migrating or in-place update.²

This is a fairly standard setup, so I decided to do these two steps in parallel.

Preparations#

Follow the "New installation" checklist.²

While it is not advised, there is a small list of packages that I have installed directly on Proxmox itself. This includes some basic packages, e.g.:

• telegraf: Monitoring of system metrics to InfluxDB
• iotop, sysstat, ledmon, htop: Local administration
• git: Tracking changes of Proxmox configuration files
• libsasl2-modules, postfix-pcre: Forward of notifications to external mailbox
• apcupsd: APC UPS Daemon
• byobu: Terminal multiplexer

I keep note of these packages, but also went through the list of all packages installed, to see if I missed any:

apt list --installed

Just to verify, also see what the check tool says:

pve6to7
pve6to7 --full

Create Backups#

This is independent of the migration and basically follows the 3-2-1 rule.³

1. Create LXC Dumps
2. Create ZFS Snapshots of VM volumes and data disks
3. Transfer/Send snapshots and backups offsite
4. Create a Proxmox config backup

For creating config backups, I used two approaches.

First, a backup script from DerDanilo will create a comprehensive *.tar.gz of all relevant files.⁴

./prox_config_backup.sh

Second, I have a local git repository on Proxmox to selectively track changes of all relevant files, synced with my private Gitlab. The main purpose here is not backup, but monitoring and annotation of changes and general reproducibility.

The setup is straightforward:

cd /root
mkdir -p config/proxmox
git init
git remote add origin git@gitlab.local.mytld.com/configs/proxmox.git
nano filelist.txt

Add files to be tracked. Click to see my list.

## container configs
/etc/pve/lxc/100.conf
/etc/pve/lxc/101.conf
/etc/pve/lxc/102.conf
/etc/pve/lxc/103.conf
/etc/pve/lxc/104.conf
/etc/pve/lxc/105.conf
/etc/pve/lxc/106.conf
/etc/pve/lxc/107.conf
/etc/pve/lxc/108.conf
/etc/pve/lxc/109.conf
/etc/pve/lxc/110.conf
/etc/pve/lxc/120.conf

## storage config
/etc/pve/storage.cfg

## network
/etc/resolv.conf
/etc/network/interfaces

## user config
/etc/pve/user.cfg

## datacenter
/etc/pve/datacenter.cfg

## nodes
/etc/pve/nodes/monkey/config

## special files
/etc/pve/.vmlist
/etc/pve/.version
/etc/pve/.members
/etc/subuid
/etc/subgid
/etc/timezone
/etc/hostname

## telegraf mod
/etc/pve/status.cfg
/etc/telegraf/telegraf.conf
/etc/sudoers

## docker mod
/etc/modules-load.d/modules.conf

## apt sources
/etc/apt/sources.list.d/influxdb.list
/etc/apt/sources.list.d/pve-enterprise.list

## root/user config
/root/.bashrc

## apcupsd mod
/etc/default/apcupsd
/etc/apcupsd/apcupsd.conf

## postfix mod
/etc/postfix/main.cf
/etc/postfix/smtp_header_checks

## tools
/root/drive_check.sh
/root/prox_config_backup.sh

The following script will read filelist.txt and copy all files, replicating full paths in the repository.

nano get-files.sh

#!/bin/bash

################################################################################
#
# Shell script to copy important configuration files from the current
# environment to this repository.
#
################################################################################

# Exit as soon as a command fails
set -e

# Accessing an empty variable will yield an error
set -u

# Full path to repo directory
REPO_PATH="/root/config/proxmox/"

# copy all files from filelist.txt, excluding 
# comments, recreate all paths on target directory
grep -e '^[^#]' "$REPO_PATH/filelist.txt" | \
    xargs cp --parents --target-directory "$REPO_PATH"

echo "Completed."

Update and commit files:

chmod +x get-files.sh
bash get-files.sh
git add .
git commit -m "Initial commit"

For the migration itself, I used a spare SSD as a working drive, which was not part of my ZFS pool.

Copy either the git files or the compressed config backups⁴ to the temporary drive:

mkdir /tmp/migration
MIGRATIONDRIVE=/dev/disk/by-id/ata-Samsung_SSD_840_EVO_1TB_S1D9NEAD808121E-part1
MIGRATIONPATH=/tmp/migration
mount -v $MIGRATIONDRIVE $MIGRATIONPATH
# git
cp /root/config/proxmox $MIGRATIONPATH
cd $MIGRATIONPATH && ls -alh
# compressed backup
PMBACKUP=proxmox_backup_root.local.mytld.com_2022-01-22.08.03.27.tar.gz
cp $PMBACKUP $MIGRATIONPATH
tar -zxvf $PMBACKUP
cd var/tmp/proxmox-DeXUKUc3
tar -xvf proxmoxetc.2022-01-22.08.03.27.tar
tar -xvf proxmoxroot.2022-01-22.08.03.27.tar
tar -xvf proxmoxpve.2022-01-22.08.03.27.tar
# unmount
umount $MIGRATIONPATH

Migration#

Shutdown all guests#

Prepare ZFS Export#

If ZFS is used, pools should be exported for the time of the migration.

• Go to PM GUI > Datacenter > Storage
• uncheck "enabled" for all ZFS storage

Then unmount and export pools:

zfs unmount tank_hdd
zfs unmount tank_ssd
zpool export tank_hdd
zpool export tank_ssd

Optionally, test import:

zpool import tank_hdd
zpool import tank_ssd
zpool status

Export again for migration.

Hardware Migration#

Now it was time for migrating the Raid1 to a ZFS Mirror.

First, I attached the two SSDs to my HBA, which would not boot Proxmox. Until I realized that it appears recommended to directly attach the root pool to SATA/SAS ports on the mainboard.⁵

Prepare Proxmox 7 Iso USB#

Flash Proxmox Iso to USB. See the docs⁶ and the download section.⁷

I used Etcher (chocolatey install with choco install etcher).

Proxmox installation#

• insert USB (Proxmox 7)
• follow install instructions⁸

I used the IPMI interface iKVM/HTML5 Console:

• note that it is tricky to copy & paste
• select English keyboard layout for iKVM/iHTML
• use the iKVM keyboard for special characters

Select your root partitions to install the ZFS root pool:

• provide the base network details (see /etc/hosts and /etc/network/interfaces):
  • eno1
  • address 192.168.10.42/24
  • proxmox.local.mytld.com
  • gateway 192.168.10.1

Configuration#

The strategy here was going step by step and in the correct order.

First, connect via ssh, using the password set during installation.

ssh root@192.168.10.42

Verify:

zpool status
lsblk

lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 465.8G  0 disk
├─sda1   8:1    0 465.8G  0 part
└─sda9   8:9    0     8M  0 part
sdb      8:16   0 465.8G  0 disk
├─sdb1   8:17   0 465.8G  0 part
└─sdb9   8:25   0     8M  0 part
...

Looks good.

Import ZFS pools

zpool import tank_hdd
zpool import tank_ssd
zpool status

Restore configs#

Mount migration drive

mkdir /tmp/migration
MIGRATIONDRIVE=/dev/disk/by-id/ata-Samsung_SSD_840_EVO_1TB_S1D9NEAD808121E-part1
MIGRATIONPATH=/tmp/migration
mount -v $MIGRATIONDRIVE $MIGRATIONPATH
cd $MIGRATIONPATH && ls -alh
# set the path for the migration steps below
BAKPATH=$MIGRATIONPATH/var/tmp/proxmox-DeXUKUc3
cd $BAKPATH

Restore home folder:

• Either restore selectively,
• or copy all, YMMV

  mv ~/.ssh ~/.ssh.bak
  mv ~/.rnd ~/.rnd.bak
  cp -avr $BAKPATH/root/. ~/
  

SSH would not work afterwards, before I realized that authorized_keys is a symlink to /etc/pve/priv/authorized_keys.

rm ~/.ssh/authorized_keys
mv ~/.ssh.bak/authorized_keys ~/.ssh/authorized_keys

Restore storage configuration#

Compare files (byobu):

nano /etc/pve/storage.cfg 
nano $BAKPATH/etc/pve/storage.cfg 

Install custom apt packages#

apt update
apt-get install sudo byobu iotop git apcupsd
apt-get install lm-sensors
sudo sensors-detect # answer all yes
# for postfix:
apt-get install libsasl2-modules
apt install postfix-pcre
...

Restore network configuration#

I have two network cards, eno1 and eno2. The first is used for the Management subnet, the second for the Service subnet(s), based on tagged VLAN traffic. I added the eno2 network configuration here, since eno1 was already setup during the Proxmox installation.

Compare/check/merge network interfaces

nano /etc/network/interfaces
nano $BAKPATH/etc/network/interfaces

Compare/check/merge user cfg

nano /etc/pve/user.cfg
nano $BAKPATH/etc/pve/user.cfg

Restore further customization#

Postfix:

cp -a $BAKPATH/etc/postfix/main.cf /etc/postfix/main.cf
cp -a $BAKPATH/etc/postfix/smtp_header_checks /etc/postfix/smtp_header_checks

Create "data user":

For historical reasons, files on my data pool are owned by a specific user (samba_user - although I don't use samba anymore). This user is mapped to www-data inside LXCs.

lxc.idmap: u 0 100000 33
lxc.idmap: g 0 100000 33
lxc.idmap: u 33 1005 1
lxc.idmap: g 33 1005 1
lxc.idmap: u 34 100034 65502
lxc.idmap: g 34 100034 65502

root:1005:1

To preserve these LXC-UID-Mappings, I wanted to restore this user, with matching user id/gid.

groupadd samba_user -g 1005 
useradd samba_user -u 1005 -g 1005

apcupsd:

/etc/default/apcupsd # type yes
/etc/apcupsd/apcupsd.conf # copy from backup

Restore remaining files#

These files did not change or needed no manual merge.

Container configs:

cp -a $BAKPATH/etc/pve/lxc/* /etc/pve/lxc/

Network/DNS:

/etc/resolv.conf # (not changed)

Datacenter:

/etc/pve/datacenter.cfg # change keyboard to de

Nodes config:

/etc/pve/nodes/monkey/config # Add ACME config

Special files

/etc/pve/.vmlist # not necessary, automatically generated
/etc/pve/.version # not necessary, automatically generated
/etc/pve/.members # (not changed)
/etc/subuid # manual merge (add one line, for LXC uid mapping)
/etc/subgid # manual merge (add one line, for LXC gid mapping)
/etc/timezone # (not changed)
/etc/hostname # (not changed)

Apt sources:

/etc/apt/sources.list.d/pve-enterprise.list # not changed (or newer)

Verify#

• reboot
• Open new Brower (e.g. firefox), head to: http://192.168.10.42:8006

Restore/Reissue subscription#

New hardware requires updating the subscription key⁹, if you have one:

• go to node in gui/subscription
• upload subscription key
• in web, click reissue
• in gui, click check

Update#

• Go to updates in gui and install all

Check#

• at this stage, all VMs appear in the list
• all storages appear in the list

SSL Setup#

I have a split-brain DNS setup, meaning that my services (Proxmox..) can only be reached through the internal subnet, through a local DNS server. Proxmox gets SSL certificates through ACME, configured with DNS credentials flow for my A record/TLD.

Restore acme.conf

cp $BAKPATH/etc/pve/priv/acme/default /etc/pve/priv/acme
cp $BAKPATH/etc/pve/priv/acme/plugins.cfg /etc/pve/priv/acme

Configs appear in gui: Check.

Check Certificate Management.

reboot

Go to Node/Certificates/Acme

• click Order Certificates now
• looks good.

Prepare VM start#

ZFS unlock/mount:

zfs mount -l tank_hdd/data
zfs mount -l tank_ssd/lxc

zpool list
> NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
> rpool      232G  1.43G   231G        -         -     0%     0%  1.00x    ONLINE  -
> tank_hdd  43.7T  5.74T  37.9T        -         -     0%    13%  1.00x    ONLINE  -
> tank_ssd   464G  69.3G   395G        -         -    11%    14%  1.00x    ONLINE  -

Looks good.

dmesg -wHT

Check if ¹⁰ applies to you:

• go through all menu items in proxmox gui and check each (datacenter and node)
• test starting an LXC
• error: "storage encrypted_zfs does not exist"

pvesm add zfspool encrypted_zfs -pool tank_ssd/lxc

• start LXC: looks good.
• start next LXC: looks good.
  • check web: looks good.
• start all other LXC and VM

Conclusion#

Any customization makes migration, backup and restore more tedious. I was not even sure whether I should publish my notes, since the process is highly custom. But maybe someone can benefit from this, as a starting point.

Additions#

Restoring the pmxcfs database file#

A friendly user ¹² pointed out that it is also possible to restore the pmxcfs database file directly, which backs all configuration files in /etc/pve¹¹:

1. Install a new proxmox server with same name
2. systemctl stop pve-cluster
3. cp config.db.bak /var/lib/pve-cluster/config.db
4. reboot

This is also described as a shortcut in the Proxmox docs¹¹, but I have not tried it so far and thus, cannot report how successful this will be. Comments?