Building Debian system images with vmdb2

1 Introduction

vmdb2 builds disk images with Debian installed. The images can be used for virtual machines, or can be written to USB flash memory devices, and hardware computers can be booted off them.

This manual is published as HTML and as PDF.

1.1 Installation

You can get vmdb2 by getting the source code from git, either author’s server or gitlab.com.

You can then run it from the source tree:

$ sudo /path/to/vmdb2/vmdb2 ...

In Debian 10 (“buster”) and its derivatives, you can also install the vmdb2 package:

$ apt install vmdb2

For any other systems, we have no instructions. If you figure it out, please tell us how.

1.2 Why vmdb2 given vmdebootstrap already existed

vmdb2 is a successor of the vmdebootstrap program, written by the same author, to fix a number of architectural problems and limitations with the old program. The new program is not compatible with the old one; that would’ve required keeping the problems, as well.

vmdebootstrap was the first attempt by it author to write a tool to build system images. It turned out to not be well designed. Specifically, it was not easily extensible to be as flexible as a tool of this sort should be.

1.3 Why vmdb2 given other tools already exist

The author likes to write tools for himself and had some free time. He sometimes prefers to write his own tools rather than spend time and energy evaluating and improving existing tools. He admits this is a character flaw.

Also, he felt ashamed of how messy vmdebootstrap turned out to be.

If nobody else likes vmdb2, that just means the author had some fun on his own.

2 Getting started

You need to make a specification file (in YAML) that tells vmdb2 what kind of image to build, and how. An example:

steps:
  - mkimg: "{{ output }}"
    size: 4G
  - mklabel: msdos
    device: "{{ output }}"
  - mkpart: primary
    device: "{{ output }}"
    start: 0%
    end: 100%
    tag: /
  - kpartx: "{{ output }}"
  - mkfs: ext4
    partition: /
  - mount: /
  - debootstrap: buster
    mirror: http://deb.debian.org/debian
    target: /
  - apt: install
    packages:
    - linux-image-amd64
    tag: /
  - fstab: /
  - grub: bios
    tag: /

The source repository of vmdb2 has more examples, which are also automatically tested, unlike the above one.

The list of steps builds the kind of image that the user wants. The specification file can easily be shared, and put under version control.

Every action in a step is provided by a plugin to vmdb2. Each action is a well-defined task, which may be parameterised by some of the key/value pairs in the step. For example, mkimg would create a raw disk image file. The image is 4 gigabytes in size. mkpart creates a partition, and mkfs an ext4 filesystem in the partition. And so on.

Steps may need to clean up after themselves. For example, a step that mounts a filesystem will need to unmount it at the end of the image creation. Also, if a later step fails, then the unmount needs to happen as well. This is called a “teardown”.

By providing well-defined steps that the user may combine as they wish, vmdb2 gives great flexibility without much complexity, but at the cost of forcing the user to write a longer specification file than a simple command line invocation.

To use this, save the specification into test.vmdb, and run the following command:

$ sudo vmdb2 test.vmdb --output test.img --verbose

Alternatively the specification can be passed in via stdin by setting the file name to -, like so:

$ cat test.vmdb | sudo vmdb2 - --output test.img --verbose

This will take a long time, mostly at the debootstrap step. See below for speeding that up by caching the result.

Due to the kinds of things vmdb2 does (such as mounting, creating device nodes, etc), it needs to be run using root privileges. For the same reason, it probably can’t be run in an unprivileged container.

Running vmdb2 in a container, privileged or not, is not supported. If it works, that is a happy accident. There are no tests to make sure that works, however, and no patches to make vmdb2 work in a container will be considered for merging. This is because vmdb2 is maintained by Lars, who has no use for containers, and does not want the maintenance burden of having code that supports that.

2.1 Setting the hostname

vmdb2 uses debootstrap, which copies the host’s /etc/hostname file into the image. You probably want to set the hostname for the image you’re creating. You can do this by overwriting the /etc/hostname file in the image, for example with the following step:

- chroot: rootfs
  shell: |
    echo myhostname > /etc/hostname

2.2 All images must be partitioned

At this time, vmdb2 does not support building partitioned images without partition, or images without a partition table. Such support may be added later. If this would be useful, do tell the authors.

2.3 Tags

Instead of device filenames, which vary from run to run, vmdb2 steps refer to block devices inside the image, and their mount points, by symbolic names called tags. Tags are any names that the user likes, and vmdb2 does not assign meaning to them. They’re just strings.

2.4 Jinja2 expansion

To refer to the filename specified with the --output or --image command line options, you can use Jinja2 templating. The variables output and image can be used.

- mkimg: "{{ output }}"
- mklabel: "{{ image }}"

The difference is that --output creates a new file, or truncates an existing file, whereas --images requires the file to already exist. The former is better for image file, the latter for real block devices.

2.5 Speed up image creation by caching

Building an image can take several minutes, and that’s with fast access to a Debian mirror and an SSD. The slowest part is typically running debootstrap, and that always results in the same output, for a given Debian release. This means its easy to cache.

vmdb2 has the two actions cache-roots and unpack-rootfs and the command line option --rootfs-tarball to allow user to cache. The user uses the option to name a file. cache-rootfs takes the root filesystem and stores it into the file as a compress tar archive (“tarball”). unpack-rootfs unpacks the tarball. This allows vmdb2 to skip running debootstrap needlessly.

The specify which steps should be skipped, the unless field can be used: unpack-rootfs sets the rootfs_unpacked flag if it actually unpacks a tarball, and unless allows checking for that flag. If the tarball doesn’t exist, the flag is not set.

- unpack-rootfs: root

- debootstrap: buster
  target: root
  unless: rootfs_unpacked

- cache-rootfs: root
  unless: rootfs_unpacked

If the tarball exists, it is unpacked, and the debootstrap and cache-rootfs steps are skipped. If the tarball doesn’t exist, the unpack step is silently skipped, and the debootstrap and caching steps are performed instead.

It’s possible to have any number of steps between the unpack and the cache steps. However, note that if you change anything within those steps, or time passes and you want to include the new packages that have made it into Debian, you need to delete the tarball so it is run again.

3 Acceptance criteria

This chapter documents the user-level acceptance criteria for vmdb2, and how they are to be verified. It’s meant to be processed with the Subplot tool, but understood by all users of and contributors to the vmdb2 software. The criteria and their verification are expressed as scenarios.

For reasons of speed, security, and reliability, these scenarios test only the core functionality of vmdb2. All the useful steps for actually building images are left out.. Those are tested by actually building images. However, those useful steps are not useful, if the core that invokes them is rotten.

3.1 A happy path

The first case we look at is one for the happy path: a specification with two echo steps, and nothing else. It’s very simple, and nothing goes wrong when executing it. In addition to the actual thing to do, each step also defines a “teardown” thing to do. We check that all the steps and teardown steps are performed, in the right order.

Note that the “echo” step is provided by vmdb2 explicitly for this kind of testing, and that the teardown field in the step is implemented by the echo step. It’s not a generic feature.

given an installed vmdb2
given file happy.vmdb
when I run vmdb2 -v happy.vmdb --output=happy.img
then exit code is 0
then stdout contains "foo\nbar\nbar_teardown\n"
steps:
- echo: foo
  teardown: foo_teardown
- echo: bar
  teardown: bar_teardown

3.2 Checking the version

Requirement: We can ask vmdb2 for its version.

given an installed vmdb2
when I run vmdb2 --version
then exit code is 0
then stdout matches regex ^\d+\.\d+$

3.3 Jinja2 templating in specification file values

vmdb2 allows values in specification files to be processed by the Jinja2 templating engine. This allows users to do thing such as write specifications that use configuration values to determine what happens. For our simple echo/error steps, we will write a rule that outputs the image file name given by the user. A more realistic specification file would instead do thing like create the file.

given an installed vmdb2
given file j2.vmdb
when I run vmdb2 -v j2.vmdb --output=foo.img
then exit code is 0
then stdout contains "image is foo.img\nbar"
steps:
- echo: "image is {{ output }}"
- echo: bar

3.4 Error handling

Sometimes things do not quite go as they should. Does vmdb2 do things in the right order then? This scenario uses the “error” step provided for testing this kind of thing.

given an installed vmdb2
given file unhappy.vmdb
when I try to run vmdb2 -v unhappy.vmdb --output=unhappy.img
then exit code is 1
then stdout contains "foo\nyikes\n"
then stdout contains "WAT?!\n"
then stdout contains "foo_teardown\n"
then stdout doesn't contain "bar_step"
then stdout contains "bar_teardown"
steps:
- echo: foo
  teardown: foo_teardown
- error: yikes
  teardown: "WAT?!"
- echo: bar
  teardown: bar_teardown

3.5 Step: ansible

Run Ansible using a provided playbook, to configure the image. vmdb2 sets up Ansible so that it treats the image as the host being configured (via the chroot connection). The image MUST have Python installed (version 2 or 3 depending on Ansible version).

Step keys:

Example (in the .vmdb file):

- apt: install
  tag: root
  packages: [python]

- ansible: root
  playbook: foo.yml
  tags: bar
  config_file: ansible.cfg
  group: AwesomeGroup
  extra_vars:
    iface:
      name: eth0

Example (foo.yml):

- hosts: image
  tasks:

    - name: "set /etc/hostname"
      shell: |
        echo "{{ hostname }}" > /etc/hostname

    - name: "unset root password"
      shell: |
        sed -i '/^root:[^:]*:/s//root::/' /etc/passwd

    - name: "configure networking"
      copy:
        content: |
          auto {{ iface.name }}
          iface {{ iface.name }} inet dhcp
          iface {{ iface.name }} inet6 auto
        dest: /etc/network/interfaces.d/wired

  vars:
    hostname: discworld

3.6 Step: apt

Install packages using apt, which needs to already have been installed.

Step keys:

Example (in the .vmdb file):

- apt: install
  tag: root
  packages:
  - python
  - linux-image-amd64

3.7 Step: cache-rootfs

Create a tarball of the root filesystem in the image.

Step keys:

Example (in the .vmdb file):

# typical use
- cache-rootfs: root
  unless: rootfs_unpacked

# create a rootfs tarball output at the end of a build process
- cache-rootfs: root
  force: true

3.8 Step: chroot

Run a shell snippet in a chroot inside the image.

Step keys:

Example (in the .vmdb file):

- chroot: root
  shell: |
      echo I am in chroot

3.9 Step: copy-dir

Recursively copy a directory from outside into the target filesystem.

Step keys:

3.10 Step: copy-file

Copy a file from outside into the target filesystem.

Step keys:

3.11 Step: create-dir

Create a directory in the target filesystem

Step keys:

3.12 Step: create-file

Create an empty file in the target filesystem.

Step keys:

3.13 Step: cryptsetup

Set up disk encryption using LUKS with the cryptsetup utility. The encryption passphrase is read from a file or from the output of a command. The encrypted disk gets opened and can be mounted using a separate tag for the cleartext view.

Step keys:

One of password, key-file, or key-cmd is REQUIRED.

Example (in the .vmdb file):

- cryptsetup: cleartext_pv0
  password: hunter2
  name: pv0

3.14 Step: debootstrap

Create a directory tree with a basic Debian installation. This does not include a boot loader.

Step keys:

Example (in the .vmdb file):

- debootstrap: buster
  target: root
  mirror: http://mirror.example.com/debian
  keyring: /etc/apt/trusted.gpg

3.15 Step: fstab

Create /etc/fstab inside the the image.

Step keys:

Example (in the .vmdb file):

- fstab: root

3.16 Step: grub

Install the GRUB bootloader to the image. Works on a PC for traditional BIOS booting, PC and ARM machines for modern UEFI booting, and PowerPC machines for IEEE1275 booting. Supports Secure Boot for amd64 UEFI.

Warning: This is the least robust part of vmdb2.

Step keys:

Example (in the .vmdb file):

- grub: bios
  tag: root

Same, but for UEFI, assuming that a FAT32 filesystem exists on the partition with tag efi:

- grub: uefi
  tag: root
  efi: efi
  console: serial

Or for IEEE1275, assuming that a partition with tag prep exists:

- grub: ieee1275
  tag: root
  prep: prep
  console: serial

Install to a real hard disk (named with the --image option):

- grub: uefi
  tag: root
  efi: efi
  image-dev: "{{ image }}"

3.17 Step: kpartx

Create loop devices for partitions in an image file. Not needed when installing to a real block device, instead of an image file.

Step keys:

Example (in the .vmdb file):

# typical use
- kpartx: "{{ output }}"

# using an image that already contains partitions containing filesystems
# that should be mounted as `/boot` and `/`
- kpartx: "{{ output }}"
  tags:
    - boot
    - root

3.18 Step: lvcreate

Create an LVM2 logical volume (LV) in an existing volume group.

Step keys:

Example (in the .vmdb file):

- lvcreate: rootvg
  name: rootfs
  size: 1G

3.19 Step: lvscan

Scans for existing LVM2 logical volumes (LVs) within a named volume group. This is useful when using an existing image as input which has been pre-populated with LVM logical volumes, allowing the volumes to be mounted. Please see the kpartx plugin documentation for further details on passing an existing image to vmdb2. This plugin adds LVM support to that use-case.

Step keys:

Example (in the .vmdb file):

- lvscan: the_volume_group
  tags:
    - lv_one
    - lv_two

3.20 Step: mkfs

Create a filesystem.

Step keys:

Example (in the .vmdb file):

- mkfs: ext4
  partition: root
  options: -O ^64bit,^metadata_csum

3.21 Step: mkimg

Create a new image file of a desired size.

Step keys:

Example (in the .vmdb file):

- mkimg: "{{ output }}"
  size: 4G

3.22 Step: mklabel

Create a partition table on a block device.

Step keys:

Example (in the .vmdb file):

- mklabel: msdos
  device: "{{ output }}"

3.23 Step: mkpart

Create a partition.

Step keys:

Example (in the .vmdb file):

- mkpart: primary
  device: "{{ output }}"
  start: 0%
  end: 100%
  tag: root

3.24 Step: mount

Mount a filesystem.

Step keys:

Example (in the .vmdb file):

- mount: root

3.25 Step: resize-rootfs

Configure the system on the image so that it automatically resizes itself to fill the actual disk, upon first boot. For this to work, the root file system MUST be the last partition on the image.

Also, the image MUST have the parted package installed for the partprobe command.

Step keys:

This is based on reading the changes by Peter Lawler to the image-specs repository to do the same thing.

Example:

- resize-rootfs: root

3.26 Step: set_part_flag

Set or clear a flag in a partition.

Step keys:

Example (in the .vmdb file):

- set_part_flag: "{{ output }}"
  tag: rootfs
  flag: bios_grub
  state: enabled

3.27 Step: shell

Run a shell snippet on the host. This is not run in a chroot, and can access the host system.

Step keys:

Example (in the .vmdb file):

- root-fs: root
  shell: |
      echo I am in NOT in chroot.

3.28 Step: unpack-rootfs

Unpack a tarball of the root filesystem to the image, and set the rootfs_unpacked condition to true. If the tarball doesn’t exist, do nothing and leave the rootfs_unpacked condition to false.

Step keys:

Example (in the .vmdb file):

- unpack-rootfs: root

3.29 Step: vgcreate

Create an LVM2 volume group (VG), and also initialise the physical volumes for it.

Step keys:

Example (in the .vmdb file):

- vgcreate: rootvg
  physical:
  - my_partition
  - other_partition

3.30 Step: virtual-filesystems

Mount the usual Linux virtual filesystems in the chroot:

They will be automatically unmounted at the end.

Often, the virtual filesystems are unnecessary, but some Debian packages won’t install without them. The grub boot loader needs them as well, but mounts what it needs itself, if necessary.

Step keys:

Example (in the .vmdb file):

- virtual-filesystems: rootfs