Hey there,

Quick post to report back on my experiment with guix on a Vultr VPS.

There are 2 options to get a Guix system running:

Custom ISO

From your Vultr panel you need to add a custom iso, simply points it toward the guix installer iso.

Then on your VPS go to server details -> and look for isos, attach your iso.

Now this is all good if you want to do things manually but what about something more automatable?

Vultr snapshot and guix images

Fortunately Vultr allows you to bring in your own snapshots.

As of 2024 Guix does not provide an equivalent of Preseed/Kickstart or answers file so that’s why we’re not going to make an iso using that feature.

The only way for you to make a pre-made system is via the guix system image api.

The workflow is like this:

  1. Generate an image with the api, in our case we will need to generate an efi-raw image as the vultr snapshot feature only accept those, which is a shame as they do take more space than qcow2

  2. Upload that image to a bucket on S3 (scaleway, aws…)

  3. Download that image on Vultr

  4. When you deploy a news vps look for snapshot, and simply choose the snapshot you downloaded, it will restart with your snapshot.

  5. Resize the partition and the filesystem of the snapshot to match with the HDD/SSD storage volume provided by vultr.

  6. Create a swapfile.

  7. Recreate the /etc/guix/acl file, if this file is missing any guix pull or guix install command will try to build everything instead of using substitutes, I do not know why this is happening so please feel free to ping me.

    For now this is easily fixed with a guix archive command and then a reconfigure.

  8. As a bonus we will explore the setup of an nginx server with tls support.

Creating the image

Here is a sample os.scm:

(use-modules (gnu))

(use-service-modules networking ssh admin virtualization sysctl web mcron certbot)

(use-package-modules bootloaders ssh certs tls python linux disk)

(define garbage-collector-job
  ;; Collect garbage 5 minutes after midnight every day.
  ;; The job's action is a shell command.
  #~(job "5 0 * * *"            ;Vixie cron syntax
       "guix gc -F 1G"))

(define %web-root
  (with-imported-modules '((guix build utils))
    #~(begin
      (use-modules (guix build utils))
      (mkdir-p "/srv/http/YOUR-SITE")

      (call-with-output-file "/srv/http/YOUR-SITE/index.html"
        (lambda (port)
          (display "\
<!DOCTYPE html>
<html>
  <head>
    <title>Welcome to nginx!</title>
    <style>
     html { color-scheme: light dark; }
     body { width: 35em; margin: 0 auto;
       font-family: Tahoma, Verdana, Arial, sans-serif; }
    </style>
  </head>
  <body>
    <h1>Welcome to nginx!</h1>
    <p>If you see this page, the nginx web server is successfully installed and
      working. Further configuration is required.</p>

    <p>For online documentation and support please refer to
      <a href='http://nginx.org/'>nginx.org</a>.<br/>
      Commercial support is available at
      <a href='http://nginx.com/'>nginx.com</a>.</p>

    <p><em>Thank you for using nginx.</em></p>
  </body>
</html>
" port))))))

(operating-system
 (bootloader
  (bootloader-configuration
   (bootloader grub-efi-bootloader)
   (targets (list "/boot/efi"))
   ;; It is very important that terminal-outputs and inputs are set to serial.
   ;; I spent a couple hours trying to debug something that made syslogd using 100% cpu.
   ;; At first I noticed that too many failed attempts when sshing into the box caused a hang.
   ;; Then when I logged into the box via a kvm console I noticed syslog taking all the cpu.
   ;; To find out why I straced the process and discovered that it was trying to write
   ;; "max failed attempts blabla" to /dev/console except that it couldnt it was full of I/O
   ;; exceptions and syslog would try indefinitely to write to /dev/console.
   ;; I also had logs about agetty complaining about ttyS0 not being a real device or something.
   ;; So that's what pointed me towards the GRUB terminal outputs and input config and after
   ;; a bit of fiddling I found a config that worked.
   (terminal-outputs '(serial))
   (terminal-inputs '(serial))
   (serial-unit 0)
   (serial-speed 115200)))

 (host-name "web")

 (sudoers-file (plain-file "sudoers" "\
 root ALL=(ALL) ALL
 %wheel ALL=(ALL) NOPASSWD: ALL
 "))

 (timezone "Etc/UTC")
 (locale "en_US.utf8")

 (swap-devices (list (swap-space
                    (target "/swapfile"))))

 (file-systems (cons*
              (file-system
               (mount-point "/boot")
               (type "vfat")
               (device "/dev/vda1"))
              (file-system
               (mount-point "/")
               (device "/dev/vda2")
               (type "ext4"))
              %base-file-systems))

 (users (cons*
       (user-account
        (name "YOUR-USER")
        (comment "guix user")
        (group "users")
        (supplementary-groups (quote ("wheel")))
        (home-directory "/home/YOUR-USER"))
       %base-user-accounts))

 (packages (cons* nss-certs gnutls python strace parted %base-packages))

 (services (append (list
                  (service dhcp-client-service-type)

                  (service unattended-upgrade-service-type)

                  (service ntp-service-type)

                  (simple-service 'make-web-root
                                  activation-service-type
                                  %web-root)

                  (simple-service 'my-cron-jobs
                                  mcron-service-type
                                  (list garbage-collector-job))

                  (service nginx-service-type
                           (nginx-configuration
                            (server-blocks
                             (list (nginx-server-configuration
                                    (listen '("443 ssl"))
                                    (server-name '("YOUR-SITE"))
                                    (ssl-certificate
                                     "/etc/certs/YOUR-SITE/fullchain.pem")
                                    (ssl-certificate-key
                                     "/etc/certs/YOUR-SITE/privkey.pem")
                                    (root "/srv/http/YOUR-SITE"))))))

                  (service certbot-service-type
                           (certbot-configuration
                            ;; (server "https://acme-staging-v02.api.letsencrypt.org/directory")
                            (email "YOUR-EMAIL")
                            (certificates
                             (list
                              (certificate-configuration
                               (domains '("YOUR-DOMAIN")))))))

                  (service openssh-service-type
                           (openssh-configuration
                            (openssh openssh-sans-x)
                            (permit-root-login #f)
                            (password-authentication? #f)
                            (authorized-keys `(("YOUR-USER" ,(plain-file "YOUR-USER.pub" "\
YOUR-PUBKEY
")))))))
                 %base-services)))

Don’t forget to replace these values:

  • YOUR-USER
  • YOUR-PUBKEY
  • YOUR-SITE
  • YOUR-EMAIL
  • YOUR-DOMAIN

Here is a sample script to create an image from scratch:

#!/usr/bin/env bash
new_name=$1
image=$(guix system image --image-type=efi-raw --image-size=4831838208 --save-provenance os.scm)

cp "$image" .

image_name=$(basename "$image")

chmod +w "$image_name"

mv "$image_name" "$new_name"

Copy it somewhere and name it build-images, make it executable and then invoke it like this:

./build-images my-awesome-vultr-snapshot

The first parameter is a human readable name of the snapshot because by default guix output the image it created into the store and that one has a hash as a name.

You’ll notice that the image size is roughly around 4.5Gb , this is because the filesystem and partition will be resized later as mentionned earlier.

–save-provenance is there so that you can find the os.scm that was used to create the snapshot in /run/current-system/configuration.scm on the running machine.

Upload the image

I chose scaleway because they offer ~70gb for free in object storage.

You can do everything by the gui or the cli(recommended for gb uploads) so not much to say there, just don’t forget to configure the visibility of the object to public or generate a temporary link to share with vultr.

Download from vultr

Products -> Orchestration -> Snapshots -> Add Snapshot -> Upload snapshot from remote machine

Do not forget to tick the UEFI box.

Then:

Products -> Compute -> Deploy new instance -> At the server image section choose your snapshot.

Resize the partition and the filesystem

You should be able to ssh into the machine, once you’re in you’ll just need to resize the partition with parted and the filesystem with resize2fs.

# https://bugs.launchpad.net/ubuntu/+source/parted/+bug/1270203
# I tried to get a command that woud launch without prompting but alas :(
# Just follow the prompt, say fix and answer 2 then 100%
sudo parted  --fix -a opt -m /dev/vda ---pretend-input-tty unit % resizepart 2
sudo echo -e "yes\n100%" | sudo parted --fix -a opt -m /dev/vda ---pretend-input-tty unit % resizepart 2
sudo resize2fs /dev/vda2

A resize filesystem service was discussed on the mailing list https://www.mail-archive.com/help-guix@gnu.org/msg17088.html but nothing merged yet.

If you want to automate this kind of thing in the future you basically have 2 options:

  • Delegate this to an external tool such as Terraform, Pulumi or Ansible…
  • Write custom code in your os.scm that will take care of that, typically this will be implemented as a one-time service, you can use the link I posted earlier for an example.

Create a swapfile

We’ll create a swap file here, but you can use a partition aswell.

It is not enough to have this snippet in your code:

(swap-devices (list (swap-space
                 (target "/swapfile"))))

You really need to create the file with the following commands:

sudo truncate -s 0 /swapfile
# Change the amount of swap according to your requirements
# I picked 512Mb because on the vultr machine I have it has 1gb of ram.
# But you can certainly do much better, there is a plethora of literature
# concerning the optimal size of a swap file.
sudo fallocate -l 512M /swapfile
sudo chmod 0600 /swapfile
sudo mkswap /swapfile

And then start the swap service or reboot.

sudo herd start swap-/swapfile

Again you can write custom code to automate this or use an external tool.

Making substitutes work again

I noticed that any guix pull or install would throw with this warning:

substitute: guix substitute: warning: ACL for archive imports seems to be uninitialized, substitutes may be unavailable

I have no idea why but the /etc/guix/acl file doesn’t get created when you restore the snapshot, this file is necessary otherwise you won’t download packages and build everything from scratch.

The workaround is simple just use the guix archive command to recreate that file and then reconfigure the system.

sudo guix archive --authorize < /run/current-system/profile/share/guix/berlin.guix.gnu.org.pub
sudo guix system reconfigure /run/current-system/configuration.scm

The reconfigure will make sure that you have the acl with the keys you defined in your os.scm or the default ones.

Deploying an nginx site with tls support

In 2023 nginx and certbot didn’t play nice together and you first needed to reconfigure once with only the certbot service defined to get the certificates and a second time with the actual nginx service.

This was needed because if nginx started without a path to a certificate file it would just error out.

Now this has been fixed and you can have the 2 services at the same time from the get go, basically certbot will generate a self signed certificate for nginx to use so that it doesn’t complain about the missing certificate and then the cronjob that run certbot every hour will try to get a new one from let’s encrypt.

If you do not want to wait for the cronjob you can force the renewal with this:

sudo herd start renew-certbot-certificates

Of course be sure to have pointed an A record to your VPS, if anything fails you’ll get the logs in /var/log/letsencrypt

Finally if you want to know when the next certbot command will be run you can check with:

sudo herd schedule mcron

Pro-tip to know which action you can use on a service use:

sudo herd doc mcron list-actions

Conclusion

You should have a guix system up and running.