We run a few Raspberry Pis as servers for a variety of standard tasks.  Some tasks are run on a hosted Raspberry Pi (shortened to RPi subsequently in this post) to circumvent the appalling Internet infrastructure BT/Openreach tosses our way in the rural Highlands. Some are run at home to provide various services.  Among the services are: email (smtp and imap), web servers, file sync, calendars etc - anything provided by the excellent Nextcloud platform, hifi music, video and music storage, xmpp chat services, video conferencing (Nextcloud Talk and xmpp jingle) and so on. Also databases, weather station software with data going back to 2011, and Nagios system monitoring.  Software for this includes apache2, lighttpd, weewx, postgresql, mariadb, sqlite, php-fpm, postfix, dovecot, coturn, mpd, minidlna, opendkim, openvpn and all the little bits and pieces that hold everything together. The services are, if you like, "home" services, but are also necessary to run my wife's business and my interests.

For some years now, various Pis and various disks have run all our services, quite successfully. We are off-grid, so power consumption is important to us. When we first started living here full-time, the lowest power-consuming type systems were Intel Atoms, but they were positively blast-furnaces in comparison with running Raspberry Pis.  An estimate would be that the Pis use a quarter to a sixth of the power of those Atoms.  These days, another option would be a laptop-based system. The Ryzen 5 laptop on which I am typing this has an at-rest power draw, excluding the screen, of less than 4w, though considerable more when it's working, and to be honest, I'd love to see a little mini-ITX board with such a chip on board. But the cost and capability of a Raspberry Pi 4 means it remains a good choice for a small system, and even with USB disks rarely draws more than around 7w peaks, and 3.5w continuously.

Until now, my server operating system platform of choice has been Debian-based, or anyway, its RPi equivalent, Raspbian (now Raspberry Pi OS). All of the above services currently run on Raspbian. Although we run OpenSUSE on our laptops, and have done for quite a few years now, I prefer the predictability of Debian-based servers.  I have in the past, however, run at a corporate level SUSE, Gentoo, Ubuntu Server, Debian and Red Hat, as well as CentOS. I have tended to be cautious about server implementation (believe it or not, including the use of Gentoo).  I use the venerable jfs filesystem, as it is, to date, the only filesystem with which I have never lost any data, having used in the past ext(2-4), xfs and reiserfs.  Elsewhere on this blog I have described a way I have found reliable of using a filesystem other than ext4 for the root of a Raspbian system.

But I scent a whiff of change in the air.  It's hard to put a finger on why, but some software decisions from within the Pi Foundation make me uneasy, which contribute to a sense of disquiet regarding the continuing use of their preferred operating system in a server, rather than desktop or IoT configuration. It's just a feeling, but one on which I feel I should act sooner rather than later. As a harder reason, my beloved jfs filesystem seems to be rather neglected these days. This last is really unfortunate, as jfs really is a great filesystem, being fast, using few resources, and making excellent use of all space on a disk. Up to now, I've argued that jfs does not need continuing code-fiddling to remain relevant; it's complete, but the reality is that it is not really a first class citizen any more, more's the pity.  The next obvious steps, if I am to make a change, would appear to be OpenSUSE as a server, and btrfs as the preferred filesystem. Both are big changes when you are used to other ways of doing things. Let's deal with btrfs first.

It is difficult to do a paper evaluation of the btrfs filesystem. The main reason for this is that information and discussions on the internet are mostly very outdated. Many also fall into the category of opinion by those who haven't used the filesystem, and some are downright flame fests, for some reason. Actually, thinking about it, among the reasons could be a certain amount of polarity among the key Linux distributions. Discussions deteriorate to shouts of "whatabout-ery" especially among those who are more embedded in their preferred distributions and their strategies. Other discussions centre around competing filesystems and their legal status within Linux. Some statements are just wrong, so make trying to find decent information very hard. 

Certainly, if it was, say, 2015, I would not consider btrfs, as the rate of change was very fast. But for some years now, OpenSUSE has defaulted to using btrfs, mostly for its more advanced capabilities, such as snapshots, and SUSE themselves now have years of using the filesystem on anything from mainframes downwards. It was time to see for myself.  While I can see the value in the advanced features, I'm not one to use features for their own sake, so I expect my use to be quite simple.  The big question I had to answer was, will btrfs eat my data?  The answer so far appears to be "no". It's a safe option.  I would prefer to make that statement after several years of use, but I have now spent several weeks doing some hard kicking of the tyres, including doing things I'd never knowingly do to a production server, and all seems fine.  I have converted my laptop to btrfs, rather than install from scratch, I've swapped a RPi installation from sd-card to USB stick to USB-connected SSD and back again, and it's not missed a beat. I've pulled plugs out. I've pulled USB cables out. I've stress tested,  I've discovered issues with the RPi USB3 implementation and worked around them. So far, so good.

But why a move to OpenSUSE, then, rather than the many other options for the RPi? Well, for a start, it seems that the ARM platform is important to the mainstream commercial SUSE.  It results in an implementation that means that, if you are sitting in front of a console, it'll take more than cosmetics to let you know you are on an ARM single board computer rather than an x86 based system or even quite possibly for all I know, a mainframe. It's all the same pedigree. I came across an example of this just yesterday. I had to implement a workaround to a particular module. On Raspbian, you have to do this by editing the /boot/cmdline.txt file, which passes on the setting to the boot process. On OpenSUSE, you place the module options into settings under /etc/modprobe.d/ the same as you do under x86. (Admittedly, you then need to run the initrd creation process, dracut, but the same applies to x86).

If you move out of the immediate Raspberry Pi Foundation-sanctioned environment, there are some aspects that are not available, such as the proprietary aspects of the video chip in utilities like vcgencmd, but on a server, these are not missed. The standard Linux sensors suite tells you things like temperatures on OpenSUSE, at least for the CPU, and the video chips temperature will not be markedly different. Also, USB boot is more problematic, at the moment, but I tend to use a modified version of that anyway, and will deal with that later. OpenSUSE is a 64-bit implementation while Raspbian remains 32bit. I state that simply, as I'm not convinced that at the level of a limited memory RPi single board computer it has a vast bearing on things. I will say, though, that some services tested so far are a little faster, such as logging in to Nextcloud. What the cause of that slight speed increase may be, I don't know.

No, the biggest problem is learning the SUSE/OpenSUSE way of running a server, as opposed to years of experience in running services the Debian way.  One example is the extensive use of /etc/sysconfig configuration files under OpenSUSE, and the use of Yast, which for me is a two-edged sword.  None of it is hard, just different, and that takes getting used to.  This morning, I installed dropbear as an ssh service.  I expected to find a config file under /etc/default/dropbear, and was taken aback when I found nothing there.  Of course, under OpenSUSE, the correct file to edit to set the parameters I wanted was /etc/sysconfig/dropbear

Anyway, with that extensive preamble, I will now note some of the aspects I am encountering with the new (to me) server platform.  Please note that these are just my notes and may not mean much to anyone without some knowledge and experience of Linux system administration.

Also please note that some of the aspects of this blog post, most notably, regarding btrfs, may not be entirely best practice, or indeed, even correct, as I don't have enough knowledge or experience with them yet. It should be clear to which sections this caveat applies.

Change the root filesystem

This is where OpenSUSE shines. As with most distros, the easiest way to install OpenSUSE for the RPi is to copy an image file to the boot sdcard.  Like many or most distros, OpenSUSE defaults to ext4 as the root filesystem for this purpose, a pragmatic change, I suspect, from their defaults for other platforms.  But of course, that's no good for my purposes.  I want btrfs (although i also tried jfs and xfs, just because it's so straight forward.)  The trick here is to use a USB stick or disk as the new root filesystem.  So shove a stick in, partition it as you like and format it. Let's assume for the purpose of the exercise that you want to use xfs - we'll come to btrfs later.  Format the stick, probably appearing as /dev/sda, so you now have a new xfs filesystem as /dev/sda1.  Mount it at /mnt/newroot.  Hooray!  Start copying your old filesystem, but you need to leave out some directories, including the ones that are created at runtime, the destination directory, of course, and the RPi boot vfat partition.  This can be done with the following incantation, but feel free to make this cleaner:
rsync -aruPh --stats / /mnt/newroot/  --exclude=/mnt/newroot--exclude=/boot/efi \
--exclude=/dev --exclude=/run --exclude=/proc --exclude=/sys
Don't forget to re-create the directories left out of the rsync:
mkdir /mnt/newroot/dev /mnt/newroot/boot/efi /mnt/newroot/dev /mnt/newroot/run /mnt/newroot/proc /mnt/newroot/sys
Now, the old Gentoo build trick. Bind-mount the required directories before chroot'ing into the new environment to act as though you;re working in the new environment already:
for i in dev run proc sys; do mount -v --bind /$i /mnt/newroot/$i; done
Now in we go:
chroot /mnt/newroot
At this stage, we have to find out the unique UUID of the new root disk. This can be found by finding the right entry in
ls -al /dev/disk/by-uuid
Edit /etc/fstab to reflect the new disk UUID and the filesystem type, which will not longer be ext4 but xfs (in this case)
Now mount the new disks:
mount -a
I tend to rebuild the initrd at this point,to ensure that it includes the required modules for our new disk:
dracut -f

Then run yast, choose Bootloader (you may have to install this yast module beforehand), and make some change - I usually change the waiting time "Bootloader Options". Choose OK, and the system creates the required grub entries and so on. (I find when doing this on x86, I should run "grub2-install" to prevent surprises, but this step seems unnecessary under ARM).

By the way, OpenSUSE on Arm is sometimes criticised for being slow to boot. The problem lies with the above settings on the boot loader, where grub pauses for a default 10 seconds before loading the main system. I reduce this to 1 or 2, saving 8 or 9 seconds on the boot speed. There's no value in having the grub options delayed for 10 seconds, as the system is headless, but on other systems, for example, if you have grub and snapshots integration, you may want to boot into a prevoius snapshot, so that time period may be helpful.

Exit the chroot and reboot, and you should find yourself booting from the /boot/efi vfat partion on the sdcard, which then hands off to the USB stick, and you root will now be the USB stick and its xfs partition.  As straight forward as that.

It is possible to do the same thing with brfs, just using it as an ordinary filesystem, but really it is probably worth laying out the disk in the way SUSE/OpenSUSE lay out the disk for other platforms. This may be worth while in case some capabilities need to be used in future, or to make future changes more in keeping with the SUSE way of doing things.  For this, I mostly used a blog post by Richard Brown of OpenSUSE, along with the procedure outlined above.  Be aware that Richard's blog post includes a few minor typos ("subvoulme" instead of "subvolume" once, and "unmount" instead of "umount") so don't wildly cut 'n' paste.

As an exercise, I back-converted the sd card root partition to brfs, and so on, as these experiences make useful knowledge for disaster recovery or to understand what the system is doing. Eventually, though, more experience was gained after setting up a USB-connected SSD.

I should add now that, while raspbian can boot off the USB disk, this is still problematic for OpenSUSE Leap (though I believe it is possible on Tumbleweed, which I would rather not use as a server).  Actually, I don't mind this. The RPi SBC is designed to boot from a vfat partition, usually on an sd card. So why not make the sd card the required vfat partition, and have done with it. This is possible under Raspbian too, but is easier under OpenSUSE as all that vfat partition does is the initial early handoff to the main /boot area, the same as under x86.   So that's what I have done. A small but entire sdcard was formatted as vfat to hold the /boot/efi contents, and the required change to the UUID made in /etc/fstab. This makes the system independent of updating the onboard bootloader.

So far the only thing I have noticed that will need getting used to is the fact that moving a directory takes time under btrfs, but is almost instant with jfs. This is obviously because of the copy-on-write nature of the filesystem, and is no big deal, but for some purposes could, I suppose, be an issue.

USB 3 issues

Some Pi peripherals don't always play nicely with USB 3 on the Pi 4.  There is a "sticky" on this subject on the RPi support site entitled:

STICKY: If you have a Raspberry Pi 4 and are getting bad speeds transferring data to/from USB3.0 SSDs, read this

This post refers to a solution to which I referred earlier.  The module usb_storage allows for a "quirks" parameter that works around certain problematic USB adapters.  I had purchased a USB->SATA SSD adapter from The Pi Hut, and noticed, when working the disk hard that the system would hang for a period of time. Plugging in to a USB2 port solved the problem, but at the cost of a drastic slowdown. Raspbian solves this by adding an entry to /boot/cmdline.txt, but the OpenSUSE boot process does not use a cmdline.txt file.  Simply add the entry somewhere in /etc/modprobe.d and re-run dracut. The entry for me, I put in /etc/modprobe.d/99-local.conf and reads:

options usb_storage quirks=152d:0562:u

Note those USB identifier numbers may be different for your system This resolves the strange hangs under USB 3, but introduces an issue. The trim facility for the SSD stops working.  Enter another blog post, about how to enable trim under these circumstances. I'll not repeat the process outlined in that blog, but, for me, it works, and I now have a trim-enabled SSD connected via USB3 that hdparm reports as being capable of over 200MB/s as opposed to 30MB/s under USB2.

I knew about the above suggestions a while back. but have been reluctant to try them out on the existing production Raspbian server, for obvious reasons - I don't want to see the disks chew the data.  But while the OpenSUSE RPi is in a stress-testing phase, it's worth while trying this.

Weewx weather service

The installation of weewx was straight forward.  This is python-based weather station software that supports a wide variety of weather stations.  I have used it for quite a number of years, and the data in the sqlite database goes back to 2011.  The data may not be appropriate for meteorologists, but it's fine as a home weather station. It's especially important for us, as we are off grid, and rely on the weather, one way or another, for our power. With raspbian, it's just a case of adding a repo, and editing the configuration to one's test.  Fortunately, with OpenSUSE, it's just the same, and it is with thanks I note that the weewx team treat OpenSUSE on an equal footing with debian-style distros.  Migration was simply a case of copying my customised skins to the relevant /etc/weewx/skins directory, copying across the weewx.conf config file and copying the database sqlite file (one advantage of sqlite) to the /var/lib/weewx directory.  I have been able to test this quite easily as I have a backup USB-connected weather station master controller, which collects data from the same sensors as the production controller.

I have come across one oddity, in that I get report generation failures, with errors that suggest an issue with the change from python 2 to python 3, but that must be misleading, because sometimes the reports run perfectly.  Once the reports are running, there is no problem. I have not yet found the cause. or a real solution, but have noticed that if I change the "skin" of the report, then change it back again, it runs perfectly.

Later: I found the problem. It was in the customised "skin" template that the report generator uses. A customised section that creates the pressure trend summary was originally clumsily written. The biggest problem was finding which section was causing the trouble.

Nextcloud

I expected some difficulty testing a migrated Nextcloud environment, as the production system runs as a top level URL rather than a sub-directory of a top level (ie, xxxxx.vasten.co.uk rather than www.vasten.co.uk/xxxxx). But actually it was very easy.  One advantage is that I have long moved from using MySQL/Mariadb as the database to using Postgresql. I originally moved to Postgresql at a time when pretty much every update to Nextcloud resulted in database errors, incorrect data types and so on,which disappeared when I moved to Postgresql. It's likely that those issues under Mariadb have been solved, but the speed and ease of Postgresql meant I'd not want to go back.  Postgresql also co-exists on a general purpose server in a much happier way than mysql, which all too often assumes the entire server is given over to running it. (I do have some mariadb/mysql configurations at hand for when it's required, though, which reduce the system demands, especially memory demands, when running on a Pi.)

So I just dumped the Postgresql database on the production server and restored it under OpenSUSE (which runs Postgresql 12, rather than 11). I then copied the nextcloud web folder to the new location on the test server.   OpenSUSE stores web server data under /srv/www/htdocs, while Debian stores it under /var/www/html, another thing that needs getting used to. After that, it was a simple case of editing the nextcloud/config/config.php file to add the test server and location to "trusted domains". Then it was a case of copying the data, which is kept outside the web server directory, and finally changing ownership of all the files.  Debian uses the owner "www-data" for web ownership, while apache under OpenSUSE uses the wwwrun owner.  This has allowed me to test the full production instance of Nextcloud, but of course I will have to change some detail once the new server goes live.

I should add that there were changes to the way php is run, between Debian and OpenSUSE too.  But the general configurations are pretty much similar so I won't document it all here. I run php-fpm rather than using an apache module for php.  There was one oddity, though, and that was that OpenSUSE does not include a php.ini for the fpm configuration. I should probably find out why, but for the purposes of getting things running, I just symlinked the version under the cli config to the fpm config.

As I mentioned earlier, Nextcloud feels a little faster under OpenSUSE, an advantage but not a critical factor. It would have been a puzzler had it been the other way around.

While I was running these tests, I also tried running Nextcloud under lighttpd.  I understand why Nextcloud the company don't provide a standard configuration for lighttpd for their product - they can't chase every web server out there - but there are advantages to running it under lighttpd.  One is that lighttpd does what it can to get out of the way of a php application.  Anyway, there are some projects, such as dietpi, gentoo and arch, who run nextcloud under lighttpd, so getting a safe and working configuration was not hard. However, I did run into the dreaded circular login problem. This was caused by the fact that lighttpd runs as the lighttpd user, rather than wwwrun. I had chown'ed the data directories, but the directory where php stores session files, /var/lib/php, was still owned by wwwrun. Once I made that change,  all was well, and nextcloud logins are 20-25% faster than under Raspbian.

However, I may be forced to go back to apache. The default installation of Nagios under OpenSUSE requires apache.  This is very frustrating, as there is not need for it - it just needs a web server.  I will see if I can get around this. (LATER: yes, I can get around this, and mailgraph demanding apache too, but it's ugly and un-SUSE-like. I had to add a zypper lock to apache and its php module, then innstall them, acceptinng the dire warnings that the package would be broken becasue of uninstalled dependencies.)

The final transfer on migration day entailed copying across the nextcloud data directory, taking a pg_dump of the live database and copying that to the new system, and, for good meaure, copying the existing nextcloud application directory. I chose to store nextclouod data in a slightly different directory, to aid subsequent data management, so had to chnage that location in nextcloud config.php. Then, on the new server, do an "su - postgres", making sure the pg_dump file was in /var/lib/pgsql, where the postgres user could read it, then do a restore, with the "create database" and "clean out the existing data" options:

pg_restore -d nextcloud -c --clean final-nextcloud_pgsql.sql

Finally, I chose to help the new innstallation by rescanning the files. In the nextclouod application directory:

sudo -u lighttpd /usr/bin/php occ files:scan --all

I was afraid the nextcloud desktop clients might sync the wrong things after this, but, while the clients checked each file, perhaps because of an attribute change, all data was retained.

After the migration: The Nextcloud implementation gets an A+ security rating on Nextcloud's securit checker, so I am happy that the lighttpd configuration works well.

Nagios

Under Raspbian, I run a small Nagios monitoring environment for a dozen or so services. This is also linked to using the sendxmpp utility to send xmpp alerts for events.  I had a lot of trouble getting the web interface configured under Raspbian/Debian, so the question was whether to try to migrate the existing configuration or start again. Actually, I eventually ran the nagios web interface under an instance of lighttpd, as configuring it was a lot easier than apache.  As mentioned, OpenSUSE make apache a requirement for Nagios, and worse, makes it the prefork option and the apache php module, a stupidly heavy requirement when a little bit of choice would be a good thing.  After managing to install the system, I then discovered that it looked as though the package that really does the work, nagios4-plugins, is not part of the standard aarch64 installation.  In fact, after some sleuthing, I found that these are called "monitoring-plugins" rather than nagios-plugins.

This was another exercise that was worth the effort.  I first set up Nagios on my home systems about 13 years ago, migrating the configuration as time went by, or at least, building on it. So to start again from the beginning was a good way of refreshing my lack of knowledge.  I do run some services on non-standard ports and have other customisations, so some configuration changes are time consuming.

For the "ping" option to work, it's necessary to "chmod +s /usr/bin/ping" as OpenSUSE usually restricts the ping command to root.

I also run the sendxmpp utility to send an alert to myself via xmpp/jabber. This failed for a while, but the log gave the answer - the SSL perl utility needs to be installed, " zypper in perl-IO-Socket-SSL"

Overall, this was indeed a worth while exercise, and it turns out that the way OpenSUSE lays out Nagios configurations seems easier to understand than the Debian equivalent, or, possibly more likely, my Debian Nagios installation has been updated and migrated to witin an inch of its life, so a new start was well past time. It's just a pity that the OpenSUSE nagios installation has those daft, unneccesary dependencies.

Wallabag

This is a Free Software equivalent of the proprietary "Pocket" service, and allows us to store web pages for subsequent use, such as when researching a topic.  It used to be a nice simple set of little php scripts, but morphed into something rather more complex using the symfony and composer framework, and pulls in vast amounts (or so it seems) of web resources that take it far from the ideal small local service.  Well, I suppose it's not that bad, but in comparison with its simplle roots, it seems so. Nonetheless, as I have quite a corpus of data stored in the system, I was keen to migrate it, rather than re-install. Migrating it is fairly straight forward, even though I had forgotten that it has caches and all kinds of things, and that it is best to run a "make install" as the web user, eg "sudo -u lighttpd make install" to generate all the correct configuration.

After a while, although I could indeed migrate the system, I decided to try a new installation, but retain the old sqlite database. This is a finicky and, to me, unintuitive process, but I have to allow, it does work.  All my old data is there, and arguably, I have a cleaner system than before, as I chose more sensible file locations, and on a newer version of wallabag.

Logging

Raspbian/Debian uses rsyslogd for logging.  A basic OpenSUSE installation uses systemd journald. I would need to change a lot about how I run servers to swap to using journald, which I think may be a step too far for me. So I installed rsyslog.  Alas, the data format at the beginning of each line was daunting, " 2021-02-24T20:26:04.194396+00:00 purplepi weewx" followed by the actual message.  A bit of search engine work and I found I could apply this template to /etc/rsyslog.conf which makes the format this: "24 Feb 21:15:21 purplepi weewx".

$template MyTemplate,"%$day% %timegenerated:1:3:date-rfc3164% %timegenerated:12:19:date-rfc3339% %HOSTNAME% %syslogtag% %msg%\n"

Then add ";MyTemplate" to the various log files defined later in that file, for example:

*.*;mail.none;news.none                 -/var/log/messages;MyTemplate

This emulates the log format I'm used to. I'm probably being a stick-in-the-mud by changing this from the defaults, but I find it easier to read.

However, I then had a fit of the obvious, and thought I would compare the Debian /etc/rsyslog.conf file with the one from OpenSUSE.  Close to the top were two comments and a single line:

# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

I transferred that line to OpenSUSE with a comment and the logs are now what I am used to. Later I found that, had I read OpenSUSE's rsyslogd.conf file thoroughly, this line is actually there already, but commented out.

(Addendum: BUT:  See the section under "Mailgraph", below. I will probably try not to be so stuck in the mud and get used to the OpenSUSE way of log dates)

I'll leave the above addendum in place, to document my wanderings around this issue, butu eventually, settled on that TraditionalFileFormat log style. This allowed Debian's version of mailgraph to work, and also is what my eyes are used to scanning. fail2ban works against that format too.

Petidomo

This was one of the little utilities that run on my server that I almost forgot about. I run a couple of small mailing lists for some closed groups with which I am involved. Most mailing list software, not unreasonably, seems to be slanted towards high end requirements, but I needed something that just does the job and gets out of the way. Years ago, after running mailman and others, I stumbled across petidomo, a pleasing play on the name of the venerable majordomo list software. Unlike most other mailing list software, this is a simple C program that just does what it needs to do with no bells or whistles.  Under OpenSUSE, it was simply a case of running "./configure" and "make", and I could migrate the lists it manages, all of which are simple text configurations. There were no compilation errors, and no drama or fuss. Also, I did not have to install any additional build tools to compile petidomo. All were already present on the system.

DHCP, dnsmasq and dnscrypt

The main server runs dhcp for the network, as well as dns services with the fabulous dnsmasq.  I have recently added a layer of comfort with dnscrypt-proxy in conjunction with dnsmasq, a doddle to set up under Debian/Raspbian.   I have quite a few custom entries in dnsmasq, which is standard enough not to foresee any major issues when I finally put the new OpenSUSE live - obviously the full dhcp parts can't be tested while the production server runs those services.

dnscypt-proxy is another matter.  For a start, the OpenSUSE package is apparently buggy - see this post. (Actually, I think now it's just old. Somme of the comments in that forum do not seem entirely correct.)  For another, the configuration file is lengthy and unintuitive in comparison with Debian.  In fact, I was initially unable to get the packaged version to work at all.  Instead, I found that there is a pre-compiled binary available at the project's github site, https://github.com/dnscrypt/dnscrypt-proxy and I could download a working arm64 example. This was far more like the working system under Debian.  There were some community compilations for OpenSUSE at https://software.opensuse.org/package/dnscrypt-proxy but none of these were for the arm64 architecture. I don't really like having to use precompiled, out-of-package-manager options like this, but it is the best way if I want to use dnscrypt. I also thought I would have to start learning the unenviable "joys" of systemd unit files, but in fact, the prebuilt binaries have a flag that builds the systemd files for you.  So the procedure was to download the binary and copy the executable to /usr/local/sbin. I then copied the example dnscrypt-proxy.toml file to /etc/dnscrypt-proxy.d and edited it to point to my preferred upstream resolver, and the local port on which I want the dns queried. Still in the /etc/dnscrypt-proxy.d directory (as this creates the correct "working directory" for the unit file) I ran "/usr/local/sbin/dnscrypt-proxy -service install" and the unit file was created.  This could now be enabled and started.  Again, I was able to re-create administration options like logging by comparing configuration files.

Later, having learnt a lot on this process, I went back to trying the packaged version, but it is badly out of date (v1.94, while the latest is 2.45) and  had trouble getting it to use the upstream servers of my choice, rather than its choice. I decided to go back to the binary version.

This was a slightly disappointing exercise, from the point of view of OpenSUSE, as dnscrypt-proxy should be easily available and configurable in this day and age. However, the people on the dnscrypt-proxy team made what could have been a difficult issue quite painless. Indeed, on the github site, they warn "installing a new DNS cache and changing the DNS settings may surprisingly be one of the most complicated tasks you will ever have to do."

Email - postfix, dovecot and all the plumbing

This, to me, was the biggest challenge.  Email is the key service which justifies running one's own server, and it is worth while getting it right.  Let's deal with the straight forward issues first.  mailgraph provides a way of visualising what is happening on a postfix mail server, allowing a glance to give you a gist of inbound, outbound volumes, spam, blocked messages etc.  As with nagios, the default installation for OpenSUSE has a requirement of apache. This is annoying, as mailgraph is just a service, with a perl cgi front end. There is no specific need for apache.  But I found the server monitoring repository when looking for some nagios plugins that also contains a different mailgraph vs mailgraph-apache package. This installed without pulling in apache.

Again, I was faced with the question of how best to migrate. I have a well-working system, but I am aware that some comments in my postfix  main.cf are dated 2006, and at least one says "Not working on Wheezy".  The correct OpenSUSE way of doing this would be to create suitable entries in the /etc/sysconfig/postfix file and let the system create the appropriate main.cf file.  But one of the joys of postfix is its flexibility, and I have numerous additions and customisations.  I decided to choose a middle ground, cleaning up the master.cf and main.cf files while also migrating the various other files that make up the service.  Dovecot was slightly easier. I mostly migrated straight across, though there were some little differences, mostly around the fact that Debian runs postfix in a chroot jail, while OpenSUSE doesn't.  I wondered whether moving away from a chroot environment was sensible, but then found a discussion in which the creator of postfix, Wietse Venema, argues against using a chroot jail. So I took that as just an implementation choice of Debian, rather than a necessary security configuration.

Elsewhere on this blog I documented the use of bogofilter for the last security ring of spam protection on a lightweight server such as a Saspberry Pi.  Bogofilter works incredibly well, and really is lightweight, so I would not want to lose it.  There are a series of scripts and configurations which tie everything together, and as they are mostly documented elsewhere I'll not repeat them here. For the sake of the migration, though, I had to ensure that all the scripts were not only copied, but in their right places too. I should have the option of re-using the spam wordlist database or training the database from scratch.  As it is now highly effective, I may simply re-use the existing one.

I can't really fully test the new system until it goes live, but I can simulate quite a lot. We run Evolution as our email clients, so I was able to install Thunderbird working against the test environment to prove that the basics work.

I have run into a problem with opendkim. I have not been able to get OpenSUSE's non-chroot postfix enviornment working with opendkim as a socket milter. It works fine with an inet connection. 

Mailgraph

Mailgraph is a helpful utility for postfix that, well, graphs mail. It consists of a perl daemon that creates an rrd database by parsing the mail log, with a cgi front-end that generates graphs showing inbound, outbound, spam, blocked, greylisted etc, mail. I find it handy for gaining an at-a-glance idea of what trends the email server is coping with, eg, high spam rates and so on. This used to be clumsy to set up under Debian - not the daemon, which just worked, but the cgi front end. Once again, the configuration is not under /etc/default but under /etc/sysconfig.  Actually mailgraph is a simple thing, and can easily be installed from the source if necessary.  But I was having a great deal of difficulty in getting it to work, coming up with numerous "WARNING: line not in rsyslog format" errors.  It turns out that the cause was my creating a template for the rsyslog date format (see "Logging", above).  Occasionally, I could run it without errors, but also without it creating the rrd database. Once or twice I could run it manually and it worked, but I could not then resproduce those circumstances. I have now spent hours on this, which I kid myself is time well spent in understanding the OpenSUSE server way.  If I copy a log file from Debian, it parses that perfectly, yet the log format I now have in place (see "Logging" above) looks to me exactly the same as on Debian. It's a complete puzzle at the moment. There is a possibility that the local emailing I am testing with at the moment do not reflect in the mailgraph, and I may decide to wait until if and and when the system goes live and the logs are conventionally populated to see what's going on.

(Much later, and the last thing to get working properly) - I now have the rrd working. I could not get it working with the OpenSUSE native version. It simply refused to create the rrd database, and outputs nothing at all on the command line, so debugging is almost impossible.  I decided to run rsyslog with the "$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat" line, as that makes the log much easier for a mere human to read. At that point, I'd given up on getting mailgraph to work at all.  But I found that the version of the main mailgraph script that runs under Raspbian/Debian works in tis configuration, though without some of the parameters the OpenSUSE version allows, such as log type (rsyslog, syslog or metalog).  I could edit the systemd unit file (systemctl edit --full mailgraph) to remove those options, after which, with great relief, I have a working mailgraph system.

Fail2ban

An internet-facing server without fail2ban is only half dressed. In this case, it was quite straight forward to populate /etc/fail2ban/jail.local with the same, but edited for readability, settings as the old server. I remembered I had made a change to the way fail2ban works with the ufw firewall. Really, when fail2ban finds an attacker, I don't want to muck around just stopping the attack on the detected service, but I want that attacker to go away. Usually, fail2ban just blocks the attacked port to the attacker.  I was able to change /etc/fail2ban/actions.d/ufw.conf from:

#actionban = [ -n "<application>" ] && app="app <application>"
#            ufw insert <insertpos> <blocktype> from <ip> to <destination> $app
#
#actionunban = [ -n "<application>" ] && app="app <application>"
#              ufw delete <blocktype> from <ip> to <destination> $app

to

actionban = ufw insert <insertpos> <blocktype> from <ip> to <destination> $app
actionunban = ufw delete <blocktype> from <ip> to <destination> $app

and this does the trick.

It also means I can see the state of atacks either via the fail2ban log, or by doing a "ufw status" or usually "ufw status numbered", which gives an indication of the severity of attacks.

OpenVPN

The configuration for this copied over quite easily.  The only issue I still need to get my head around is that the systemd service file for this seems complicated, in that it is called openvpn@openvpn.service.  That odd ampersand was also in the dnscrypt-proxy installation, and obviously means something which I need to research. Debian allows the configuration file to be anything you wish as long as it ends dot-conf, but OpenSUSE wants a configuration file called openvpn.conf.  My use of OpenVPN is largely restricted to some accesses to managed systems on other networks, although occasionally I might use it from my laptop.   I should really take this opportunity too refresh the keys for this service, and may do so once the dust has settled.

Firewalling

While on the subject of OpenVPN , I don't fancy learning yet another complicated firewalling solution in the standard firewalld to which OpenSUSE defaults. However, the simpler ufw, which really does live up to its name of being uncomplicated, installs from the repos just fine.  I have found that ufw works beautifully with fail2ban scripts as well as the various rules required for OpenVPN to work. Again, perhaps I should look at firewalld a little closer, but I moved to ufw from Shorewall way back in the day, which is much more like firewalld, and I think those push factors, for a relatively simple setup, still apply.

In the end I stuck with ufw.

SysAdmin Scripts

The /usr/local/bin directory on the production server is full of scripts, from back-up scripts to little utilities.  On closer inspection, many of these I simply do not use any more and should long ago have cleaned them up. Some need tweaking, like the script for easy access to the system log, filtering out what I don't need to see and going far back enough to be useful. That points to Debian's syslog file, while the new one will have to point to /var/log/messages. This is just tedious process of ensuring all the necessary scripts are there and working (see the email section above) and working through issues as they get hit.

Letsencrypt and certbot

As I was no longer using apache, I had to re-issue the letsencrypt certificates using the --webroot option. I was not expecting any issues, but the virtual host directive in lighttpd for nextcloud's configuration resulted in a verification error when --dry-run'ning the certificate.  This was because the webroot differs from the nextcloud server root.  It took me a while to wrap my head around the issue, but I renew the certificate using a script anyway, as I only open port 80 when renewing LE certificates, so I just added two extra steps which remove (mask, really) the nextcloud vhost, runs certbot, when all verifications pass, then re-enables the vhost.

Oh no! No OpenSUSE equivalent.

So far, the only utility I run on Debian, that I can't find for OpenSUSE, and which is hardly a dealbreaker, is xtide. This is a system for generating tide tables for various locations, and as we are so near the sea, we have found this very useful.  I can run this on one of my raspbian based Pis if we really have to have it, or find a way later of building it for OpenSUSE, although these days it is difficult to get the required data files. (Addendum: Resolved: I have a trivial once-a-day script that generates the information as a local web page. I have migrated this to the little Pi on which we run mpd through a HiFiBerry Hifi connector, and which does little else. It'll not add to the workload of that Pi.)

Final Disk Copy

Part of the process all described above is to use a pre-production server to gain experience in administering a different distribution to Debian/Raspbian, with which I am so familiar.  While most tasks are the same, there will inevitably be different ways, different options, and different preferences for doing admin tasks. Another big aspect with which I knew I needed experience was, of course, the btrfs filesystem.  I have been running two test and pre-production environments for about two weeks now, doing unspeakable things to them to try to simulate the worst that could happen to a filesystem. So far I haven't found any issues.  That's a two-edged sword, as I would prefer it if Something Bad happened from which I could learn to recover.  Hopefully, as my days of administering systems for large corporates are thankfully over, and therefore the demands on services are not in the same league, I will never get to experience such an issue.

But I still didn't know how best to get the system I have been developing transferred to the newly-purchased new disk. With a traditional filesystem, I'd just create the new filesystem, rsync the data across, run a chroot environment as described above to bring the new disk into use and reboot.  Under btrfs, I knew I would have to re-create a lot of the steps to set up new subvolumes and so on.  There also seem to be utilities that carry out this task for you.  But my development system was just a 120G ssd, so I thought I'd choose the easy route: use the excellent dd tool to create a copy of the partition to a newly created partition on the new disk.  Here, sdb is the old disk and sda is the new.

dd if=/dev/sdb3 of=/dev/sda3 bs=4M iflag=fullblock oflag=direct status=progress

After this finished, I unmounted the old disk, used gdisk to delete and re-create the sda3 partition on the new disk (to ensure it covered the full remaining space on the disk), mounted the new partition to /mnt/tmp and ran

btrfs filesystem resize max /mnt/tmp/

A second later I had the full capacity I was expecting, the UUID of the root was the same, though as I had made a new, bigger swap partition, I changed the UUID of that in the new /etc/fstab, and the system then booted sweetly.  I hafe created a small vfat partition on the disk, because, although I understand that OpenSUSE Leap 15.2 currently needs to boot off the sd-card, this i only for what OpenSUSE sees as /boot/efi. In this case, the sd-card only contains a vfat parition with the contents of /boot/efi

It's all worked well, but see the next section, as eventually I decided to use this development phase to see what disaster recovery on a new disk would look like. So although the above was a good exercise, I ended up following a DR process to give me more confidence in how best to handle things if the worst comes to the worst.

btrfs and DR, backups and restores

I needed to answer the question about the essential aspects of running any filesystem, as regards btrfs: how do you go about setting up a decent backup and restore regime.  I could find plenty online about using snapshots to do backups, and I can see, in a larger environment, how attractive that aspect of btrfs can be.  But in a simpler environment, where one is more concerned that the primary, or only, disk, might fail or become corrupt, restoring is closer to the requirements for a full disaster recovery rebuild scenario. So my question was, how do I take periodic full copies of the system, more frequent full copies of the data, and how do I go about starting from scratch if there is a total failure of the main system?

Yes, it is possible to reconfigure all the services from scratch, but that would be terribly time consuming.  Yes, it is possible to use snapshots to do this, but the management of snapshots is a lot more complex (and quite likely prone to error as a result) than simply taking copies of the data from time to time. It's not as though we have multiple terrabytes or more of data on the server.

I decided, at least initially, to build on the simple backup regime I currently have in place. Occasionally (say, once a quarter, or when I remember) take a full rsync copy of the entire filesystem.  Weekly, take compressed tarballs of the main data stores, /home, maildirs and nextcloud data. (Actually in our case home directories change infrequently.)

If the worst comes to the worst, it should be possible rebuild on a new disk like this: Download the RPi4 image from OpenSUSE and create  a new SD card that allows a fresh boot. Plug in the new disk.  Use this DR script as a template for what needs to be done. It may or may not work, but seeks to automates the processes Richard Brown describes, and as done manually at the top of this lengthy post.  Don't just run this script. Don't trust it. Understand what it does and decide accordingly. You have been warned. (Having said that, I was having some trouble in a test system, and found the omissions ont he original script the hard way. This more or less works now, but still, the onus is on you.)


#!/bin/bash


############ See https://rootco.de/2018-01-19-opensuse-btrfs-subvolumes/
###  Start from scratch creation of a btrfs root disk
## This _SHOULD_ automate the processes in Richard Brown's post
## altered for ARM rather than x86

##########################################################
### DO NOT WILDLY RUN THIS SCRIPT ########################
##########################################################

### Rather understand what it does, and of necessary tease it out IF
## you have to so a full disk recovery

## parameters are:
## $1 = device name the bt AFTER /dev, eg sdb3
## $2 = the mount point WITHOUT the trailling slash - usually /mnt

## Also assumes a compressed mount.


##################################################
## Do this bit manually to make sure it starts OK:

##  mkfs.btrfs /dev/$1
## mount /dev/$1 $2 eg mount /dev/sdb3 /mnt
#################################################

mount -o compress=zstd /dev/$1 $2/

echo "Create the subvolumes"

btrfs subvolume create /$2/@
btrfs subvolume create /$2/@/.snapshots
mkdir /$2/@/.snapshots/1
btrfs subvolume create /$2/@/.snapshots/1/snapshot
mkdir -p /$2/@/boot/grub2/
btrfs subvolume create /$2/@/boot/grub2/arm64-efi
btrfs subvolume create /$2/@/home
btrfs subvolume create /$2/@/opt
btrfs subvolume create /$2/@/root
btrfs subvolume create /$2/@/srv
btrfs subvolume create /$2/@/tmp
mkdir /$2/@/usr/
btrfs subvolume create /$2/@/usr/local
btrfs subvolume create /$2/@/var

## create snapshot info
touch /$2/@/.snapshots/1/info.xml

echo "<?xml version=\"1.0\"?>" > /$2/@/.snapshots/1/info.xml
echo "<snapshot>"  > /$2/@/.snapshots/1/info.xml
echo "  <type>single</type>"  > /$2/@/.snapshots/1/info.xml
echo "  <num>1</num>"  > /$2/@/.snapshots/1/info.xml
echo "  <date>`date +%F" "%T`</date>"  > /$2/@/.snapshots/1/info.xml
echo "  <description>first root filesystem</description>" > /$2/@/.snapshots/1/info.xml
echo "</snapshot>"  > /$2/@/.snapshots/1/info.xml

##########################################

btrfs subvolume set-default $(btrfs subvolume list /$2 | grep "@/.snapshots/1/snapshot" | grep -oP '(?<=ID )[0-9]+') /$2
umount $2
# mount - o compress=zstd /dev/$1 $2
mount -o compress=zstd /dev/$1 $2/

######################################################

echo "Make directories...."

mkdir /$2/.snapshots
mkdir -p /$2/boot/grub2/arm64-efi
mkdir /$2/home
mkdir /$2/opt
mkdir /$2/root
mkdir /$2/srv
mkdir /$2/tmp
mkdir -p /$2/usr/local
mkdir /$2/var


echo "mount directories..."

mount /dev/$1 /$2/.snapshots -o subvol=@/.snapshots
mount /dev/$1 /$2/boot/grub2/arm64-efi -o subvol=@/boot/grub2/arm64-efi
mount /dev/$1 /$2/home -o subvol=@/home
mount /dev/$1 /$2/opt -o subvol=@/opt
mount /dev/$1 /$2/root -o subvol=@/root
mount /dev/$1 /$2/srv -o subvol=@/srv
mount /dev/$1 /$2/tmp -o subvol=@/tmp
mount /dev/$1 /$2/usr/local -o subvol=@/usr/local
mount /dev/$1 /$2/var -o subvol=@/var

######################################################

echo "Bind mounting for new disk chroot"

mkdir /$2/dev
mkdir /$2/proc
mkdir /$2/sys
mkdir /$2/run

for i in dev proc sys run;
    do mount -v --bind /$i /$2/$i;
    done

echo "If that looked OK, go into chroot"
echo "Once in, correct fstab, add the mount point as above, but with UUID"
echo "don't forget to add /boot/efi"
echo " and 'mount -a' before running yast boot loader config"

echo "Next step is to chroot into the new disk"

####################################

Now it should be possible to copy the Full backup that's been taken most recently onto the disk, run yast's bootloader config, and hopefully all will be well.  Actually this is where the Pi makes life easier. The boot loader seems to be much more a configuration manager rather than having also to install grub correctly, the way that x86 requires.

Having done such a full disk, start-from-scratch configuration, I am much happier that btrfs can be used in a way that suits my usage, rather than having to adapt to snapshots and their management demands.

General Notes

I did the above development work on an 8GB version of the Raspberry Pi 4. But my production Pi is just the 2GB version, which has been fine for our use, rarely using more than 500MB of memory and usually hovering around the 425-475 mark, under Raspbian, anyway.  But obviously I would need to ensure that the OpenSUSE installation runs within a 2GB capability.  So I swapped over the two systems, in preparation for the final stretch towards putting OpenSUSE live. Imagine my disappointment when the server was unpingable when it booted.  The problem was a silly one - persistent rules for the network device.  Removing the entries in /etc/udev/rules.d/70-persistent-net.rules resolved that. My omission rather than a mistake.  At first I thought it might be a filesystem problem, as in a further "test" of the filesystem, I whipped the Pi apart while it was still running.  But it all came up smoothly after all.

Postscript

Now, after the migration, I think some aspects of my expectations during the development process were too unrealistic. In particular, I was trying to chase the OpenSUSE way of doing things, but strived for too much purity in this. An example is postfix, where I thought I should use the OpenSUSE configuration engine via /etc/sysconfig/postfix, but, in fact, the customisations that suit our use of postfix were probably more valid than squeezing those customisations into a meta-configuration file.  The auto-create method is almost certainly for simpler, perhaps non-server, use cases. Once I understood this, I relaxed into the development more. My main concern, having done this type of thing for quite a while, was to make sure I don't do something that adversely affects what will happen come upgrade or update time. That means sticking to the rules where possible, and divergin in predictable ways where not.

The outcome has been good. The OpenSUSE system uses around 10% less memory than Raspbian, which surprises me, as I thought it would be the other way around. It also seems to be running slightly cooler. The Pi is in a wrap-around aluminium case/heat sink, and ran at around 39-40°C, whereasit now runs at around 35-36°C. I create tarballs of the email store and nextcloud data once a week for off-system backup. I could now think about using btrfs snapshots for this, but will continue my tried and tested way for a while.  Interestingly, this weekend's backup creation took about 15% longer than under Raspbian.  I had suspected that the reason lay with btrfs vs jfs, but then recalled that the old system ran on two disks, the data being read from one, and the tarball created on the other, and that may account for the regression.

The biggest outcome was surely the fact that I chose to redo or rework most of the services, rather than bluntly copy the migration. One forgets, over time, the little changes and configurations carried out when installing services, and as these all had to be re-done, it meant refreshing my memory, no bad thing. I was able to tidy up some implementation choices that might have been difficult on a live system.

I would very much like to re-visit the mailgraph issue, and maybe I should raise a bug report, as it cleary does not work as it should. The issue with dnscrypt-proxy is more difficult, as the defaut is an old version, and that may be the biggest issue. The only thing that I feel I need to monitor closely is the chnage to using lighttpd rather than apache. I don't mean the issues regarding the dependencies on apache that some services, like nagios, claim, which would have to be over-ridden anway (who uses mod-apache-php these days when fastcgi is available?) but rather I have no experience of its stability. Lighttpd is slow-moving software, which I quite like, but I would like the next version up, which supports http2. On our little system, it's unlikely that there qwould be much performance difference between apache and lighttpd, but the one thing that I do like about lighttpd is that configuration is not only easier, but much easier to understand.

I've long enjoyed the solidity of OpenSUSE on my laptops and desktops. I'm looking forward to that same sense of sturdiness on our new server too.

Addendum:  Now, a month or so after the migration, things are going well and I am pleased all the above work has proven to be worth while. I did notice, though, that the system seems to spend a lot of time at maximum cpu frequency.  It's a server, it got a lot of things to do, but I would still expect it to calm down fromm time to time.  The proprietary software needed to get the Pi performance details that are present on raspberry Pi OS are not available with OpenSUSE, but detail scan be approximated closely enough using the "sensors" package.  I then run a script (run it with "watch") that gives those approimated details:

## See https://www.pragmaticlinux.com/2020/06/check-the-raspberry-pi-cpu-temperature/

TEMP=`head -n 1 /sys/class/thermal/thermal_zone0/temp | xargs -I{} awk "BEGIN {printf \"%.1f\n\", {}/1000}"`

echo "----------------------------------------"
echo "`cat /etc/hostname` - Temperatures & Processor States"
echo ""

#################
# Alternative way with more immediate detail
cpuTemp0=$(cat /sys/class/thermal/thermal_zone0/temp)
cpuTemp1=$(($cpuTemp0/1000))
cpuTemp2=$(($cpuTemp0/100))
cpuTempM=$(($cpuTemp2 % $cpuTemp1))

echo CPU temp: $cpuTemp1"."$cpuTempM"°C"
echo -n "GPU approx temp: "$TEMP°C
echo ""

#cpuinfo_cur_freq
####################
echo ""
echo "Processor core speeds"
for cpunum in 0 1 2 3
do
#hz=$(cat /sys/devices/system/cpu/cpu$cpunum/cpufreq/scaling_cur_freq)
hz=$(cat /sys/devices/system/cpu/cpu$cpunum/cpufreq/cpuinfo_cur_freq)
hz=$(($hz+1))
MHz=$(($hz/1000))
#do cat /sys/devices/system/cpu/cpu$cpunum/cpufreq/scaling_cur_freq;
printf $MHz;printf " ";
done

echo " "
echo "Scaling core speeds"
for cpunum in 0 1 2 3
do
hz=$(cat /sys/devices/system/cpu/cpu$cpunum/cpufreq/scaling_cur_freq)
#hz=$(cat /sys/devices/system/cpu/cpu$cpunum/cpufreq/cpuinfo_cur_freq)
#hz=$(($hz+1))
MHz=$(($hz/1000))
#do cat /sys/devices/system/cpu/cpu$cpunum/cpufreq/scaling_cur_freq;
printf $MHz;printf " ";
done

echo ""
echo "---------------------------------------"

I then found this page discussing the use of cpupower utility to check and set the governer.  I dound that the "ondemand" giverner was set.  However, the "schedutil" governer is also available.  When I tested this, it seemed to calm things down a great deal, and it was no longer a case of simplistic high or low frequencies.  The page also recommends a systemd servcie file at /etc/systemd/system/cpupower.service, containing the following:

# /etc/systemd/system/cpupower.service
#
[Unit]
Description=Set cpupower to performance for cpu0

[Service]
Type=oneshot
# ExecStart=/bin/sh -c "/usr/bin/cpupower --cpu 0 frequency-set -g performance"
ExecStart=/bin/sh -c "/usr/bin/cpupower frequency-set -g schedutil"

[Install]
WantedBy=multi-user.target