Back in 2009, I wrote an article for the much-missed Tectonic online magazine about Linux services without mains power.  The article mostly deals with design decisions when trying to squeeze quarts out of electrical pint pots, and a few years later, I wrote an update regarding the evolution on our systems.  A recent tweet regarding the software choices of a web site, Low-Tech Magazine, at the extreme edge of power supply prompt a revisit to the issue. The article, it seemed to me, is right in suggesting that design choices are key to getting by with less, though it also seemed to me that their approach makes too few compromises  from the ideal when some  pragmatism might make for more comfort.

Anyway, the last few years have seen the further development and considered replacement of parts of our power system, and we rarely have to be too concerned about how we use power, although that is also because power-consuming items we have are all purchased with their requirements in mind.  Look at the "Off-grid Power" category if this interests you.  I mention  this here, because most of the comments to the original article were not about the IT, but rather about how we get our power.  This really seems to interest folk, especially the partners of visitors to Helen's dye-shed, who love to be shown around the battery shed while their other halves are choosing yarn.  I must admit there are times when the fact that we produce our own electricity supply does indeed seem rather magical.  Over the summer months, when more than enough power comes in from the solar panels, we have been enjoying bread from our breadmaker (600w max) every second day, and we have also made extensive use of a small slow cooker (about 45-70w, for eight hours or more). So these days we don't feel too strapped for power, although recently, when we were having our hair cut, and the hairdresser said to Helen "And how are we drying our hair today?",  the hairdresser's look of disbelief that there could be a household without an electric hair dryer was a sight to behold.  We are now heading into the part of the year when the solar panels are least effective, but which tends to have more wind, but we may need to go back to baking bread in the gas cooker's oven from time to time.

But, my goodness, it was 4.5 years ago that I last wrote about our IT systems. By that time, we reluctantly ran an Atom-based mini-itx system as our main server. I say reluctantly, as it is hard to understand how Atom systems were marketed as low power parts. That system used 15-18w of power, as I recall, but I ended that blog post saying that maybe, just maybe, a Raspberry Pi could cope with our requirements.

So it not a surprise that a little while later - perhaps two years ago, we moved from the Atom-based mini-itx system to a simple Raspberry Pi. I wrote about that here. That required mental gymnastics for an old techie, as the mains storage is USB-connected, and everything in my techie make-up said using USB for main storage was wrong.  Other than sheer performance, which is not really an issue, that prejudice was unfounded.  The Raspberry Pi model 3 even allows booting off a USB disk or stick, meaning one can bypass the real storage compromise, the micro-SD card.

So now we are at the point where we need to design a system not only for low power, but also to fit in to where the Raspberry Pi is sweetest, rather than assuming it is merely a weak "proper" system.  Other than changes forced on us by time and bit-rot, the original design decisions still largely hold good.

The Hardware

We use a standard Raspberry Pi model 3.  We could go for the latest 3B+ model for a little extra power, but it has not been necessary, and, I must admit, I burnt out the early model I bought in the days after it was launched through rather exuberant play. We use an official Raspberry Pi power supply, as it seems there can be issues around using sub-standard power on the Pi.  There is a microSD card,  but only the /boot partition exists on it.  This is a bit of a legacy, to allow us to run the system on a Raspberry Pi 2 if a "disaster" happens, as the Pi 2 cannot boot from USB directly.  There is a 64GB USB stick attached, to which key backups occur (email, databases). The main storage is a USB 2TB disk.  From time to time, I back up the entire system, most of which consists of our entertainment movie and music files. Also attached is our old weather station, a TFA 932 (Irox, Honeywell etc).

The main disk, which is both the root and data storage, is formatted with the venerable jfs filesystem.  I have little experience with btrfs, but other than that, jfs remains the only filesystem with which I have not lost any data. Its performance is excellent, especially with the thousands of little files dovecot generates, and it reputedly makes low hardware demands. Up until recently, this meant creating an init file for booting which included the jfs modules, as the Raspbian kernel does not have built-in support for jfs, or anything other than vfat or ext4, I think.  But a while back the Raspbian developers put some effort into this, and now a kernel update automatically generates a correct initramfs file. I know this choice of jfs is a little bit greybeardy, but honestly, don't knock jfs until you have tried it.  There have also been benchmarks of postgresql against jfs suggesting it performs especially well, but that is no real consideration in our environment. (I even run jfs on my laptop with OpenSUSE and on an SSD disk. It has for some years supported the discard mount option for SSDs, and I find it as solid as ever.)

Current Software

Email remains postfix and dovecot.  Regarding email security, I got tired of the poor performance and high resource use of spamassassin, and moved to the superb, fast and accurate dspam system, but that was dropped from Debian after the maintainers lost interest.  I moved back to spamassassin reluctantly, but then settled on bogofilter, a rarely-used bit of software for server-based spam hunting, but which as met the requirement of being lightweight and accurate, if not wildly fast. The move to bogofilter is documented here.  Fail2ban picks up smtp and IMAP attacks and deals with them effectively.  I run fail2ban against the UFW firewall rather than generating iptables rules directly, which makes other firewalling and access limiting easier.  As before, postfix is set up to do as many security and antispam checks as possible, arranged largely in order of CPU and network expense, and this remains so effective that bogofilter only has to pick up the last few percent of nastiness aimed our way. The system handles the email for three domains. I have now added opendkim to the system as well.

There are a few changes that have more or less been forced on us.  9 years ago we used eGroupware for collaboration, calendars etc, but that product was going "open source core", which means going proprietary with a veneer of software freedom, but the Owncloud, then Nextcloud projects easily eclipsed eGroupware's utility.  Initially we ran owncloud against mysql, but two things changed. One was Owncloud became a nightmare to update, and the second was that mysql, or the mariadb variant Debian uses, is not very kind on resource on a Raspberry Pi.  Actually all database systems software seems stuck in an old world timewarp, as they seem to expect that only the database would be running on the system, rather than playing nicely and sharing.  But while still on Owncloud, before the move to Nextcloud resolved many of the issues we faced, I found that using Postgresql as the database was much more robust than the mysql option.  And as it happens, Postgresql is far less demanding on scarce hardware resources, both cpu and memory, than mariadb.  The best I can manage paring down mariadb to the minimum is using about 10% of the Pi's under-1GB of memory.  Postgresql, without any further tuning, uses less than half of that. Performance is about equal, but Postgresql somehow feels more solid.  So now we just run Postgresql on the main server (but see later...)

We still run Nagios to monitor various systems.  At its height a few months back, the little Pi, on top of all its other duties, was watching over 36 services spread across 7 machines at 4 locations.

The weather station software we ran became a problem.  While we have run weather station software since 2008, we moved to the fast, efficient and generally wonderful weatherview software.  But the developer has not updated the software since 2014, and although, being Free Software, I could have compiled the source, I did not fancy trying to keep software that old running.  Fortunately I was not the only one with this problem, and a group of erstwhile weatherview users, who have programming skills, started developing weewx.  I documented the moved form weatherview to weewx here.  The key things to note here was that Free Software meant I could build on the years of data we had accumulated, rather than start from scratch.  Weewx uses more system resources than weatherview, but considering weatherview is written in C and weewx in python, it is still very manageable by the little Pi.

Right now, the load averages on our little Pi are 0.08: 0.18: 0.13. The 5-minute figure is probably because the weewx software fires every five minutes, and updates both the local web server and two remote servers.  There are two active users, running between 6 and 8 devices against the services on the Pi, a mix of laptops, android phones and android tablets.  Generally, memory use is around 150MB at start-up, before the various services start using what they need, and is generally around 190-200MB, except when some Nextcloud functions are required, when php-fpm makes that climb as high as 400MB against the 973MB available.

But we run some other services too.  We have a Raspberry Pi Zero connected to the TV set. That Pi runs OpenELEC, which is just enough operating system for the Kodi entertainment system. Movies and music are kept on the main server and are access via NFS. The Pi Zero uses less than a watt of power most of the time, so we don't even think about it these days.  Kodi itself is controlled either via a web page, or via our mobile phones running Kore.  We also run a Raspberry Pi model B, together with a HiFiBerry DAC board, connected to our main hifi stereo amp and large freestanding speakers.  This runs mpd, with Rompr as a controlling web page, or else we use various mpd clients on our phones (I think the latest one is called M.A.L.P). Again, it accesses our music as FLAC files on the server via NFS, and also streams our favourite radio stations, like Cape Town's Fine Music Radio, Radio Venice International, SomaFM Left Coast 70s, and others, and all with astonishing audio quality.

This last Raspberry Pi, which we call Alpinoni (geddit?), sits idle most of the time, and rarely uses more than 40MB of the 490-odd MB available.  As it's a model B, it uses a watt or so of power.  As the CPU demands, with the hifiberry board, are very low, I have installed MariaDB on the machine, as a test environment for our hosted systems, as well as for when I need or want a mysql/mariadb database server.  The database increases the memory demand to around 25% of the system's memory, but as it is largely idle, it's no big deal.

I think that about wraps up the latest update on our off-grid IT life.  It will be clear that power requirement is a key driver of many of the purchasing decisions we make, such as when we bought a new laptop for Helen, but I hope that these stories are interesting for on-grid people too, as they show choices which can drastically reduce a household's, or even a business's need for power.  I prefer not to enable comments on this blog, but feel free to contact me at quibble @ (you know how to contract the email address) if you have questions or comments.