In the unlikely event that anyone has read my tech-related posts on this blog, it will be clear that I thoroughly enjoy getting enough from little, quarts from pint pots. This is partly because we live off-grid, so we can't throw power at a technology service, but mostly because it's much more fun delivering services when there are constraints. The pleasure of achievement, I find, is a pretty good reward.

I've discussed my choice of the lighttpd web server, rather then the more-usual apache or nginx, in previous posts too.   I find lighttpd configuration far easier to understand than either of those two stalwarts, and I believe, therefore, that I can be more certain of achieving a given outcome as a result. With lighttpd, I can configure it with greater understanding than just following a script, and hoping that I've cut and pasted correctly. And although I have no reason not to believe the claims that nginx performs best under heavy load, we are talking about a Raspberry Pi, which simply can't handle a heavy load, so nginx is not a specified need.  Yes, apache is such a lovely standard, reliable workhorse, and can be configured for leanness, but lighttpd appeals in other areas too.

One service that has become more important over the last few years is Nextcloud.  More recently, we have begun using Nextcloud Talk for weekly chats with friends, and, after a shaky start, we now find it a solid alternative to commercial offerings, with excellent quality. It means one is less concerned with the technology while chatting, and more about the chat itself. 

But our poor connection to the Internet here in the rural world pretty much abandoned by the monopoly provider, and even more abandoned by the Scottish Government's failed and failing R100 fibre roll-out for the north of Scotland, which looks to be as distant as ever, means that using a home-based internet service remotely from the home network can be slow.  Recently, after two years of badgering, BT have condescended to update out exchange to the ADSL2 specification, but our distance from the exchange means that the improvements do not extend to the upstream link. And BY have made it clear that this is our lot; we'll not get even fibre-to-the-cabinet, let alone fibre to the house, here.  So while we get old waiting for the Scottish Government's R100 fibre roll-out, which may or may not include our area, we just have to make do.

Our friends were reporting having to wait over five minutes after logging in to Nextcloud Talk.  I thought this was odd, as my tests on my phone were slow, yes, but not that slow.  Further tests showed that yes, it really was that slow.  But lighttpd was not compressing any of the data it was sending to the remote client.  using firefox's performance tools, I could see that Nextcloud Talk only needs to send about 14MB of data to the client.  But 14MB over a 0.35MB upstream connection makes that 5 minutes of waiting seem quite good.  Quickly I found I had to enable the compression module in lighttpd.  Compression options have changed over the years, and the old compress.conf configuration file is now deflate.conf. Sure enough, this dramatically improved things, and now only 4MB had to be uploaded to the client.  Suddenly Talk became more usable over our connection.

But further reading up about how lighttpd does all this revealed that a real gain was possible using a newer version of lighttpd than comes with openSUSE Leap 15.3, which is the operating system we run on the server.  Indeed, this languishes at version 1.4.55, several years old now, while the latest version, as of now, is 1.4.60.  There is a repository one can add to make a more recent version available, but this repository only included the x86_64 version, no good on the ARM-based Raspberry Pi.  Versions of lighttpd from 1.4.56 include Google's brotli compression technique.  Pretty much all browsers support brotli these days, and it is usually produces much smaller output than trusty old gzip, which lighttpd has long supported. It could be worth the effort to run a brotli-enabled version of lighttpd.  Also crucially is that the older versions of lighttpd do not support HTTP/2, another aid to lightness across the wire.

No matter. Some refresher reading, as I haven't done this for years, and I was able to rebuild a source RPM file.  Alas, it made no difference, and I spent ages convincing myself it was my configuration.  But then I did a check on the build, to find that brotli was not included at all. A check of the SPEC file and I could see why:  the flag --with-brotli was missing.  So another compile later and we had brotli compression.

This was a big difference. The 4MB gzipped version of the Talk main screen was now just 2.4MB.  But it took several minutes for the poor little Raspberry Pi to compress all the little snippets of javascript, text, style sheets, fonts and the rest.  After that first compression it was fast, but any new screen pretty much stopped the server, to the extent that for those minutes lighttpd could handle no further requests.  The main problem was solved, and to make life easier for our friends, I simply made sure I was logged in before they were, as the cache of brotli-compressed files would simply be used for their access.

But then I started reading about how brotli really wasn't as bad a performer as is often made out. Hah, I thought. What I can see happening is hardly my imagination.  It did occur to me that it could be the compression level that could be pouring all system resources into producing the smallest possible brotli file. But the lighttpd configuration says, at that section, not to play with the defaults unless you have a specific need, which sounded like good advice.  Further reading made it unclear whether the standard brotli implementation used compression level 11 (smallest files, highest resource use) as the default, or compression level 5 (balance between file size and resource use).  So  I added that explicitly:
deflate.compression-level = 5  
(Note that version 1.4.60 uses a different way of managing compression levels)

I cleared the cache and tested again. This time there was no page creation lag discernable, the compression achieved meant only 2.5MB of data had to be uploaded.  That's a result.  Almost the same compression outcome, with barely any additional load on the server, much better than gzip, and possibly bearing out some claims that brotli can be faster as well as compress better than gzip.

So I have changed my other servers, running raspberry Pi OS (Raspbian). and which do already offer a newer version of lighttpd, to this configuration. I think I would do this now even if we were not bandwidth-constrained.

I have not  checked apache's brotli capability, but nginx, at least on openSUSE, requires the installation of the brotli module. I would imagine that the same outcome can be configured for those.

We have recently been away, and our weekly chat with friends coincided with our being away. We found Nextcloud Talk with these configuration options very workable, and experienced no trouble at all with the session.  It is likely that, with more powerful hardware on a wider bandwidth upstream pipe would never reveal these issues, but it is heartening that they can be resolved when one has these constraints.

Addendum: I created the installable RPM that resulted in this blog post using version 1.4.58, which was the latest version I could get to compile properly.  I saw that the Opensuse Build Service was also having trouble compiling version 1.4.59 and 1.4.60, so was relieved it wasn't just me. After a while, like a terrier with a bone, I went back to the problem, which was a failure of the test process near the end of the compilation, and found a way of circumventing the test failure. But I knew it was a horrible hack.  So I registered with the Lighttpd forum to let the developers know about the issue and the workaround, which I knew was a "wrong" workaround.  Quickly a developer came back with a fix, then when I hit another similar snag, another fix.  These worked perfectly.  I noticed that the changes were immediately committed to the newer versions of Lighttpd, suggesting that others might benefit from this. Truly, the joys of Free and Open Source software, where my problem was taken seriously and resolved, and where one can experience the pleasure of making some really tiny contribution to the improvement of the software.

See the bug report here