Create the VPS instance

• Spin up a new server. Make note of its IP address
• Make DNS entries for the new server. There might be multiple entries, one for each $DOMAIN to be served. Keep the TTL value at 5 minutes or less for now; the TTL can be increased to 24 hours after everything has been set up and the system is stable.
• "ssh" into the new server as root. Example: ssh root@$DOMAIN

Lock down SSHD

• Edit /etc/ssh/sshd_config
  • Disable password authentication
  • Review other default configuration settings and make prudent adjustments to ensure your server is secure.
• rc-service sshd restart
• Testing: log out, then reconnect to verify that you didn't break anything.

Fix nano

• Edit /etc/nanorc and fix the key bindings for ^F and ^B so that they work as in Emacs:

bind ^F forward main
bind ^B back main

Create the "www" user and setup /home/www as a chroot jail

• adduser www
• mkdir -p /home/www/dev /home/www/proc
• → In the following, mknod values are from Alpine but also match Ubuntu. I think they are generic to Linux
• mknod /home/www/dev/null c 1 3
• mknod /home/www/dev/random c 1 8
• mknod /home/www/dev/urandom c 1 9
• chown root.root /home/www/dev/*
• `chmoe 0666 /home/www/dev/*
• chmod 0555 /home/www/proc
• mount -t proc /proc /home/www/proc
• → Edit /etc/fstab to append a line:
  • /proc /home/www/proc none rw,bind 0 2

Install Fossil, both inside and outside the chroot jail

• mkdir -p /home/www/usr/bin
• chown www /home/www/logs /home/www/Fossils
• → Upload a static fossil binary into /home/www/usr/bin
  • TO DO: add a link here to instructions on how to build a static Fossil binary.
• ln -s /home/www/usr/bin/fossil /usr/local/bin/fossil
• Testing:
  • which fossil
  • fossil version -v

Set up files for the website:

• mkdir -p /home/www/logs /home/www/Fossils /home/www/sbin
• chown www /home/www/logs /home/www/Fossils
• → Upload or clone zero or more Fossil repositories into the new /home/www/Fossils dir.
• → For each $DOMAIN to be served:
  • mkdir /home/www/$DOMAIN.website
  • chown www /home/www/$DOMAIN.website
  • Create static an dynamic content files in $DOMAIN. Examples shown below.
• Example static content file ".../index.html": <p>Hello, world!</p>
• Example dynamic content file ".../src" for invoking Fossil as a CGI. Make this file executable. Adjust pathnames to point to the Fossil repository, from the point of view of the chroot jail (omitting the /home/www prefix):

#!/usr/bin/fossil
repository: /Fossils/sqlite.fossil
errorlog: /logs/errors.txt

Install and test althttpd

• Upload a static althttpd binary into /usr/local/bin
  • TO DO: add a link here on how to build a static althttpd binary.
  • Need a version dated 2025-04-04 19:23Z or later
• /usr/local/bin/althttpd -root /home/www -user www -port 80
• Verify that the website is accessible
• Ctrl-C the "althttpd" above to stop the test server.

Make althttpd a service that starts automatically

• mkdir /home/www/ipshun
• chown www /home/www/ipshun
• Create file /etc/init.d/althttpd with content like this:

#!/sbin/openrc-run
# docs: https://wiki.alpinelinux.org/wiki/Writing_Init_Scripts
# https://github.com/OpenRC/openrc/blob/master/service-script-guide.md
name=$RC_SVCNAME
description="the althttpd web server"
command=/usr/local/bin/althttpd
command_args="-root /home/www
              -port 80
              -tls-port 443
              -certdir /etc/letsencrypt/live/main
              -logfile /logs/http.log
              -ipshun /ipshun"
command_background=true
pidfile="/run/${RC_SVCNAME}.pid"
depend() {
  need net
}

• Note: The althttpd server is launched to serve both unencrypted requests on port 80 and TLS-encrypted requests on port 443. However, the cert does not yet exist, so the 443 port option will be silently ignore for now.
• rc-update add althttpd default
• rc-service althttpd start
• Verify that the website is accessible (via "http:" only) and works. Since no encrypt is yet enabled, do not enter passwords or query sensitive information.
• Reboot the server and verify that everything comes back up correctly.

Install certbot

• apk add certbot
• Construct a script to generate a new cert in /home/www/sbin/cert-update.sh. This script should read something like the following, except adjusting the -w and -d arguments appropriately for the domains to be served.

certbot certonly --cert-name main --webroot $* \
   -w /home/www/a1_sqlite_org.website -d a1.sqlite.org \
   -w /home/www/a1_fossil_scm_org.website -d a1.fossil-scm.org

• Note: The --cert-name must be "main" because of the "/main/" in the pathname of the cert files in the /etc/init.d/althttpd.
• Test the script using: sh ./certbot-update --dry-run
• sh ./certbot-update.sh
• Create a cert renewal script in /home/www/sbin/cert-renew.sh and make it executable.

#!/bin/sh
certbot renew
rc-service althttpd restart

• Symbolic link from /home/www/sbin/cert-renew.sh into /etc/periodic/daily. This will cause a renewal attempt to happen once per day.
• /etc/periodic/daily/cert-renew.sh ↑ This should cause the cert to be obtained and the althttpd server to restart and begin serving TLS-encrypted content.

Install and activate OpenSMPTD

• apk add opensmtpd
• rc-update add smtpd default
• rc-service smtpd start
• Notes: The default configuration for the opensmtpd package is to operate in MTA (mail transfer agent) mode only. It will not accept incoming email from other machine, only from the local machine. It then relays the mail to its final recipient. This is exactly want is wanted for email notification in Fossil, hence no adjustments to the /etc/smptd/smptd.conf file are necessary.

Testing OpenSMTPD

• Create a email message in a text file "msg.txt" that is something like the following:

To: your-email@gmail.com
From: root@$DOMAIN
Subject: OpenSMTPD test

This is a test message from the new machine, $DOMAIN.

• Inject the message thusly: sendmail -f root@$DOMAIN -t <msg.txt
• Verify that the email arrives at your in-box.

Install wapptclsh (optional)

• Note: This step is only needed if wapptclsh is used by CGIs within the web content area. Skip it if you are not using wapptclsh CGIs.
• Upload a static build of wapptclsh to /home/www/usr/bin
  • TO DO: document how to generate a static wapptclsh
• Add a symbolic link: ln -s /home/www/usr/bin/wapptclsh /usr/local/bin
• Verify using: wapptclsh -v

Install TCL and the SQLite extension for TCL

• Note: This is a prerequisite for the script in the next section that transfers Fossil email notificates out of the jail and over to OpenSMTPD for delivery.
• apk add tcl
• apk add sqlite-tcl
• Verify using: echo 'puts [package require sqlite3]' | tclsh

Set up the script used to relay Fossil notification emails out of the jail

• apk add sqlite
• mkdir /home/www/data
• chown www /home/www/data
• Upload the script at https://fossil-scm.org/home/file/tools/email-sender.tcl into /home/www/sbin. Edit to change the DBFILE to "/home/www/data/emailqueue.db"
• Configure one or more of the Fossil repositories for sending email notifications using the "Store in a database" method. Make the database file "/data/emailqueue.db".
• Send one or more Fossil notification messages to yourself. Verify that those messages land in the /home/www/data/emailqueue.db file.
• Run: tclsh /home/www/sbin/email-sender.tcl. Let that command run for a few seconds to give enough time to transfer all of the messages in emailqueue.db over to OpenSMTPD, then stop it using Ctrl-C.
• Check your email to make sure the messages got delivered.

Configure email-sender.tcl to run automatically

• Create a new file "/etc/init.d/email-sender" that looks like the follows:

#!/sbin/openrc-run
# docs: https://wiki.alpinelinux.org/wiki/Writing_Init_Scripts
# https://github.com/OpenRC/openrc/blob/master/service-script-guide.md
name=$RC_SVCNAME
description="script to export emails out of the chroot jail"
command=/usr/bin/tclsh
command_args="/home/www/sbin/email-sender.tcl"
command_background=true
pidfile="/run/${RC_SVCNAME}.pid"</pre>

• rc-update add email-sender default
• rc-service email-sender start
• Test by using visiting the website and sending Fossil notifications to yourself.

Installing rspamd (for DKIM signing of outgoing email)

• Resource: https://prefetch.eu/blog/2020/email-server/
• apk add rspamd
• rc-update add rspamd default
• mkdir /etc/dkim-keys
• Puts public and privates keys in /etc/dkim-keys/mail.pub and /etc/dkim-keys/mail.key, respectively
• Leave the default rspamd configuration unchanged, except to add the file "/etc/rspamd/local.d/dkim_signing.conf":

allow_username_mismatch = true;
domain {
   pikchr.org {
      path = "/etc/dkim-keys/mail.key";
      selector = "mail";
   }
   sqlite.org {
      path = "/etc/dkim-keys/mail.key";
      selector = "mail";
   }
   fossil-scm.org {
      path = "/etc/dkim-keys/mail.key";
      selector = "mail";
   }
}

• Ensure the correct "mail._domainkey" TXT entries for each domain are in DNS

Connecting rspamd up to OpenSMTPD

• Resources:
  • https://github.com/poolpOrg/filter-rspamd
  • https://prefetch.eu/blog/2020/email-server/
• apk add opensmtpd-filter-rspamd

---

Maintenance Procedures

Adding a new domain

• Create and populate the domain folder /home/www/$DOMAIN.website
• Set up DNS for $DOMAIN to point to the server
• Verify page content using http:
• Edit /home/www/sbin/certbot-update.sh to add the new domain: Add a -w entry for /home/www/$DOMAIN.website followed by a -d $DOMAIN.
• sh /home/www/sbin/certbot-update.sh
• rc-service althttpd restart
• Verify using https:

Updating the althttpd binary to a new version

• Build a fully static binary on the desktop using procedures to be published.
• Verify: ldd ./althttpd
• Verify: ./althttpd -version
• `scp althttpd root@$SERVER:
• ssh root@$SERVER
• mv althttpd /usr/local/bin
• rc-service althttpd restart

Updating the fossil binary to a new version

• Build a fully static binary on the desktop using procedures to be documented more carefully later.
• Verify: ldd ./fossil
• Verify: ./fossil version -v
• scp fossil root@$SERVER:
• ssh root@$SERVER
• `mv fossil /home/www/usr/bin

Taking a website down for maintenance

• Do this, for example, before migrating from one server to another, first take the original website down so that it does not get modified while the transfer is in progress.
• mkdir /home/www/down.website
• chown www /home/www/down.website
• Create file "/home/www/down.website/not-found.html` with text like the following, but with the date/time adjust appropriately:

<h1>Website Down For Maintenance<h1>
<p>Expect return to service on or before 2025-04-04 00:00Z</p>

• mv $SITE.website $SITE.off; ln -s down.website $SITE.webiste