Advanced PiBuilder

This document explains how to customise PiBuilder to your needs.

Overview
Getting started
PiBuilder installation options
- Per-host installation options
- Environment variable overrides
  - about Git options
Script scaffolding
The PiBuilder Patching System
Existing customisation points
Using your custom branch in a build
- Original build method still works
Keeping in sync with GitHub

Overview

PiBuilder's main goal is to tailor a Raspberry Pi OS system to support IOTstack. If you are a first-time user, running the PiBuilder scripts and (implicitly) accepting all defaults will get you a stable "server" platform optimised for running your Docker containers.

As time goes on and you make changes to your Raspberry Pi, you may find yourself wondering what would happen if your Raspberry Pi failed (corrupted SD card; magic smoke; operator error) and you needed to rebuild it.

Here's an example of the kind of problem you might encounter. PiBuilder installs IOTstackBackup. Suppose you decide to take advantage of that. You follow the IOTstackBackup README. You choose the RCLONE option and set up a connection with Dropbox so your backups are saved in the cloud. And you finish off by creating a cron job to run iotstack_backup once a day.

If you ever have to rebuild your Raspberry Pi from scratch, PiBuilder will still install IOTstackBackup but you won't be able to run iotstack_restore to restore your IOTstack as of the last backup. Two other components are required:

Your RCLONE configuration. This contains your Dropbox token and is stored in ~/.config/rclone.
Your IOTstackBackup configuration. That tells IOTstackBackup to use RCLONE and Dropbox for backup and restore operations. It is stored in ~/.config/iotstack_backup.

It's actually a chicken-and-egg problem. Those files aren't included in any backup and, even if they were, that wouldn't help because you would still need the configuration files to be in the right place on your Raspberry Pi before you could fetch the backup files and extract the configuration files.

The solution is to add the IOTstackBackup and RCLONE configurations to PiBuilder. Then, the configuration files will already be in the right place at the end of the PiBuilder run and you will be able to run iotstack_restore without further ado.

Adding those configuration files to PiBuilder also means you won't have to go through the IOTstackBackup setup procedure on the newly-rebuilt Raspberry Pi before you can run iotstack_backup. Sort of win, win, win.

Customising PiBuilder doesn't just help with IOTstackBackup configuration files. You can include add your own packages to be installed via apt. Custom configuration files in /etc. Whatever you want, really.

Getting started

The examples here assume you will be working at the command line but you can also use desktop tools.

Start by cloning PiBuilder onto your support host:

$ git clone https://github.com/Paraphraser/PiBuilder.git ~/PiBuilder

PiBuilder does not have to be located in your home directory. It can be anywhere. Just substitute the appropriate path wherever you see ~/PiBuilder.

Create a custom branch to keep your own changes separate from the main repository on GitHub. A custom branch makes it a bit simpler to manage merging if a change you make conflicts with a change coming from GitHub.

$ cd ~/PiBuilder
$ git switch -c custom

You don't have to call your branch "custom". You can choose any name you like.

PiBuilder installation options

Use a text editor to open:

~/PiBuilder/boot/scripts/support/pibuilder/options.sh

The file supplied with PiBuilder looks like this:

 # this file is "sourced" in all build scripts. In the release version,
 # all variables are commented-out and shown with their default values.

 # - skip full upgrade in the 01 script.
 #SKIP_FULL_UPGRADE=false

 # - skip firmware in the 01 script.
 #SKIP_EEPROM_UPGRADE=false

 # - preference for kernel. Only applies to 32-bit installations. If
 #   true, adds "arm_64bit=1" to /boot/config.txt
 #PREFER_64BIT_KERNEL=false

 # - preference for handling virtual memory swapping. Three options:
 #      VM_SWAP=disable
 #         turns off swapping. You should consider this on any Pi
 #         that boots from SD.
 #      VM_SWAP=automatic
 #         same as "disable" if the Pi is running from SD. Otherwise,
 #         changes /etc/dphys-swapfile configuration so that swap size
 #         is twice real RAM, with a maximum limit of 2GB. In practice,
 #         this will usually result in 2GB of swap space. You should
 #         consider this if your Pi boots from SSD.
 #      VM_SWAP=custom
 #         applies whatever patching instructions are found in:
 #            ./support/etc/dphys-swapfile.patch
 #         Same as "automatic" but does not check if running from SD.
 #      VM_SWAP=default
 #         the Raspberry Pi OS defaults apply. In practice, this means
 #         swap is enabled and the swap space is 100MB.
 #   if VM_SWAP is not defined but the old DISABLE_VM_SWAP=true then
 #   that combination is interpreted as VM_SWAP=disable
 #VM_SWAP=automatic

 # - default language
 #   Whatever you change this to must be in your list of active locales
 #   (set via ~/PiBuilder/boot/scripts/support/etc/locale.gen.patch)
 #LOCALE_LANG="en_GB.UTF-8"

 # - Raspberry Pi ribbon-cable camera control
 #   Options are: disabled, "false", "true" and "legacy"
 #ENABLE_PI_CAMERA=false

 # - Handling options for .bashrc and .profile
 #   Options are: "append" (default), "replace" and "skip"
 #   See PiBuilder "login" tutorial
 #DOT_BASHRC_ACTION=append
 #DOT_PROFILE_ACTION=append

The defaults are appropriate for most first-time builds. However, you can uncomment any variable and set its right hand side as follows:

SKIP_FULL_UPGRADE to true. This prevents the 01 script from performing a "full upgrade". It may be appropriate if you want to test against a base release of Raspberry Pi OS.
SKIP_EEPROM_UPGRADE to true. This prevents the 01 script from updating your Raspberry Pi's firmware. Otherwise, the 01 script runs:
```
 $ rpi-eeprom-update
```
If and only if the response includes "UPDATE AVAILABLE" is a firmware update applied. The EEPROM is updated during the reboot at the end of the 01 script. The process adds extra time to the normal reboot cycle so please be patient.
PREFER_64BIT_KERNEL to true. This only applies to 32-bit versions of Raspbian. The overall effect is a 64-bit kernel with a 32-bit user mode.
VM_SWAP to:
- disable to disable virtual memory (VM) swapping. This is appropriate if your Raspberry Pi boots from SD and has limited RAM.
- automatic:
  - If the Pi is running from an SD card, this is the same as disable.
  - If the Pi is not running from an SD card, the script changes the swap configuration in /etc/dphys-swapfile so that swap size is calculated in two steps:
    1. The amount of real RAM is doubled (eg a 2GB Raspberry Pi 4 will be doubled to 4GB);
    2. A maximum limit of 2GB is applied.
    This calculation will result in a 2GB swap file for any Raspberry Pi with 1GB or more of real RAM. This is the recommended option if your Raspberry Pi boots from SSD or HD.
    
    Rules 1 and 2 are implemented by the ./etc/dphys-swapfile.patch supplied with PiBuilder. If you change or override that file then whatever rules your patch imposes will be implemented by automatic.
- custom is equivalent to automatic but it does not check if your system is running from SD. If you want to enable swap on an SD system, this or "default" are the options to use.
- default makes no changes to the virtual memory system. The current Raspberry Pi OS defaults enable virtual memory swapping with a swap file size of 100MB. This is perfectly workable on systems with 4GB of RAM or more.
If VM_SWAP is not set, it defaults to automatic.

Running out of RAM causes swapping to occur and that, in turn, has both a performance penalty (because SD cards are quite slow) and increases the wear and tear on the SD card (leading to a heightened risk of failure). There are two main causes of limited RAM:
- Insufficient physical memory. A good example is a Raspberry Pi Zero W2 which only has 512MB to start with; and/or
- Expecting your Raspberry Pi to do too much work, such as running a significant number of containers which either have large memory footprints, or cause a lot of I/O and consume cache buffers, or both.
If you disable VM swapping by setting VM_SWAP to disable, but you later decide to re-enable swapping, run these commands:
```
 $ sudo systemctl enable dphys-swapfile.service
 $ sudo reboot
```
You can always check if swapping is enabled using the swapon -s command. Silence means swapping is disabled.

VM swapping is not bad. Please don't disable swapping without giving it some thought. If you can afford to add an SSD, you'll get a better result with swapping enabled than if you stick with the SD and disable swapping.
LOCALE_LANG to a valid language descriptor but any value you set here must also be enabled via a locale patch. See setting localisation options tutorial. "en_GB.UTF-8" is the default language and I recommend leaving that enabled in any locale patch that you create.
ENABLE_PI_CAMERA controls whether the Raspberry Pi ribbon-cable camera support is enabled at boot time:
- false (or undefined) means "do not attempt to enable the camera".
- true means "enable the camera in the mode that is native for the version of Raspberry Pi OS that is running".
- legacy, if the Raspberry Pi is running:
  - Buster, then legacy is identical to true;
  - Bullseye the legacy camera system is loaded rather than the native version. In other words, Bullseye's camera system behaves like Buster and earlier. This is the setting to use if downstream applications have not been updated to use Bullseye's native camera system.
DOT_BASHRC_ACTION and DOT_PROFILE_ACTION both default to append. Allowable values if uncommented are append, replace and skip. See Login Profiles tutorial for more information on how to use these options.

Per-host installation options

Changes you make to the following file apply to all your hosts:

~/PiBuilder/boot/scripts/support/pibuilder/options.sh

You can also create a variant of the options file which is specific to a given host. You do that by appending @ followed by the host name. For example, if your Raspberry Pi uses the name "iot-hub", its host-specific options file would be:

~/PiBuilder/boot/scripts/support/pibuilder/options.sh@iot-hub

If both a host-specific and a general options file exist, the host-specific file is given precedence and the general file is ignored.

Environment variable overrides

Some of PiBuilder's scripts support additional customisation by setting environment variables that are not listed in the default options.sh. You can apply overrides in one of three ways:

Adding the environment variable to your options.sh; or

Specifying the override inline on the call to the script. For example:

$ IOTSTACK="$HOME/MySpecialIOTstack" ./PiBuilder/boot/scripts/03_setup.sh

Exporting the override before calling the script. Example:

$ export IOTSTACK="$HOME/MySpecialIOTstack"
$ ./PiBuilder/boot/scripts/03_setup.sh

The variables supported in this fashion are summarised below.

variable	script(s)	default
`GIT_CLONE_OPTIONS`	03	`--filter=tree:0`
`IOTSTACK`	03, 04	`$HOME/IOTstack`
`IOTSTACK_URL`	03	`https://github.com/SensorsIot/IOTstack.git`
`IOTSTACK_BRANCH`	03	`master`
`IOTSTACKALIASES_URL`	03	`https://github.com/Paraphraser/IOTstackAliases.git`
`IOTSTACKALIASES_BRANCH`	03	`master`
`IOTSTACKBACKUP_URL`	03	`https://github.com/Paraphraser/IOTstackBackup.git`
`IOTSTACKBACKUP_BRANCH`	03	`master`

The variables with _URL and _BRANCH suffixes are intended to make it easy to clone those repositories from your own custom clones, forks and branches.

Note:

If you change the IOTSTACK variable, you must be consistent and use it for both the 03 and 04 scripts, otherwise PiBuilder will raise an error.

about Git options

The default value of GIT_CLONE_OPTIONS is consistent with the IOTstack install.sh script, save that it is also applied to cloning the IOTstackAliases and IOTstackBackup repositories.

These are your options for invoking the 03 script. They are ranked in increasing order of the load placed on GitHub:

Shallow clone (least expensive):
```
 $ GIT_CLONE_OPTIONS="--depth=1" ./PiBuilder/boot/scripts/03_setup.sh
```
This is the "cheapest" download but it constrains your options (eg your ability to switch between the IOTstack old and new menu systems) quite severely. Not really recommended.
Treeless clone (the PiBuilder default):
```
 $ ./PiBuilder/boot/scripts/03_setup.sh
```
This passes the --filter=tree:0 option to git clone. It only downloads from GitHub what is essential to running IOTstack on your machine. The downloading of additional components is deferred until it is actually necessary which, in many installations, could easily be "never".
Blobless clone:
```
 $ GIT_CLONE_OPTIONS="--filter=blob:none" ./PiBuilder/boot/scripts/03_setup.sh
```
This download all reachable commits and trees, but only downloads blobs when necessary.
Full clone (most expensive):
```
 $ GIT_CLONE_OPTIONS= ./PiBuilder/boot/scripts/03_setup.sh
```
This is the more traditional clone which downloads a complete copy of each repository from GitHub.

Note:

You can use GIT_CLONE_OPTIONS= to pass any supported options to the git clone command. Fairly obviously, you are responsible for passing valid options!

Script scaffolding

Every script has the same basic scaffolding:

source the common functions from functions.sh
invoke run_pibuilder_prolog which:
- sources the installation options from either:
  - options.sh@$HOSTNAME or
  - options.sh
- sources a script-specific user-defined prolog, if one exists
perform the installation steps defined in the script
invoke run_pibuilder_epilog which sources a script-specific user-defined epilog, if one exists
either reboot your Raspberry Pi or logout, as is appropriate.

Note:

When used in the context of shell scripts, the words "source", "sourcing" and "sourced" mean that the associated file is processed, inline, as though it were part of the original calling script. It is analogous to an "include" file.

The PiBuilder Patching System

How PiBuilder scripts search for files, folders and patches

Search function

PiBuilder's search function is called supporting_file(). Despite the name, it can search for both files and folders.

In most cases, supporting_file() is used like this:

TARGET="/etc/resolv.conf"
if SOURCE="$(supporting_file "$TARGET")" ; then
   
   # do something like copy $SOURCE to $TARGET

fi

Here's a walkthrough. supporting_file() takes a single argument which is always a path beginning with a /. The path to the support directory is prepended so the argument so you wind up with an absolute path like this:

/home/pi/PiBuilder/boot/scripts/support/etc/resolv.conf

That path is considered to be the general path. A host-specific is constructed from the general path by appending @ plus the $HOSTNAME environment variable. For example, if HOSTNAME had the value "iot-hub" the host-specific path would be:

/home/pi/PiBuilder/boot/scripts/support/etc/resolv.conf@iot-hub

If the host-specific path exists, the general path is ignored. The general path is only used if the host-specific path does not exist.

If whichever path emerges from the preceding step is:

a file of non-zero length; or
a folder containing at least one visible component (file or sub-folder),

then supporting_file() returns that path and sets its result code to mean that the path can be used. Otherwise the result code is set to mean that no path was found.

So, assuming the if test succeeds:

SOURCE will be the absolute path inside the PiBuilder folder to either a host-specific or general path containing your customisations; and
TARGET will be an absolute path on the Raspberry Pi to the file to be replaced or otherwise manipulated.

If the conditional code within the scope of the if were, say:

cp "$SOURCE" "$TARGET"

the effect would be to replace the default version of resolv.conf supplied with your Raspberry Pi, with the version provided by you in PiBuilder.

Patch function

The try_patch() function takes two or three arguments:

A path beginning with a / which has the same definition as for supporting_file().
A comment string summarising the purpose of the patch.
An optional boolean. If "true", it instructs the function to ignore patching errors. Defaults to false if omitted.

For example:

try_patch "/etc/resolv.conf" "this is an example"

The patch algorithm first checks whether the target (the file to be patched in the running system) actually exists. If it does not then, returns "success" if the third argument is true, otherwise returns "fail".

If the target exists, the patch algorithm appends .patch to the path supplied in the first argument and then invokes supporting_file():

supporting_file "/etc/resolv.conf.patch"

Calling supporting_file() implies both host-specific and general candidates will be considered, with the host-specific form given precedence.

If supporting_file() returns a candidate, the patching algorithm will assume it is a valid patch file and attempt to apply it to the target file. The function sets its result code to mean "success" if either:

the patch was applied successfully; or
the patch failed, in whole or in part, and the third argument is true.

Otherwise the function result code is set to mean "fail".

The try_patch() function has two common use patterns:

unconditional invocation where there are no actions that depend on the success of the patch. For example:
```
 try_patch "/etc/dhcpcd.conf" "allowinterfaces eth*,wlan*"
```

conditional invocation where subsequent actions depend on the success of the patch. For example:

 if try_patch "/etc/dphys-swapfile" "setting swap to max(2*physRAM,2048) GB" ; then
 	sudo dphys-swapfile setup
 fi

conditional invocation where subsequent actions should occur as long as the patch was attempted (the third optional "true" argument). For example:
```
 if try_patch "/etc/locale.gen" "setting locales (ignore errors)" true ; then
 	sudo dpkg-reconfigure -f noninteractive locales
 fi
```

File edit function

The try_edit() function takes two arguments:

A path beginning with a / which has the same definition as for supporting_file().
A comment string summarising the purpose of the edit.

For example:

try_edit "/etc/dphys-swapfile" "revert swapfile to defaults"

The algorithm first checks whether the target (the file to be patched in the running system) actually exists. If it does not then the function returns "fail".

If the target exists, the algorithm appends .sed to the path supplied in the first argument and then invokes supporting_file():

supporting_file "/etc/dphys-swapfile.sed"

Calling supporting_file() implies both host-specific and general candidates will be considered, with the host-specific form given precedence.

If supporting_file() returns a candidate, the algorithm will assume it is a file containing valid sed editing instructions and will attempt to apply it to the target file. The function sets its result code to mean "success" if the editing instructions could be applied and the edited file compares-different from the original.

Otherwise the function result code is set to mean "fail".

The try_edit() function has two common use patterns:

unconditional invocation where there are no actions that depend on the success of the patch. For example:
```
 try_edit "/etc/dphys-swapfile" "revert swapfile to defaults"
```

conditional invocation where subsequent actions depend on the success of the patch. For example:

 if try_edit "/etc/dphys-swapfile" "revert swapfile to defaults" ; then
 	sudo dphys-swapfile setup
 fi

Folder merge function

The try_merge() function takes two arguments:

A path beginning with a / which has the same definition as for supporting_file().
A comment string summarising the purpose of the merge.

For example:

try_merge "/etc/network" "set up custom interfaces"

The merge algorithm invokes supporting_file() to see if the source path can be found. Calling supporting_file() implies both host-specific and general candidates will be considered, with the host-specific form given precedence.

supporting_file() will return successfully if the above path exists and is either a file or a non-empty directory. However, try_merge() then insists that both the source and target paths lead to directories. If both are directories then rsync is called to perform a non-overwriting merge. The result code returned by rsync becomes the result code returned by try_merge().

If any of the preliminary tests fail, rsync is not called and the result code is set to indicate failure.

The try_merge() function has two common use patterns:

unconditional invocation where there are no actions that depend on the success of the merge. For example:
```
 try_merge "/etc/network" "set up custom interfaces"
```

conditional invocation where subsequent actions depend on the success of the merge. For example:

 if try_merge "/etc/network" "set up custom interfaces" ; then
 	sudo service networking restart
 fi

Preparing your own patches

PiBuilder can apply patches for you, but you still need to create each patch.

Tools overview: diff and patch

Understanding how patching works will help you to develop and test patches before handing them to PiBuilder. Assume:

an «original» file (the original supplied as part of Raspbian); and
a «final» file (after your editing to make configuration changes).

To create a «patch» file, you use the diff tool which is part of Unix:

$ diff «original» «final» > «patch»

Subsequently, given:

a fresh Raspbian install where only «original» exists; plus
your «patch» file,

you use the patch tool which is also part of Unix:

$ patch -bfnz.bak -i «patch» «original»

That patch command will:

copy «original» to «original».bak; and
apply «patch» to «original» to convert it to «final».

Basic process

The basic process for creating a patch file for use in PiBuilder is:

Make sure you have a baseline version of the file you want to change. The baseline version of a «target» file should always be whatever was in the Raspbian image you downloaded from the web. Typically, there are two situations:
- You have run PiBuilder and PiBuilder has already applied a patch to the «target» file. In that case, «target».bak is a copy of whatever was in the Raspbian image you downloaded from the web. That means «target».bak is your baseline and you don't need to do anything else.
- The «target» file has never been changed. The currently-active file is your baseline so you need to preserve it by making a copy before you start changing anything. The most likely place where you will be working is the /etc directory so sudo is usually appropriate:
```
 $ sudo cp «target» «target».bak
```
Note:
- One of PiBuilder's first actions in the 01 script is to make a copy of /etc as /etc-baseline. PiBuilder does this before it makes any changes. If you make some changes in the /etc directory and only then realise that you forgot to save a baseline copy, you can always fetch a copy of the original file from /etc-baseline.
Make whatever changes you need to make to the «target». Sometimes this will involve using sudo and a text editor. Other times, you will be able to run a configuration tool like raspi-config and it will change the «target» file(s) for you.
Create a «patch» file using the diff tool. For any given patch file, you always have two options:
- If the patch file should apply to a specific Raspberry Pi, generate the patch file like this:
```
 $ diff «target».bak «target» > «target».patch@$HOSTNAME
```
- If the patch file should apply to all of your Raspberry Pis each time they are built, generate the patch file like this:
```
 $ diff «target».bak «target» > «target».patch
```
You can do both. A host-specific patch always takes precedence over a general patch.
Place the «patch» file in its proper location in the PiBuilder structure on your support host (Mac/PC).

For example, suppose you have prepared a patch that will be applied to the following file on your Raspberry Pi:
```
/etc/resolvconf.conf
```
Remove the file name, leaving the path component:
```
/etc
```
The path to the support folder in your PiBuilder structure on your support host is:
```
~/PiBuilder/boot/scripts/support
```
Append the path component ("/etc") to the path to the support folder:
```
~/PiBuilder/boot/scripts/support/etc
```
That folder is where your patch files should be placed. The patch file you prepared will have one of the following names:
```
resolvconf.conf.patch@«hostname»
resolvconf.conf.patch
```
The proper location for the patch file in the PiBuilder structure structure on your support host is one of the following paths:
```
~/PiBuilder/boot/scripts/support/etc/resolvconf.conf.patch@«hostname»
~/PiBuilder/boot/scripts/support/etc/resolvconf.conf.patch
```

Configure home directory

PiBuilder assumes «username» equals "pi". If you choose a different «username», you might need to take special care with the following folder and its contents:

~/PiBuilder/boot/scripts/support/home/pi/

This is the default structure:

└── home
    └── pi
        ├── .bashrc
        ├── .config
        │   ├── iotstack_backup
        │   │   └── config.yml
        │   └── rclone
        │       └── rclone.conf
        ├── .gitconfig
        ├── .gitignore_global
        └── crontab

Let's suppose that, instead of "pi", you decide to use "me" for your «username». What you might need to do is make a copy of the "pi" directory, as in:

$ cd ~/PiBuilder/boot/scripts/support/home
$ cp -a pi me

If you have followed the instructions about creating a custom branch to hold your changes, your next step would be:

$ git add me
$ git commit -m "clone default home directory structure"

Note:

This duplication is optional, not essential. If PiBuilder is not able to find a specific home folder for «username», it falls back to using "pi" as the source of files being copied into the /home/«username» folder on your Raspberry Pi.

`.bashrc`

The contents of this file are appended to the ~/.bashrc provided automatically by Raspberry Pi OS. The additions:

source IOTstackAliases;
enable DOCKER_BUILDKIT; and
define COMPOSE_PROFILES to be a synonym for HOSTNAME.

See also DOT_BASHRC_ACTION which explains how to instruct PiBuilder to replace your .bashrc with a fully custom file.

You can find more information about using compose profiles in this gist.

`.config/iotstack_backup/config.yml`

This is a placeholder. If you decide to set up IOTstackBackup then you should replace this placeholder with your working configuration.

`.config/rclone/rclone.conf`

This is a placeholder. If you decide to configure IOTstackBackup to use the RCLONE option (eg so your backups are stored in Dropbox), you should replace this placeholder with your working RCLONE configuration.

`.gitconfig`

This is (mostly) a template. At the very least, you should:

Replace "Your Name"; and
Replace "email@domain.com"

If you have not created a key for signing commits, remove the signingkey line, otherwise uncomment it and set the correct value.

Hint:

You may find it simpler to replace .gitconfig with whatever is in .gitconfig in your home directory on your support host.

You should only need to change .gitconfig in PiBuilder if you also change .gitconfig your home directory on your support host. Otherwise, the configuration can be re-used for all of your Raspberry Pis.

`.gitignore_global`

This file has a base set of ignore patterns. You can use it as-is or tailor it to your needs.

`crontab`

This is a placeholder containing comments on how to set up cron jobs. PiBuilder will use whatever you supply here to initialise your crontab.

Existing customisation points

DHCP client daemon

Patch file: /etc/dhcpcd.conf.patch
Note: not attempted if Network Manager is running.

The patch file supplied with PiBuilder adds the line:

allowinterfaces eth*,wlan*

Explicitly allowing interface participation in DHCP has the side-effect of excluding all other interfaces from DHCP participation. IOTstack uses this approach to prevent the virtual interfaces created by Docker from participating in host DHCP. If those interfaces are allowed to participate in DHCP, it can have the effect of freezing the Raspberry Pi as it comes up after a reboot. Docker assigns IP addresses to all virtual interfaces it creates so DHCP participation is not actually necessary.

You can also use this patch file to assign a static IP address to an interface. For example:

interface eth0
static ip_address=192.168.132.55/24
static routers=192.168.132.1

Note that this only works in systems where Network Manager is not running (ie Bullseye and earlier). See Network Manager customisation for an example of setting a static IP address.

Docker daemon

Source file: /etc/docker/daemon.json

If the source file (in general or host-specific form) exists in the support directory, it is copied into place. One useful thing you can do with this file is to limit the size of your logs:

{
  "log-driver": "local",
  "log-opts": {
    "max-size": "1m"
  }
}

System swap-file

Controlling variable: VM_SWAP
Edits file: /etc/dphys-swapfile.sed

The edits file supplied with PiBuilder sets the conditions such that the default for swap space is twice the amount of physical RAM, capped at a limit of 2GB. This will be 2GB for any Raspberry Pi with 1GB or more of real RAM. You can, however, change this arrangement to suit your needs, either by altering the supplied edits file (refer to the try_edit() function) or by providing a host-specific override.

If VM_SWAP is set to:

disable, no swapping occurs. This may be appropriate if your Raspberry Pi boots from SD and you want to avoid wear and tear on the card.
automatic:
- If the Pi is running from an SD card, this is the same as disable.
- Otherwise the patched version of /etc/dphys-swapfile is implemented. This is the recommended option if your Raspberry Pi boots from SSD or HD.
custom is equivalent to automatic but it does not check if your system is running from SD. If you want to enable swap on an SD system, this or default are the options to use.
default makes no changes to the virtual memory system. The current Raspberry Pi OS defaults enable virtual memory swapping with a swap file size of 100MB. This is perfectly workable on systems with 4GB of RAM or more.

If VM_SWAP is not set, it defaults to automatic.

Running out of RAM causes swapping to occur and that, in turn, has both a performance penalty (because SD cards are quite slow) and increases the wear and tear on the SD card (leading to a heightened risk of failure). There are two main causes of limited RAM:

Insufficient physical memory. A good example is a Raspberry Pi Zero W2 which only has 512MB to start with; and/or
Expecting your Raspberry Pi to do too much work, such as running a significant number of containers which either have large memory footprints, or cause a lot of I/O and consume cache buffers, or both.

If you disable VM swapping by setting VM_SWAP to disable, but you later decide to re-enable swapping, run these commands:

$ sudo systemctl enable dphys-swapfile.service
$ sudo reboot

You can always check if swapping is enabled using the swapon -s command. Silence means swapping is disabled.

It is important to appreciate that VM swapping is not bad. Please don't disable swapping without giving it some thought. If you can afford to add an SSD, you'll get a better result with swapping enabled than if you stick with the SD and disable swapping.

`/etc/dphys-swapfile` deprecated

Previously, /etc/dphys-swapfile was edited via a patch file. The Raspberry Pi Foundation changed the contents of the default file such that patching (using the Unix patch command) became less reliable that editing (using the Unix sed command). If PiBuilder senses a patch file, it will display a deprecation notice and force VM_SWAP=default which means "no change from OS defaults".

GRUB

Configuration directory: /etc/default/grub.d

Raspberry Pi OS does not use GRUB so you should ignore this section if you are using PiBuilder on a Raspberry Pi.

However, GRUB (Grand Unified Bootloader) is common in other environments such as Debian native or Debian-in-Proxmox. In such cases, the contents of the PiBuilder configuration directory are merged with its equivalent on the system under construction, and then update-grub is invoked.

Locales

Configuration file: /etc/locale.conf

Example:

 [enable]
 en_AU ISO-8859-1
 en_AU.UTF-8 UTF-8
 en_US.UTF-8 UTF-8

locale.conf has a simple syntax and the script that parses it has no sanity checking so you would be unwise to push it too far. Rules:

Everything should be left-aligned.
Lines starting with a hash are treated as comments.
Blank lines are ignored.
Two "directives" are supported ([enable] and [disable]). Everything else is considered to be a locale name.
The script assumes "[enable]" mode on entry so a simple list of locales will be treated as if [enable] was present.
No existing locale name in /etc/locale.gen contains a slash (/). The script relies on that when generating sed commands and will misbehave if this rule is broken.
Given a locale to be activated, a sed command is generated to replace an inactive form with an active form but if and only if the locale is not already active.
Given a locale to be deactivated, a sed command is generated to replace all active forms of the locale with inactive forms.
Nothing happens if a locale does not actually exist in /etc/locale.gen. In other words, this mechanism can't be used to add new locales.

Notes:

Do not deactivate "en_GB.UTF-8" if you are running on a Raspberry Pi. If you really want to remove that locale then you should use raspi-config after PiBuilder has finished. This is not an issue on native Debian installs.
If LOCALE_LANG is defined and contains a value which is active after /etc/locale.gen has been modified, then that will be made the active locale.

Deprecated mechanism

Patch file: /etc/locale.gen.patch

If a patch file is present, PiBuilder will attempt to apply it but will issue a deprecation warning.

Patch files are reasonably safe providing you are using a single platform (eg Raspberry Pi) and a single OS build (eg Bookworm) but the locale.conf mechanism should prove more reliable in the long term.

Network interfaces

Configuration directory: /etc/network

PiBuilder does not include a default directory. If you supply a general or host-specific directory, its contents will be merged with /etc/network. Network definitions are almost always highly host-specific so you should probably think in those terms.

Network interface monitoring

NetworkManager already takes care of keeping interfaces alive so the mechanism discussed in this section is not installed on systems where NetworkManager is running.

See Do your Raspberry Pi's Network Interfaces freeze? for the background to this.

Patch file: /etc/rc.local.patch
Support script: /usr/bin/isc-dhcp-fix.sh

Several preconditions need to be met before this mechanism will be installed:

NetworkManager must be inactive.
/etc/rc.local must be world-executable and have non-zero length. Debian and Ubuntu typically create an empty rc.local and without execute permission.
PiBuilder/boot/scripts/support/usr/bin/usr/bin/isc-dhcp-fix.sh (or a host-specific version) must exist. It exists in the PiBuilder release but might be removed in customised versions.

If the preconditions are met:

isc-dhcp-fix.sh is copied into place in /usr/bin; then
If /etc/rc.local.patch is found, it is used to patch /etc/rc.local. The default patch adds this line:
```
# /usr/bin/isc-dhcp-fix.sh &
```
If that inactive line is found in the patched /etc/rc.local then PiBuilder checks eth0 and wlan0, adds each active interface to the command, and removes the comment mark. For example, if both interfaces are active, the result will be:
```
/usr/bin/isc-dhcp-fix.sh eth0 wlan0 &
```
If neither interface exists (which may well be the case on non-Raspberry Pi systems), the comment is left in place.

If you don't want any of this to happen, you can either remove /usr/bin/isc-dhcp-fix.sh (or replace it with a do-nothing script) or remove the line added by the patch in step 2.

Network Manager customisation

Two hook points are provided for customising network manager:

/etc/NetworkManager/dispatcher.d - if this folder exists then its contents are merged with the corresponding folder in the system under construction. Example:
```
 $ cat PiBuilder/boot/scripts/support/etc/NetworkManager/dispatcher.d/00-sysctl
```
```
 #!/bin/sh

 # refer https://bbs.archlinux.org/viewtopic.php?id=282819
 # (path to sysctl amended)
 # this file should be owned root:root with mode 755

 /usr/sbin/sysctl --system

 exit 0
```
If present, the effect of this file is to enforce options set in /etc/sysctl.conf and /etc/sysctl.d after each NetworkManager configuration change. Numerous web recipes mention these files so it is useful for the two ecosystems to coexist.
/etc/NetworkManager/custom_settings.sh - if this file exists, it is executed. Here is an example of how to set a static IP address. First, start with a shebang:
```
 #!/usr/bin/env bash
```
Next:
- if you know the connection name, define it:
```
 CONN="Wired connection 1"
```
- alternatively, if you only know the interface name, you can ask Network Manager to lookup the corresponding connection name:
```
 PHY=eth0
 CONN=$(nmcli -g GENERAL.CONNECTION dev show "$PHY" 2>/dev/null)
```
Then, set the static IP address on the connection:
```
 STATIC="203.0.132.100/24"
 GATEWAY="203.0.132.1"
 if [ -n "$CONN" ] ; then
    sudo nmcli con mod "$CONN" \
       ipv4.addresses "$STATIC" \
       ipv4.gateway "$GATEWAY" \
       ipv4.method "manual"
    echo "Note: $PHY->$CONN set to static IP address $STATIC"
 else
    echo "Warning: Unable to set static IP address $STATIC for $PHY"
 fi
```
Remember to give the script execute permission:
```
 $ chmod +x custom_settings.sh
```
PiBuilder will apply the changes when the 02 script runs, and the changes will take effect on the reboot at the end of the 02 script.

DNS resolver

Patch file: /etc/resolvconf.conf.patch

There is no default patch. If you supply a general or host-specific patch file, you can achieve things like:

Add a default search domain:
```
search_domains=my.domain.com
```
Tell a host to use itself for DNS resolution (eg running BIND9 or PiHole), with a fallback to Google:
```
name_servers="127.0.0.1 8.8.8.8"
resolv_conf_local_only=NO
```

Samba (SMB)

Configuration file: /etc/samba/smb.conf

PiBuilder does not include a default configuration file for SAMBA. If you provide a general or host-specific configuration file then PiBuilder will install and activate SAMBA for you.

Secure Shell (SSH)

Zipped replacement directory: /etc/ssh/etc-ssh-backup.tar.gz@$HOSTNAME

If the .gz is found, it is unpacked and the contents used to replace /etc/ssh. This lets you preserve a host's SSH identity across builds. It is particularly useful if you use SSH certificates. See also Some words about SSH.

Kernel parameters

Merge folder: /etc/sysctl.d (new method, recommended)

The recommended method is files (not patches) placed in /etc/sysctl.d. The default supplied with PiBuilder contains instructions to disable IPv6. You can either add to that file or supply additional .conf files of your own.

Journal control

Patch file: /etc/systemd/journald.conf.patch

The default patch file changes the system logging level to reduce endless docker-runtime mount messages.

Time synchronisation

Patch file: /etc/systemd/timesyncd.conf.patch

There is no default patch. If you supply a general or host-specific patch, it can be used to set up a more geographically-appropriate source from which your Raspberry Pi can obtain its time.

For more information, see Network Time Protocol - setting your closest servers.

Dynamic device management (UDEV)

Configuration directory: /etc/udev/rules.d

PiBuilder provides an empty rules.d folder. If you place any UDEV rules files in this folder, or if you provide a host-specific folder, the contents of the folder will be copied onto the target system.

The copy is done without replacement. In other words, if a rule file of the same name already exists on the target system, it won't be replaced with the version from PiBuilder.

Using your custom branch in a build

When you want to use your customised version of PiBuildet, instead of cloning PiBuilder from GitHub, clone your customised version from your support host. The basic syntax is:

$ git clone -b «branch» «user»@«host»:«remotePath» ~/PiBuilder

Here's an example. Assume:

The «branch» you are using in PiBuilder to hold your changes is called "custom".
Your «user» name on your support host is "edmund".
Your «support» host is named "everest" and can be reached via:
- The IP address 192.168.1.100 ; or
- The multicast DNS (mDNS) name "everest.local" ; or
- The fully-qualified domain name (FQDN) "everest.my.domain.com"
The PiBuilder directory on "everest" is located in Edmund's home directory.

Any of the following commands should work:

via IP address:

 $ git clone -b custom edmund@192.168.1.100:PiBuilder ~/PiBuilder

via mDNS name:

 $ git clone -b custom edmund@everest.local:PiBuilder ~/PiBuilder

via FQDN:

 $ git clone -b custom edmund@everest.my.domain.com:PiBuilder ~/PiBuilder

In each case:

:PiBuilder is interpreted as relative to Edmund's home directory on "everest". Alternatives:
- In a sub-folder of Edmund's home directory: :path/to/PiBuilder; or
- An absolute path on "everest": :/path/to/PiBuilder.
~/PiBuilder is the path on the local host (ie the Raspberry Pi) where the clone will be placed.

Notes:

SSH will probably present a TOFU (Trust On First Use) challenge; and then
Ask for Edmund's password on "everest".

Original build method still works

The original PiBuilder build method still works on the Raspberry Pi but there are differences depending on whether you are installing Raspberry Pi OS Bullseye (or earlier), or Raspberry Pi OS Bookworm.

The steps are:

Image your media (SD/SSD). Although you can change the default, Raspberry Pi Imager normally ejects the media at the end of the process.
Mount the boot partition on your support host. This can be as simple as physically removing and re-connecting the media and waiting for the operating system on your support host to mount the media.
Identify the name of the boot partition. If you are building a system based on:
- Bullseye (or earlier), the boot partition has the name "boot".
- Bookworm, the boot partition has the name "bootfs".
Copy the contents of the PiBuilder boot directory to the boot partition. If your support host is:
- macOS, you can perform the copying operation by running:
```
 $ ./setup_boot_volume.sh
```
  On macOS, the script detects whether /Volumes/boot or /Volumes/bootfs has mounted and adapts accordingly.
- Linux, you will need to pass the correct path to the boot partition. Example:
```
 $ ./setup_boot_volume.sh path/to/boot-or-bootfs-partition
```
- Windows, the setup_boot_volume.sh script will not run. You need to copy the contents of the boot directory to the drive where the boot partition has mounted.
Move the media to your Raspberry Pi and apply power.
Connect to your Pi via SSH and run the scripts. If you are building a system based on:
- Bullseye (or earlier), you can run the first script like this:
```
 $ /boot/scripts/01_setup.sh «newHostName»
```
- Bookworm, you can run the first script like this:
```
 $ /boot/firmware/scripts/01_setup.sh «newHostName»
```

You can use this older method with either a clean clone of PiBuilder from GitHub or with a local repository containing your own customisations.

The reason why the PiBuilder documentation now focuses on the newer method is because it will also work in situations where the boot partition does not exist (or you can't get to it easily), such as Proxmox VE, or starting with a Debian install on non-Pi hardware, or starting with a non-Raspberry Pi OS on Raspberry Pi hardware.

Keeping in sync with GitHub

The instructions in Getting Started recommended that you create a Git branch ("custom") to hold your customisations. If you did not do that, please do so now:

$ cd ~/PiBuilder
$ git checkout -b custom

Notes:

any changes you may have made before creating the "custom" branch will become part of the "custom" branch. You won't lose anything. After you "add" and "commit" your changes on the "custom" branch, the "master" branch will be a faithful copy of the PiBuilder repository on GitHub at the moment you first cloned it.
once the "custom" branch becomes your working branch, there should be no need to switch branches inside the PiBuilder repository. The instructions in this section assume you are always in the "custom" branch.

From time to time as you make changes, you should run:

$ git status

Add any new or modified files or folders using:

$ git add «path»

Note:

You can't add an empty folder to a Git repository. A folder must contain at least one file before Git will consider it for inclusion.

Whenever you reach a logical milestone, commit your changes:

$ get commit -m "added a patch for something or other"

naturally, you will want to use a far more informative commit message!

Periodically, you will want to check for updates to PiBuilder on GitHub:

$ git fetch origin master:master

That pulls changes into the master branch. Next, you will want to merge those changes into your "custom" branch:

$ git merge master --no-commit

If the merge:

succeeds, you will see:

 Automatic merge went well; stopped before committing as requested

is blocked before it completes, you will see one or more messages like this:
```
 CONFLICT (content): Merge conflict in «filename»
```
That tells you that the problem is in «filename». For each file mentioned in such a message:
1. Open the file using your favourite text editor.
2. Search for <<<<<<<. You are looking for a pattern like this:
```
<<<<<<< HEAD
one or more lines of your own text
=======
one or more lines of text coming from PiBuilder on GitHub
>>>>>>> master
```
3. To resolve the conflict, you just need to decide what the file should look like and remove the conflict markers:
  - If you want to preserve your own text and discard the PiBuilder lines, reduce the above to just:
```
 one or more lines of your own text
```
  - If you want the lines coming from the PiBuilder to replace your own, reduce the above to just:
```
 one or more lines of text coming from PiBuilder on GitHub
```
  - If you want to preserve material from both:
```
 one or more lines of your own text
 one or more lines of text coming from PiBuilder on GitHub
```
    or:
```
 one or more lines of my own text merged with one or more lines from GitHub
```
4. Don't forget that a file may have more than one area of conflict so go back to step 2 and repeat the search until you are sure all the conflicts have been found and resolved.
5. Once you are sure you have resolved all of the conflicts in a file, tell git by:
```
$ git add «filename»
```
6. If more than one file was marked as being in conflict, start over from step 1. You can always refresh your memory on which files are still in conflict by:
```
$ git status

…
Changes to be committed:
	modified:   file1.txt

Unmerged paths:
	both modified:   file2.txt
…
```
  In the above, file1.txt is no longer in conflict but file2.txt still needs to be checked.

It does not matter whether the merge succeeded immediately or if it was blocked and you had to resolve conflicts, the next step is to run:

$ git status

For each file mentioned in the status list that is not in the "Changes to be committed" list, run:

$ git add «filename»

The last step is to commit the merged changes to your own branch:

$ git commit -m "merged with GitHub updates"

Now you are in sync with GitHub.

Files

README-ADVANCED.md

Latest commit

History

README-ADVANCED.md

File metadata and controls

Advanced PiBuilder

Contents

Overview

Getting started

PiBuilder installation options

Per-host installation options

Environment variable overrides

about Git options

Script scaffolding

The PiBuilder Patching System

How PiBuilder scripts search for files, folders and patches

Search function

Patch function

File edit function

Folder merge function

Preparing your own patches

Tools overview: diff and patch

Basic process

Configure home directory

.bashrc

.config/iotstack_backup/config.yml

.config/rclone/rclone.conf

.gitconfig

.gitignore_global

crontab

Existing customisation points

DHCP client daemon

Docker daemon

System swap-file

/etc/dphys-swapfile deprecated

GRUB

Locales

Deprecated mechanism

Network interfaces

Network interface monitoring

Network Manager customisation

DNS resolver

Samba (SMB)

Secure Shell (SSH)

Kernel parameters

Journal control

Time synchronisation

Dynamic device management (UDEV)

Using your custom branch in a build

Original build method still works

Keeping in sync with GitHub

`.bashrc`

`.config/iotstack_backup/config.yml`

`.config/rclone/rclone.conf`

`.gitconfig`

`.gitignore_global`

`crontab`

`/etc/dphys-swapfile` deprecated