State Management

In addition to robust OTA updates, Rugix Ctrl also provides state management functionality.

Managing and persisting system state can be quite challenging. In particular, if state needs to persist through updates. Rugix Ctrl provides a robust state management mechanism, simplifying state management and improving a system’s resiliency.

✨ Feature Highlights ✨

✅ Straightforward selective persistence of system state through updates.

✅ Protection against accidental state and system partition corruption.
✅ Off-the-shelf factory reset functionality.

Rugix Ctrl’s state management mechanism has been inspired by Docker. The basic idea is to construct a virtual writable root filesystem by layering a writeable overlay on top of a read-only system partition. Instead of modifying the system partition, any writes will go to the overlay stored on the data partition. To persist state, certain files or directories can be explicitly bind mounted from the data partition, thereby ensuring that their contents persists through updates. The overlay itself is typically ephemeral and will be discarded whenever the system boots. This ensures that no accidental state survives a reboot. The following diagram visualizes the idea:

           ╭─────────────────────────────────────────────────────────────╮
           │                  Writable Root Filesystem                   │
           ╰─────────────────────────────────────────────────────────────╯
                          ▲                               ▲
                          │ reads                         │ reads/writes
                          │                               ▼
           ╭──────────────────────────────╮  ╭───────────────────────────╮
Storage    │  Read-Only System Partition  │  │  Writable Data Partition  │
           ╰──────────────────────────────╯  ╰───────────────────────────╯

If you are familiar with Docker, you may view your system as running in an ephemeral low-level container¹ using bind mounts from the data partition to persist the state of your system, such as your database, application data, and user settings.

This approach has several advantages:

• It simplifies system development by providing a writable root filesystem. While you can run Linux on a read-only root filesystem, this usually comes with some caveats as some tools assume directories such as /etc to be writeable. Also, at least /var needs to be writable and thus needs to live on a separate partition anyway. By using a writable overlay, you do not have to worry about any of these issues and everything should work as it normally would.
• As the system partition is read-only, it will stay in a known-good, well-defined state. This also adds an extra layer of protection against power outages or bugs that could otherwise corrupt the system. You will get a truly immutable system ensuring that your devices run a well-defined, consistent software stack, always and everywhere.
• Discarding the overlay on each boot means that only explicitly declared state is selectively preserved. This prevents issues with accidental state. After a reboot you can be sure that any modifications to the root filesystem, that are not meant to persist and that may have caused problems, have been discarded. In addition, it reduces the risk for data loss during an update, as you will notice very early when the state that would be preserved by an update is insufficient.
• Controlling the state of the system enables Rugix Ctrl to provide off-the-shelf state management functionality such as factory resets. Instead of wasting your engineers’ time to implement such functionality, you can rely on Rugix Ctrl.

Rugix Ctrl’s state management mechanism is configurable. You can also configure it to keep the overlay, to put the overlay into main memory instead of on the data partition, or to disable the writable overlay altogether falling back to an actual read-only root filesystem and only setting up bind mounts for the files and directories you want to persist on the data partition.

Selective Persistent State

As explain above, state that should persist through updates (and reboots) must be explicitly declared and is then managed by Rugix Ctrl. Rugix Ctrl can be configured to use bind mounts to persist certain directories and files of the root filesystem on the data partition. To this end, state configuration files must be placed in /etc/rugix/state.

For instance, to persist the home directory of the root user the following state configuration file may be added:

#:schema https://raw.githubusercontent.com/rugix/rugix/refs/tags/v0.8.0/schemas/rugix-ctrl-state.schema.json

[[persist]]
directory = "/root"

This instructs Rugix Ctrl to bind mount /root to a persistent and writable directory on the data partition, thereby persisting the entire home directory of the root user. The directory on the data partition is automatically determined based on the path in the system. In this case, it would be /var/rugix/state/persist/root. While the state directory /var/rugix/state persists through updates and reboots, this directory is managed by Rugix Ctrl. Your application can also use /var/rugix/state/app to directly store any data as part of Rugix Ctrl’s managed state, without requiring an additional bind mount.

Note that you can put multiple [[persist]] sections into a single file and also use a section with file = "/path/to/file" to persist a file instead of a directory. You will find the full schema for these configuration files below.

Factory Reset

As the state is managed by Rugix Ctrl, a factory reset is simply done with:

rugix-ctrl state reset

This command will reboot the system and throw away any state replacing it with factory defaults. These factory defaults are taken from the system partition. Persisted directories and files are initially copied from the system partition, if they exist.

Overlay Configuration

By default, the overlay is discarded to prevent accidental state from corrupting the system and to give you an early indication that not all necessary state is declared. You can, however, make the overlay persistent with the following configuration:

#:schema https://raw.githubusercontent.com/rugix/rugix/refs/tags/v0.8.0/schemas/rugix-ctrl-state.schema.json

overlay = "persist"

Note that a persistent overlay that may exist when an update is installed to a system is deleted prior to installing the update. To avoid the overlay from being discarded, use the --keep-overlay option when installing the update with rugix-ctrl update. Please be aware that this may lead to incompatibilities between the overlay and the freshly installed system.

For development or debugging purposes, you can also force persistency of the overlay at runtime. To this end, run:

rugix-ctrl state overlay force-persist true

To disable force persistency of the overlay, run:

rugix-ctrl state overlay force-persist false

Note that this will discard the overlay and thereby any modifications which have been made with persistency set to true.

In addition to persisting the overlay, the overlay option can also be set to the following values:

• discard: Discards the overlay on each boot (default setting).
• in-memory: Puts the overlay in the main memory instead of on the data partition.
• disabled: Disables the overlay completely.

Disabling State Management

If you do not want to use the state management feature, do not configure your system to run rugix-ctrl as the init system. To disable only the writable overlay, use the disabled setting as described above.

Configuration Reference

For reference, here is the complete schema for state configuration files:

You will find the most recent version of this schema on GitHub.