Customizing RHCOS with CoreOS Layering to Add the `rsyslog` Package

Introduction

Red Hat Enterprise Linux CoreOS (RHCOS) is the immutable, container-optimized operating system that powers OpenShift Container Platform nodes. While its immutability provides stability and security, there are scenarios where you might need to add additional packages or custom configurations. OpenShift’s CoreOS Layering feature allows you to extend the base RHCOS image by adding RPM packages, creating a customized OS layer.

This guide provides a step-by-step walkthrough of how to use CoreOS layering to build a custom RHCOS image that includes the rsyslog package. This is useful for environments that require traditional log forwarding mechanisms or need to integrate with existing logging infrastructure that relies on syslog. We will also install plymouth to ensure boot-time logs are captured correctly.

For more details, refer to the official OpenShift documentation on CoreOS layering.

Prerequisites

Before you begin, ensure you have the following:

A running OpenShift Container Platform 4.18 cluster.
Administrative access to the cluster with the oc CLI tool.
A Red Hat Enterprise Linux (RHEL) 9 host to build the custom image. The RHEL version must match the RHCOS version.
Podman installed on the RHEL build host.
A container registry (like Quay.io, Harbor, or Docker Hub) to store the custom image.
A pull secret for your container registry.

Step 1: Identify the Base RHCOS Image

First, you need to determine the exact base RHCOS image used by your cluster. This ensures your custom image is built on the correct foundation.

Execute the following command to get the image reference:


        # Get the current RHCOS image digest from your cluster
        
        oc adm release info --image-for rhel-coreos
        
        # Example Output:
        
        # quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cbe2965ad3a408286fe06e4a7a58e006621f33847c17f8e8ff84504dbeebe666

Take note of the full image URL with the SHA256 digest. You will use this as the FROM image in your Dockerfile.

Step 2: Build the Custom RHCOS Image on a RHEL Host

Next, prepare a build environment on your RHEL host to create the layered image.

1. Prepare the Build Directory and RPMs

Create a working directory and download the necessary RPM packages and their dependencies. We need rsyslog for logging, plymouth to help write logs to /var/log/boot.log during startup, and other required libraries.


        # Create a working directory
        
        mkdir -p /data/rhcos-build
        cd /data/rhcos-build
        
        # Download rsyslog, plymouth, and their dependencies
        
        dnf download --resolve --destdir=. rsyslog libestr libfastjson logrotate plymouth plymouth-core-libs plymouth-scripts

2. Create the Dockerfile

Create a Dockerfile (or Containerfile) to define the build process for your custom image. This file will start from the base RHCOS image, copy the downloaded RPMs, install them, and then commit the changes to the OS layer.


        # Define the base image using the output from Step 1
        
        RHCOS_IMAGE="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cbe2965ad3a408286fe06e4a7a58e006621f33847c17f8e8ff84504dbeebe666"
        
        # Create the Dockerfile
        
        tee rhcos.dockerfile << EOF
        FROM ${RHCOS_IMAGE}
        
        # Copy the downloaded RPMs into the container
        
        COPY *.rpm /root/
        
        # Install the RPMs using dnf and commit the layer to ostree
        
        RUN cd /root && \
            dnf install --nobest -y ./*.x86_64.rpm && \
            ostree container commit
        
        # Clean up the RPM files
        
        RUN rm -f /root/*.rpm
        
        # Optional: Add custom configurations
        
        # For example, to add a custom rsyslog configuration file:
        
        # ADD remote.conf /etc/rsyslog.d/remote.conf
        
        EOF

3. Build and Push the Image

Now, build the image using podman and push it to your container registry. Make sure you are logged into your registry and have your pull secret available.


        # Define your custom image tag
        
        CUSTOM_IMAGE="quay.io/wangzheng422/qimgs:rhcos-4.18-rsyslog-2025.09.03-v02"
        
        # Build the image
        
        # Replace /path/to/pull-secret.json with the actual path to your pull secret
        
        podman build --authfile /path/to/pull-secret.json -t ${CUSTOM_IMAGE} -f rhcos.dockerfile .
        
        # Push the image to your registry
        
        podman push ${CUSTOM_IMAGE}

Step 3: Apply the Custom Image to the Cluster

With the custom image available in your registry, you can now create a MachineConfig to apply it to your OpenShift nodes. This example targets the master nodes.

This MachineConfig does two things:

It points the nodes to the new custom osImageURL.
It creates a configuration file at /etc/tmpfiles.d/rsyslog.conf. This file instructs systemd-tmpfiles to create the /var/lib/rsyslog directory on boot, which is required for rsyslog to function correctly.


        # Create the MachineConfig manifest
        
        tee machine-config-rsyslog.yaml << 'EOF'
        ---
        apiVersion: machineconfiguration.openshift.io/v1
        kind: MachineConfig
        metadata:
          labels:
            machineconfiguration.openshift.io/role: master
          name: 99-master-layered-rhcos
        spec:
          config:
            ignition:
              version: 3.4.0
            storage:
              files:
                # This section defines a file to be created on the node.
                # It ensures the rsyslog working directory exists.
                - path: /etc/tmpfiles.d/rsyslog.conf
                  mode: 0644
                  overwrite: true
                  contents:
                    # The content is base64 encoded.
                    # The original text is: "d /var/lib/rsyslog 0755 root root -"
                    # This line instructs systemd-tmpfiles to create the directory.
                    source: "data:text/plain;charset=utf-8;base64,ZCAvdmFyL2xpYi9yc3lzbG9nIDA3NTUgcm9vdCByb290IC0K"
                - path: /etc/logrotate.d/rsyslog
                  mode: 0644
                  overwrite: true
                  contents:
                    # content of this file
                    # /var/log/cron
                    # /var/log/maillog
                    # /var/log/messages
                    # /var/log/secure
                    # /var/log/spooler
                    # {
                    #     missingok
                    #     sharedscripts
                    #     postrotate
                    #         /usr/bin/systemctl -s HUP kill rsyslog.service >/dev/null 2>&1 || true
                    #     endscript
                    # }
                    source: "data:text/plain;charset=utf-8;base64,L3Zhci9sb2cvY3JvbgovdmFyL2xvZy9tYWlsbG9nCi92YXIvbG9nL21lc3NhZ2VzCi92YXIvbG9nL3NlY3VyZQovdmFyL2xvZy9zcG9vbGVyCnsKICAgIG1pc3NpbmdvawogICAgc2hhcmVkc2NyaXB0cwogICAgcG9zdHJvdGF0ZQogICAgICAgIC91c3IvYmluL3N5c3RlbWN0bCAtcyBIVVAga2lsbCByc3lzbG9nLnNlcnZpY2UgPi9kZXYvbnVsbCAyPiYxIHx8IHRydWUKICAgIGVuZHNjcmlwdAp9Cg=="
                - path: /etc/rsyslog.d/remote.conf
                  mode: 0644
                  overwrite: true
                  contents:
                    # content of this file
                    # if $programname == 'crio' or $programname == 'kubenswrapper' then {
                    #     stop
                    # }
                    source: "data:text/plain;charset=utf-8;base64,aWYgJHByb2dyYW1uYW1lID09ICdjcmlvJyBvciAkcHJvZ3JhbW5hbWUgPT0gJ2t1YmVuc3dyYXBwZXInIHRoZW4gewogICAgc3RvcAp9Cg=="
          # Set the osImageURL to your custom-built image
          osImageURL: quay.io/wangzheng422/qimgs:rhcos-4.18-rsyslog-2025.09.03-v02
        EOF
        
        # Apply the MachineConfig to the cluster
        
        oc apply -f machine-config-rsyslog.yaml

After applying the MachineConfig, the Machine Config Operator (MCO) will detect the change and begin a rolling update of the master nodes to apply the new configuration and boot from the custom OS image.

Step 4: Verify the Changes

Once the nodes have been updated, you can verify that rsyslog is running and generating logs.

Check Node Status: Monitor the nodes to ensure they reboot successfully and return to a Ready state. bash oc get nodes
Inspect a Node: SSH into one of the updated master nodes. bash oc debug node/<node-name>

Check for Log Files: List the contents of /var/log. With rsyslog running, you should see classic log files like messages, secure, and boot.log.

# Inside the debug pod on the node
        chroot /host
        ls -l /var/log

You should see an output similar to this, confirming that the log files are being created:

total 53604
        lrwxrwxrwx.  1 root        root                  39 Sep  1 04:58 README -> ../../usr/share/doc/systemd/README.logs
        drwx------.  2 root        root                  23 Sep  1 04:58 audit
        -rw-------.  1 root        root                8055 Sep  3 13:20 boot.log
        -rw-rw----.  1 root        utmp                   0 Sep  1 04:58 btmp
        ...
        -rw-------.  1 root        root            24620123 Sep  3 13:28 messages
        ...
        -rw-------.  1 root        root                5169 Sep  3 13:20 secure
        ...

4. Verify System Services and OS Layer

After the node has rebooted, it’s crucial to verify that the new OS layer is active and the rsyslog service is running as expected.

a. Check the `rpm-ostree` Status

The rpm-ostree status command confirms which OS deployment is currently active. The output should show that the system is booted into your custom-layered image.


        # Verify the active OS deployment
        
        rpm-ostree status

Example Output Analysis:

State: idle
        Deployments:
        ● ostree-unverified-registry:quay.io/wangzheng422/qimgs:rhcos-4.18-rsyslog-2025.09.03-v02
                           Digest: sha256:51221e7c01b3bb77cb53d5cc77eb7a089e900b2d0186454a9964b04b11d6c491
                          Version: 418.94.202508060022-0 (2025-09-03T13:12:38Z)

The ● symbol indicates that the currently running deployment is the one from your custom image (quay.io/wangzheng422/qimgs:rhcos-4.18-rsyslog-2025.09.03-v02).
The State: idle confirms that there are no ongoing rpm-ostree operations.

b. Check the `rsyslog` Service Status

Next, verify that the rsyslog service was started automatically and is running without errors.


        # Check the status of the rsyslog service
        
        systemctl status rsyslog

Example Output Analysis:

● rsyslog.service - System Logging Service
             Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; preset: enabled)
             Active: active (running) since Wed 2025-09-03 14:20:16 UTC; 34min ago
        ...
           Main PID: 2034 (rsyslogd)
        ...
        Sep 03 14:20:16 master-03-demo systemd[1]: Starting System Logging Service...
        Sep 03 14:20:16 master-03-demo rsyslogd[2034]: [origin software="rsyslogd" swVersion="8.2412.0-1.el9" x-pid="2034" x-info="https://www.rsyslog.com"] start
        Sep 03 14:20:16 master-03-demo systemd[1]: Started System Logging Service.
        Sep 03 14:20:17 master-03-demo rsyslogd[2034]: message too long (8461) with configured size 8096, begin of message is: time="2025-09-03 13:49:49.187181490Z" level=info msg="Current CRI-O c>
        Sep 03 14:20:18 master-03-demo rsyslogd[2034]: imjournal from <master-03-demo:kubenswrapper>: begin to drop messages due to rate-limiting
        Sep 03 14:20:18 master-03-demo rsyslogd[2034]: imjournal: journal files changed, reloading...  [v8.2412.0-1.el9 try https://www.rsyslog.com/e/0 ]
        Sep 03 14:30:21 master-03-demo rsyslogd[2034]: imjournal: 20204 messages lost due to rate-limiting (20000 allowed within 600 seconds)

Active: active (running) confirms the service is operational.
The log entries show the service starting successfully.
Important: The output also reveals potential issues that may need attention in a production environment:
- message too long (8461) with configured size 8096: This indicates that some log messages from CRI-O are larger than the default rsyslog message size limit. You may need to adjust the $MaxMessageSize setting in /etc/rsyslog.conf if you need to capture these large messages in their entirety.
- imjournal: ... messages lost due to rate-limiting: This warning means that the journal is producing logs faster than rsyslog is configured to process them, causing some messages to be dropped. To mitigate this, you can adjust the rate-limiting settings in /etc/rsyslog.conf (e.g., by increasing $imjournalRatelimit.Burst and $imjournalRatelimit.Interval).

This level of verification ensures not only that the package is installed but also that the service is functioning correctly and is configured appropriately for the cluster’s workload.

Conclusion

You have successfully customized an RHCOS image by layering the rsyslog package, pushed it to a container registry, and applied it to your OpenShift cluster using a MachineConfig. This process demonstrates the power of CoreOS layering for extending the functionality of OpenShift nodes while maintaining the core principles of an immutable infrastructure.

Customizing RHCOS with CoreOS Layering to Add the rsyslog Package

Introduction

Prerequisites

Step 1: Identify the Base RHCOS Image

Step 2: Build the Custom RHCOS Image on a RHEL Host

1. Prepare the Build Directory and RPMs

2. Create the Dockerfile

3. Build and Push the Image

Step 3: Apply the Custom Image to the Cluster

Step 4: Verify the Changes

4. Verify System Services and OS Layer

a. Check the rpm-ostree Status

b. Check the rsyslog Service Status

Conclusion

Customizing RHCOS with CoreOS Layering to Add the `rsyslog` Package

a. Check the `rpm-ostree` Status

b. Check the `rsyslog` Service Status