Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-59551

Privileged RHEL 10 UBI Pod with hostNetwork Fails to Bind Socket Due to 'Permission Denied' on MicroShift

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • 4
    • None
    • None
    • None
    • None
    • uShift Sprint 275, uShift Sprint 276
    • 2
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      When attempting to deploy a Kea DHCP server, running inside a RHEL 10 UBI container, the application fails to bind to a privileged port (67) on a host network interface. The process is being blocked by a "Permission denied" error from the kernel. This failure occurs despite the Pod being configured with hostNetwork: true, having the privileged Security Context Constraint (SCC) granted to its service account, and with the host node's SELinux being fully disabled. The issue points to a low-level security policy in the MicroShift host environment that is preventing even a fully privileged container from performing a necessary network operation.    

      Version-Release number of selected component (if applicable):

      RHEL 9.4 
      MicroShift 4.18.11
      KEA DHCP 2.6.3 on UBI10    

      How reproducible:

      Always. The failure occurs consistently on every pod startup attempt. 

      Steps to Reproduce:

      1.  Prepare a MicroShift Node:**

            * Ensure a node has a network interface with a static IP (e.g., `enp7s0` with IP `192.168.200.1`).
            * (just for verification proposes) On the MicroShift host, edit `/etc/selinux/config` to set `SELINUX=disabled` and *reboot the node* to ensure the policy is fully unloaded. Verify with `sestatus`.

      2.  Build the Kea DHCP Container Image:**

            * Create a `Containerfile` with the following content to build the RHEL 10 UBI-based image.
              ```dockerfile
              # Use Red Hat Universal Base Image 10
              FROM registry.access.redhat.com/ubi10/ubi:latest

              # Install Kea DHCP and necessary tools
              RUN dnf install -y kea kea-ctrl-agent procps-ng net-tools iproute && \
                  dnf clean all

              # Expose DHCP and API ports
              EXPOSE 67/udp
              EXPOSE 8000/tcp

              # Create necessary directories
              RUN mkdir -p /run/kea && \
                  chown kea:kea /run/kea && \
                  mkdir -p /var/lib/kea && \
                  chown kea:kea /var/lib/kea

              # Set the user to run Kea
              USER kea

              # Entrypoint script to start Kea
              CMD ["/usr/sbin/kea-dhcp4", "-c", "/etc/kea/kea-dhcp4.conf"]
              ```
            * Build and push the image to a registry accessible by MicroShift (e.g., `quay.io/youruser/kea-dhcp:latest`).

      3.  Create Kubernetes Project and Permissions:**

            * `oc new-project kea-dhcp`
            * Grant the `privileged` SCC to the default service account:
              ```bash
              oc adm policy add-scc-to-user privileged -z default -n kea-dhcp
              ```

      4.  Deploy Configuration and the Application:**

            * Apply a `ConfigMap` with a minimal, valid Kea configuration:
              (You can use the same ConfigMap from the previous response)
            * Apply the following `Deployment` YAML to run the container on MicroShift.
              ```yaml
              apiVersion: apps/v1
              kind: Deployment
              metadata:
                name: kea-dhcp-server
                namespace: kea-dhcp
                labels:
                  app: kea-dhcp-server
              spec:
                replicas: 1
                selector:
                  matchLabels:
                    app: kea-dhcp-server
                template:
                  metadata:
                    labels:
                      app: kea-dhcp-server
                  spec:
                    # Use the host's network namespace
                    hostNetwork: true
                    serviceAccountName: default
                    containers:
                      - name: kea-dhcp-server
                        # Replace with your image location
                        image: quay.io/youruser/kea-dhcp:latest
                        securityContext:
                          # Run as root to allow binding to privileged ports
                          # even though the Kea process itself drops to the 'kea' user
                          runAsUser: 0
                          capabilities:
                            add:
                              - "NET_BIND_SERVICE"
                              - "NET_ADMIN"
                              - "NET_RAW"
                        ports:
                          - containerPort: 67
                            name: dhcpv4
                            protocol: UDP
                        volumeMounts:
                          - name: kea-config-volume
                            mountPath: /etc/kea
                    volumes:
                      - name: kea-config-volume
                        configMap:
                          name: kea-dhcp-config
              ```

      5.  Observe Logs:**

            * Check the logs of the newly created pod.
              ```bash
              oc logs -n kea-dhcp -l app=kea-dhcp-server -f
              ```

      Actual results:

      The Kea process inside the RHEL 10 UBI container fails. The logs show a fatal error when trying to bind to the socket on the host interface:
      `WARN [kea-dhcp4.dhcpsrv] DHCPSRV_OPEN_SOCKET_FAIL failed to open socket: Failed to open socket on interface enp7s0, reason: failed to bind fallback socket to address 192.168.200.1, port 67, reason: Permission denied`    

      Expected results:

      The Kea DHCP server pod should start successfully, bind to port 67 on the host's `enp7s0` interface, and be ready to serve DHCP requests on the `192.168.200.0/24` network.    

      Additional info:

      Extensive troubleshooting has been performed, ruling out all common causes for this error:
      - No Port Conflict: ss -lunp | grep ":67" on the host node confirms the port is free.
      - Privileged SCC: The privileged SCC is correctly applied.
      - hostNetwork: The pod is confirmed to be running with hostNetwork: true.
      - SELinux Disabled: sestatus confirms SELinux is disabled.
          

              eslutsky Evgeny Slutsky
              rhn-support-arolivei Arthur Oliveira
              None
              None
              John George John George
              None
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: