Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-34888

[Backport 4.12] Ingress operator performs spurious updates in response to API's defaulting of router deployment's router container's ports' hostPort field when using HostNetwork

XMLWordPrintable

    • Low
    • No
    • 1
    • Sprint 254
    • 1
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, the Ingress Operator would specify the `spec.template.spec.hostNetwork: true` on a router deployment without specifying the `spec.template.spec.containers[*].ports[*].hostPort`. This caused the API server to set a default value for each port's `hostPort` filed, which the Ingress Operator would then detect as an external update and attempt to revert it. Now, the Ingress Operator no longer incorrectly performs these updates resolving the issue. (link:https://issues.redhat.com/browse/OCPBUGS-34409[*OCPBUGS-34409*]) Backport to 4.13.
      Show
      * Previously, the Ingress Operator would specify the `spec.template.spec.hostNetwork: true` on a router deployment without specifying the `spec.template.spec.containers[*].ports[*].hostPort`. This caused the API server to set a default value for each port's `hostPort` filed, which the Ingress Operator would then detect as an external update and attempt to revert it. Now, the Ingress Operator no longer incorrectly performs these updates resolving the issue. (link: https://issues.redhat.com/browse/OCPBUGS-34409 [* OCPBUGS-34409 *]) Backport to 4.13.
    • Bug Fix
    • In Progress

      This is a clone of issue OCPBUGS-34409. The following is the description of the original issue:

      This is a clone of issue OCPBUGS-14995. The following is the description of the original issue:

      Description of problem

      When the ingress operator creates or updates a router deployment that specifies spec.template.spec.hostNetwork: true, the operator does not set spec.template.spec.containers[*].ports[*].hostPort. As a result, the API sets each port's hostPort field to the port's containerPort field value. The operator detects this as an external update and attempts to revert it. The operator should not update the deployment in response to API defaulting.

      Version-Release number of selected component (if applicable)

      I observed this in CI for OCP 4.14 and was able to reproduce the issue on OCP 4.11.37. The problematic code was added in https://github.com/openshift/cluster-ingress-operator/pull/694/commits/af653f9fa7368cf124e11b7ea4666bc40e601165 in OCP 4.11 to implement NE-674.

      How reproducible

      Easily.

      Steps to Reproduce

      1. Create an IngressController that specifies the "HostNetwork" endpoint publishing strategy type:

      oc create -f - <<EOF
      apiVersion: operator.openshift.io/v1
      kind: IngressController
      metadata:
        name: example-hostnetwork
        namespace: openshift-ingress-operator
      spec:
        domain: example.xyz
        endpointPublishingStrategy:
          type: HostNetwork
      EOF
      

      2. Check the ingress operator's logs:

      oc -n openshift-ingress-operator logs -c ingress-operator deployments/ingress-operator
      

      Actual results

      The ingress operator logs "updated router deployment" multiple times for the "example-hostnetwork" IngressController, such as the following:

      2023-06-15T02:11:47.229Z        INFO    operator.ingress_controller     ingress/deployment.go:131       updated router deployment       {"namespace": "openshift-ingress", "name": "router-example-hostnetwork", "diff": "  &v1.Deployment{\n  \tTypeMeta:   {},\n  \tObjectMeta: {Name: \"router-example-hostnetwork\", Namespace: \"openshift-ingress\", UID: \"d7c51022-460e-4962-8521-e00255f649c3\", ResourceVersion: \"3356177\", ...},\n  \tSpec: v1.DeploymentSpec{\n  \t\tReplicas: &2,\n  \t\tSelector: &{MatchLabels: {\"ingresscontroller.operator.openshift.io/deployment-ingresscontroller\": \"example-hostnetwork\"}},\n  \t\tTemplate: v1.PodTemplateSpec{\n  \t\t\tObjectMeta: {Labels: {\"ingresscontroller.operator.openshift.io/deployment-ingresscontroller\": \"example-hostnetwork\", \"ingresscontroller.operator.openshift.io/hash\": \"b7c697fd\"}, Annotations: {\"target.workload.openshift.io/management\": `{\"effect\": \"PreferredDuringScheduling\"}`, \"unsupported.do-not-use.openshift.io/override-liveness-grace-period-seconds\": \"10\"}},\n  \t\t\tSpec: v1.PodSpec{\n  \t\t\t\tVolumes: []v1.Volume{\n  \t\t\t\t\t{Name: \"default-certificate\", VolumeSource: {Secret: &{SecretName: \"router-certs-example-hostnetwork\", DefaultMode: &420}}},\n  \t\t\t\t\t{\n  \t\t\t\t\t\tName: \"service-ca-bundle\",\n  \t\t\t\t\t\tVolumeSource: v1.VolumeSource{\n  \t\t\t\t\t\t\t... // 16 identical fields\n  \t\t\t\t\t\t\tFC:        nil,\n  \t\t\t\t\t\t\tAzureFile: nil,\n  \t\t\t\t\t\t\tConfigMap: &v1.ConfigMapVolumeSource{\n  \t\t\t\t\t\t\t\tLocalObjectReference: {Name: \"service-ca-bundle\"},\n  \t\t\t\t\t\t\t\tItems:                {{Key: \"service-ca.crt\", Path: \"service-ca.crt\"}},\n- \t\t\t\t\t\t\t\tDefaultMode:          &420,\n+ \t\t\t\t\t\t\t\tDefaultMode:          nil,\n  \t\t\t\t\t\t\t\tOptional:             &false,\n  \t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\tVsphereVolume: nil,\n  \t\t\t\t\t\t\tQuobyte:       nil,\n  \t\t\t\t\t\t\t... // 8 identical fields\n  \t\t\t\t\t\t},\n  \t\t\t\t\t},\n  \t\t\t\t\t{\n  \t\t\t\t\t\tName: \"stats-auth\",\n  \t\t\t\t\t\tVolumeSource: v1.VolumeSource{\n  \t\t\t\t\t\t\t... // 3 identical fields\n  \t\t\t\t\t\t\tAWSElasticBlockStore: nil,\n  \t\t\t\t\t\t\tGitRepo:              nil,\n  \t\t\t\t\t\t\tSecret: &v1.SecretVolumeSource{\n  \t\t\t\t\t\t\t\tSecretName:  \"router-stats-example-hostnetwork\",\n  \t\t\t\t\t\t\t\tItems:       nil,\n- \t\t\t\t\t\t\t\tDefaultMode: &420,\n+ \t\t\t\t\t\t\t\tDefaultMode: nil,\n  \t\t\t\t\t\t\t\tOptional:    nil,\n  \t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\tNFS:   nil,\n  \t\t\t\t\t\t\tISCSI: nil,\n  \t\t\t\t\t\t\t... // 21 identical fields\n  \t\t\t\t\t\t},\n  \t\t\t\t\t},\n  \t\t\t\t\t{\n  \t\t\t\t\t\tName: \"metrics-certs\",\n  \t\t\t\t\t\tVolumeSource: v1.VolumeSource{\n  \t\t\t\t\t\t\t... // 3 identical fields\n  \t\t\t\t\t\t\tAWSElasticBlockStore: nil,\n  \t\t\t\t\t\t\tGitRepo:              nil,\n  \t\t\t\t\t\t\tSecret: &v1.SecretVolumeSource{\n  \t\t\t\t\t\t\t\tSecretName:  \"router-metrics-certs-example-hostnetwork\",\n  \t\t\t\t\t\t\t\tItems:       nil,\n- \t\t\t\t\t\t\t\tDefaultMode: &420,\n+ \t\t\t\t\t\t\t\tDefaultMode: nil,\n  \t\t\t\t\t\t\t\tOptional:    nil,\n  \t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\tNFS:   nil,\n  \t\t\t\t\t\t\tISCSI: nil,\n  \t\t\t\t\t\t\t... // 21 identical fields\n  \t\t\t\t\t\t},\n  \t\t\t\t\t},\n  \t\t\t\t},\n  \t\t\t\tInitContainers: nil,\n  \t\t\t\tContainers: []v1.Container{\n  \t\t\t\t\t{\n  \t\t\t\t\t\t... // 3 identical fields\n  \t\t\t\t\t\tArgs:       nil,\n  \t\t\t\t\t\tWorkingDir: \"\",\n  \t\t\t\t\t\tPorts: []v1.ContainerPort{\n  \t\t\t\t\t\t\t{\n  \t\t\t\t\t\t\t\tName:          \"http\",\n- \t\t\t\t\t\t\t\tHostPort:      80,\n+ \t\t\t\t\t\t\t\tHostPort:      0,\n  \t\t\t\t\t\t\t\tContainerPort: 80,\n  \t\t\t\t\t\t\t\tProtocol:      \"TCP\",\n  \t\t\t\t\t\t\t\tHostIP:        \"\",\n  \t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\t{\n  \t\t\t\t\t\t\t\tName:          \"https\",\n- \t\t\t\t\t\t\t\tHostPort:      443,\n+ \t\t\t\t\t\t\t\tHostPort:      0,\n  \t\t\t\t\t\t\t\tContainerPort: 443,\n  \t\t\t\t\t\t\t\tProtocol:      \"TCP\",\n  \t\t\t\t\t\t\t\tHostIP:        \"\",\n  \t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\t{\n  \t\t\t\t\t\t\t\tName:          \"metrics\",\n- \t\t\t\t\t\t\t\tHostPort:      1936,\n+ \t\t\t\t\t\t\t\tHostPort:      0,\n  \t\t\t\t\t\t\t\tContainerPort: 1936,\n  \t\t\t\t\t\t\t\tProtocol:      \"TCP\",\n  \t\t\t\t\t\t\t\tHostIP:        \"\",\n  \t\t\t\t\t\t\t},\n  \t\t\t\t\t\t},\n  \t\t\t\t\t\tEnvFrom:       nil,\n  \t\t\t\t\t\tEnv:           {{Name: \"DEFAULT_CERTIFICATE_DIR\", Value: \"/etc/pki/tls/private\"}, {Name: \"DEFAULT_DESTINATION_CA_PATH\", Value: \"/var/run/configmaps/service-ca/service-ca.crt\"}, {Name: \"RELOAD_INTERVAL\", Value: \"5s\"}, {Name: \"ROUTER_ALLOW_WILDCARD_ROUTES\", Value: \"false\"}, ...},\n  \t\t\t\t\t\tResources:     {Requests: {s\"cpu\": {i: {...}, s: \"100m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n  \t\t\t\t\t\tVolumeMounts:  {{Name: \"default-certificate\", ReadOnly: true, MountPath: \"/etc/pki/tls/private\"}, {Name: \"service-ca-bundle\", ReadOnly: true, MountPath: \"/var/run/configmaps/service-ca\"}, {Name: \"stats-auth\", ReadOnly: true, MountPath: \"/var/lib/haproxy/conf/metrics-auth\"}, {Name: \"metrics-certs\", ReadOnly: true, MountPath: \"/etc/pki/tls/metrics-certs\"}},\n  \t\t\t\t\t\tVolumeDevices: nil,\n  \t\t\t\t\t\tLivenessProbe: &v1.Probe{\n  \t\t\t\t\t\t\tProbeHandler: v1.ProbeHandler{\n  \t\t\t\t\t\t\t\tExec: nil,\n  \t\t\t\t\t\t\t\tHTTPGet: &v1.HTTPGetAction{\n  \t\t\t\t\t\t\t\t\tPath:        \"/healthz\",\n  \t\t\t\t\t\t\t\t\tPort:        {IntVal: 1936},\n  \t\t\t\t\t\t\t\t\tHost:        \"localhost\",\n- \t\t\t\t\t\t\t\t\tScheme:      \"HTTP\",\n+ \t\t\t\t\t\t\t\t\tScheme:      \"\",\n  \t\t\t\t\t\t\t\t\tHTTPHeaders: nil,\n  \t\t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\t\tTCPSocket: nil,\n  \t\t\t\t\t\t\t\tGRPC:      nil,\n  \t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\tInitialDelaySeconds:           0,\n  \t\t\t\t\t\t\tTimeoutSeconds:                1,\n- \t\t\t\t\t\t\tPeriodSeconds:                 10,\n+ \t\t\t\t\t\t\tPeriodSeconds:                 0,\n- \t\t\t\t\t\t\tSuccessThreshold:              1,\n+ \t\t\t\t\t\t\tSuccessThreshold:              0,\n- \t\t\t\t\t\t\tFailureThreshold:              3,\n+ \t\t\t\t\t\t\tFailureThreshold:              0,\n  \t\t\t\t\t\t\tTerminationGracePeriodSeconds: nil,\n  \t\t\t\t\t\t},\n  \t\t\t\t\t\tReadinessProbe: &v1.Probe{\n  \t\t\t\t\t\t\tProbeHandler: v1.ProbeHandler{\n  \t\t\t\t\t\t\t\tExec: nil,\n  \t\t\t\t\t\t\t\tHTTPGet: &v1.HTTPGetAction{\n  \t\t\t\t\t\t\t\t\tPath:        \"/healthz/ready\",\n  \t\t\t\t\t\t\t\t\tPort:        {IntVal: 1936},\n  \t\t\t\t\t\t\t\t\tHost:        \"localhost\",\n- \t\t\t\t\t\t\t\t\tScheme:      \"HTTP\",\n+ \t\t\t\t\t\t\t\t\tScheme:      \"\",\n  \t\t\t\t\t\t\t\t\tHTTPHeaders: nil,\n  \t\t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\t\tTCPSocket: nil,\n  \t\t\t\t\t\t\t\tGRPC:      nil,\n  \t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\tInitialDelaySeconds:           0,\n  \t\t\t\t\t\t\tTimeoutSeconds:                1,\n- \t\t\t\t\t\t\tPeriodSeconds:                 10,\n+ \t\t\t\t\t\t\tPeriodSeconds:                 0,\n- \t\t\t\t\t\t\tSuccessThreshold:              1,\n+ \t\t\t\t\t\t\tSuccessThreshold:       
            0,\n- \t\t\t\t\t\t\tFailureThreshold:              3,\n+ \t\t\t\t\t\t\tFailureThreshold:              0,\n  \t\t\t\t\t\t\tTerminationGracePeriodSeconds: nil,\n  \t\t\t\t\t\t},\n  \t\t\t\t\t\tStartupProbe: &v1.Probe{\n  \t\t\t\t\t\t\tProbeHandler: v1.ProbeHandler{\n  \t\t\t\t\t\t\t\tExec: nil,\n  \t\t\t\t\t\t\t\tHTTPGet: &v1.HTTPGetAction{\n  \t\t\t\t\t\t\t\t\tPath:        \"/healthz/ready\",\n  \t\t\t\t\t\t\t\t\tPort:        {IntVal: 1936},\n  \t\t\t\t\t\t\t\t\tHost:        \"localhost\",\n- \t\t\t\t\t\t\t\t\tScheme:      \"HTTP\",\n+ \t\t\t\t\t\t\t\t\tScheme:      \"\",\n  \t\t\t\t\t\t\t\t\tHTTPHeaders: nil,\n  \t\t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\t\tTCPSocket: nil,\n  \t\t\t\t\t\t\t\tGRPC:      nil,\n  \t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\tInitialDelaySeconds:           0,\n  \t\t\t\t\t\t\tTimeoutSeconds:                1,\n  \t\t\t\t\t\t\tPeriodSeconds:                 1,\n- \t\t\t\t\t\t\tSuccessThreshold:              1,\n+ \t\t\t\t\t\t\tSuccessThreshold:              0,\n  \t\t\t\t\t\t\tFailureThreshold:              120,\n  \t\t\t\t\t\t\tTerminationGracePeriodSeconds: nil,\n  \t\t\t\t\t\t},\n  \t\t\t\t\t\tLifecycle:              nil,\n  \t\t\t\t\t\tTerminationMessagePath: \"/dev/termination-log\",\n  \t\t\t\t\t\t... // 6 identical fields\n  \t\t\t\t\t},\n  \t\t\t\t},\n  \t\t\t\tEphemeralContainers: nil,\n  \t\t\t\tRestartPolicy:       \"Always\",\n  \t\t\t\t... // 31 identical fields\n  \t\t\t},\n  \t\t},\n  \t\tStrategy:        {Type: \"RollingUpdate\", RollingUpdate: &{MaxUnavailable: &{Type: 1, StrVal: \"25%\"}, MaxSurge: &{}}},\n  \t\tMinReadySeconds: 30,\n  \t\t... // 3 identical fields\n  \t},\n  \tStatus: {ObservedGeneration: 1, Replicas: 2, UpdatedReplicas:
      2, UnavailableReplicas: 2, ...},\n  }\n"}
      

      Note the following in the diff:

                                                      Ports: []v1.ContainerPort{                                                                                                                                                                                                                                                                                                                                                               
                                                              {                                                                                                                                                                                                                                                                                                                                                                                
                                                                      Name:          \"http\",                                                                                                                                                                                                                                                                                                                                                 
      -                                                               HostPort:      80,                                                                                                                                                                                                                                                                                                                                                       
      +                                                               HostPort:      0,                                                                                                                                                                                                                                                                                                                                                        
                                                                      ContainerPort: 80,                                                                                                                                                                                                                                                                                                                                                       
                                                                      Protocol:      \"TCP\",                                                                                                                                                                                                                                                                                                                                                  
                                                                      HostIP:        \"\",                                                                                                                                                                                                                                                                                                                                                     
                                                              },                                                                                                                                                                                                                                                                                                                                                                               
                                                              {
                                                                      Name:          \"https\",
      -                                                               HostPort:      443,
      +                                                               HostPort:      0,
                                                                      ContainerPort: 443,
                                                                      Protocol:      \"TCP\",
                                                                      HostIP:        \"\",
                                                              },
                                                              {
                                                                      Name:          \"metrics\",
      -                                                               HostPort:      1936,
      +                                                               HostPort:      0,
                                                                      ContainerPort: 1936,
                                                                      Protocol:      \"TCP\",
                                                                      HostIP:        \"\",
                                                              },
                                                      },
      

      Expected results

      The operator should ignore updates by the API that only set default values. The operator should not perform these unnecessary updates to the router deployment.

              cholman@redhat.com Candace Holman
              openshift-crt-jira-prow OpenShift Prow Bot
              Shudi Li Shudi Li
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: