Loading...

Type: Bug
Resolution: Unresolved
Priority: Normal
Fix Version/s: None
Affects Version/s: 4.17
Component/s: Networking / router
Labels:
- ne-triaged
- triaged

Severity:
Moderate
Regression:
None
Sprint:
Hypershift Sprint 258
sprint_count:
1
Release Blocker:
Rejected
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Target Version:

4.18.0
Target Backport Versions:

4.17.z

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Description of problem:

The openshift-ingress/router-default never stops reconciling in the ingress operator.

2024-08-22T15:59:22.789Z	INFO	operator.ingress_controller	controller/controller.go:114	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2024-08-22T15:59:22.799Z	INFO	operator.status_controller	controller/controller.go:114	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2024-08-22T15:59:22.868Z	INFO	operator.ingress_controller	ingress/deployment.go:135	updated router deployment	{"namespace": "openshift-ingress", "name": "router-default", "diff": "  &v1.Deployment{\n  \tTypeMeta:   {},\n  \tObjectMeta: {Name: \"router-default\", Namespace: \"openshift-ingress\", UID: \"6cf98392-8782-4741-b5c9-ce63fb77879a\", ResourceVersion: \"6244\", ...},\n  \tSpec: v1.DeploymentSpec{\n  \t\tReplicas: &1,\n  \t\tSelector: &{MatchLabels: {\"ingresscontroller.operator.openshift.io/deployment-ingresscontroller\": \"default\"}},\n  \t\tTemplate: {ObjectMeta: {Labels: {\"ingresscontroller.operator.openshift.io/deployment-ingresscontroller\": \"default\", \"ingresscontroller.operator.openshift.io/hash\": \"9c69cc8d\"}, Annotations: {\"target.workload.openshift.io/management\": `{\"effect\": \"PreferredDuringScheduling\"}`}}, Spec: {Volumes: {{Name: \"default-certificate\", VolumeSource: {Secret: &{SecretName: \"default-ingress-cert\", DefaultMode: &420}}}, {Name: \"service-ca-bundle\", VolumeSource: {ConfigMap: &{LocalObjectReference: {Name: \"service-ca-bundle\"}, Items: {{Key: \"service-ca.crt\", Path: \"service-ca.crt\"}}, DefaultMode: &420, Optional: &false}}}, {Name: \"stats-auth\", VolumeSource: {Secret: &{SecretName: \"router-stats-default\", DefaultMode: &420}}}, {Name: \"metrics-certs\", VolumeSource: {Secret: &{SecretName: \"router-metrics-certs-default\", DefaultMode: &420}}}}, Containers: {{Name: \"router\", Image: \"registry.build05.ci.openshift.org/ci-op-vxbb8hxy/stable@sha256:f\"..., Ports: {{Name: \"http\", ContainerPort: 80, Protocol: \"TCP\"}, {Name: \"https\", ContainerPort: 443, Protocol: \"TCP\"}, {Name: \"metrics\", ContainerPort: 1936, Protocol: \"TCP\"}}, Env: {{Name: \"DEFAULT_CERTIFICATE_DIR\", Value: \"/etc/pki/tls/private\"}, {Name: \"DEFAULT_DESTINATION_CA_PATH\", Value: \"/var/run/configmaps/service-ca/service-ca.crt\"}, {Name: \"RELOAD_INTERVAL\", Value: \"5s\"}, {Name: \"ROUTER_ALLOW_WILDCARD_ROUTES\", Value: \"false\"}, ...}, ...}}, RestartPolicy: \"Always\", TerminationGracePeriodSeconds: &3600, ...}},\n  \t\tStrategy: v1.DeploymentStrategy{\n- \t\t\tType:          \"RollingUpdate\",\n+ \t\t\tType:          \"\",\n- \t\t\tRollingUpdate: s\"&RollingUpdateDeployment{MaxUnavailable:25%,MaxSurge:25%,}\",\n+ \t\t\tRollingUpdate: nil,\n  \t\t},\n  \t\tMinReadySeconds:      30,\n  \t\tRevisionHistoryLimit: &10,\n  \t\t... // 2 identical fields\n  \t},\n  \tStatus: {ObservedGeneration: 1, Replicas: 1, UpdatedReplicas: 1, ReadyReplicas: 1, ...},\n  }\n"}
2024-08-22T15:59:22.884Z	ERROR	operator.ingress_controller	controller/controller.go:114	got retryable error; requeueing	{"after": "1m0s", "error": "IngressController is degraded: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.)"}
2024-08-22T15:59:24.820Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:103	queueing ingress	{"name": "default", "related": ""}
2024-08-22T15:59:24.820Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:103	queueing ingress	{"name": "default", "related": ""}
2024-08-22T15:59:24.820Z	INFO	operator.ingress_controller	controller/controller.go:114	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2024-08-22T15:59:24.887Z	INFO	operator.ingress_controller	ingress/deployment.go:135	updated router deployment	{"namespace": "openshift-ingress", "name": "router-default", "diff": "  &v1.Deployment{\n  \tTypeMeta:   {},\n  \tObjectMeta: {Name: \"router-default\", Namespace: \"openshift-ingress\", UID: \"6cf98392-8782-4741-b5c9-ce63fb77879a\", ResourceVersion: \"7194\", ...},\n  \tSpec: v1.DeploymentSpec{\n  \t\tReplicas: &1,\n  \t\tSelector: &{MatchLabels: {\"ingresscontroller.operator.openshift.io/deployment-ingresscontroller\": \"default\"}},\n  \t\tTemplate: {ObjectMeta: {Labels: {\"ingresscontroller.operator.openshift.io/deployment-ingresscontroller\": \"default\", \"ingresscontroller.operator.openshift.io/hash\": \"9c69cc8d\"}, Annotations: {\"target.workload.openshift.io/management\": `{\"effect\": \"PreferredDuringScheduling\"}`}}, Spec: {Volumes: {{Name: \"default-certificate\", VolumeSource: {Secret: &{SecretName: \"default-ingress-cert\", DefaultMode: &420}}}, {Name: \"service-ca-bundle\", VolumeSource: {ConfigMap: &{LocalObjectReference: {Name: \"service-ca-bundle\"}, Items: {{Key: \"service-ca.crt\", Path: \"service-ca.crt\"}}, DefaultMode: &420, Optional: &false}}}, {Name: \"stats-auth\", VolumeSource: {Secret: &{SecretName: \"router-stats-default\", DefaultMode: &420}}}, {Name: \"metrics-certs\", VolumeSource: {Secret: &{SecretName: \"router-metrics-certs-default\", DefaultMode: &420}}}}, Containers: {{Name: \"router\", Image: \"registry.build05.ci.openshift.org/ci-op-vxbb8hxy/stable@sha256:f\"..., Ports: {{Name: \"http\", ContainerPort: 80, Protocol: \"TCP\"}, {Name: \"https\", ContainerPort: 443, Protocol: \"TCP\"}, {Name: \"metrics\", ContainerPort: 1936, Protocol: \"TCP\"}}, Env: {{Name: \"DEFAULT_CERTIFICATE_DIR\", Value: \"/etc/pki/tls/private\"}, {Name: \"DEFAULT_DESTINATION_CA_PATH\", Value: \"/var/run/configmaps/service-ca/service-ca.crt\"}, {Name: \"RELOAD_INTERVAL\", Value: \"5s\"}, {Name: \"ROUTER_ALLOW_WILDCARD_ROUTES\", Value: \"false\"}, ...}, ...}}, RestartPolicy: \"Always\", TerminationGracePeriodSeconds: &3600, ...}},\n  \t\tStrategy: v1.DeploymentStrategy{\n- \t\t\tType:          \"RollingUpdate\",\n+ \t\t\tType:          \"\",\n- \t\t\tRollingUpdate: s\"&RollingUpdateDeployment{MaxUnavailable:25%,MaxSurge:25%,}\",\n+ \t\t\tRollingUpdate: nil,\n  \t\t},\n  \t\tMinReadySeconds:      30,\n  \t\tRevisionHistoryLimit: &10,\n  \t\t... // 2 identical fields\n  \t},\n  \tStatus: {ObservedGeneration: 1, Replicas: 1, UpdatedReplicas: 1, ReadyReplicas: 1, ...},\n  }\n"}
2024-08-22T15:59:24.911Z	INFO	operator.route_metrics_controller	controller/controller.go:114	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2024-08-22T15:59:24.911Z	INFO	operator.status_controller	controller/controller.go:114	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2024-08-22T15:59:24.911Z	INFO	operator.certificate_controller	controller/controller.go:114	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2024-08-22T15:59:24.911Z	INFO	operator.ingressclass_controller	controller/controller.go:114	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2024-08-22T15:59:24.913Z	INFO	operator.ingress_controller	controller/controller.go:114	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2024-08-22T15:59:24.924Z	INFO	operator.status_controller	controller/controller.go:114	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2024-08-22T15:59:24.984Z	INFO	operator.ingress_controller	ingress/deployment.go:135	updated router deployment	{"namespace": "openshift-ingress", "name": "router-default", "diff": "  &v1.Deployment{\n  \tTypeMeta:   {},\n  \tObjectMeta: {Name: \"router-default\", Namespace: \"openshift-ingress\", UID: \"6cf98392-8782-4741-b5c9-ce63fb77879a\", ResourceVersion: \"7194\", ...},\n  \tSpec: v1.DeploymentSpec{\n  \t\tReplicas: &1,\n  \t\tSelector: &{MatchLabels: {\"ingresscontroller.operator.openshift.io/deployment-ingresscontroller\": \"default\"}},\n  \t\tTemplate: {ObjectMeta: {Labels: {\"ingresscontroller.operator.openshift.io/deployment-ingresscontroller\": \"default\", \"ingresscontroller.operator.openshift.io/hash\": \"9c69cc8d\"}, Annotations: {\"target.workload.openshift.io/management\": `{\"effect\": \"PreferredDuringScheduling\"}`}}, Spec: {Volumes: {{Name: \"default-certificate\", VolumeSource: {Secret: &{SecretName: \"default-ingress-cert\", DefaultMode: &420}}}, {Name: \"service-ca-bundle\", VolumeSource: {ConfigMap: &{LocalObjectReference: {Name: \"service-ca-bundle\"}, Items: {{Key: \"service-ca.crt\", Path: \"service-ca.crt\"}}, DefaultMode: &420, Optional: &false}}}, {Name: \"stats-auth\", VolumeSource: {Secret: &{SecretName: \"router-stats-default\", DefaultMode: &420}}}, {Name: \"metrics-certs\", VolumeSource: {Secret: &{SecretName: \"router-metrics-certs-default\", DefaultMode: &420}}}}, Containers: {{Name: \"router\", Image: \"registry.build05.ci.openshift.org/ci-op-vxbb8hxy/stable@sha256:f\"..., Ports: {{Name: \"http\", ContainerPort: 80, Protocol: \"TCP\"}, {Name: \"https\", ContainerPort: 443, Protocol: \"TCP\"}, {Name: \"metrics\", ContainerPort: 1936, Protocol: \"TCP\"}}, Env: {{Name: \"DEFAULT_CERTIFICATE_DIR\", Value: \"/etc/pki/tls/private\"}, {Name: \"DEFAULT_DESTINATION_CA_PATH\", Value: \"/var/run/configmaps/service-ca/service-ca.crt\"}, {Name: \"RELOAD_INTERVAL\", Value: \"5s\"}, {Name: \"ROUTER_ALLOW_WILDCARD_ROUTES\", Value: \"false\"}, ...}, ...}}, RestartPolicy: \"Always\", TerminationGracePeriodSeconds: &3600, ...}},\n  \t\tStrategy: v1.DeploymentStrategy{\n- \t\t\tType:          \"RollingUpdate\",\n+ \t\t\tType:          \"\",\n- \t\t\tRollingUpdate: s\"&RollingUpdateDeployment{MaxUnavailable:25%,MaxSurge:25%,}\",\n+ \t\t\tRollingUpdate: nil,\n  \t\t},\n  \t\tMinReadySeconds:      30,\n  \t\tRevisionHistoryLimit: &10,\n  \t\t... // 2 identical fields\n  \t},\n  \tStatus: {ObservedGeneration: 1, Replicas: 1, UpdatedReplicas: 1, ReadyReplicas: 1, ...},\n  }\n"}
2024-08-22T15:59:43.457Z	INFO	operator.ingress_controller	controller/controller.go:114	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2024-08-22T15:59:43.539Z	INFO	operator.ingress_controller	ingress/deployment.go:135	updated router deployment	{"namespace": "openshift-ingress", "name": "router-default", "diff": "  &v1.Deployment{\n  \tTypeMeta:   {},\n  \tObjectMeta: {Name: \"router-default\", Namespace: \"openshift-ingress\", UID: \"6cf98392-8782-4741-b5c9-ce63fb77879a\", ResourceVersion: \"7194\", ...},\n  \tSpec: v1.DeploymentSpec{\n  \t\tReplicas: &1,\n  \t\tSelector: &{MatchLabels: {\"ingresscontroller.operator.openshift.io/deployment-ingresscontroller\": \"default\"}},\n  \t\tTemplate: {ObjectMeta: {Labels: {\"ingresscontroller.operator.openshift.io/deployment-ingresscontroller\": \"default\", \"ingresscontroller.operator.openshift.io/hash\": \"9c69cc8d\"}, Annotations: {\"target.workload.openshift.io/management\": `{\"effect\": \"PreferredDuringScheduling\"}`}}, Spec: {Volumes: {{Name: \"default-certificate\", VolumeSource: {Secret: &{SecretName: \"default-ingress-cert\", DefaultMode: &420}}}, {Name: \"service-ca-bundle\", VolumeSource: {ConfigMap: &{LocalObjectReference: {Name: \"service-ca-bundle\"}, Items: {{Key: \"service-ca.crt\", Path: \"service-ca.crt\"}}, DefaultMode: &420, Optional: &false}}}, {Name: \"stats-auth\", VolumeSource: {Secret: &{SecretName: \"router-stats-default\", DefaultMode: &420}}}, {Name: \"metrics-certs\", VolumeSource: {Secret: &{SecretName: \"router-metrics-certs-default\", DefaultMode: &420}}}}, Containers: {{Name: \"router\", Image: \"registry.build05.ci.openshift.org/ci-op-vxbb8hxy/stable@sha256:f\"..., Ports: {{Name: \"http\", ContainerPort: 80, Protocol: \"TCP\"}, {Name: \"https\", ContainerPort: 443, Protocol: \"TCP\"}, {Name: \"metrics\", ContainerPort: 1936, Protocol: \"TCP\"}}, Env: {{Name: \"DEFAULT_CERTIFICATE_DIR\", Value: \"/etc/pki/tls/private\"}, {Name: \"DEFAULT_DESTINATION_CA_PATH\", Value: \"/var/run/configmaps/service-ca/service-ca.crt\"}, {Name: \"RELOAD_INTERVAL\", Value: \"5s\"}, {Name: \"ROUTER_ALLOW_WILDCARD_ROUTES\", Value: \"false\"}, ...}, ...}}, RestartPolicy: \"Always\", TerminationGracePeriodSeconds: &3600, ...}},\n  \t\tStrategy: v1.DeploymentStrategy{\n- \t\t\tType:          \"RollingUpdate\",\n+ \t\t\tType:          \"\",\n- \t\t\tRollingUpdate: s\"&RollingUpdateDeployment{MaxUnavailable:25%,MaxSurge:25%,}\",\n+ \t\t\tRollingUpdate: nil,\n  \t\t},\n  \t\tMinReadySeconds:      30,\n  \t\tRevisionHistoryLimit: &10,\n  \t\t... // 2 identical fields\n  \t},\n  \tStatus: {ObservedGeneration: 1, Replicas: 1, UpdatedReplicas: 1, ReadyReplicas: 1, ...},\n  }\n"}
2024-08-22T16:01:07.866Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:103	queueing ingresscontroller	{"name": "default"}
2024-08-22T16:01:07.866Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:103	queueing ingresscontroller	{"name": "default"}
2024-08-22T16:01:07.866Z	INFO	operator.route_metrics_controller	controller/controller.go:114	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2024-08-22T16:01:07.870Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:103	queueing ingresscontroller	{"name": "default"}
2024-08-22T16:01:07.870Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:103	queueing ingresscontroller	{"name": "default"}
2024-08-22T16:01:07.870Z	INFO	operator.route_metrics_controller	controller/controller.go:114	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2024-08-22T16:01:07.899Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:103	queueing ingresscontroller	{"name": "default"}
2024-08-22T16:01:07.899Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:103	queueing ingresscontroller	{"name": "default"}
2024-08-22T16:01:07.899Z	INFO	operator.route_metrics_controller	controller/controller.go:114	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2024-08-22T16:01:08.957Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:103	queueing ingresscontroller	{"name": "default"}
2024-08-22T16:01:08.957Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:103	queueing ingresscontroller	{"name": "default"}
2024-08-22T16:01:08.957Z	INFO	operator.route_metrics_controller	controller/controller.go:114	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}

Version-Release number of selected component (if applicable):

4.17

The diff is:

❯ cat /tmp/msg.json | jq -r '.diff'
  &v1.Deployment{
  	TypeMeta:   {},
  	ObjectMeta: {Name: "router-default", Namespace: "openshift-ingress", UID: "6cf98392-8782-4741-b5c9-ce63fb77879a", ResourceVersion: "6244", ...},
  	Spec: v1.DeploymentSpec{
  		Replicas: &1,
  		Selector: &{MatchLabels: {"ingresscontroller.operator.openshift.io/deployment-ingresscontroller": "default"}},
  		Template: {ObjectMeta: {Labels: {"ingresscontroller.operator.openshift.io/deployment-ingresscontroller": "default", "ingresscontroller.operator.openshift.io/hash": "9c69cc8d"}, Annotations: {"target.workload.openshift.io/management": `{"effect": "PreferredDuringScheduling"}`}}, Spec: {Volumes: {{Name: "default-certificate", VolumeSource: {Secret: &{SecretName: "default-ingress-cert", DefaultMode: &420}}}, {Name: "service-ca-bundle", VolumeSource: {ConfigMap: &{LocalObjectReference: {Name: "service-ca-bundle"}, Items: {{Key: "service-ca.crt", Path: "service-ca.crt"}}, DefaultMode: &420, Optional: &false}}}, {Name: "stats-auth", VolumeSource: {Secret: &{SecretName: "router-stats-default", DefaultMode: &420}}}, {Name: "metrics-certs", VolumeSource: {Secret: &{SecretName: "router-metrics-certs-default", DefaultMode: &420}}}}, Containers: {{Name: "router", Image: "registry.build05.ci.openshift.org/ci-op-vxbb8hxy/stable@sha256:f"..., Ports: {{Name: "http", ContainerPort: 80, Protocol: "TCP"}, {Name: "https", ContainerPort: 443, Protocol: "TCP"}, {Name: "metrics", ContainerPort: 1936, Protocol: "TCP"}}, Env: {{Name: "DEFAULT_CERTIFICATE_DIR", Value: "/etc/pki/tls/private"}, {Name: "DEFAULT_DESTINATION_CA_PATH", Value: "/var/run/configmaps/service-ca/service-ca.crt"}, {Name: "RELOAD_INTERVAL", Value: "5s"}, {Name: "ROUTER_ALLOW_WILDCARD_ROUTES", Value: "false"}, ...}, ...}}, RestartPolicy: "Always", TerminationGracePeriodSeconds: &3600, ...}},
  		Strategy: v1.DeploymentStrategy{
- 			Type:          "RollingUpdate",
+ 			Type:          "",
- 			RollingUpdate: s"&RollingUpdateDeployment{MaxUnavailable:25%,MaxSurge:25%,}",
+ 			RollingUpdate: nil,
  		},
  		MinReadySeconds:      30,
  		RevisionHistoryLimit: &10,
  		... // 2 identical fields
  	},
  	Status: {ObservedGeneration: 1, Replicas: 1, UpdatedReplicas: 1, ReadyReplicas: 1, ...},
  }

links to

openshift/cluster-ingress-operator#1134: OCPBUGS-38871: ingress: deployment: explicitly set DeploymentStrategy in SingleReplica case

RHEA-2024:6122 OpenShift Container Platform 4.18.z bug fix update

Details

Description

Attachments

Issue Links

Activity

People

Dates