Loading...

Type: Bug
Resolution: Unresolved
Priority: Major
Fix Version/s: 4.20.0
Affects Version/s: 4.19, 4.20
Component/s: Networking / router
Labels:

Activity Type:
Quality / Stability / Reliability
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Story Points:
None
Severity:
Important
Regression:
None

Target Backport Versions:
None
Target Version:

4.20
Release Blocker:
Rejected
Sprint:
NI&D Sprint 275
sprint_count:
1

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

PX Impact Score:

Release Note Status:
In Progress
Release Note Type:
Release Note Not Required
Release Note Text:
None

Escape Reason:
None
Escape Impact:
None
Corrective Measures:
None
SDLC stage when should've been found:
None

Description of problem

CI is flaky because of test failures such as the following:

=== RUN   TestAll/serial/TestGatewayAPI/testGatewayAPIDNS/gatewayListenersWithOverlappingHostname
    util_gatewayapi_test.go:623: Found gateway openshift-ingress/gw3 as Accepted
    gateway_api_test.go:562: Observed that gateway openshift-ingress/gw3 has been accepted: {Addresses:[] Conditions:[{Type:Accepted Status:True ObservedGeneration:1 LastTransitionTime:2025-07-08 23:08:52 +0000 UTC Reason:Accepted Message:Resource accepted} {Type:Programmed Status:False ObservedGeneration:1 LastTransitionTime:2025-07-08 23:08:52 +0000 UTC Reason:AddressNotAssigned Message:Failed to assign to any requested addresses: no instances found for hostname "gw3-openshift-default.openshift-ingress.svc.cluster.local"}] Listeners:[{Name:http SupportedKinds:[{Group:0xc000bfc3f0 Kind:HTTPRoute} {Group:0xc000bfc400 Kind:GRPCRoute}] AttachedRoutes:0 Conditions:[{Type:Accepted Status:True ObservedGeneration:1 LastTransitionTime:2025-07-08 23:08:52 +0000 UTC Reason:Accepted Message:No errors found} {Type:Conflicted Status:False ObservedGeneration:1 LastTransitionTime:2025-07-08 23:08:52 +0000 UTC Reason:NoConflicts Message:No errors found} {Type:Programmed Status:True ObservedGeneration:1 LastTransitionTime:2025-07-08 23:08:52 +0000 UTC Reason:Programmed Message:No errors found} {Type:ResolvedRefs Status:True ObservedGeneration:1 LastTransitionTime:2025-07-08 23:08:52 +0000 UTC Reason:ResolvedRefs Message:No errors found}]}]}
    util_gatewayapi_test.go:623: Found gateway openshift-ingress/gw4 as Accepted
    gateway_api_test.go:562: Observed that gateway openshift-ingress/gw4 has been accepted: {Addresses:[] Conditions:[{Type:Accepted Status:True ObservedGeneration:1 LastTransitionTime:2025-07-08 23:08:52 +0000 UTC Reason:Accepted Message:Resource accepted} {Type:Programmed Status:False ObservedGeneration:1 LastTransitionTime:2025-07-08 23:08:52 +0000 UTC Reason:AddressNotAssigned Message:Assigned to service(s) gw4-openshift-default.openshift-ingress.svc.cluster.local:80, but failed to assign to all requested addresses: address pending for hostname "gw4-openshift-default.openshift-ingress.svc.cluster.local"}] Listeners:[{Name:http SupportedKinds:[{Group:0xc0009ffd00 Kind:HTTPRoute} {Group:0xc0009ffd10 Kind:GRPCRoute}] AttachedRoutes:0 Conditions:[{Type:Accepted Status:True ObservedGeneration:1 LastTransitionTime:2025-07-08 23:08:52 +0000 UTC Reason:Accepted Message:No errors found} {Type:Conflicted Status:False ObservedGeneration:1 LastTransitionTime:2025-07-08 23:08:52 +0000 UTC Reason:NoConflicts Message:No errors found} {Type:Programmed Status:True ObservedGeneration:1 LastTransitionTime:2025-07-08 23:08:52 +0000 UTC Reason:Programmed Message:No errors found} {Type:ResolvedRefs Status:True ObservedGeneration:1 LastTransitionTime:2025-07-08 23:08:52 +0000 UTC Reason:ResolvedRefs Message:No errors found}]}]}
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "*.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw4) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "*.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw4) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "*.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw4) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "*.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw4) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "*.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw4) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "*.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw4) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "*.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw4) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "*.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw4) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "*.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw4) is expected to be present but was not found; retrying...
    util_gatewayapi_test.go:756: DNSRecord for hostname "qwe.gws.ci-op-kxm3184y-43abb.origin-ci-int-aws.dev.rhcloud.com." (gateway: gw3) is expected to be present but was not found; retrying...
    gateway_api_test.go:569: dnsRecord expectations not met: failed to observe expected DNSRecords: context deadline exceeded

This particular failure comes from https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-ingress-operator/1227/pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-operator/1942690958306971648. Another failure is https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-ingress-operator/1241/pull-ci-openshift-cluster-ingress-operator-release-4.19-e2e-azure-operator/1941128347979354112.

Version-Release number of selected component (if applicable)

4.20 and 4.19.

How reproducible

Presently, search.ci shows two failures, with the following impact:

pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-operator (all) - 11 runs, 73% failed, 13% of failures match = 9% impact
pull-ci-openshift-cluster-ingress-operator-release-4.19-e2e-azure-operator (all) - 3 runs, 67% failed, 50% of failures match = 33% impact

Steps to Reproduce

1. Post a PR and have bad luck.
2. Check search.ci.

Actual results

CI fails.

Expected results

CI passes, or fails on some other test failure.

Additional info

The failures include both Azure and AWS, which is evidence that the infrastructure is not the issue and that the test needs to be more robust.

links to

openshift/cluster-ingress-operator#1271: OCPBUGS-59139: Increase assertExpectedDNSRecords timeouts

Details

Description

Description of problem

Version-Release number of selected component (if applicable)

How reproducible

Steps to Reproduce

Actual results

Expected results

Additional info

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates

Hide