Loading...

Type: Bug
Resolution: Unresolved
Priority: Normal
Fix Version/s: None
Affects Version/s: 4.21
Component/s: Networking / router
Labels:

Activity Type:
None
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Story Points:
2
Severity:
Low
Regression:
No

Target Backport Versions:
None
Target Version:

4.22.0
Release Blocker:
Rejected
Sprint:
NI&D Sprint 281
sprint_count:
1

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

PX Impact Score:

Release Note Status:
In Progress
Release Note Type:
Release Note Not Required
Release Note Text:
N/A

Escape Reason:
None
Escape Impact:
None
Corrective Measures:
None
SDLC stage when should've been found:
None

Description of problem

CI can fail because of a panic in the TestGatewayAPI/testGatewayAPIDNSListenerWithNoHostname and TestGatewayAPI/testGatewayAPIDNSListenerUpdate tests if either test succeeds in creating the gateway but the gateway is never accepted/programmed.

Version-Release number of selected component (if applicable)

OpenShift 4.19 and later.

How reproducible

100%.

Steps to Reproduce

1. Scale the ingress-operator to 0 replicas:

oc patch clusterversions/version --type=json --patch='[{"op":"add","path":"/spec/overrides","value":[{"kind":"Deployment","group":"apps","name":"ingress-operator","namespace":"openshift-ingress-operator","unmanaged":true}]}]' && oc -n openshift-ingress-operator scale deploy/ingress-operator --replicas=0

2. Run the tests:

make test-e2e TEST='TestGatewayAPI/(testGatewayAPIDNSListenerWithNoHostname|testGatewayAPIDNSListenerUpdate)

Actual results

The tests panic (note: I shortened some timeouts so that I could reproduce the problem before my cluster expired):

% make test-e2e TEST='TestGatewayAPI/testGatewayAPIDNSListenerWithNoHostname'
go generate ./pkg/manifests
CGO_ENABLED=1 GO111MODULE=on GOFLAGS=-mod=vendor go test -timeout 1.5h -count 1 -v -tags e2e -run "TestGatewayAPI/testGatewayAPIDNSListenerWithNoHostname" ./test/e2e
=== RUN   TestGatewayAPI
=== RUN   TestGatewayAPI/testGatewayAPIDNSListenerWithNoHostname
[...]
    gateway_api_test.go:701: failed to accept/program gateway test-nohost-gateway: gateway openshift-ingress/test-nohost-gateway does not have all expected conditions, last recorded status messages status messages for Accepted: "", Programmed: ""
--- FAIL: TestGatewayAPI (7.98s)
    --- FAIL: TestGatewayAPI/testGatewayAPIDNSListenerWithNoHostname (3.34s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered, repanicked]
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x1e74dca]

goroutine 89 [running]:
testing.tRunner.func1.2({0x22333e0, 0x3ebc940})
	/home/mmasters/sdk/go1.25.0/src/testing/testing.go:1872 +0x237
testing.tRunner.func1()
	/home/mmasters/sdk/go1.25.0/src/testing/testing.go:1875 +0x35b
panic({0x22333e0?, 0x3ebc940?})
	/home/mmasters/sdk/go1.25.0/src/runtime/panic.go:783 +0x132
github.com/openshift/cluster-ingress-operator/test/e2e.testGatewayAPIDNSListenerWithNoHostname.func1()
	/home/mmasters/src/github.com/openshift/cluster-ingress-operator/test/e2e/gateway_api_test.go:695 +0x8a
testing.(*common).Cleanup.func1()
	/home/mmasters/sdk/go1.25.0/src/testing/testing.go:1308 +0x109
testing.(*common).runCleanup(0xc000502fc0, 0xc000382fc0?)
	/home/mmasters/sdk/go1.25.0/src/testing/testing.go:1572 +0xea
testing.tRunner.func2()
	/home/mmasters/sdk/go1.25.0/src/testing/testing.go:1928 +0x25
runtime.Goexit()
	/home/mmasters/sdk/go1.25.0/src/runtime/panic.go:615 +0x5e
testing.(*common).FailNow(0xc000502fc0)
	/home/mmasters/sdk/go1.25.0/src/testing/testing.go:1013 +0x4a
testing.(*common).Fatalf(0xc000502fc0, {0x2684141?, 0x11?}, {0xc0006f3f48?, 0x13?, 0x11?})
	/home/mmasters/sdk/go1.25.0/src/testing/testing.go:1219 +0x59
github.com/openshift/cluster-ingress-operator/test/e2e.testGatewayAPIDNSListenerWithNoHostname(0xc000502fc0)
	/home/mmasters/src/github.com/openshift/cluster-ingress-operator/test/e2e/gateway_api_test.go:701 +0x2dd
testing.tRunner(0xc000502fc0, 0x27a7de0)
	/home/mmasters/sdk/go1.25.0/src/testing/testing.go:1934 +0xea
created by testing.(*T).Run in goroutine 84
	/home/mmasters/sdk/go1.25.0/src/testing/testing.go:1997 +0x465
FAIL	github.com/openshift/cluster-ingress-operator/test/e2e	8.987s
FAIL
make: *** [Makefile:62: test-e2e] Error 1
zsh: exit 2     make test-e2e TEST='TestGatewayAPI/testGatewayAPIDNSListenerWithNoHostname'
make test-e2e TEST='TestGatewayAPI/testGatewayAPIDNSListenerWithNoHostname'  6.23s user 0.79s system 58% cpu 12.044 total

% make test-e2e TEST='TestGatewayAPI/testGatewayAPIDNSListenerUpdate'
go generate ./pkg/manifests
CGO_ENABLED=1 GO111MODULE=on GOFLAGS=-mod=vendor go test -timeout 1.5h -count 1 -v -tags e2e -run "TestGatewayAPI/testGatewayAPIDNSListenerUpdate" ./test/e2e
=== RUN   TestGatewayAPI
=== RUN   TestGatewayAPI/testGatewayAPIDNSListenerUpdate
    gateway_api_test.go:609: Created gateway test-gateway-update with multiple hostnames
[...]
    gateway_api_test.go:622: failed to accept/program gateway test-gateway-update: gateway openshift-ingress/test-gateway-update does not have all expected conditions, last recorded status messages status messages for Accepted: "", Programmed: ""
--- FAIL: TestGatewayAPI (8.40s)
    --- FAIL: TestGatewayAPI/testGatewayAPIDNSListenerUpdate (3.62s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered, repanicked]
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x1e7461c]

goroutine 83 [running]:
testing.tRunner.func1.2({0x22333e0, 0x3ebc940})
	/home/mmasters/sdk/go1.25.0/src/testing/testing.go:1872 +0x237
testing.tRunner.func1()
	/home/mmasters/sdk/go1.25.0/src/testing/testing.go:1875 +0x35b
panic({0x22333e0?, 0x3ebc940?})
	/home/mmasters/sdk/go1.25.0/src/runtime/panic.go:783 +0x132
github.com/openshift/cluster-ingress-operator/test/e2e.testGatewayAPIDNSListenerUpdate.func1()
	/home/mmasters/src/github.com/openshift/cluster-ingress-operator/test/e2e/gateway_api_test.go:616 +0x9c
testing.(*common).Cleanup.func1()
	/home/mmasters/sdk/go1.25.0/src/testing/testing.go:1308 +0x109
testing.(*common).runCleanup(0xc000583c00, 0xc000583a40?)
	/home/mmasters/sdk/go1.25.0/src/testing/testing.go:1572 +0xea
testing.tRunner.func2()
	/home/mmasters/sdk/go1.25.0/src/testing/testing.go:1928 +0x25
runtime.Goexit()
	/home/mmasters/sdk/go1.25.0/src/runtime/panic.go:615 +0x5e
testing.(*common).FailNow(0xc000583c00)
	/home/mmasters/sdk/go1.25.0/src/testing/testing.go:1013 +0x4a
testing.(*common).Fatalf(0xc000583c00, {0x2684109?, 0x11?}, {0xc0008b9c48?, 0x13?, 0x11?})
	/home/mmasters/sdk/go1.25.0/src/testing/testing.go:1219 +0x59
github.com/openshift/cluster-ingress-operator/test/e2e.testGatewayAPIDNSListenerUpdate(0xc000583c00)
	/home/mmasters/src/github.com/openshift/cluster-ingress-operator/test/e2e/gateway_api_test.go:622 +0x41c
testing.tRunner(0xc000583c00, 0x27a7dd8)
	/home/mmasters/sdk/go1.25.0/src/testing/testing.go:1934 +0xea
created by testing.(*T).Run in goroutine 78
	/home/mmasters/sdk/go1.25.0/src/testing/testing.go:1997 +0x465
FAIL	github.com/openshift/cluster-ingress-operator/test/e2e	9.751s
FAIL
make: *** [Makefile:62: test-e2e] Error 1
zsh: exit 2     make test-e2e TEST='TestGatewayAPI/testGatewayAPIDNSListenerUpdate'
make test-e2e TEST='TestGatewayAPI/testGatewayAPIDNSListenerUpdate'  3.52s user 0.86s system 37% cpu 11.694 total

Expected results

The tests should fail but not panic:

% make test-e2e TEST='TestGatewayAPI/(testGatewayAPIDNSListenerWithNoHostname|testGatewayAPIDNSListenerUpdate)'
go generate ./pkg/manifests
CGO_ENABLED=1 GO111MODULE=on GOFLAGS=-mod=vendor go test -timeout 1.5h -count 1 -v -tags e2e -run "TestGatewayAPI/(testGatewayAPIDNSListenerWithNoHostname|testGatewayAPIDNSListenerUpdate)" ./test/e2e
=== RUN   TestGatewayAPI
=== RUN   TestGatewayAPI/testGatewayAPIDNSListenerUpdate
    gateway_api_test.go:609: Created gateway test-gateway-update with multiple hostnames
    util_gatewayapi_test.go:663: [2025-12-23 15:55:05] Not all expected gateway conditions are found, checking gateway service...
    util_gatewayapi_test.go:670: Failed to get gateway service openshift-ingress/test-gateway-update-openshift-default: services "test-gateway-update-openshift-default" not found; retrying...
[...]
    util_gatewayapi_test.go:637: Failed to get gateway openshift-ingress/test-gateway-update: context deadline exceeded; retrying...
[...]
    gateway_api_test.go:621: failed to accept/program gateway test-gateway-update: gateway openshift-ingress/test-gateway-update does not have all expected conditions, last recorded status messages status messages for Accepted: "Waiting for controller", Programmed: "Waiting for controller"
=== RUN   TestGatewayAPI/testGatewayAPIDNSListenerWithNoHostname
[...]
    util_gatewayapi_test.go:663: [2025-12-23 15:58:05] Not all expected gateway conditions are found, checking gateway service...
    util_gatewayapi_test.go:670: Failed to get gateway service openshift-ingress/test-nohost-gateway-openshift-default: services "test-nohost-gateway-openshift-default" not found; retrying...
[...]
    util_gatewayapi_test.go:637: Failed to get gateway openshift-ingress/test-nohost-gateway: context deadline exceeded; retrying...
[...]
    gateway_api_test.go:699: failed to accept/program gateway test-nohost-gateway: gateway openshift-ingress/test-nohost-gateway does not have all expected conditions, last recorded status messages status messages for Accepted: "Waiting for controller", Programmed: "Waiting for controller"
--- FAIL: TestGatewayAPI (365.55s)
    --- FAIL: TestGatewayAPI/testGatewayAPIDNSListenerUpdate (180.53s)
    --- FAIL: TestGatewayAPI/testGatewayAPIDNSListenerWithNoHostname (180.39s)
FAIL
FAIL	github.com/openshift/cluster-ingress-operator/test/e2e	366.554s
FAIL
make: *** [Makefile:62: test-e2e] Error 1
zsh: exit 2     make test-e2e
make test-e2e   4.05s user 0.74s system 1% cpu 6:08.40 total

Additional info

A similar issue was reported as OCPBUGS-64675.

is caused by

NE-1957 Test DNS Records creation for Gateways with unique and overlapping hostnames

Closed

is related to

OCPBUGS-64675 CI fails on TestGatewayAPI/testGatewayAPIDNS

Verified

links to

openshift/cluster-ingress-operator#1330: OCPBUGS-70212: testGatewayAPIDNS: Fix nil gateway in cleanup

Details

Description

Description of problem

Version-Release number of selected component (if applicable)

How reproducible

Steps to Reproduce

Actual results

Expected results

Additional info

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates