Loading...

XML

Word

Printable

Type: Bug
Resolution: Unresolved
Priority: Major
Fix Version/s: None
Affects Version/s: 4.17.z
Component/s: kube-apiserver
Labels:
None

Activity Type:
Quality / Stability / Reliability
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Story Points:
None
Severity:
Important
Regression:
None

Target Backport Versions:
None
Target Version:
None
Release Blocker:
None
Sprint:
None

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

PX Impact Score:

Release Note Status:
None
Release Note Type:
None
Release Note Text:
None

Escape Reason:
None
Escape Impact:
None
Corrective Measures:
None
SDLC stage when should've been found:
None

Hi,

Description of problem:
Since upgrade from 4.16.9 to 4.17.9, openapi/v2 path is returning the following error

    E0102 14:02:11.107847      17 handler.go:160] Error in OpenAPI handler: failed to build merge specs: unable to merge: duplicated path /apis/route.openshift.io/v1/routes

This cause Console's CRDs Schema tab to not show (getting 503 on api/kubernetes/openapi/v2 in Network tab) and also ArgoCD from GitOps Operator to not work properly.

Note that openapi/v3 works.

Version-Release number of selected component (if applicable):

    quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7de4bf32ba888017c2ef4149d927b9c28b25f9592c4673bb44ba3fdb1d8ce801

"io.openshift.build.commit.url":"https://github.com/openshift/kubernetes/commit/df0195cb1143b34da26b6aa490ab812036e77898"

How reproducible:

I wasn't able to reproduce it yet but you can find a similar behavior of the issue with other apis on this github issue :

https://github.com/kubernetes/kubernetes/issues/122668

Steps to Reproduce:

    1.
    2.
    3.

Actual results:
Please find full stacktrace

E0102 14:01:21.638285      17 handler.go:160] Error in OpenAPI handler: failed to build merge specs: unable to merge: duplicated path /apis/route.openshift.io/v1/routes
I0102 14:01:21.638571      17 httplog.go:138] "HTTP" verb="GET" URI="/openapi/v2" latency="997.587204ms" userAgent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:126.0) Gecko/20100101 Firefox/126.0" audit-ID="38eb18e3-c75c-4271-9810-7fc7bce32320" srcIP="10.17.1.52:36670" apf_pl="global-default" apf_fs="global-default" apf_iseats=1 apf_fseats=0 apf_additionalLatency="0s" apf_execution_time="997.216694ms" resp=503 statusStack=<
	
	goroutine 69675 [running]:
	k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc01f581760, 0xc0268ca3c0?)
		k8s.io/apiserver/pkg/server/httplog/httplog.go:331 +0xfa
	k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc01f581760, 0x1f7)
		k8s.io/apiserver/pkg/server/httplog/httplog.go:311 +0x1d
	k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc01fd645d0, 0x1f7)
		k8s.io/apiserver/pkg/server/filters/timeout.go:239 +0x1cc
	k8s.io/apiserver/pkg/endpoints/filters.(*auditResponseWriter).WriteHeader(0xc03a541ec0, 0x1f7)
		k8s.io/apiserver/pkg/endpoints/filters/audit.go:253 +0x34
	github.com/NYTimes/gziphandler.(*GzipResponseWriter).startPlain(0xc05f2e78f0)
		github.com/NYTimes/gziphandler@v1.1.1/gzip.go:188 +0x35
	github.com/NYTimes/gziphandler.(*GzipResponseWriter).Close(0xc02fca3c20?)
		github.com/NYTimes/gziphandler@v1.1.1/gzip.go:233 +0xb0
	github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1({0x5764a60, 0xc01fd648d0}, 0xc02fca3c20)
		github.com/NYTimes/gziphandler@v1.1.1/gzip.go:344 +0x24e
	net/http.HandlerFunc.ServeHTTP(0xc073109ef4?, {0x5764a60?, 0xc01fd648d0?}, 0xb?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc01e9b0680, {0x5764a60, 0xc01fd648d0}, 0xc02fca3c20)
		k8s.io/apiserver/pkg/server/mux/pathrecorder.go:242 +0x1dd
	k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc073109ef4?, {0x5764a60?, 0xc01fd648d0?}, 0x1?)
		k8s.io/apiserver/pkg/server/mux/pathrecorder.go:235 +0x66
	k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x4ea1e7a?, 0xc075111f80?}, 0xc0138b06c0?, 0xc014496460?}, {0x5764a60, 0xc01fd648d0}, 0xc02fca3c20)
		k8s.io/apiserver/pkg/server/handler.go:154 +0x6f1
	k8s.io/kubernetes/openshift-kube-apiserver/openshiftkubeapiserver.BuildHandlerChain.func1.withOAuthInfo.1({0x5764a60?, 0xc01fd648d0?}, 0x41f9300?)
		k8s.io/kubernetes/openshift-kube-apiserver/openshiftkubeapiserver/patch_handlerchain.go:65 +0xa7
	net/http.HandlerFunc.ServeHTTP(0x188724f?, {0x5764a60?, 0xc01fd648d0?}, 0x5728f10?)
		net/http/server.go:2171 +0x29
	k8s.io/kubernetes/openshift-kube-apiserver/openshiftkubeapiserver.BuildHandlerChain.func1.WithAPIRequestCountLogging.2({0x5764a60?, 0xc01fd648d0?}, 0xc02fca3c20)
		k8s.io/kubernetes/openshift-kube-apiserver/filters/apirequestcount_filter.go:17 +0x2cb
	net/http.HandlerFunc.ServeHTTP(0x1aafeaf?, {0x5764a60?, 0xc01fd648d0?}, 0x5728f10?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func27({0x5764a60, 0xc01fd648d0}, 0xc02fca3c20)
		k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:110 +0x177
	net/http.HandlerFunc.ServeHTTP(0x57794f8?, {0x5764a60?, 0xc01fd648d0?}, 0x4?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/endpoints/filters.withAuthorization.func1({0x5764a60, 0xc01fd648d0}, 0xc02fca3c20)
		k8s.io/apiserver/pkg/endpoints/filters/authorization.go:78 +0x626
	net/http.HandlerFunc.ServeHTTP(0xc072802540?, {0x5764a60?, 0xc01fd648d0?}, 0xc0001564d0?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x5764a60, 0xc01fd648d0}, 0xc02fca3c20)
		k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:84 +0x192
	net/http.HandlerFunc.ServeHTTP(0x1aafeaf?, {0x5764a60?, 0xc01fd648d0?}, 0x5728f10?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func28({0x5764a60, 0xc01fd648d0}, 0xc02fca3c20)
		k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:110 +0x177
	net/http.HandlerFunc.ServeHTTP(0xc03c8731c0?, {0x5764a60?, 0xc01fd648d0?}, 0x20?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/server/filters.(*priorityAndFairnessHandler).Handle.func9()
		k8s.io/apiserver/pkg/server/filters/priority-and-fairness.go:292 +0xd9
	k8s.io/apiserver/pkg/util/flowcontrol.(*configController).Handle.func2()
		k8s.io/apiserver/pkg/util/flowcontrol/apf_filter.go:192 +0x257
	k8s.io/apiserver/pkg/util/flowcontrol/fairqueuing/queueset.(*request).Finish.func1(0xc073281590?, 0x70?, 0x4c52180?)
		k8s.io/apiserver/pkg/util/flowcontrol/fairqueuing/queueset/queueset.go:391 +0x56
	k8s.io/apiserver/pkg/util/flowcontrol/fairqueuing/queueset.(*request).Finish(0xc073281590, 0xc05f2e7880)
		k8s.io/apiserver/pkg/util/flowcontrol/fairqueuing/queueset/queueset.go:392 +0x39
	k8s.io/apiserver/pkg/util/flowcontrol.(*configController).Handle(0xc0003472c0, {0x57795a0, 0xc05f2e7570}, {0xc01f5816b0, {0x57791b0, 0xc03c873140}}, 0x0?, 0x0?, 0xc03c873180?, 0xc03c873180)
		k8s.io/apiserver/pkg/util/flowcontrol/apf_filter.go:179 +0x74b
	k8s.io/apiserver/pkg/server/filters.(*priorityAndFairnessHandler).Handle.func10(0xc00b5585f0, {0x57794f8?, 0xc01fd64810?}, {0xc01f5816b0?, {0x57791b0?, 0xc03c873140?}}, 0xc075110d80, 0xc01fd64900, 0xc0407002a0, 0xc03c873180)
		k8s.io/apiserver/pkg/server/filters/priority-and-fairness.go:298 +0xeb
	k8s.io/apiserver/pkg/server/filters.(*priorityAndFairnessHandler).Handle(0xc00b5585f0, {0x5764a60, 0xc01fd648d0}, 0xc02fca3c20)
		k8s.io/apiserver/pkg/server/filters/priority-and-fairness.go:299 +0xb25
	net/http.HandlerFunc.ServeHTTP(0xc00d774d08?, {0x5764a60?, 0xc01fd648d0?}, 0xc0001564d0?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x5764a60, 0xc01fd648d0}, 0xc02fca3c20)
		k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:84 +0x192
	net/http.HandlerFunc.ServeHTTP(0x1aafeaf?, {0x5764a60?, 0xc01fd648d0?}, 0x5728f10?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func29({0x5764a60, 0xc01fd648d0}, 0xc02fca3c20)
		k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:110 +0x177
	net/http.HandlerFunc.ServeHTTP(0xc01fd64630?, {0x5764a60?, 0xc01fd648d0?}, 0xab932f?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.WithImpersonation.func5({0x5764a60, 0xc01fd648d0}, 0xc02fca3c20)
		k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x1b0
	net/http.HandlerFunc.ServeHTTP(0xc00d775490?, {0x5764a60?, 0xc01fd648d0?}, 0xc0001564d0?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x5764a60, 0xc01fd648d0}, 0xc02fca3c20)
		k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:84 +0x192
	net/http.HandlerFunc.ServeHTTP(0x1aafeaf?, {0x5764a60?, 0xc01fd648d0?}, 0x5728f10?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func30({0x5764a60, 0xc01fd648d0}, 0xc02fca3c20)
		k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:110 +0x177
	net/http.HandlerFunc.ServeHTTP(0xc0017977f0?, {0x5764a60?, 0xc01fd648d0?}, 0x57794f8?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.WithAudit.func7({0x5764a60, 0xc01fd64600}, 0xc02fca3c20)
		k8s.io/apiserver/pkg/endpoints/filters/audit.go:115 +0x395
	net/http.HandlerFunc.ServeHTTP(0x0?, {0x5764a60?, 0xc01fd64600?}, 0x41403b?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x5764a60, 0xc01fd64600}, 0xc02fca3c20)
		k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:84 +0x192
	net/http.HandlerFunc.ServeHTTP(0x3?, {0x5764a60?, 0xc01fd64600?}, 0xc00d775900?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.WithStartupEarlyAnnotation.withStartupEarlyAnnotation.func31({0x5764a60, 0xc01fd64600}, 0xc02fca3c20)
		k8s.io/apiserver/pkg/server/filters/with_early_late_annotations.go:127 +0x2a7
	net/http.HandlerFunc.ServeHTTP(0x1aafeaf?, {0x5764a60?, 0xc01fd64600?}, 0x5728f10?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func33({0x5764a60, 0xc01fd64600}, 0xc02fca3c20)
		k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:110 +0x177
	net/http.HandlerFunc.ServeHTTP(0x57794f8?, {0x5764a60?, 0xc01fd64600?}, 0x5720828?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/endpoints/filters.withAuthentication.func1({0x5764a60, 0xc01fd64600}, 0xc02fca3c20)
		k8s.io/apiserver/pkg/endpoints/filters/authentication.go:120 +0x79f
	net/http.HandlerFunc.ServeHTTP(0x57794f8?, {0x5764a60?, 0xc01fd64600?}, 0x5728f10?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x5764a60, 0xc01fd64600}, 0xc02fca38c0)
		k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:94 +0x354
	net/http.HandlerFunc.ServeHTTP(0xc01fd64630?, {0x5764a60?, 0xc01fd64600?}, 0xc03e263e60?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/server/filters.WithCORS.func1({0x5764a60, 0xc01fd64600}, 0xc02fca38c0)
		k8s.io/apiserver/pkg/server/filters/cors.go:61 +0x630
	net/http.HandlerFunc.ServeHTTP(0xc02fca2ea0?, {0x5764a60?, 0xc01fd64600?}, 0xc005863ae0?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.WithWarningRecorder.func12({0x5764a60, 0xc01fd64600}, 0xc02fca2ea0)
		k8s.io/apiserver/pkg/endpoints/filters/warning.go:35 +0xb9
	net/http.HandlerFunc.ServeHTTP(0xc006da4720?, {0x5764a60?, 0xc01fd64600?}, 0xc03e263fd0?)
		net/http/server.go:2171 +0x29
	k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1()
		k8s.io/apiserver/pkg/server/filters/timeout.go:115 +0x62
	created by k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP in goroutine 14729
		k8s.io/apiserver/pkg/server/filters/timeout.go:101 +0x198
 >

Expected results:

Additional info:
This seems to have been a change in the way the handler deals with duplicate path that breaks my kube-apiserver.

 % oc version
Client Version: 4.17.0-202412120604.p0.gf4525b8.assembly.stream.el9-f4525b8
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: 4.17.9
Kubernetes Version: v1.30.6

 % kubectl get --raw /openapi/v3 | jq | grep route    
"apis/route.openshift.io": {
      "serverRelativeURL": "/openapi/v3/apis/route.openshift.io?hash=041EE0C2C97B4625E50B9CB4AF6EEA3AF777DFD1B6589899D8E82F14D0FA9C0394E0C52B7DA3C9F374A1309E7FA14C83400D584C59B481520C538498668A2BCC"
    "apis/route.openshift.io/v1": {
      "serverRelativeURL": "/openapi/v3/apis/route.openshift.io/v1?hash=F7151C8023EF2F909D7578C5FF7F1DECE307715FD6B2A9BD0CC4870ABE07AF26FBE08AA78FB7469A710B1EC72EC0E656C31A7BA3063943D6C278BAC85BA4E577"

Assignee:: Unassigned

Reporter:: Alexandre KNECHT (Inactive)

Need Info From:: None

Contributors:: None

QA Contact:: None

Doc Contact:: None

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Created:: 2025/01/02 3:06 PM

Updated:: 2025/07/17 1:23 PM

Details

Description

Attachments

Easy Agile Planning Poker

Activity

People

Dates

Hide