Loading...

XML

Word

Printable

Type: Bug
Resolution: Won't Do
Priority: Undefined
Fix Version/s: None
Affects Version/s: 4.12.z
Component/s: route-controller-manager
Labels:

Regression:
No
Story Points:
1
Sprint:
NE Sprint 260
sprint_count:
1
Release Blocker:
Rejected
Blocked:
False
Blocked Reason:

Hide

None

Show
None

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Description of problem:
Observed in 4.12.19 ROSA cluster and raised as a ClusterOperatorDown alert for the openshift-controller-manager. Issue appears to have began shortly after installation completed; the cluster was potentially never healthy.

Upon investigating, it was found that all route-controller-manager pods were in a CrashLoopBackoff state. Each of their logs contained only the following:

$ oc logs route-controller-manager-7c6d8d8b66-nqxhk -n openshift-route-controller-manager -p
I1213 20:30:25.500136       1 controller_manager.go:26] Starting controllers on 0.0.0.0:8443 (4.12.0-202305101515.p0.g9e74d17.assembly.stream-9e74d17)
unexpected fault address 0xcc001427540
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0xcc001427540 pc=0x221fad4]

goroutine 1 [running]:
runtime.throw({0x2f287c5?, 0xc00091ed78?})
        runtime/panic.go:1047 +0x5d fp=0xc00091ed60 sp=0xc00091ed30 pc=0x109821d
runtime.sigpanic()
        runtime/signal_unix.go:842 +0x2c5 fp=0xc00091edb0 sp=0xc00091ed60 pc=0x10af465
k8s.io/apiserver/pkg/authentication/token/cache.newStripedCache(0x20, 0x3089620, 0xc00091ee78)
        k8s.io/apiserver@v0.25.2/pkg/authentication/token/cache/cache_striped.go:37 +0xf4 fp=0xc00091ee30 sp=0xc00091edb0 pc=0x221fad4
k8s.io/apiserver/pkg/authentication/token/cache.newWithClock({0x3298a60?, 0xc000226380}, 0x0, 0x45d964b800, 0x45d964b800, {0x32be670, 0x4756188})
        k8s.io/apiserver@v0.25.2/pkg/authentication/token/cache/cached_token_authenticator.go:112 +0xd4 fp=0xc00091eea0 sp=0xc00091ee30 pc=0x221ff94
k8s.io/apiserver/pkg/authentication/token/cache.New(...)
        k8s.io/apiserver@v0.25.2/pkg/authentication/token/cache/cached_token_authenticator.go:91
github.com/openshift/route-controller-manager/pkg/cmd/controller/route.newRemoteAuthenticator({0x32a3f28, 0xc00043e600}, 0xc0005c4ea0, 0x30872f0?)
        github.com/openshift/route-controller-manager/pkg/cmd/controller/route/apiserver_authenticator.go:33 +0x1b9 fp=0xc00091f0f8 sp=0xc00091eea0 pc=0x28e8f79
github.com/openshift/route-controller-manager/pkg/cmd/controller/route.RunControllerServer({{{0x2f32887, 0xc}, {0x2f27140, 0x3}, {{0x2f62ce1, 0x25}, {0x2f62d06, 0x25}}, {0x2f7010f, 0x2b}, ...}, ...}, ...)
        github.com/openshift/route-controller-manager/pkg/cmd/controller/route/standalone_apiserver.go:45 +0x18d fp=0xc00091f5f8 sp=0xc00091f0f8 pc=0x28ebfcd
github.com/openshift/route-controller-manager/pkg/cmd/route-controller-manager.RunRouteControllerManager(0xc00052ed80, 0x4?, {0x32b8bd0, 0xc00043c1c0})
        github.com/openshift/route-controller-manager/pkg/cmd/route-controller-manager/controller_manager.go:28 +0x24b fp=0xc00091f8a0 sp=0xc00091f5f8 pc=0x28f6a4b
github.com/openshift/route-controller-manager/pkg/cmd/route-controller-manager.(*RouteControllerManager).StartControllerManager(0xc0008dc940, {0x32b8bd0, 0xc00043c1c0})
        github.com/openshift/route-controller-manager/pkg/cmd/route-controller-manager/cmd.go:119 +0x3f2 fp=0xc00091fa00 sp=0xc00091f8a0 pc=0x28f6752
github.com/openshift/route-controller-manager/pkg/cmd/route-controller-manager.NewRouteControllerManagerCommand.func1(0xc0008bd900?, {0x2f279cd?, 0x2?, 0x2?})
        github.com/openshift/route-controller-manager/pkg/cmd/route-controller-manager/cmd.go:49 +0xc5 fp=0xc00091fb20 sp=0xc00091fa00 pc=0x28f5f65
github.com/spf13/cobra.(*Command).execute(0xc0008bd900, {0xc0008dcce0, 0x2, 0x2})
        github.com/spf13/cobra@v1.4.0/command.go:860 +0x663 fp=0xc00091fbf8 sp=0xc00091fb20 pc=0x14a7483
github.com/spf13/cobra.(*Command).ExecuteC(0xc0008bc000)
        github.com/spf13/cobra@v1.4.0/command.go:974 +0x3bd fp=0xc00091fcb0 sp=0xc00091fbf8 pc=0x14a7b9d
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.4.0/command.go:902
k8s.io/component-base/cli.run(0xc0008bc000)
        k8s.io/component-base@v0.25.2/cli/run.go:146 +0x317 fp=0xc00091fd70 sp=0xc00091fcb0 pc=0x24c8917
k8s.io/component-base/cli.Run(0x32b8bd0?)
        k8s.io/component-base@v0.25.2/cli/run.go:46 +0x1d fp=0xc00091fdf0 sp=0xc00091fd70 pc=0x24c84fd
main.main()
        github.com/openshift/route-controller-manager/cmd/route-controller-manager/main.go:28 +0x17f fp=0xc00091ff80 sp=0xc00091fdf0 pc=0x28f735f
runtime.main()
        runtime/proc.go:250 +0x212 fp=0xc00091ffe0 sp=0xc00091ff80 pc=0x109ae52
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc00091ffe8 sp=0xc00091ffe0 pc=0x10cdb41

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc000084fb0 sp=0xc000084f90 pc=0x109b216
runtime.goparkunlock(...)
        runtime/proc.go:369
runtime.forcegchelper()
        runtime/proc.go:302 +0xad fp=0xc000084fe0 sp=0xc000084fb0 pc=0x109b0ad
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc000084fe8 sp=0xc000084fe0 pc=0x10cdb41
created by runtime.init.6
        runtime/proc.go:290 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc000085790 sp=0xc000085770 pc=0x109b216
runtime.goparkunlock(...)
        runtime/proc.go:369
runtime.bgsweep(0x0?)
        runtime/mgcsweep.go:297 +0xd7 fp=0xc0000857c8 sp=0xc000085790 pc=0x1083df7
runtime.gcenable.func1()
        runtime/mgc.go:178 +0x26 fp=0xc0000857e0 sp=0xc0000857c8 pc=0x1078986
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0000857e8 sp=0xc0000857e0 pc=0x10cdb41
created by runtime.gcenable
        runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc0000b2000?, 0x328cad0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc000085f70 sp=0xc000085f50 pc=0x109b216
runtime.goparkunlock(...)
        runtime/proc.go:369
runtime.(*scavengerState).park(0x47246a0)
        runtime/mgcscavenge.go:389 +0x53 fp=0xc000085fa0 sp=0xc000085f70 pc=0x1081dd3
runtime.bgscavenge(0x0?)
        runtime/mgcscavenge.go:622 +0x65 fp=0xc000085fc8 sp=0xc000085fa0 pc=0x10823e5
runtime.gcenable.func2()
        runtime/mgc.go:179 +0x26 fp=0xc000085fe0 sp=0xc000085fc8 pc=0x1078926
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0x10cdb41
created by runtime.gcenable
        runtime/mgc.go:179 +0xaa

goroutine 5 [finalizer wait]:
runtime.gopark(0x4725820?, 0xc000009860?, 0x0?, 0x0?, 0xc000084770?)
        runtime/proc.go:363 +0xd6 fp=0xc000084628 sp=0xc000084608 pc=0x109b216
runtime.goparkunlock(...)
        runtime/proc.go:369
runtime.runfinq()
        runtime/mfinal.go:180 +0x10f fp=0xc0000847e0 sp=0xc000084628 pc=0x1077a0f
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0000847e8 sp=0xc0000847e0 pc=0x10cdb41
created by runtime.createfing
        runtime/mfinal.go:157 +0x45

goroutine 7 [GC worker (idle)]:
runtime.gopark(0x1061c3d?, 0xc0003af980?, 0xa0?, 0x67?, 0xc0000867a8?)
        runtime/proc.go:363 +0xd6 fp=0xc000086750 sp=0xc000086730 pc=0x109b216
runtime.gcBgMarkWorker()
        runtime/mgc.go:1235 +0xf1 fp=0xc0000867e0 sp=0xc000086750 pc=0x107aad1
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0000867e8 sp=0xc0000867e0 pc=0x10cdb41
created by runtime.gcBgMarkStartWorkers
        runtime/mgc.go:1159 +0x25

goroutine 8 [GC worker (idle)]:
runtime.gopark(0xd92381ca853?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc000086f50 sp=0xc000086f30 pc=0x109b216
runtime.gcBgMarkWorker()
        runtime/mgc.go:1235 +0xf1 fp=0xc000086fe0 sp=0xc000086f50 pc=0x107aad1
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc000086fe8 sp=0xc000086fe0 pc=0x10cdb41
created by runtime.gcBgMarkStartWorkers
        runtime/mgc.go:1159 +0x25

goroutine 22 [GC worker (idle)]:
runtime.gopark(0xd92381ca42e?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc000080750 sp=0xc000080730 pc=0x109b216
runtime.gcBgMarkWorker()
        runtime/mgc.go:1235 +0xf1 fp=0xc0000807e0 sp=0xc000080750 pc=0x107aad1
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0000807e8 sp=0xc0000807e0 pc=0x10cdb41
created by runtime.gcBgMarkStartWorkers
        runtime/mgc.go:1159 +0x25

goroutine 23 [GC worker (idle)]:
runtime.gopark(0xd9238f02c33?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc000080f50 sp=0xc000080f30 pc=0x109b216
runtime.gcBgMarkWorker()
        runtime/mgc.go:1235 +0xf1 fp=0xc000080fe0 sp=0xc000080f50 pc=0x107aad1
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc000080fe8 sp=0xc000080fe0 pc=0x10cdb41
created by runtime.gcBgMarkStartWorkers
        runtime/mgc.go:1159 +0x25

goroutine 50 [GC worker (idle)]:
runtime.gopark(0xd9238f09c5d?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc000586750 sp=0xc000586730 pc=0x109b216
runtime.gcBgMarkWorker()
        runtime/mgc.go:1235 +0xf1 fp=0xc0005867e0 sp=0xc000586750 pc=0x107aad1
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0005867e8 sp=0xc0005867e0 pc=0x10cdb41
created by runtime.gcBgMarkStartWorkers
        runtime/mgc.go:1159 +0x25

goroutine 9 [GC worker (idle)]:
runtime.gopark(0xd9238f03015?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc000087750 sp=0xc000087730 pc=0x109b216
runtime.gcBgMarkWorker()
        runtime/mgc.go:1235 +0xf1 fp=0xc0000877e0 sp=0xc000087750 pc=0x107aad1
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0000877e8 sp=0xc0000877e0 pc=0x10cdb41
created by runtime.gcBgMarkStartWorkers
        runtime/mgc.go:1159 +0x25

goroutine 10 [GC worker (idle)]:
runtime.gopark(0xd9238f0d9c7?, 0xc00058a000?, 0x18?, 0x14?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc000087f50 sp=0xc000087f30 pc=0x109b216
runtime.gcBgMarkWorker()
        runtime/mgc.go:1235 +0xf1 fp=0xc000087fe0 sp=0xc000087f50 pc=0x107aad1
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc000087fe8 sp=0xc000087fe0 pc=0x10cdb41
created by runtime.gcBgMarkStartWorkers
        runtime/mgc.go:1159 +0x25

goroutine 11 [GC worker (idle)]:
runtime.gopark(0xd9238f0392d?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc000582750 sp=0xc000582730 pc=0x109b216
runtime.gcBgMarkWorker()
        runtime/mgc.go:1235 +0xf1 fp=0xc0005827e0 sp=0xc000582750 pc=0x107aad1
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0005827e8 sp=0xc0005827e0 pc=0x10cdb41
created by runtime.gcBgMarkStartWorkers
        runtime/mgc.go:1159 +0x25

goroutine 29 [select, locked to thread]:
runtime.gopark(0xc0005897a8?, 0x2?, 0x0?, 0x0?, 0xc0005897a4?)
        runtime/proc.go:363 +0xd6 fp=0xc000589618 sp=0xc0005895f8 pc=0x109b216
runtime.selectgo(0xc0005897a8, 0xc0005897a0, 0x0?, 0x0, 0x1?, 0x1)
        runtime/select.go:328 +0x7bc fp=0xc000589758 sp=0xc000589618 pc=0x10ab53c
runtime.ensureSigM.func1()
        runtime/signal_unix.go:991 +0x1b0 fp=0xc0005897e0 sp=0xc000589758 pc=0x10af9b0
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0005897e8 sp=0xc0005897e0 pc=0x10cdb41
created by runtime.ensureSigM
        runtime/signal_unix.go:974 +0xbd

goroutine 30 [syscall]:
runtime.notetsleepg(0x0?, 0x0?)
        runtime/lock_futex.go:236 +0x34 fp=0xc000588fa0 sp=0xc000588f68 pc=0x10687b4
os/signal.signal_recv()
        runtime/sigqueue.go:152 +0x2f fp=0xc000588fc0 sp=0xc000588fa0 pc=0x10ca0ef
os/signal.loop()
        os/signal/signal_unix.go:23 +0x19 fp=0xc000588fe0 sp=0xc000588fc0 pc=0x144cd59
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc000588fe8 sp=0xc000588fe0 pc=0x10cdb41
created by os/signal.Notify.func1.1
        os/signal/signal.go:151 +0x2a

goroutine 31 [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc000582f00 sp=0xc000582ee0 pc=0x109b216
runtime.chanrecv(0xc0008e7680, 0x0, 0x1)
        runtime/chan.go:583 +0x49b fp=0xc000582f90 sp=0xc000582f00 pc=0x1062e9b
runtime.chanrecv1(0x0?, 0x0?)
        runtime/chan.go:442 +0x18 fp=0xc000582fb8 sp=0xc000582f90 pc=0x1062998
k8s.io/apiserver/pkg/server.SetupSignalContext.func1()
        k8s.io/apiserver@v0.25.2/pkg/server/signal.go:48 +0x2b fp=0xc000582fe0 sp=0xc000582fb8 pc=0x24c82eb
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc000582fe8 sp=0xc000582fe0 pc=0x10cdb41
created by k8s.io/apiserver/pkg/server.SetupSignalContext
        k8s.io/apiserver@v0.25.2/pkg/server/signal.go:47 +0xe5

goroutine 32 [select]:
runtime.gopark(0xc0005837a0?, 0x2?, 0x0?, 0x0?, 0xc000583764?)
        runtime/proc.go:363 +0xd6 fp=0xc0005835e0 sp=0xc0005835c0 pc=0x109b216
runtime.selectgo(0xc0005837a0, 0xc000583760, 0x0?, 0x0, 0x0?, 0x1)
        runtime/select.go:328 +0x7bc fp=0xc000583720 sp=0xc0005835e0 pc=0x10ab53c
k8s.io/klog/v2.(*flushDaemon).run.func1()
        k8s.io/klog/v2@v2.80.1/klog.go:1135 +0x11e fp=0xc0005837e0 sp=0xc000583720 pc=0x118f27e
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0005837e8 sp=0xc0005837e0 pc=0x10cdb41
created by k8s.io/klog/v2.(*flushDaemon).run
        k8s.io/klog/v2@v2.80.1/klog.go:1131 +0x17b

Attempting to reproduce this in other workloads by deploying images which also invoke cache.New has failed: so far no other pod on the cluster is known to be crashlooping.

SRE attempted to restart the nodes running the affected pods. After booting, the route-controller-manager pod was able to run for a short time, but eventually re-entered a CrashLoopBackoff state. Logs did not change after rebooting.

Version-Release number of selected component (if applicable):

4.12.19

How reproducible:

Unsure: only observed on a single cluster, but very reproducible on that cluster

Steps to Reproduce:

    1. Install cluster
    2. Observe route-controller-manager pods crashlooping

Actual results:

$ oc get po -n openshift-route-controller-manager
NAME                                            READY   STATUS             RESTARTS         AGE
route-controller-manager-7c6d8d8b66-nqxhk       0/1     CrashLoopBackOff   26 (4m13s ago)   113m
route-controller-manager-7c6d8d8b66-qspnx       0/1     CrashLoopBackOff   26 (4m18s ago)   113m
route-controller-manager-7c6d8d8b66-twtm8       0/1     CrashLoopBackOff   26 (4m7s ago)    113m

Expected results:

All route-controller-manager pods are running

Assignee:: Filip Krepinsky

Reporter:: Trevor Nierman

QA Contact:: Unassigned

Votes:: 0 Vote for this issue

Watchers:: 6 Start watching this issue

Created:: 2023/12/13 8:36 PM

Updated:: 2024/10/15 8:19 AM

Resolved:: 2024/10/01 8:44 PM

Details

Description

Attachments

Easy Agile Planning Poker

Activity

People

Dates