-
Bug
-
Resolution: Done
-
Major
-
oadp 1.0.7
-
False
-
-
False
-
QE - Ack
-
oadp-velero-container-1.0.7-4
-
Passed
-
-
-
0
-
0
-
Very Likely
-
0
-
None
-
Unset
-
Unknown
-
Yes
Description of problem: Backup got stuck InProgress status after debugging it looks like the velero pod is getting restarted due to fatal error, see actual results.
Version-Release number of selected component (if applicable):
OADP 1.0.7
How reproducible:
Always
Steps to Reproduce:
1. Create an empty namespace
2. Create a backup
Actual results: Velero pod got restarted due to fatal error resulting backup stuck at InProgress status.
$ oc get pods -n openshift-adp NAME READY STATUS RESTARTS AGE oadp-ts-dpa-1-gcp-registry-5c68f98d5b-7nfhf 1/1 Running 0 16m openshift-adp-controller-manager-5d86d6bc57-464m6 1/1 Running 0 60m restic-7vp4q 1/1 Running 0 16m restic-r9szn 1/1 Running 0 16m restic-w2xkn 1/1 Running 0 16m velero-57bcb4f849-2wvrg 1/1 Running 1 (12m ago) 16m
$ oc logs velero-57bcb4f849-2wvrg --previous time="2023-01-16T11:20:37Z" level=info msg="Backed up a total of 23 items" backup=openshift-adp/test logSource="/remote-source/velero/app/pkg/backup/backup.go:423" progress= time="2023-01-16T11:20:37Z" level=info msg="Setting up backup store to persist the backup" backup=openshift-adp/test logSource="/remote-source/velero/app/pkg/controller/backup_controller.go:654" Failed to write to log, flate: closed writer unexpected fault address 0x0 fatal error: fault [signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x46935f]goroutine 1450 [running]: runtime.throw({0x1ff6611?, 0x1?}) /usr/lib/golang/src/runtime/panic.go:1047 +0x5d fp=0xc0015723f0 sp=0xc0015723c0 pc=0x437ebd runtime.sigpanic() /usr/lib/golang/src/runtime/signal_unix.go:842 +0x2c5 fp=0xc001572440 sp=0xc0015723f0 pc=0x44e6e5 aeshashbody() /usr/lib/golang/src/runtime/asm_amd64.s:1366 +0x39f fp=0xc001572448 sp=0xc001572440 pc=0x46935f runtime.mapiternext(0xc0004a4c40) /usr/lib/golang/src/runtime/map.go:936 +0x2eb fp=0xc0015724b8 sp=0xc001572448 pc=0x4100ab runtime.mapiterinit(0x9148a2?, 0xc001045ba0?, 0xc00099760a?) /usr/lib/golang/src/runtime/map.go:863 +0x236 fp=0xc0015724d8 sp=0xc0015724b8 pc=0x40fd76 reflect.mapiterinit(0x9296d6?, 0x1d28960?, 0xc0004a4c40?) /usr/lib/golang/src/runtime/map.go:1375 +0x19 fp=0xc001572500 sp=0xc0015724d8 pc=0x4659b9 github.com/modern-go/reflect2.(*UnsafeMapType).UnsafeIterate(...) /remote-source/velero/deps/gomod/pkg/mod/github.com/modern-go/reflect2@v1.0.1/unsafe_map.go:112 github.com/json-iterator/go.(*sortKeysMapEncoder).IsEmpty(0x11?, 0x1ad89d0?) /remote-source/velero/deps/gomod/pkg/mod/github.com/json-iterator/go@v1.1.11/reflect_map.go:333 +0x28 fp=0xc001572540 sp=0xc001572500 pc=0x91caa8 github.com/json-iterator/go.(*placeholderEncoder).IsEmpty(0x40ae6f?, 0xc000816020?) /remote-source/velero/deps/gomod/pkg/mod/github.com/json-iterator/go@v1.1.11/reflect.go:336 +0x22 fp=0xc001572560 sp=0xc001572540 pc=0x914902 github.com/json-iterator/go.(*structFieldEncoder).IsEmpty(0xc001016c00, 0x1d1ecd7?) /remote-source/velero/deps/gomod/pkg/mod/github.com/json-iterator/go@v1.1.11/reflect_struct_encoder.go:118 +0x42 fp=0xc001572580 sp=0xc001572560 pc=0x929882 github.com/json-iterator/go.(*structEncoder).Encode(0xc001016d50, 0x1?, 0xc000f8e3c0) /remote-source/velero/deps/gomod/pkg/mod/github.com/json-iterator/go@v1.1.11/reflect_struct_encoder.go:148 +0x565 fp=0xc001572668 sp=0xc001572580 pc=0x929ee5 github.com/json-iterator/go.(*structFieldEncoder).Encode(0xc000ef00c0, 0x1b53fde?, 0xc000f8e3c0) /remote-source/velero/deps/gomod/pkg/mod/github.com/json-iterator/go@v1.1.11/reflect_struct_encoder.go:110 +0x56 fp=0xc0015726e0 sp=0xc001572668 pc=0x9296d6 github.com/json-iterator/go.(*structEncoder).Encode(0xc000ef0210, 0x0?, 0xc000f8e3c0) /remote-source/velero/deps/gomod/pkg/mod/github.com/json-iterator/go@v1.1.11/reflect_struct_encoder.go:158 +0x765 fp=0xc0015727c8 sp=0xc0015726e0 pc=0x92a0e5 github.com/json-iterator/go.(*OptionalEncoder).Encode(0xc000a90460?, 0x0?, 0x0?) /remote-source/velero/deps/gomod/pkg/mod/github.com/json-iterator/go@v1.1.11/reflect_optional.go:70 +0xb0 fp=0xc001572818 sp=0xc0015727c8 pc=0x921230 github.com/json-iterator/go.(*onePtrEncoder).Encode(0xc00041e6e0, 0xc000816000, 0xc0010237d0?) /remote-source/velero/deps/gomod/pkg/mod/github.com/json-iterator/go@v1.1.11/reflect.go:219 +0x82 fp=0xc001572850 sp=0xc001572818 pc=0x913e62 github.com/json-iterator/go.(*Stream).WriteVal(0xc000f8e3c0, {0x1fb18e0, 0xc000816000}) /remote-source/velero/deps/gomod/pkg/mod/github.com/json-iterator/go@v1.1.11/reflect.go:98 +0x166 fp=0xc0015728c0 sp=0xc001572850 pc=0x913186 github.com/json-iterator/go.(*frozenConfig).Marshal(0xc000a90460, {0x1fb18e0, 0xc000816000}) /remote-source/velero/deps/gomod/pkg/mod/github.com/json-iterator/go@v1.1.11/config.go:299 +0xc9 fp=0xc001572958 sp=0xc0015728c0 pc=0x90a289 github.com/json-iterator/go.(*frozenConfig).MarshalIndent(0xc0001cf900, {0x1fb18e0, 0xc000816000}, {0x0?, 0x1c70d60?}, {0x1ff414f, 0x2}) /remote-source/velero/deps/gomod/pkg/mod/github.com/json-iterator/go@v1.1.11/config.go:320 +0x131 fp=0xc0015729d0 sp=0xc001572958 pc=0x90a571 k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).doEncode(0x1ae0146?, {0x23a5aa0?, 0xc000816000?}, {0x239c060, 0xc000a420f0}) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/runtime/serializer/json/json.go:319 +0x111 fp=0xc001572a68 sp=0xc0015729d0 pc=0xd786b1 k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Encode(0xc000403950, {0x23a5aa0, 0xc000816000}, {0x239c060, 0xc000a420f0}) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/runtime/serializer/json/json.go:301 +0xfc fp=0xc001572ac8 sp=0xc001572a68 pc=0xd7853c k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*codec).doEncode(0xc000a903c0, {0x23a5aa0?, 0xc000816000}, {0x239c060, 0xc000a420f0}) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/runtime/serializer/versioning/versioning.go:244 +0x946 fp=0xc001572e50 sp=0xc001572ac8 pc=0xd7d5e6 k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*codec).Encode(0xc000a903c0, {0x23a5aa0, 0xc000816000}, {0x239c060, 0xc000a420f0}) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/runtime/serializer/versioning/versioning.go:184 +0x106 fp=0xc001572eb0 sp=0xc001572e50 pc=0xd7cc46 github.com/vmware-tanzu/velero/pkg/util/encode.EncodeTo({0x23a5aa0, 0xc000816000}, {0x1ff56ad?, 0xffffffffffffffff?}, {0x239c060, 0xc000a420f0}) /remote-source/velero/app/pkg/util/encode/encode.go:50 +0x6e fp=0xc001572f00 sp=0xc001572eb0 pc=0x14f4c4e github.com/vmware-tanzu/velero/pkg/controller.persistBackup(0xc0002e9540, 0xc000482530, 0xc000482528, {0x23cb868, 0xc000f93680}, {0xc0008853f8?, 0xc00011e288?}, {0x0, 0x0, 0x0}, ...) /remote-source/velero/app/pkg/controller/backup_controller.go:700 +0x92 fp=0xc0015730b0 sp=0xc001572f00 pc=0x19d5792 github.com/vmware-tanzu/velero/pkg/controller.(*backupController).runBackup(0xc000764fc0, 0xc0002e9540) /remote-source/velero/app/pkg/controller/backup_controller.go:660 +0x1676 fp=0xc0015736b0 sp=0xc0015730b0 pc=0x19d4f56 github.com/vmware-tanzu/velero/pkg/controller.(*backupController).processBackup(0xc000764fc0, {0xc0007e7c98, 0x12}) /remote-source/velero/app/pkg/controller/backup_controller.go:274 +0x7e5 fp=0xc001573c60 sp=0xc0015736b0 pc=0x19d0c45 github.com/vmware-tanzu/velero/pkg/controller.(*backupController).processBackup-fm({0xc0007e7c98?, 0x5e1136?}) <autogenerated>:1 +0x34 fp=0xc001573c88 sp=0xc001573c60 pc=0x19f9714 github.com/vmware-tanzu/velero/pkg/controller.(*genericController).processNextWorkItem(0xc000a29b00) /remote-source/velero/app/pkg/controller/generic_controller.go:132 +0xeb fp=0xc001573e60 sp=0xc001573c88 pc=0x19e5b8b github.com/vmware-tanzu/velero/pkg/controller.(*genericController).runWorker(0xc000a04ea8?) /remote-source/velero/app/pkg/controller/generic_controller.go:119 +0x25 fp=0xc001573e80 sp=0xc001573e60 pc=0x19e5a65 github.com/vmware-tanzu/velero/pkg/controller.(*genericController).runWorker-fm() <autogenerated>:1 +0x26 fp=0xc001573e98 sp=0xc001573e80 pc=0x19f9f26 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000b24d80?) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:155 +0x3e fp=0xc001573eb8 sp=0xc001573e98 pc=0x1174a9e k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x23bcdd8?, {0x239f460, 0xc000f08420}, 0x1, 0xc000a86fc0) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:156 +0xb6 fp=0xc001573f38 sp=0xc001573eb8 pc=0x1174976 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x23bcdd8?, 0x3b9aca00, 0x0, 0xe0?, 0x19662c0?) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:133 +0x89 fp=0xc001573f88 sp=0xc001573f38 pc=0x1174869 k8s.io/apimachinery/pkg/util/wait.Until(...) .s:1594 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0x46be21 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1159 +0x25goroutine 17 [GC worker (idle)]: runtime.gopark(0xaf14d5f28b6?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/lib/golang/src/runtime/proc.go:363 +0xd6 fp=0xc000072750 sp=0xc000072730 pc=0x43aad6 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1235 +0xf1 fp=0xc0000727e0 sp=0xc000072750 pc=0x41c031 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000727e8 sp=0xc0000727e0 pc=0x46be21 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1159 +0x25goroutine 18 [GC worker (idle)]: runtime.gopark(0xaf14d24b1e5?, 0x3?, 0xfd?, 0xa0?, 0x0?) /usr/lib/golang/src/runtime/proc.go:363 +0xd6 fp=0xc000072f50 sp=0xc000072f30 pc=0x43aad6 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1235 +0xf1 fp=0xc000072fe0 sp=0xc000072f50 pc=0x41c031 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000072fe8 sp=0xc000072fe0 pc=0x46be21 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1159 +0x25goroutine 10 [GC worker (idle)]: runtime.gopark(0xaf14d5f30e3?, 0x3?, 0x3d?, 0x5b?, 0x0?) /usr/lib/golang/src/runtime/proc.go:363 +0xd6 fp=0xc000504750 sp=0xc000504730 pc=0x43aad6 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1235 +0xf1 fp=0xc0005047e0 sp=0xc000504750 pc=0x41c031 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005047e8 sp=0xc0005047e0 pc=0x46be21 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1159 +0x25goroutine 1400 [select]: runtime.gopark(0xc000b61f58?, 0x2?, 0x0?, 0x0?, 0xc000b61f14?) /usr/lib/golang/src/runtime/proc.go:363 +0xd6 fp=0xc000b61d90 sp=0xc000b61d70 pc=0x43aad6 runtime.selectgo(0xc000b61f58, 0xc000b61f10, 0x1cd24a0?, 0x1, 0xc000563700?, 0x1) /usr/lib/golang/src/runtime/select.go:328 +0x7bc fp=0xc000b61ed0 sp=0xc000b61d90 pc=0x44a97c k8s.io/client-go/tools/cache.(*processorListener).pop(0xc000c81400) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/shared_informer.go:742 +0x119 fp=0xc000b61f98 sp=0xc000b61ed0 pc=0x14e3cf9 k8s.io/client-go/tools/cache.(*processorListener).pop-fm() <autogenerated>:1 +0x26 fp=0xc000b61fb0 sp=0xc000b61f98 pc=0x14e9e06 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:73 +0x5a fp=0xc000b61fe0 sp=0xc000b61fb0 pc=0x11746da runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000b61fe8 sp=0xc000b61fe0 pc=0x46be21 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:71 +0x85goroutine 1536 [select, 3 minutes]: runtime.gopark(0xc0007c9f98?, 0x3?, 0xb8?, 0x50?, 0xc0007c9ef2?) /usr/lib/golang/src/runtime/proc.go:363 +0xd6 fp=0xc0007c9d50 sp=0xc0007c9d30 pc=0x43aad6 runtime.selectgo(0xc0007c9f98, 0xc0007c9eec, 0x0?, 0x0, 0x14dbe40?, 0x1) /usr/lib/golang/src/runtime/select.go:328 +0x7bc fp=0xc0007c9e90 sp=0xc0007c9d50 pc=0x44a97c k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2() /remote-source/velero/deps/gomod/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:373 +0x130 fp=0xc0007c9fe0 sp=0xc0007c9e90 pc=0x14dcdd0 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0007c9fe8 sp=0xc0007c9fe0 pc=0x46be21 created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch /remote-source/velero/deps/gomod/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:367 +0x39fgoroutine 1504 [sync.Cond.Wait, 3 minutes]: runtime.gopark(0xc000f77bd0?, 0xc000f77c00?, 0x3d?, 0xac?, 0x1e023a0?) /usr/lib/golang/src/runtime/proc.go:363 +0xd6 fp=0xc000877b68 sp=0xc000877b48 pc=0x43aad6 runtime.goparkunlock(...) /usr/lib/golang/src/runtime/proc.go:369 sync.runtime_notifyListWait(0xc000b534d0, 0x0) /usr/lib/golang/src/runtime/sema.go:517 +0x14c fp=0xc000877bb0 sp=0xc000877b68 pc=0x4680ac sync.(*Cond).Wait(0x23acf30?) /usr/lib/golang/src/sync/cond.go:70 +0x8c fp=0xc000877be8 sp=0xc000877bb0 pc=0x47422c k8s.io/client-go/util/workqueue.(*Type).Get(0xc000a29380) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/client-go@v0.22.2/util/workqueue/queue.go:151 +0x9e fp=0xc000877c58 sp=0xc000877be8 pc=0x117a85e k8s.io/client-go/util/workqueue.(*delayingType).Get(0xc0006bed60?) <autogenerated>:1 +0x2a fp=0xc000877c70 sp=0xc000877c58 pc=0x117c16a k8s.io/client-go/util/workqueue.(*rateLimitingType).Get(0xc000cbb8c0?) <autogenerated>:1 +0x2a fp=0xc000877c88 sp=0xc000877c70 pc=0x117cc8a github.com/vmware-tanzu/velero/pkg/controller.(*genericController).processNextWorkItem(0xc000a29560) /remote-source/velero/app/pkg/controller/generic_controller.go:124 +0x5b fp=0xc000877e60 sp=0xc000877c88 pc=0x19e5afb github.com/vmware-tanzu/velero/pkg/controller.(*genericController).runWorker(0xc000846ea8?) /remote-source/velero/app/pkg/controller/generic_controller.go:119 +0x25 fp=0xc000877e80 sp=0xc000877e60 pc=0x19e5a65 github.com/vmware-tanzu/velero/pkg/controller.(*genericController).runWorker-fm() <autogenerated>:1 +0x26 fp=0xc000877e98 sp=0xc000877e80 pc=0x19f9f26 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0004c8810?) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:155 +0x3e fp=0xc000877eb8 sp=0xc000877e98 pc=0x1174a9e k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x23bcdd8?, {0x239f460, 0xc0009073e0}, 0x1, 0xc000a86fc0) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:156 +0xb6 fp=0xc000877f38 sp=0xc000877eb8 pc=0x1174976 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x23bcdd8?, 0x3b9aca00, 0x0, 0xe0?, 0x19662c0?) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:133 +0x89 fp=0xc000877f88 sp=0xc000877f38 pc=0x1174869 k8s.io/apimachinery/pkg/util/wait.Until(...) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:90 github.com/vmware-tanzu/velero/pkg/controller.(*genericController).Run.func2() /remote-source/velero/app/pkg/controller/generic_controller.go:92 +0x6e fp=0xc000877fe0 sp=0xc000877f88 pc=0x19e582e runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000877fe8 sp=0xc000877fe0 pc=0x46be21 created by github.com/vmware-tanzu/velero/pkg/controller.(*genericController).Run /remote-source/velero/app/pkg/controller/generic_controller.go:91 +0x45agoroutine 1696 [select]: runtime.gopark(0xc00196ff00?, 0x2?, 0x1e?, 0x0?, 0xc00196fed4?) /usr/lib/golang/src/runtime/proc.go:363 +0xd6 fp=0xc00196fd60 sp=0xc00196fd40 pc=0x43aad6 runtime.selectgo(0xc00196ff00, 0xc00196fed0, 0x556c25?, 0x0, 0xc001140000?, 0x1) /usr/lib/golang/src/runtime/select.go:328 +0x7bc fp=0xc00196fea0 sp=0xc00196fd60 pc=0x44a97c google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00061fe00, 0x1) /remote-source/velero/deps/gomod/pkg/mod/google.golang.org/grpc@v1.40.0/internal/transport/controlbuf.go:406 +0x115 fp=0xc00196ff30 sp=0xc00196fea0 pc=0x1531095 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00032e8a0) /remote-source/velero/deps/gomod/pkg/mod/google.golang.org/grpc@v1.40.0/internal/transport/controlbuf.go:533 +0x85 fp=0xc00196ff88 sp=0xc00196ff30 pc=0x15317a5 google.golang.org/grpc/internal/transport.newHTTP2Client.func3() /remote-source/velero/deps/gomod/pkg/mod/google.golang.org/grpc@v1.40.0/internal/transport/http2_client.go:405 +0x6a fp=0xc00196ffe0 sp=0xc00196ff88 pc=0x153adca runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00196ffe8 sp=0xc00196ffe0 pc=0x46be21 created by google.golang.org/grpc/internal/transport.newHTTP2Client /remote-source/velero/deps/gomod/pkg/mod/google.golang.org/grpc@v1.40.0/internal/transport/http2_client.go:403 +0x1f94goroutine 1446 [sync.Cond.Wait, 1 minutes]: runtime.gopark(0xc000d82900?, 0x23a9a88?, 0xe0?, 0x1c?, 0x239f2a0?) /usr/lib/golang/src/runtime/proc.go:363 +0xd6 fp=0xc0009b9a30 sp=0xc0009b9a10 pc=0x43aad6 runtime.goparkunlock(...) /usr/lib/golang/src/runtime/proc.go:369 sync.runtime_notifyListWait(0xc000894c48, 0x3) /usr/lib/golang/src/runtime/sema.go:517 +0x14c fp=0xc0009b9a78 sp=0xc0009b9a30 pc=0x4680ac sync.(*Cond).Wait(0x7b?) /usr/lib/golang/src/sync/cond.go:70 +0x8c fp=0xc0009b9ab0 sp=0xc0009b9a78 pc=0x47422c golang.org/x/net/http2.(*pipe).Read(0xc000894c30, {0xc000d98601, 0x5ff, 0x5ff}) /remote-source/velero/deps/gomod/pkg/mod/golang.org/x/net@v0.0.0-20220722155237-a158d28d115b/http2/pipe.go:76 +0xeb fp=0xc0009b9b18 sp=0xc0009b9ab0 pc=0x99984b golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc000d98601?, 0x2?, 0x1ffe8e4?}) /remote-source/velero/deps/gomod/pkg/mod/golang.org/x/net@v0.0.0-20220722155237-a158d28d115b/http2/transport.go:2406 +0x85 fp=0xc0009b9bb0 sp=0xc0009b9b18 pc=0x9a74a5 encoding/json.(*Decoder).refill(0xc000002b40) /usr/lib/golang/src/encoding/json/stream.go:165 +0x188 fp=0xc0009b9c00 sp=0xc0009b9bb0 pc=0x5dcb28 encoding/json.(*Decoder).readValue(0xc000002b40) /usr/lib/golang/src/encoding/json/stream.go:140 +0xbb fp=0xc0009b9c50 sp=0xc0009b9c00 pc=0x5dc71b encoding/json.(*Decoder).Decode(0xc000002b40, {0x1d25060, 0xc000a4aa08}) /usr/lib/golang/src/encoding/json/stream.go:63 +0x78 fp=0xc0009b9c80 sp=0xc0009b9c50 pc=0x5dc358 k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000aeed20, {0xc0000d2800, 0x400, 0x400}) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/framer/framer.go:152 +0x19c fp=0xc0009b9cd8 sp=0xc0009b9c80 pc=0xd76a7c k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000f80d70, 0xc0009dc120?, {0x23a9970, 0xc000cd7780}) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7 fp=0xc0009b9d80 sp=0xc0009b9cd8 pc=0x118b1a7 k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000fae120) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/client-go@v0.22.2/rest/watch/decoder.go:49 +0x4f fp=0xc0009b9de0 sp=0xc0009b9d80 pc=0x118b5af k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc00012eac0) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/watch/streamwatcher.go:105 +0xe5 fp=0xc0009b9fc8 sp=0xc0009b9de0 pc=0x9b4785 k8s.io/apimachinery/pkg/watch.NewStreamWatcher.func1() /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/watch/streamwatcher.go:76 +0x26 fp=0xc0009b9fe0 sp=0xc0009b9fc8 pc=0x9b44e6 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0009b9fe8 sp=0xc0009b9fe0 pc=0x46be21 created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/watch/streamwatcher.go:76 +0x130goroutine 1555 [select, 3 minutes]: runtime.gopark(0xc000b14f98?, 0x3?, 0xd8?, 0xc7?, 0xc000b14ef2?) /usr/lib/golang/src/runtime/proc.go:363 +0xd6 fp=0xc000b14d50 sp=0xc000b14d30 pc=0x43aad6 runtime.selectgo(0xc000b14f98, 0xc000b14eec, 0x32ef580?, 0x0, 0x14dbe40?, 0x1) /usr/lib/golang/src/runtime/select.go:328 +0x7bc fp=0xc000b14e90 sp=0xc000b14d50 pc=0x44a97c k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2() /remote-source/velero/deps/gomod/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:373 +0x130 fp=0xc000b14fe0 sp=0xc000b14e90 pc=0x14dcdd0 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000b14fe8 sp=0xc000b14fe0 pc=0x46be21 created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch /remote-source/velero/deps/gomod/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:367 +0x39fgoroutine 1497 [chan receive, 3 minutes]: runtime.gopark(0xc000764ea0?, 0xc0009b9e40?, 0x3f?, 0xd8?, 0x0?) /usr/lib/golang/src/runtime/proc.go:363 +0xd6 fp=0xc000898dc0 sp=0xc000898da0 pc=0x43aad6 runtime.chanrecv(0xc000a86fc0, 0x0, 0x1) /usr/lib/golang/src/runtime/chan.go:583 +0x49b fp=0xc000898e50 sp=0xc000898dc0 pc=0x406dbb runtime.chanrecv1(0x23?, 0xc0006bee10?) /usr/lib/golang/src/runtime/chan.go:442 +0x18 fp=0xc000898e78 sp=0xc000898e50 pc=0x4068b8 github.com/vmware-tanzu/velero/pkg/controller.(*genericController).Run(0xc000a29b00, {0x23bcdd8?, 0xc0008cca80}, 0x1) /remote-source/velero/app/pkg/controller/generic_controller.go:111 +0x405 fp=0xc000898f30 sp=0xc000898e78 pc=0x19e55a5 github.com/vmware-tanzu/velero/internal/util/managercontroller.Runnable.func1({0x23bcdd8?, 0xc0008cca80?}) /remote-source/velero/app/internal/util/managercontroller/managercontroller.go:34 +0x39 fp=0xc000898f60 sp=0xc000898f30 pc=0x19fd579 sigs.k8s.io/controller-runtime/pkg/manager.RunnableFunc.Start(0xc000a04fb0?, {0x23bcdd8?, 0xc0008cca80?}) /remote-source/velero/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/manager/manager.go:297 +0x2c fp=0xc000898f80 sp=0xc000898f60 pc=0x196634c sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1() /remote-source/velero/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/manager/internal.go:696 +0x7d fp=0xc000898fe0 sp=0xc000898f80 pc=0x196625d runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000898fe8 sp=0xc000898fe0 pc=0x46be21 created by sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable /remote-source/velero/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/manager/internal.go:694 +0xadgoroutine 1662 [select, 1 minutes]: runtime.gopark(0xc000ed2f58?, 0x2?, 0x63?, 0x6f?, 0xc000ed2f4c?) /usr/lib/golang/src/runtime/proc.go:363 +0xd6 fp=0xc000ed2dd8 sp=0xc000ed2db8 pc=0x43aad6 runtime.selectgo(0xc000ed2f58, 0xc000ed2f48, 0x65623176223a226e?, 0x0, 0x223a22656d616e22?, 0x1) /usr/lib/golang/src/runtime/select.go:328 +0x7bc fp=0xc000ed2f18 sp=0xc000ed2dd8 pc=0x44a97c github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).Recv(0x3a226e6f69737265?) /remote-source/velero/deps/gomod/pkg/mod/github.com/hashicorp/go-plugin@v0.0.0-20190610192547-a1bc61569a26/grpc_broker.go:231 +0x67 fp=0xc000ed2f88 sp=0xc000ed2f18 pc=0x15e4567 github.com/hashicorp/go-plugin.(*GRPCBroker).Run(0xc0000a32c0) /remote-source/velero/deps/gomod/pkg/mod/github.com/hashicorp/go-plugin@v0.0.0-20190610192547-a1bc61569a26/grpc_broker.go:411 +0x44 fp=0xc000ed2fc8 sp=0xc000ed2f88 pc=0x15e52c4 github.com/hashicorp/go-plugin.newGRPCClient.func1() /remote-source/velero/deps/gomod/pkg/mod/github.com/hashicorp/go-plugin@v0.0.0-20190610192547-a1bc61569a26/grpc_client.go:56 +0x26 fp=0xc000ed2fe0 sp=0xc000ed2fc8 pc=0x15e5fe6 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000ed2fe8 sp=0xc000ed2fe0 pc=0x46be21 created by github.com/hashicorp/go-plugin.newGRPCClient /remote-source/velero/deps/gomod/pkg/mod/github.com/hashicorp/go-plugin@v0.0.0-20190610192547-a1bc61569a26/grpc_client.go:56 +0x285goroutine 1410 [select, 1 minutes]: runtime.gopark(0xc000ac9980?, 0x3?, 0xb?, 0x0?, 0xc000ac974a?) /usr/lib/golang/src/runtime/proc.go:363 +0xd6 fp=0xc000ac9508 sp=0xc000ac94e8 pc=0x43aad6 runtime.selectgo(0xc000ac9980, 0xc000ac9744, 0xc0001cefa0?, 0x0, 0xc000154e40?, 0x1) /usr/lib/golang/src/runtime/select.go:328 +0x7bc fp=0xc000ac9648 sp=0xc000ac9508 pc=0x44a97c k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc0009600e0, {0x0?, 0x0?, 0x341b000?}, {0x23a9bf0?, 0xc00031ca00}, 0xc000ac9bc8, 0xc0003d8b40, 0xc0004d8120) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:468 +0x1b0 fp=0xc000ac9a88 sp=0xc000ac9648 pc=0x14de290 k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0009600e0, 0xc0004d8120) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:428 +0x6d6 fp=0xc000ac9d78 sp=0xc000ac9a88 pc=0x14dc676 k8s.io/client-go/tools/cache.(*Reflector).Run.func1() /remote-source/velero/deps/gomod/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:221 +0x26 fp=0xc000ac9da8 sp=0xc000ac9d78 pc=0x14dbe66 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x14?) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:155 +0x3e fp=0xc000ac9dc8 sp=0xc000ac9da8 pc=0x1174a9e k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000724ec0?, {0x239f440, 0xc000b27450}, 0x1, 0xc0004d8120) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:156 +0xb6 fp=0xc000ac9e48 sp=0xc000ac9dc8 pc=0x1174976 k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0009600e0, 0xc0004d8120) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:220 +0x225 fp=0xc000ac9f78 sp=0xc000ac9e48 pc=0x14dbc65 k8s.io/client-go/tools/cache.(*Reflector).Run-fm(0x0?) <autogenerated>:1 +0x2c fp=0xc000ac9f98 sp=0xc000ac9f78 pc=0x14e93ac k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1() /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:56 +0x22 fp=0xc000ac9fb0 sp=0xc000ac9f98 pc=0x1174462 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:73 +0x5a fp=0xc000ac9fe0 sp=0xc000ac9fb0 pc=0x11746da runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000ac9fe8 sp=0xc000ac9fe0 pc=0x46be21 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:71 +0x85goroutine 1505 [select, 3 minutes]: runtime.gopark(0xc000a00f18?, 0x2?, 0x3f?, 0xd8?, 0xc000a00f04?) /usr/lib/golang/src/runtime/proc.go:363 +0xd6 fp=0xc000a00d88 sp=0xc000a00d68 pc=0x43aad6 runtime.selectgo(0xc000a00f18, 0xc000a00f00, 0x30?, 0x0, 0x30?, 0x1) /usr/lib/golang/src/runtime/select.go:328 +0x7bc fp=0xc000a00ec8 sp=0xc000a00d88 pc=0x44a97c k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x56656761726f7473?, {0x239f460, 0xc000907380}, 0x1, 0xc000a86fc0) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:167 +0x135 fp=0xc000a00f48 sp=0xc000a00ec8 pc=0x11749f5 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x6e222c22223a2265?, 0x45d964b800, 0x0, 0x22?, 0x726f74617265704f?) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:133 +0x89 fp=0xc000a00f98 sp=0xc000a00f48 pc=0x1174869 k8s.io/apimachinery/pkg/util/wait.Until(...) /remote-source/velero/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:90 github.com/vmware-tanzu/velero/pkg/controller.(*genericController).Run.func3() /remote-source/velero/app/pkg/controller/generic_controller.go:106 +0x54 fp=0xc000a00fe0 sp=0xc000a00f98 pc=0x19e5794 runtime.goexit()
Expected results:
Backup should be completed successfully.
Additional info:
