Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-48111

ARO cluster (OVN) adding egress IP causes CNCC(openshift-cloud-network-config-controller) panic. Caused by missing permissions

    • None
    • SDN Sprint 267, SDN Sprint 268, CORENET Sprint 269
    • 3
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      How reproducible:

      Steps to Reproduce:

      • Create a CloudPrivateIPConfig

      Actual results:

       

      2025-01-06T22:29:20.517217043Z I0106 22:29:20.517183       1 cloudprivateipconfig_controller.go:352] CloudPrivateIPConfig: "xx.xx.xx.xx" will be added to node: "<hostname>"
      2025-01-06T22:29:20.517283023Z I0106 22:29:20.517183       1 cloudprivateipconfig_controller.go:352] CloudPrivateIPConfig: "xx.xx.xx.xx" will be added to node: "<hostname>"
      2025-01-06T22:29:20.518399247Z I0106 22:29:20.518379       1 cloudprivateipconfig_controller.go:352] CloudPrivateIPConfig: "xx.xx.xx.xx" will be added to node: "<hostname>"
      
      2025-01-06T22:29:20.527426582Z I0106 22:29:20.527401       1 azure.go:140] Acquiring node lock for assigning ip address, node: <hostname>, ip: xx.xx.xx.xx 
      2025-01-06T22:29:20.528282519Z I0106 22:29:20.528259       1 azure.go:140] Acquiring node lock for assigning ip address, node: <hostname>, ip: xx.xx.xx.xx 
      2025-01-06T22:29:20.528297571Z I0106 22:29:20.528278       1 azure.go:140] Acquiring node lock for assigning ip address, node: <hostname>, ip: xx.xx.xx.xx 
      
      2025-01-06T22:29:21.469837845Z E0106 22:29:21.469789       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)  2025-01-06T22:29:21.469837845Z goroutine 165 [running]:
      2025-01-06T22:29:21.469837845Z k8s.io/apimachinery/pkg/util/runtime.logPanic({0x2932e60?, 0x4b3ff80})
      2025-01-06T22:29:21.469837845Z     /go/src/github.com/openshift/cloud-network-config-controller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x85
      2025-01-06T22:29:21.469837845Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000b303a0?})
      2025-01-06T22:29:21.469837845Z     /go/src/github.com/openshift/cloud-network-config-controller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x6b
      2025-01-06T22:29:21.469837845Z panic({0x2932e60?, 0x4b3ff80?})
      2025-01-06T22:29:21.469837845Z     /usr/lib/golang/src/runtime/panic.go:914 +0x21f
      2025-01-06T22:29:21.469837845Z github.com/openshift/cloud-network-config-controller/pkg/cloudprovider.(*Azure).AssignPrivateIP(0xc000816000, {0xc0008d0330, 0x10, 0x10}, 0xc000622000)
      2025-01-06T22:29:21.469837845Z     /go/src/github.com/openshift/cloud-network-config-controller/pkg/cloudprovider/azure.go:186 +0xe6a
      2025-01-06T22:29:21.469837845Z github.com/openshift/cloud-network-config-controller/pkg/controller/cloudprivateipconfig.(*CloudPrivateIPConfigController).SyncHandler(0xc0002f54a0, {0xc000aa0740, 0xb})
      2025-01-06T22:29:21.469837845Z     /go/src/github.com/openshift/cloud-network-config-controller/pkg/controller/cloudprivateipconfig/cloudprivateipconfig_controller.go:397 +0x1d89
      2025-01-06T22:29:21.469837845Z github.com/openshift/cloud-network-config-controller/pkg/controller.(*CloudNetworkConfigController).processNextWorkItem.func1(0xc000b06780, {0x26a57a0?, 0xc000b303a0})
      2025-01-06T22:29:21.469837845Z     /go/src/github.com/openshift/cloud-network-config-controller/pkg/controller/controller.go:152 +0x10c 

       

       

      Expected results:

      No Panic....

      Additional info:

      Affected Platforms: OVNK + Azure + egressIP

      Customer Issue: EgressIP + Azure cloud causes cloud network config controller to panic.

      Root Cause:

      CredentialsRequest openshift-cloud-network-config-controller-azure is missing permission: Microsoft.Network/load-balancers/backendAddressPools/read

      Customer Has Confirmation that manually appending this permission to the workload identity has resolved the panic..

      https://github.com/openshift/cloud-network-config-controller/blob/8ceee78810756afaa2efb75f60bef0358502834f/pkg/cloudprovider/azure.go#L186

      Ironically is in the error handling branch. A quick look reveals that pool.Name is the only variable that is dereferenced without checking if it is not nil.

       

       

       

       

              rravaiol@redhat.com Riccardo Ravaioli
              rhn-support-tidawson Tim Dawson
              Jean Chen Jean Chen
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated: