Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-4964

Managed Cluster config-policy-controller crashing with `panic: interface conversion`

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Undefined Undefined
    • None
    • None
    • GRC
    • None
    • 1
    • False
    • None
    • False
    • GRC Sprint 2023-07
    • -
    • No

      Description of problem:

      config-policy-controller pod on the managed cluster is persistently crashing due to a panic encountered during the handling of object templates. Managed clusters are not having their policies enforced as a result of the issue.

      2023-04-14T19:32:21.994166875Z 2023-04-14T19:32:21.993Z    info    configuration-policy-controller    controllers/configurationpolicy_controller.go:844    Handling the object template for the relevant namespace    {"policy": "deploy-prometheusrules", "namespace": "openshift-cluster-version", "desiredName": "adp-cluster-version-operator", "index": 0}
      2023-04-14T19:32:21.999634603Z 2023-04-14T19:32:21.999Z    info    configuration-policy-controller    controllers/configurationpolicy_controller.go:844    Handling the object template for the relevant namespace    {"policy": "deploy-prometheusrules", "namespace": "openshift-ingress-operator", "desiredName": "adp-ingress-operator", "index": 1}
      2023-04-14T19:32:22.006100107Z 2023-04-14T19:32:22.006Z    info    configuration-policy-controller    logr@v1.2.2/logr.go:252    Processing the policies    {"count": 41}
      2023-04-14T19:32:22.007155176Z panic: interface conversion: interface {} is nil, not map[string]interface {}
      2023-04-14T19:32:22.007155176Z 
      2023-04-14T19:32:22.007155176Z goroutine 476 [running]:
      2023-04-14T19:32:22.007165793Z open-cluster-management.io/config-policy-controller/controllers.fmtMetadataForCompare(0x169d8e0?, 0xc0013ffad0?)
      2023-04-14T19:32:22.007165793Z     /remote-source/app/controllers/configurationpolicy_utils.go:385 +0x578
      2023-04-14T19:32:22.007165793Z open-cluster-management.io/config-policy-controller/controllers.handleSingleKey({0xc00082a500, 0x8}, {0xc00082a53c?}, 0xc000358148, {0xc0009631d0, 0x8})
      2023-04-14T19:32:22.007176484Z     /remote-source/app/controllers/configurationpolicy_controller.go:2110 +0x79f
      2023-04-14T19:32:22.007181693Z open-cluster-management.io/config-policy-controller/controllers.(*ConfigurationPolicyReconciler).checkAndUpdateResource(0xc001698f60?, {0xc001a166c0, {{0xc000046ba0, 0x15}, {0xc00043cd44, 0x2}, {0xc0008bb390, 0xf}}, 0xc000358148, {0xc001e82c78, ...}, ...}, ...)
      2023-04-14T19:32:22.007200474Z     /remote-source/app/controllers/configurationpolicy_controller.go:2209 +0x52f
      2023-04-14T19:32:22.007209784Z open-cluster-management.io/config-policy-controller/controllers.(*ConfigurationPolicyReconciler).handleSingleObj(0xc00022e240, {0xc001a166c0, {{0xc000046ba0, 0x15}, {0xc00043cd44, 0x2}, {0xc0008bb390, 0xf}}, 0xc000358148, {0xc001e82c78, ...}, ...}, ...)
      2023-04-14T19:32:22.007223463Z     /remote-source/app/controllers/configurationpolicy_controller.go:1361 +0x87f
      2023-04-14T19:32:22.007228492Z open-cluster-management.io/config-policy-controller/controllers.(*ConfigurationPolicyReconciler).handleObjects(0xc00022e240, 0xc000d81130, {0xc000fb5380, 0x1a}, {{0xc000aed720, 0xe}, {0xc001e82c78, 0x14}, {0xc000fb5380, 0x1a}, ...}, ...)
      2023-04-14T19:32:22.007242124Z     /remote-source/app/controllers/configurationpolicy_controller.go:1195 +0xe50
      2023-04-14T19:32:22.007251130Z open-cluster-management.io/config-policy-controller/controllers.(*ConfigurationPolicyReconciler).handleObjectTemplates(_, {{{0x154783b, 0x13}, {0xc000fc42a0, 0x24}}, {{0xc000022f48, 0x16}, {0x0, 0x0}, {0xc0009631b0, ...}, ...}, ...})
      2023-04-14T19:32:22.007264816Z     /remote-source/app/controllers/configurationpolicy_controller.go:851 +0x22ee
      2023-04-14T19:32:22.007269616Z open-cluster-management.io/config-policy-controller/controllers.(*ConfigurationPolicyReconciler).handlePolicyWorker(0x8e172a?, 0x199e4c8?, 0xc000668f88?)
      2023-04-14T19:32:22.007279439Z     /remote-source/app/controllers/configurationpolicy_controller.go:230 +0x125
      2023-04-14T19:32:22.007279439Z created by open-cluster-management.io/config-policy-controller/controllers.(*ConfigurationPolicyReconciler).PeriodicallyExecConfigPolicies
      2023-04-14T19:32:22.007279439Z     /remote-source/app/controllers/configurationpolicy_controller.go:189 +0x51f

      Version-Release number of selected component (if applicable):

      2.6.3

      How reproducible:

      Occurring on multiple managed clusters in this environment, not seen elsewhere to my knowledge.

      Steps to Reproduce:

      1.  
      2.  
      3. ...

      Actual results:

      Expected results:

      Additional info:

      Must-gathers linked in comments.

            jkulikau@redhat.com Justin Kulikauskas
            rhn-support-jayoung James Young
            Derek Ho Derek Ho
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: