XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • rhos-18.0.4
    • None
    • test-operator
    • None
    • 4
    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • ?
    • ?
    • ?
    • None
    • Important

      When loading an AnsibleTest CR like this:

      apiVersion: test.openstack.org/v1beta1
      kind: AnsibleTest
      metadata:
        creationTimestamp: "2024-11-04T17:34:57Z"
        generation: 1
        name: ansibletest
        namespace: openstack
        resourceVersion: "6215831"
        uid: 71534195-6d0f-4db8-be79-9b1e8fcaf195
      spec:
        ansibleCollections: ""
        ansibleExtraVars: ""
        ansibleGitRepo: https://github.com/NFV18/ansible-nfv
        ansibleInventory: |
          localhost ansible_connection=local ansible_python_interpreter=python3
          undercloud-0 ansible_connection=local ansible_python_interpreter=python3
          compute-0 ansible_host=192.168.122.100 ansible_user=cloud-admin ansible_ssh_private_key_file=/var/lib/ansible/.ssh/compute_id
          compute-1 ansible_host=192.168.122.101 ansible_user=cloud-admin ansible_ssh_private_key_file=/var/lib/ansible/.ssh/compute_id
          [compute]
          compute-0
          compute-1
          [local]
          localhost
          [undercloud]
          undercloud-0
        ansiblePlaybookPath: playbooks/packet_gen/trex/performance_scenario.yml
        ansibleVarFiles: |
          ---
          # Use exist cloud resources
          cloud_resources: external
          # DUT compute
          dut_compute: compute-1
          # Trex config
          trex_lcores: "2-11"
          # testpmd path (for rhel 8.4 vm)
          testpmd_bin: "/root/dpdk/build/app/dpdk-testpmd"
          # Disable search rates higher than the specified
          binary_search_disable_upward_search: true
          # Used to balance queues
          binary_search_warmup_trial_runtime: 120
          discover_instance_external_ip: true
          ssh_key: test_keypair.key
          dynamic_instances:
            - name: trex
              group: trex
              user: cloud-user
            - name: testpmd-dpdk-dut
              group: dpdk_dut
              user: cloud-user
            - name: testpmd-sriov-vf-dut
              group: sriov_dut
              user: cloud-user
        backoffLimit: 0
        computeSSHKeySecretName: dataplane-ansible-ssh-private-key-secret
        containerImage: quay.io/podified-master-centos9/openstack-ansible-tests:current-podified
        debug: true
        openStackConfigMap: openstack-config
        openStackConfigSecret: openstack-config-secret
        privileged: true
        storageClass: lvms-local-storage
        workflow:
        - ansibleExtraVars: ' -e manual_run=false -e binary_perf_log=/var/lib/AnsibleTests/external_files/dpdk_performance.log
            -e dut_group=dpdk_dut -e dut_type=dpdk -e testpmd_lcores=3,4,5 -e trex_rate=2
            -e emc_insert_inv_prob=100 -e clone_traffic_gen_repo=false -e private_key_fetch_location=~/
            -e python_interperter=/usr/bin/python3 '
          backoffLimit: 0
          computeSSHKeySecretName: ""
          containerImage: ""
          openStackConfigMap: openstack-config
          openStackConfigSecret: openstack-config-secret
          stepName: dpdk
          storageClass: local-storage
          workloadSSHKeySecretName: ""
        - ansibleExtraVars: ' -e manual_run=false -e binary_perf_log=/var/lib/AnsibleTests/external_files/sriov_performance.log
            -e dut_group=sriov_dut -e dut_type=sriov -e testpmd_lcores=1,2,3 -e trex_rate=15
            -e clone_traffic_gen_repo=false -e private_key_fetch_location=~/ -e python_interperter=/usr/bin/python3 '
          backoffLimit: 0
          computeSSHKeySecretName: ""
          containerImage: ""
          openStackConfigMap: openstack-config
          openStackConfigSecret: openstack-config-secret
          stepName: sriov
          storageClass: local-storage
          workloadSSHKeySecretName: ""
        workloadSSHKeySecretName: open-tofu-ssh-keys

      We hit this error:

      [zuul@controller-0 ~]$ oc logs -f pod/test-operator-controller-manager-bd7d49858-wgvwt
      ...
      2024-11-04T17:41:03Z    INFO    Starting workers        {"controller": "ansibletest", "controllerGroup": "test.openstack.org", "controllerKind": "AnsibleTest", "worker count": 1}
      2024-11-04T17:41:04Z    INFO    Controllers.AnsibleTest Can not acquire lock    {"controller": "ansibletest", "controllerGroup": "test.openstack.org", "controllerKind": "AnsibleTest", "AnsibleTest": {"name":"ansibletest","namespace":"openstack"}, "namespace": "openstack", "name": "ansibletest", "reconcileID": "a3409fcd-92cc-4223-bc56-bec66c179a9c"}
      2024-11-04T17:42:04Z    INFO    Controllers.AnsibleTest Can not acquire lock    {"controller": "ansibletest", "controllerGroup": "test.openstack.org", "controllerKind": "AnsibleTest", "AnsibleTest": {"name":"ansibletest","namespace":"openstack"}, "namespace": "openstack", "name": "ansibletest", "reconcileID": "b987f984-23ce-4b8d-a261-c4004df61474"}
      2024-11-04T17:42:59Z    INFO    ansibletest-resource    default {"name": "ansibletest"}
      2024-11-04T17:42:59Z    INFO    ansibletest-resource    validate create {"name": "ansibletest"}
      2024-11-04T17:42:59Z    INFO    Controllers.AnsibleTest Lock acquired   {"controller": "ansibletest", "controllerGroup": "test.openstack.org", "controllerKind": "AnsibleTest", "AnsibleTest": {"name":"ansibletest","namespace":"openstack"}, "namespace": "openstack", "name": "ansibletest", "reconcileID": "11971c22-136a-48b3-876b-61af8fa12467"}
      2024-11-04T17:42:59Z    INFO    Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference        {"controller": "ansibletest", "controllerGroup": "test.openstack.org", "controllerKind": "AnsibleTest", "AnsibleTest": {"name":"ansibletest","namespace":"openstack"}, "namespace": "openstack", "name": "ansibletest", "reconcileID": "11971c22-136a-48b3-876b-61af8fa12467"}
      panic: runtime error: invalid memory address or nil pointer dereference [recovered]
              panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x170ba9c]
      goroutine 316 [running]:
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
              /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.6/pkg/internal/controller/controller.go:116 +0x1e5
      panic({0x19076c0?, 0x2b0bba0?})
              /usr/lib/golang/src/runtime/panic.go:914 +0x21f
      github.com/openstack-k8s-operators/test-operator/pkg/ansibletest.GetVolumeMounts(0x1, 0xc000cbd180, 0x0)
              /remote-source/app/pkg/ansibletest/volumes.go:208 +0x6fc
      github.com/openstack-k8s-operators/test-operator/pkg/ansibletest.Job(0xc000cbd180, 0xc0012830b0, {0xc0009e32a0, 0x20}, {0xc0009cc9c0, 0x13}, 0x0?, 0x0?, 0x0?, 0x0, ...)
              /remote-source/app/pkg/ansibletest/job.go:65 +0x412
      github.com/openstack-k8s-operators/test-operator/controllers.(*AnsibleTestReconciler).Reconcile(0xc000128f40, {0x1e043c0?, 0xc000cde9c0}, {{{0xc000c97ae0?, 0x5?}, {0xc000c97ad0?, 0xc002dcbd08?}}})
              /remote-source/app/controllers/ansibletest_controller.go:274 +0x1665
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1e07720?, {0x1e043c0?, 0xc000cde9c0?}, {{{0xc000c97ae0?, 0xb?}, {0xc000c97ad0?, 0x0?}}})
              /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.6/pkg/internal/controller/controller.go:119 +0xb7
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00068ef00, {0x1e043f8, 0xc000250000}, {0x19b8180?, 0xc00068b3a0?})
              /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.6/pkg/internal/controller/controller.go:316 +0x3cc
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00068ef00, {0x1e043f8, 0xc000250000})
              /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.6/pkg/internal/controller/controller.go:266 +0x1af
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
              /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.6/pkg/internal/controller/controller.go:227 +0x79
      created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 147
              /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.6/pkg/internal/controller/controller.go:223 +0x565
      

      As a result the ansibletest resource keeps waiting forever:

      [zuul@controller-0 ci-framework]$ oc get ansibletests.test.openstack.org ansibletest
      NAME          STATUS    MESSAGE
      ansibletest   Unknown   Deployment not started

      The problem seems to be here: ExtraConfigmapsMounts is expected. So if the AnsibleTest doesn't have extraConfigmapsMounts we hit the above error.

      Adding this to workflow items as a workaround seems to work:

          extraConfigmapsMounts:
            - name: dummy-ansibletest-cm
              mountPath: /mnt
              subPath: dummy

              eshulman Ella Shulman
              rdiazcam@redhat.com Ricardo Diaz Campos
              rhos-dfg-nfv
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: