Upgrade automation is failing to get the flowcollector object

1. Run the upgrade scenario as mentioned in test-case https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-59746

Operator fails to get flowcollector object for NetObserv1.4.2

Operator should upgrade successfully

 • [FAILED] [210.211 seconds]
  [sig-netobserv] Network_Observability [It] NonPreRelease-Author:aramesha-High-59746-NetObserv upgrade testing [Serial]
  /Users/amoghrd/repos/openshift-tests-private/test/extended/netobserv/test_flowcollector.go:548


    Timeline >>
    STEP: Creating a kubernetes client @ 12/06/23 16:31:30.879
    Dec  6 16:31:31.277: INFO: configPath is now "/var/folders/mr/y1c7j5913sg2y9pt2lx3yvp80000gn/T/configfile2680306466"
    Dec  6 16:31:31.277: INFO: The user is now "e2e-test-netobserv-2hjbz-user"
    Dec  6 16:31:31.277: INFO: Creating project "e2e-test-netobserv-2hjbz"
    Dec  6 16:31:31.364: INFO: Waiting on permissions in project "e2e-test-netobserv-2hjbz" ...
    Dec  6 16:31:31.416: INFO: Waiting for ServiceAccount "default" to be provisioned...
    Dec  6 16:31:31.543: INFO: Waiting for ServiceAccount "builder" to be provisioned...
    Dec  6 16:31:31.667: INFO: Waiting for ServiceAccount "deployer" to be provisioned...
    Dec  6 16:31:31.793: INFO: Waiting for RoleBinding "system:image-pullers" to be provisioned...
    Dec  6 16:31:31.841: INFO: Waiting for RoleBinding "system:image-builders" to be provisioned...
    Dec  6 16:31:31.903: INFO: Waiting for RoleBinding "system:deployers" to be provisioned...
    Dec  6 16:31:31.946: INFO: Project "e2e-test-netobserv-2hjbz" has been fully provisioned.
    Dec  6 16:31:31.947: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get -n openshift-marketplace catalogsource qe-app-registry'
    STEP: Subscribe operators to stable channel @ 12/06/23 16:31:32.201
    Dec  6 16:31:32.201: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get namespace openshift-netobserv-operator'
    NAME                           STATUS   AGE
    openshift-netobserv-operator   Active   4h4m
    Dec  6 16:31:32.389: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get sub netobserv-operator -n openshift-netobserv-operator'
    NAME                 PACKAGE              SOURCE            CHANNEL
    netobserv-operator   netobserv-operator   qe-app-registry   stable
    Dec  6 16:31:32.734: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get sub netobserv-operator -n openshift-netobserv-operator -o=jsonpath={.status.installedCSV}'
    Dec  6 16:31:42.923: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get csv network-observability-operator.v1.5.0 -n openshift-netobserv-operator -o=jsonpath={.status.phase}'
    Dec  6 16:31:43.259: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get infrastructure cluster -o=jsonpath={.status.platformStatus.type}'
    Dec  6 16:31:45.623: INFO: the required cpu is: 6000, and the required memory is: 10737418240000
    Dec  6 16:31:45.623: INFO: the remaining cpu is: 20903, and the remaning memory is: 86762020864000
    STEP: Deploy loki operator @ 12/06/23 16:31:45.623
    Dec  6 16:31:45.623: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get namespace openshift-operators-redhat'
    NAME                         STATUS   AGE
    openshift-operators-redhat   Active   6h30m
    Dec  6 16:31:45.843: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get sub loki-operator -n openshift-operators-redhat'
    NAME            PACKAGE         SOURCE            CHANNEL
    loki-operator   loki-operator   qe-app-registry   stable
    Dec  6 16:31:46.215: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get sub loki-operator -n openshift-operators-redhat -o=jsonpath={.status.installedCSV}'
    Dec  6 16:31:56.444: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get csv loki-operator.v5.9.0 -n openshift-operators-redhat -o=jsonpath={.status.phase}'
    STEP: Deploy lokiStack @ 12/06/23 16:31:56.688
    Dec  6 16:31:56.712: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get infrastructure cluster -o=jsonpath={.status.platformStatus.type}'
    Dec  6 16:31:56.914: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get infrastructure/cluster -o=jsonpath={.status.infrastructureName}'
    Dec  6 16:31:57.107: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get infrastructure cluster -o=jsonpath={.status.platformStatus.aws.region}'
    Dec  6 16:31:57.309: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig extract secret/aws-creds -n kube-system --confirm --to=/tmp/e2e-test-netobserv-2hjbz-creds'
    Dec  6 16:31:58.871: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get infrastructure cluster -o=jsonpath={.status.platformStatus.aws.region}'
    Dec  6 16:31:59.088: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig extract secret/aws-creds -n kube-system --confirm --to=/tmp/e2e-test-netobserv-2hjbz-creds'
    Dec  6 16:31:59.295: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig create secret generic objectstore-secret --from-file=access_key_id=/tmp/e2e-test-netobserv-2hjbz-creds/aws_access_key_id --from-file=access_key_secret=/tmp/e2e-test-netobserv-2hjbz-creds/aws_secret_access_key --from-literal=region=us-east-2 --from-literal=bucketnames=netobserv-loki-aramesha-14-d4mlk --from-literal=endpoint=https://s3.us-east-2.amazonaws.com -n e2e-test-netobserv-2hjbz'
    secret/objectstore-secret created
    Dec  6 16:32:02.489: INFO: Running 'oc --namespace=e2e-test-netobserv-2hjbz --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig process -f /var/folders/mr/y1c7j5913sg2y9pt2lx3yvp80000gn/T/fixture-testdata-dir2237131599/test/extended/testdata/netobserv/loki/lokistack-simple.yaml -n e2e-test-netobserv-2hjbz -p NAME=lokistack NAMESPACE=e2e-test-netobserv-2hjbz SIZE=1x.demo SECRET_NAME=objectstore-secret STORAGE_TYPE=s3 STORAGE_CLASS=gp3-csi'
    Dec  6 16:32:02.721: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig apply -f /tmp/e2e-test-netobserv-2hjbz-khl0kdwo.json -n e2e-test-netobserv-2hjbz'
    lokistack.loki.grafana.com/lokistack created
    Dec  6 16:32:06.277: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get -n e2e-test-netobserv-2hjbz lokistack lokistack'
    Dec  6 16:32:06.496: INFO: Find lokistack lokistack
    Dec  6 16:32:11.532: INFO: Waiting for full availability of lokistack-distributor deployment (0/1)


    Dec  6 16:32:16.522: INFO: Waiting for full availability of lokistack-distributor deployment (0/1)


    Dec  6 16:32:21.524: INFO: Waiting for full availability of lokistack-distributor deployment (0/1)


    Dec  6 16:32:26.527: INFO: Deployment lokistack-distributor available (1/1)


    Dec  6 16:32:31.575: INFO: Deployment lokistack-gateway available (2/2)


    Dec  6 16:32:36.610: INFO: Deployment lokistack-querier available (1/1)


    Dec  6 16:32:41.637: INFO: Deployment lokistack-query-frontend available (1/1)


    Dec  6 16:32:46.667: INFO: statefulset lokistack-compactor available (1/1)


    Dec  6 16:32:51.700: INFO: statefulset lokistack-index-gateway available (1/1)


    Dec  6 16:32:56.727: INFO: Waiting for full availability of lokistack-ingester statefulset (0/1)


    Dec  6 16:33:01.765: INFO: Waiting for full availability of lokistack-ingester statefulset (0/1)


    Dec  6 16:33:06.730: INFO: Waiting for full availability of lokistack-ingester statefulset (0/1)


    Dec  6 16:33:11.726: INFO: Waiting for full availability of lokistack-ingester statefulset (0/1)


    Dec  6 16:33:16.725: INFO: statefulset lokistack-ingester available (1/1)


    STEP: Uninstall operator deployed by BeforeEach and delete operator NS @ 12/06/23 16:33:16.749
    Dec  6 16:33:16.749: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig delete -n openshift-netobserv-operator subscription netobserv-operator'
    Dec  6 16:33:19.986: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get -n openshift-netobserv-operator subscription netobserv-operator'
    Dec  6 16:33:20.212: INFO: Error running /usr/local/bin/oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get -n openshift-netobserv-operator subscription netobserv-operator:
    Error from server (NotFound): subscriptions.operators.coreos.com "netobserv-operator" not found
    Dec  6 16:33:20.212: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig delete -n openshift-netobserv-operator csv -l operators.coreos.com/netobserv-operator.openshift-netobserv-operator='
    clusterserviceversion.operators.coreos.com "network-observability-operator.v1.5.0" deleted
    Dec  6 16:33:20.547: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig delete namespace openshift-netobserv-operator'
    namespace "openshift-netobserv-operator" deleted
    Dec  6 16:33:26.680: INFO: Deleted namespace/openshift-netobserv-operator, err: <nil>
    STEP: Deploy older version of netobserv operator @ 12/06/23 16:33:26.68
    STEP: Subscribe operators to stable channel @ 12/06/23 16:33:26.68
    Dec  6 16:33:26.680: INFO: Creating Netobserv operator namespace
    Dec  6 16:33:29.682: INFO: showInfo is true
    Dec  6 16:33:29.682: INFO: Running 'oc --namespace=e2e-test-netobserv-2hjbz --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig process --ignore-unknown-parameters=true -f /var/folders/mr/y1c7j5913sg2y9pt2lx3yvp80000gn/T/fixture-testdata-dir2237131599/test/extended/testdata/netobserv/subscription/namespace.yaml'
    Dec  6 16:33:29.878: INFO: the file of resource is /tmp/e2e-test-netobserv-2hjbz-0ofmkyoaconfig.json.stdout
    Dec  6 16:33:29.878: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig apply -f /tmp/e2e-test-netobserv-2hjbz-0ofmkyoaconfig.json.stdout'
    namespace/openshift-netobserv-operator created
    Dec  6 16:33:30.454: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get -n openshift-netobserv-operator og'
    Dec  6 16:33:33.661: INFO: Running 'oc --namespace=e2e-test-netobserv-2hjbz --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig process -n openshift-netobserv-operator -f /var/folders/mr/y1c7j5913sg2y9pt2lx3yvp80000gn/T/fixture-testdata-dir2237131599/test/extended/testdata/netobserv/subscription/allnamespace-og.yaml -p OG_NAME=openshift-netobserv-operator NAMESPACE=openshift-netobserv-operator'
    Dec  6 16:33:38.904: INFO: Running 'oc --namespace=e2e-test-netobserv-2hjbz --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig apply -f /tmp/e2e-test-netobserv-2hjbz-gs65p1kt.json -n openshift-netobserv-operator'
    Dec  6 16:33:39.444: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get sub -n openshift-netobserv-operator netobserv-operator'
    Dec  6 16:33:39.673: INFO: Error running /usr/local/bin/oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get sub -n openshift-netobserv-operator netobserv-operator:
    Error from server (NotFound): subscriptions.operators.coreos.com "netobserv-operator" not found
    Dec  6 16:33:44.673: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get -n openshift-marketplace packagemanifests -l catalog=redhat-operators'
    Dec  6 16:33:47.947: INFO: Running 'oc --namespace=e2e-test-netobserv-2hjbz --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig process -n openshift-netobserv-operator -f /var/folders/mr/y1c7j5913sg2y9pt2lx3yvp80000gn/T/fixture-testdata-dir2237131599/test/extended/testdata/netobserv/subscription/sub-template.yaml -p PACKAGE_NAME=netobserv-operator NAMESPACE=openshift-netobserv-operator CHANNEL=stable SOURCE=redhat-operators SOURCE_NAMESPACE=openshift-marketplace'
    Dec  6 16:33:53.141: INFO: Running 'oc --namespace=e2e-test-netobserv-2hjbz --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig apply -f /tmp/e2e-test-netobserv-2hjbz-67v7zfqs.json -n openshift-netobserv-operator'
    Dec  6 16:34:03.730: INFO: Waiting for pod with label app=netobserv-operator to be ready...


    Dec  6 16:34:13.726: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get namespace openshift-netobserv-operator'
    NAME                           STATUS   AGE
    openshift-netobserv-operator   Active   44s
    Dec  6 16:34:13.935: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get sub netobserv-operator -n openshift-netobserv-operator'
    NAME                 PACKAGE              SOURCE             CHANNEL
    netobserv-operator   netobserv-operator   redhat-operators   stable
    Dec  6 16:34:14.147: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get sub netobserv-operator -n openshift-netobserv-operator -o=jsonpath={.status.installedCSV}'
    Dec  6 16:34:24.409: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get csv network-observability-operator.v1.4.2 -n openshift-netobserv-operator -o=jsonpath={.status.phase}'
    Dec  6 16:34:24.662: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get crd -o jsonpath='{.items[*].spec.names.kind}''
    STEP: Deploy FlowCollector @ 12/06/23 16:34:25.432
    Dec  6 16:34:28.433: INFO: showInfo is true
    Dec  6 16:34:28.433: INFO: Running 'oc --namespace=e2e-test-netobserv-2hjbz --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig process --ignore-unknown-parameters=true -f /var/folders/mr/y1c7j5913sg2y9pt2lx3yvp80000gn/T/fixture-testdata-dir2237131599/test/extended/testdata/netobserv/flowcollector_v1beta1_template.yaml -p Namespace=e2e-test-netobserv-2hjbz LokiURL=https://lokistack-gateway-http.e2e-test-netobserv-2hjbz.svc.cluster.local:8080/api/logs/v1/network/ LokiTLSCertName=lokistack-gateway-ca-bundle LokiNamespace=e2e-test-netobserv-2hjbz Template=/var/folders/mr/y1c7j5913sg2y9pt2lx3yvp80000gn/T/fixture-testdata-dir2237131599/test/extended/testdata/netobserv/flowcollector_v1beta1_template.yaml'
    Dec  6 16:34:28.635: INFO: the file of resource is /tmp/e2e-test-netobserv-2hjbz-kwn7fzelconfig.json.stdout
    Dec  6 16:34:28.635: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig apply -f /tmp/e2e-test-netobserv-2hjbz-kwn7fzelconfig.json.stdout -n e2e-test-netobserv-2hjbz'
    flowcollector.flows.netobserv.io/cluster created
    Dec  6 16:34:30.175: INFO: Deploy ClusterRoleBinding in Forward mode
    Dec  6 16:34:33.175: INFO: showInfo is true
    Dec  6 16:34:33.175: INFO: Running 'oc --namespace=e2e-test-netobserv-2hjbz --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig process --ignore-unknown-parameters=true -f /var/folders/mr/y1c7j5913sg2y9pt2lx3yvp80000gn/T/fixture-testdata-dir2237131599/test/extended/testdata/netobserv/clusterRoleBinding-FORWARD.yaml -p NAMESPACE=e2e-test-netobserv-2hjbz SERVICE_ACCOUNT_NAME=flowlogs-pipeline'
    Dec  6 16:34:33.391: INFO: the file of resource is /tmp/e2e-test-netobserv-2hjbz-sf3lm2qkconfig.json.stdout
    Dec  6 16:34:33.391: INFO: Running 'oc --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig apply -f /tmp/e2e-test-netobserv-2hjbz-sf3lm2qkconfig.json.stdout -n e2e-test-netobserv-2hjbz'
    clusterrole.rbac.authorization.k8s.io/netobserv-reader unchanged
    clusterrole.rbac.authorization.k8s.io/netobserv-writer unchanged
    clusterrolebinding.rbac.authorization.k8s.io/netobserv-writer-flp configured
    clusterrolebinding.rbac.authorization.k8s.io/netobserv-reader configured
    Dec  6 16:34:45.238: INFO: Running 'oc --namespace=e2e-test-netobserv-2hjbz --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get pods -n e2e-test-netobserv-2hjbz --output=template --template='{{- range .items -}}{{- range .status.conditions -}}{{- if ne .reason "PodCompleted" -}}{{- if eq .type "Ready" -}}{{- .status}} {{" "}}{{- end -}}{{- end -}}{{- end -}}{{- end -}}''
    Dec  6 16:34:55.519: INFO: Running 'oc --namespace=e2e-test-netobserv-2hjbz --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get pods -n e2e-test-netobserv-2hjbz-privileged --output=template --template='{{- range .items -}}{{- range .status.conditions -}}{{- if ne .reason "PodCompleted" -}}{{- if eq .type "Ready" -}}{{- .status}} {{" "}}{{- end -}}{{- end -}}{{- end -}}{{- end -}}''
    Dec  6 16:35:00.783: INFO: Running 'oc --namespace=e2e-test-netobserv-2hjbz --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get flowcollector -o jsonpath='{.items[*].status.conditions[0].reason}''
    Dec  6 16:35:00.990: INFO: Error running /usr/local/bin/oc --namespace=e2e-test-netobserv-2hjbz --kubeconfig=/Users/amoghrd/Downloads/flexy-clusters/250830/kubeconfig get flowcollector -o jsonpath='{.items[*].status.conditions[0].reason}':
    ''Error from server (NotFound): Unable to list "flows.netobserv.io/v1beta2, Resource=flowcollectors": the server could not find the requested resource (get flowcollectors.flows.netobserv.io)
    [FAILED] in [It] - /Users/amoghrd/repos/openshift-tests-private/test/extended/util/assert.go:30 @ 12/06/23 16:35:00.99 

oc get flowcollector
Error from server (NotFound): Unable to list "flows.netobserv.io/v1beta2, Resource=flowcollectors": the server could not find the requested resource (get flowcollectors.flows.netobserv.io)