Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-2850

Upgrading from ACM 2.6 to 2.7 failed

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • ACM 2.7.0
    • Submariner 0.14.1, ACM 2.7.0
    • Multicluster Networking
    • None
    • True
    • None
    • False
    • No
    • Moderate

      Description of problem:

      Upgrading from ACM 2.6 (with Submariner 0.13) ACM 2.7 (with Submariner 0.14) has failed.

      Upgrade procedure was according to https://docs.openshift.com/container-platform/4.11/operators/admin/olm-adding-operators-to-cluster.html

      Version-Release number of selected component (if applicable):

      Base version: ACM 2.6.4 + Submariner 0.13.3

      Upgraded version ACM 2.7.0 + Submariner 0.14.1

       

      # Cloud platform: Amazon
# OCP version: 4.11.23
# ACM version: 2.7.0
Name:           4.11.23
Digest:         sha256:24a273f879d6eed8d6cce1426798a623ebbe095e01a881e58aa78c4e9eff0c8b
Created:        2023-01-11T13:57:38Z
OS/Arch:        linux/amd64
Manifests:      594
Metadata files: 1
Pull From: quay.io/openshift-release-dev/ocp-release@sha256:24a273f879d6eed8d6cce1426798a623ebbe095e01a881e58aa78c4e9eff0c8b
# Cloud platform: Google
# OCP version: 4.10.48
Name:      4.10.48
Digest:    sha256:bb7e12e9dd638f584fee587b6fae07c29100143ad5428560f114df11704627fa
Created:   2023-01-10T17:45:01Z
OS/Arch:   linux/amd64
Manifests: 544
Pull From: quay.io/openshift-release-dev/ocp-release@sha256:bb7e12e9dd638f584fee587b6fae07c29100143ad5428560f114df11704627fa

###################################################################
#       ImageStream Tags (in namespace submariner-operator)       #
###################################################################

### lighthouse-agent-rhel8:v0.13.3 Image-Stream tag ###
build-date=2023-01-16T14:49:34
name=rhacm2/lighthouse-agent-rhel8
release=7
url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/lighthouse-agent-rhel8/images/v0.13.3-7 
version=v0.13.3
### lighthouse-agent-rhel8:v0.14.1 Image-Stream tag ###
build-date=2023-01-16T13:51:16
name=rhacm2/lighthouse-agent-rhel8
release=6
url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/lighthouse-agent-rhel8/images/v0.14.1-6 
version=v0.14.1
### lighthouse-coredns-rhel8:v0.13.3 Image-Stream tag ###
build-date=2023-01-16T14:50:37
name=rhacm2/lighthouse-coredns-rhel8
release=7
url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/lighthouse-coredns-rhel8/images/v0.13.3-7 
version=v0.13.3
### lighthouse-coredns-rhel8:v0.14.1 Image-Stream tag ###
build-date=2023-01-16T13:51:06
name=rhacm2/lighthouse-coredns-rhel8
release=6
url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/lighthouse-coredns-rhel8/images/v0.14.1-6 
version=v0.14.1
### nettest-rhel8:v0.14.1 Image-Stream tag ###
build-date=2023-01-16T13:51:07
name=rhacm2/nettest-rhel8
release=6
url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/nettest-rhel8/images/v0.14.1-6 
version=v0.14.1
### submariner-gateway-rhel8:v0.13.3 Image-Stream tag ###
build-date=2023-01-16T14:50:47
name=rhacm2/submariner-gateway-rhel8
release=6
url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/submariner-gateway-rhel8/images/v0.13.3-6 
version=v0.13.3
### submariner-gateway-rhel8:v0.14.1 Image-Stream tag ###
build-date=2023-01-16T13:51:15
name=rhacm2/submariner-gateway-rhel8
release=6
url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/submariner-gateway-rhel8/images/v0.14.1-6 
version=v0.14.1
### submariner-globalnet-rhel8:v0.13.3 Image-Stream tag ###
build-date=2023-01-16T14:49:42
name=rhacm2/submariner-globalnet-rhel8
release=7
url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/submariner-globalnet-rhel8/images/v0.13.3-7 
version=v0.13.3
### submariner-globalnet-rhel8:v0.14.1 Image-Stream tag ###
build-date=2023-01-16T13:51:06
name=rhacm2/submariner-globalnet-rhel8
release=9
url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/submariner-globalnet-rhel8/images/v0.14.1-9 
version=v0.14.1
### submariner-networkplugin-syncer-rhel8:v0.13.3 Image-Stream tag ###
build-date=2023-01-16T14:50:51
name=rhacm2/submariner-networkplugin-syncer-rhel8
release=6
url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/submariner-networkplugin-syncer-rhel8/images/v0.13.3-6 
version=v0.13.3
### submariner-networkplugin-syncer-rhel8:v0.14.1 Image-Stream tag ###
build-date=2023-01-16T13:51:09
name=rhacm2/submariner-networkplugin-syncer-rhel8
release=6
url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/submariner-networkplugin-syncer-rhel8/images/v0.14.1-6 
version=v0.14.1
### submariner-operator-bundle:v0.13.3 Image-Stream tag ###
build-date=2023-01-16T15:09:43
name=rhacm2/submariner-operator-bundle
release=9
url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/submariner-operator-bundle/images/v0.13.3-9 
version=v0.13.3
### submariner-operator-bundle:v0.14.1 Image-Stream tag ###
build-date=2023-01-16T14:10:23
name=rhacm2/submariner-operator-bundle
release=11
url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/submariner-operator-bundle/images/v0.14.1-11 
version=v0.14.1
### submariner-operator-bundle-index:v0.14.1 Image-Stream tag ###
build-date=2022-12-07T04:31:54
name=openshift/ose-operator-registry
release=202212070335.p0.g6094188.assembly.stream
url =https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-operator-registry/images/v4.11.0-202212070335.p0.g6094188.assembly.stream 
version=v4.11.0
### submariner-rhel8-operator:v0.13.3 Image-Stream tag ###
build-date=2023-01-16T14:50:43
name=rhacm2/submariner-rhel8-operator
release=6
url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/submariner-rhel8-operator/images/v0.13.3-6 
version=v0.13.3
### submariner-rhel8-operator:v0.14.1 Image-Stream tag ###
build-date=2023-01-16T13:50:25
name=rhacm2/submariner-rhel8-operator
release=8
url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/submariner-rhel8-operator/images/v0.14.1-8 
version=v0.14.1
### submariner-route-agent-rhel8:v0.13.3 Image-Stream tag ###
build-date=2023-01-16T14:49:41
name=rhacm2/submariner-route-agent-rhel8
release=7
url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/submariner-route-agent-rhel8/images/v0.13.3-7 
version=v0.13.3
### submariner-route-agent-rhel8:v0.14.1 Image-Stream tag ###
build-date=2023-01-16T13:51:14
name=rhacm2/submariner-route-agent-rhel8
release=6
url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2/submariner-route-agent-rhel8/images/v0.14.1-6 
version=v0.14.1

       

      How reproducible:

      Sometimes

      Steps to Reproduce:

      1.  Install ACM 2.6.4 + Submariner 0.13.3
      2. Upgrade to ACM 2.7.0 + Submariner 0.14.1
      3. Run Submariner E2E tests

      Actual results:

      Base version deployment:

      https://qe-jenkins-csb-skynet.apps.ocp-c1.prod.psi.redhat.com/job/ACM-2.6.4-Submariner-0.13.3-AWS-GCP-Globalnet/35/Test-Report/

      Upgraded version:

      https://qe-jenkins-csb-skynet.apps.ocp-c1.prod.psi.redhat.com/job/ACM-Upgrade-2.7.0-Submariner-0.14.1-AWS-GCP-Globalnet/5/

       

      All Service Discovery E2E tests failed on:

      Failed to retrieve ServiceExport. ServiceExport Synced condition status not found Unexpected error:  <*errors.errorString | 0xc00032c110>: {  s: "timed out waiting for the condition",  }  timed out waiting for the condition occurred /home/jenkins/go/src/github.com/submariner-io/lighthouse/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:562

       
      Full results:

      https://qe-jenkins-csb-skynet.apps.ocp-c1.prod.psi.redhat.com/blue/organizations/jenkins/ACM-Upgrade-2.7.0-Submariner-0.14.1-AWS-GCP-Globalnet/detail/ACM-Upgrade-2.7.0-Submariner-0.14.1-AWS-GCP-Globalnet/5/tests

       

      Expected results:

      All E2E should pass.

      Additional info:

      $ subctl gather --dir submariner-gather_"gcp-nmanos-c1" --kubecontexts admin
Cluster "gcp-nmanos-c1"
Gathering information from cluster "gcp-nmanos-c1"
 • Gathering connectivity logs  ...
 ✓ Gathering connectivity logs
 ✓ Found 1 pods matching label selector "app=submariner-gateway"
 ✓ Found 7 pods matching label selector "app=submariner-routeagent"
 ✓ Found 1 pods matching label selector "app=submariner-globalnet"
 ✓ Found 0 pods matching label selector "app=submariner-networkplugin-syncer"
 • Gathering connectivity resources  ...
 ✓ Gathering connectivity resources
 ✓ Gathering CNI data from 7 pods matching label selector "app=submariner-routeagent"
 ✓ Gathering CNI data from 1 pods matching label selector "app=submariner-gateway"
 ✓ Gathering globalnet data from 1 pods matching label selector "app=submariner-globalnet"
 ✓ Gathering cable driver data from 1 pods matching label selector "app=submariner-gateway"
 ✓ Found 2 endpoints in namespace "submariner-operator"
 ✓ Found 2 clusters in namespace "submariner-operator"
 ✓ Found 1 gateways in namespace "submariner-operator"
 ✓ Found 1 clusterglobalegressips in namespace ""
 ✓ Found 0 globalegressips in namespace ""
 ✓ Found 2 globalingressips in namespace ""
 • Gathering service-discovery logs  ...
 ✓ Gathering service-discovery logs
 ✓ Found 3 pods matching label selector "component=submariner-lighthouse"
 ✓ Found 6 pods matching label selector "dns.operator.openshift.io/daemonset-dns=default"
 • Gathering service-discovery resources  ...
 ✓ Gathering service-discovery resources
 ✓ Found 2 serviceexports in namespace ""
 ✓ Found 2 serviceimports in namespace ""
 ✓ Found 2 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""
 ✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"
 ✓ Found 1 configmaps by field selector "metadata.name=dns-default" in namespace "openshift-dns"
 ✓ Found 1 services by label selector "submariner.io/exportedServiceRef" in namespace ""
 • Gathering broker logs  ...
 ✓ Gathering broker logs
 • Gathering broker resources  ...
 ✓ Gathering broker resources
 ✓ Found 2 endpoints in namespace "test-submariner-set1-broker"
 ✓ Found 2 clusters in namespace "test-submariner-set1-broker"
 ✓ Found 1 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace "test-submariner-set1-broker"
 ✓ Found 2 serviceimports in namespace "test-submariner-set1-broker"
 • Gathering operator logs  ...
 ✓ Gathering operator logs
 ✓ Found 0 pods matching label selector "name=submariner-operator"
 • Gathering operator resources  ...
 ✓ Gathering operator resources
 ✓ Found 1 submariners in namespace "submariner-operator"
 ✓ Found 1 servicediscoveries in namespace "submariner-operator"
 ✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"
 ✓ Found 0 deployments by label selector "app=submariner-networkplugin-syncer" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"
Files are stored under directory "submariner-gather_gcp-nmanos-c1"

        1. submariner-gather_2023-01-17.zip
          1.53 MB
        2. SubmarinerUpgrade.png
          SubmarinerUpgrade.png
          270 kB
        3. upgradeACM.pdf
          1.95 MB

            mbabushk@redhat.com Maxim Babushkin
            nmanos@redhat.com Noam Manos
            Noam Manos Noam Manos
            ACM QE Team
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: