Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-57518

ocp upgrade 4.18.17->4.19.0 fails - crashing the console

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Undefined Undefined
    • None
    • 4.19
    • Management Console
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • None
    • None
    • None
    • Proposed
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

          ocp upgrade 4.18.17->4.19.0 fails 

      Version-Release number of selected component (if applicable):

          

      How reproducible:

          

      Steps to Reproduce:

          1.
          2.
          3.
          

      Actual results:

          
      console pod crashloop,
      Error building Content Security Policy directives: invalid CSP directive:
      NAME                        READY   STATUS             RESTARTS          AGE
      console-76cf96587b-5mjpk    0/1     CrashLoopBackOff   213 (18s ago)     18h
      console-76cf96587b-wdnzw    0/1     CrashLoopBackOff   213 (2m19s ago)   18h
      downloads-5ff79cb8b-4m9p6   1/1     Running            0                 18h
      downloads-5ff79cb8b-7cbsr   1/1     Running            0                 18h
      [kni@ocp-edge77 ~]$ 
      
      [kni@ocp-edge77 ~]$ oc get pods -n openshift-console console-76cf96587b-wdnzw -ojson | jq -r ".status.containerStatuses[0].lastState.terminated.message"
      0617 12:14:54.279104       1 main.go:221]  - networking-console-plugin
      W0617 12:14:54.279122       1 authoptions.go:112] Flag inactivity-timeout is set to less then 300 seconds and will be ignored!
      I0617 12:14:54.302212       1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
      I0617 12:14:54.302244       1 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
      I0617 12:14:54.302250       1 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
      I0617 12:14:54.302254       1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
      I0617 12:14:54.356425       1 server.go:553] The following console endpoints are now proxied to these services:
      I0617 12:14:54.356546       1 server.go:556]  - /api/proxy/plugin/mce/console/ -> https://console-mce-console.multicluster-engine.svc.cluster.local:3000
      I0617 12:14:54.356624       1 server.go:556]  - /api/proxy/plugin/acm/console/ -> https://console-chart-console-v2.rhacm.svc.cluster.local:3000
      I0617 12:14:54.357172       1 main.go:656] Binding to [::]:8443...
      I0617 12:14:54.357254       1 main.go:658] using TLS
      I0617 12:14:57.359838       1 metrics.go:133] serverconfig.Metrics: Update ConsolePlugin metrics...
      I0617 12:14:57.395215       1 metrics.go:143] serverconfig.Metrics: Update ConsolePlugin metrics: &map[acm:map[enabled:1] mce:map[enabled:1] monitoring:map[enabled:1] networking:map[enabled:1]] (took 35.299768ms)
      E0617 12:14:58.925844       1 middleware.go:51] TOKEN_REVIEW: 'GET /metrics' unauthorized, invalid user token, [invalid bearer token, token lookup failed]
      I0617 12:14:59.357641       1 metrics.go:80] usage.Metrics: Count console users...
      I0617 12:14:59.868422       1 metrics.go:156] usage.Metrics: Update console users metrics: 1 kubeadmin, 0 cluster-admins, 0 developers, 0 unknown/errors (took 510.544352ms)
      F0617 12:15:02.835107       1 server.go:709] Error building Content Security Policy directives: invalid CSP directive: ConnectSrc
      
      
      
      4 cluster operators that staied in the old version, all the rest moved to 4.19.0 :
      [kni@ocp-edge77 ~]$ oc get co -A | grep -v "4.19.0"
      NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      dns                                        4.18.17   True        False         False      25h     
      machine-config                             4.18.17   True        False         False      25h     
      network                                    4.18.17   True        False         False      25h     
      olm                                        4.18.17   True        False         False      21h     
      
      
      
      [kni@ocp-edge77 ~]$ oc adm upgrade Failing=True:  Reason: ClusterOperatorNotAvailable   Message: Cluster operator console is not availableinfo: An upgrade is in progress. Unable to apply 4.19.0: the cluster operator console is not availableUpgradeable=False  Reason: UpdateInProgress   Message: An update is already in progress and the details are in the Progressing conditionUpstream is unset, so the cluster will use an appropriate default. Channel: stable-4.14 warning: Cannot display available updates:   Reason: VersionNotFound   Message: Unable to retrieve available updates: currently reconciling cluster version 4.19.0 not found in the "stable-4.14" channel[kni@ocp-edge77 ~]$  

      Expected results:

          

      Additional info:

          must gather : https://drive.google.com/file/d/1bsbCnn53T0Qil9ueLHvI6ab5w1UpylGH/view?usp=sharing

       

              jhadvig@redhat.com Jakub Hadvig
              rhn-support-gamado Gal Amado
              None
              None
              YaDan Pei YaDan Pei
              None
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: