Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-122

Error: open /etc/nsswitch.conf: permission denied and Error: open ./db-609956243: permission denied

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Done
    • 4.12.0
    • None
    • OLM
    • None
    • Important
    • Approved
    • Hide

      None

      Show
      None
    • Rebase: Bug Fixes Only
    • Documented as Resolved Issue

    Description

      Description of problem:

      The SQL-based index image created by old opm failed to run in 4.12 even if added the `privileged` permission to the namespace.

       

      MacBook-Pro:~ jianzhang$ oc get pods
      NAME                   READY   STATUS             RESTARTS     AGE
      jian-operators-4g5ln   0/1     CrashLoopBackOff   1 (2s ago)   11s
      MacBook-Pro:~ jianzhang$ oc logs jian-operators-4g5ln 
      Error: open /etc/nsswitch.conf: permission denied 

       

      PS: the SQL-based index created by the new opm version doesn't have this issue.

       

      opm version
      Version: version.Version{OpmVersion:"e41024eb3", GitCommit:"e41024eb37c721bc43e8b3df226dd30c0589aee7", BuildDate:"2022-08-16T01:50:17Z", GoOs:"darwin", GoArch:"amd64"} 

       

       

      Version-Release number of selected component (if applicable):

      OCP 4.12

       

      MacBook-Pro:~ jianzhang$ oc get clusterversion NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS version   4.12.0-0.nightly-2022-08-15-150248   True        False         3h25m   Cluster version is 4.12.0-0.nightly-2022-08-15-150248 

       

      How reproducible:

      always

      Steps to Reproduce:
      1. Deploy OCP 4.12

      2, Deploy a CatalogSource in the `openshift-marketplace` namespace.

       

      MacBook-Pro:~ jianzhang$ oc get ns openshift-marketplace -o yaml
      apiVersion: v1
      kind: Namespace
      metadata:
        annotations:
          capability.openshift.io/name: marketplace
          include.release.openshift.io/ibm-cloud-managed: "true"
          include.release.openshift.io/self-managed-high-availability: "true"
          include.release.openshift.io/single-node-developer: "true"
          openshift.io/node-selector: ""
          openshift.io/sa.scc.mcs: s0:c16,c10
          openshift.io/sa.scc.supplemental-groups: 1000260000/10000
          openshift.io/sa.scc.uid-range: 1000260000/10000
          workload.openshift.io/allowed: management
        creationTimestamp: "2022-08-15T23:15:27Z"
        labels:
          kubernetes.io/metadata.name: openshift-marketplace
          olm.operatorgroup.uid/1b776321-2714-4c1f-95ba-2ddff49c4efe: ""
          openshift.io/cluster-monitoring: "true"
          pod-security.kubernetes.io/audit: baseline
          pod-security.kubernetes.io/enforce: baseline
          pod-security.kubernetes.io/warn: baseline
        name: openshift-marketplace
        ownerReferences:
        - apiVersion: config.openshift.io/v1
          kind: ClusterVersion
          name: version
          uid: cd81594b-4f6c-46d6-9369-75deef542ec8
        resourceVersion: "8617"
        uid: 1c35352e-3636-4f2b-a3b1-c84ebc6681e0
      spec:
        finalizers:
        - kubernetes
      status:
        phase: Active 
      
      

      3, Check the CatalogSource pod status, crashed.

       

       

      
      MacBook-Pro:~ jianzhang$ oc get catalogsource -n openshift-marketplace jian-operators -o yaml
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      metadata:
        creationTimestamp: "2022-08-16T02:24:20Z"
        generation: 1
        name: jian-operators
        namespace: openshift-marketplace
        resourceVersion: "106145"
        uid: 6a75ecc9-7b88-4411-bcf5-e34618f9b3cd
      spec:
        displayName: Jian Operators
        image: quay.io/olmqe/etcd-index:v1
        priority: -100
        publisher: Jian
        sourceType: grpc
        updateStrategy:
          registryPoll:
            interval: 10m0s
      status:
        connectionState:
          address: jian-operators.openshift-marketplace.svc:50051
          lastConnect: "2022-08-16T03:12:28Z"
          lastObservedState: TRANSIENT_FAILURE
        latestImageRegistryPoll: "2022-08-16T02:34:21Z"
        registryService:
          createdAt: "2022-08-16T02:24:20Z"
          port: "50051"
          protocol: grpc
          serviceName: jian-operators
          serviceNamespace: openshift-marketplace
      
      MacBook-Pro:~ jianzhang$ oc get pods -n openshift-marketplace
      NAME                                                              READY   STATUS             RESTARTS       AGE
      28bb83ea022e9728d25570ab0adbe09a31d6a0a606917488e0ddb00f925mnfw   0/1     Completed          0              3h23m
      7049ea48beb27a712fa506b76ad672be201ce5d3a6a93d627a0091e0fesvdlj   0/1     Completed          0              3h23m
      certified-operators-ftt2n                                         1/1     Running            0              3h49m
      community-operators-27dx9                                         1/1     Running            0              3h49m
      jian-operators-5zq7d                                              0/1     CrashLoopBackOff   12 (71s ago)   38m
      jian-operators-gpg4v                                              0/1     CrashLoopBackOff   14 (57s ago)   48m
      marketplace-operator-9c8496b58-2jfmv                              1/1     Running            0              3h56m
      qe-app-registry-rqrrv                                             1/1     Running            0              141m
      redhat-marketplace-s6zrj                                          1/1     Running            0              3h49m
      redhat-operators-54cqr                                            1/1     Running            0              3h49m
      
      MacBook-Pro:~ jianzhang$ oc -n openshift-marketplace logs jian-operators-gpg4v 
      Error: open /etc/nsswitch.conf: permission denied
      Usage:
        opm registry serve [flags]
      
      
      Flags:
        -d, --database string          relative path to sqlite db (default "bundles.db")
            --debug                    enable debug logging
        -h, --help                     help for serve
        -p, --port string              port number to serve on (default "50051")
            --skip-migrate             do  not attempt to migrate to the latest db revision when starting
        -t, --termination-log string   path to a container termination log file (default "/dev/termination-log")
            --timeout-seconds string   Timeout in seconds. This flag will be removed later. (default "infinite")
      
      
      Global Flags:
            --skip-tls   skip TLS certificate verification for container image registries while pulling bundles or index 

       

      4. Create a namespace with the `privileged` permission.

       

      MacBook-Pro:~ jianzhang$ oc get ns debug -o yaml
      apiVersion: v1
      kind: Namespace
      metadata:
        annotations:
          openshift.io/sa.scc.mcs: s0:c30,c10
          openshift.io/sa.scc.supplemental-groups: 1000890000/10000
          openshift.io/sa.scc.uid-range: 1000890000/10000
        creationTimestamp: "2022-08-16T02:46:41Z"
        labels:
          kubernetes.io/metadata.name: debug
          pod-security.kubernetes.io/audit: privileged
          pod-security.kubernetes.io/enforce: privileged
          pod-security.kubernetes.io/warn: privileged
          security.openshift.io/scc.podSecurityLabelSync: "false"
        name: debug
        resourceVersion: "95718"
        uid: bdf93839-6c42-4365-a65c-d9c0b9fe0504
      spec:
        finalizers:
        - kubernetes
      status:
        phase: Active 

       
      5. Deploy a CatalogSource as above step 2. Still crashed.

       

       

      MacBook-Pro:~ jianzhang$ oc get pods -n debug
      NAME                   READY   STATUS             RESTARTS        AGE
      jian-operators-4g5ln   0/1     CrashLoopBackOff   10 (114s ago)   28m
      jian-operators-wn766   0/1     CrashLoopBackOff   8 (2m25s ago)   18m
      MacBook-Pro:~ jianzhang$ oc -n debug logs jian-operators-wn766
      Error: open /etc/nsswitch.conf: permission denied
      Usage:
        opm registry serve [flags]
      
      
      Flags:
        -d, --database string          relative path to sqlite db (default "bundles.db")
            --debug                    enable debug logging
        -h, --help                     help for serve
        -p, --port string              port number to serve on (default "50051")
            --skip-migrate             do  not attempt to migrate to the latest db revision when starting
        -t, --termination-log string   path to a container termination log file (default "/dev/termination-log")
            --timeout-seconds string   Timeout in seconds. This flag will be removed later. (default "infinite")
      
      
      Global Flags:
            --skip-tls   skip TLS certificate verification for container image registries while pulling bundles or index 

       

       

      Actual results:

      The sql-based index image created by the old opm version cannot be run.

       

      MacBook-Pro:~ jianzhang$ oc -n debug logs jian-operators-wn766 Error: open /etc/nsswitch.conf: permission denied 

       

       

      Expected results:

      The old SQL-based index image runs well. Or we have a workaround for it.

       

      Additional info:

      I changed another old sql-based image and have a try, get another permission issue.

       

      MacBook-Pro:~ jianzhang$ oc get catalogsource
      NAME             DISPLAY          TYPE   PUBLISHER   AGE
      jian-operators   Jian Operators   grpc   Jian        37m
      xia-operators    Xia Operators    grpc   Xia         101s
      MacBook-Pro:~ jianzhang$ oc get catalogsource xia-operators -o yaml
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      metadata:
        creationTimestamp: "2022-08-16T03:22:38Z"
        generation: 1
        name: xia-operators
        namespace: debug
        resourceVersion: "110629"
        uid: 8be42e68-43be-4fd4-9b67-c74edc5e6353
      spec:
        displayName: Xia Operators
        image: quay.io/olmqe/ditto-index:test-xzha-1
        priority: -100
        publisher: Xia
        sourceType: grpc
        updateStrategy:
          registryPoll:
            interval: 10m0s
      status:
        connectionState:
          address: xia-operators.debug.svc:50051
          lastConnect: "2022-08-16T03:24:18Z"
          lastObservedState: CONNECTING
        registryService:
          createdAt: "2022-08-16T03:22:38Z"
          port: "50051"
          protocol: grpc
          serviceName: xia-operators
          serviceNamespace: debug
      
      MacBook-Pro:~ jianzhang$ oc project
      Using project "debug" on server "https://api.qe-daily-412-0816.ibmcloud.qe.devcluster.openshift.com:6443".
      MacBook-Pro:~ jianzhang$ oc get pods
      NAME                   READY   STATUS             RESTARTS         AGE
      jian-operators-4g5ln   0/1     CrashLoopBackOff   11 (3m41s ago)   35m
      jian-operators-wn766   0/1     CrashLoopBackOff   9 (4m13s ago)    25m
      xia-operators-6wgjt    0/1     CrashLoopBackOff   1 (8s ago)       13s
      MacBook-Pro:~ jianzhang$ oc logs xia-operators-6wgjt 
      time="2022-08-16T03:22:43Z" level=warning msg="\x1b[1;33mDEPRECATION NOTICE:\nSqlite-based catalogs and their related subcommands are deprecated. Support for\nthem will be removed in a future release. Please migrate your catalog workflows\nto the new file-based catalog format.\x1b[0m"
      Error: open ./db-609956243: permission denied
      Usage:
        opm registry serve [flags]
      
      
      Flags:
        -d, --database string          relative path to sqlite db (default "bundles.db")
            --debug                    enable debug logging
      
       

      Even if that namespace is `privileged`.

      MacBook-Pro:~ jianzhang$ oc get ns debug -o yaml
      apiVersion: v1
      kind: Namespace
      metadata:
        annotations:
          openshift.io/sa.scc.mcs: s0:c30,c10
          openshift.io/sa.scc.supplemental-groups: 1000890000/10000
          openshift.io/sa.scc.uid-range: 1000890000/10000
        creationTimestamp: "2022-08-16T02:46:41Z"
        labels:
          kubernetes.io/metadata.name: debug
          pod-security.kubernetes.io/audit: privileged
          pod-security.kubernetes.io/enforce: privileged
          pod-security.kubernetes.io/warn: privileged
          security.openshift.io/scc.podSecurityLabelSync: "false"
        name: debug
        resourceVersion: "95718"
        uid: bdf93839-6c42-4365-a65c-d9c0b9fe0504
      spec:
        finalizers:
        - kubernetes
      status:
        phase: Active 

      But, both of them work well in the 4.11 cluster. As follows,

       

      MacBook-Pro:~ jianzhang$ oc get clusterversion
      NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.11.0-0.nightly-2022-08-15-152346   True        False         91m     Cluster version is 4.11.0-0.nightly-2022-08-15-152346
      MacBook-Pro:~ jianzhang$ oc get catalogsource
      NAME                  DISPLAY               TYPE   PUBLISHER   AGE
      certified-operators   Certified Operators   grpc   Red Hat     106m
      community-operators   Community Operators   grpc   Red Hat     106m
      jian-operators        Jian Operators        grpc   Jian        48m
      redhat-marketplace    Red Hat Marketplace   grpc   Red Hat     106m
      redhat-operators      Red Hat Operators     grpc   Red Hat     106m
      xia-operators         Xia Operators         grpc   Xia         6s
      MacBook-Pro:~ jianzhang$ oc get pods
      NAME                                   READY   STATUS    RESTARTS   AGE
      certified-operators-fsjc8              1/1     Running   0          107m
      community-operators-9qvzt              1/1     Running   0          107m
      jian-operators-n5s8c                   1/1     Running   0          48m
      marketplace-operator-7b777f747-22rwq   1/1     Running   0          109m
      redhat-marketplace-2mgrl               1/1     Running   0          107m
      redhat-operators-72q6z                 1/1     Running   0          107m
      xia-operators-ngq86                    1/1     Running   0          23s
      MacBook-Pro:~ jianzhang$ oc get catalogsource jian-operators -o yaml
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      metadata:
        creationTimestamp: "2022-08-16T02:39:52Z"
        generation: 1
        name: jian-operators
        namespace: openshift-marketplace
        resourceVersion: "58565"
        uid: 481a6fbe-00a5-4af5-86f7-d7413c658db3
      spec:
        displayName: Jian Operators
        image: quay.io/olmqe/etcd-index:v1
        priority: -100
        publisher: Jian
        sourceType: grpc
        updateStrategy:
          registryPoll:
            interval: 10m0s
      status:
        connectionState:
          address: jian-operators.openshift-marketplace.svc:50051
          lastConnect: "2022-08-16T02:44:45Z"
          lastObservedState: READY
        latestImageRegistryPoll: "2022-08-16T03:24:54Z"
        registryService:
          createdAt: "2022-08-16T02:39:52Z"
          port: "50051"
          protocol: grpc
          serviceName: jian-operators
          serviceNamespace: openshift-marketplace
      MacBook-Pro:~ jianzhang$ oc get catalogsource xia-operators -o yaml
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      metadata:
        creationTimestamp: "2022-08-16T03:28:07Z"
        generation: 1
        name: xia-operators
        namespace: openshift-marketplace
        resourceVersion: "59886"
        uid: a270f665-ee0b-49a5-badb-d3394c7a9344
      spec:
        displayName: Xia Operators
        image: quay.io/olmqe/ditto-index:test-xzha-1
        priority: -100
        publisher: Xia
        sourceType: grpc
        updateStrategy:
          registryPoll:
            interval: 10m0s
      status:
        connectionState:
          address: xia-operators.openshift-marketplace.svc:50051
          lastConnect: "2022-08-16T03:28:27Z"
          lastObservedState: READY
        registryService:
          createdAt: "2022-08-16T03:28:07Z"
          port: "50051"
          protocol: grpc
          serviceName: xia-operators
          serviceNamespace: openshift-marketplace
      
      MacBook-Pro:~ jianzhang$ oc get ns openshift-marketplace -o yaml
      apiVersion: v1
      kind: Namespace
      metadata:
        annotations:
          capability.openshift.io/name: marketplace
          include.release.openshift.io/ibm-cloud-managed: "true"
          include.release.openshift.io/self-managed-high-availability: "true"
          include.release.openshift.io/single-node-developer: "true"
          openshift.io/node-selector: ""
          openshift.io/sa.scc.mcs: s0:c16,c5
          openshift.io/sa.scc.supplemental-groups: 1000250000/10000
          openshift.io/sa.scc.uid-range: 1000250000/10000
          workload.openshift.io/allowed: management
        creationTimestamp: "2022-08-16T01:38:10Z"
        labels:
          kubernetes.io/metadata.name: openshift-marketplace
          olm.operatorgroup.uid/24dae571-2843-445b-b09f-5a4631cb25ba: ""
          openshift.io/cluster-monitoring: "true"
          pod-security.kubernetes.io/audit: baseline
          pod-security.kubernetes.io/warn: baseline
        name: openshift-marketplace
        ownerReferences:
        - apiVersion: config.openshift.io/v1
          kind: ClusterVersion
          name: version
          uid: 470d072e-37d9-4203-bc5a-c675800d593c
        resourceVersion: "6981"
        uid: 554a5ceb-8343-46f4-ae69-af36ee45d7fe
      spec:
        finalizers:
        - kubernetes
      status:
        phase: Active 

      Attachments

        Issue Links

          Activity

            People

              anik120 Anik Bhattacharjee
              rhn-support-jiazha Jian Zhang
              Jian Zhang Jian Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: