Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-1039 Image push to openshift internal image registry fails after restoring the application on new cluster using datamover
  3. OADP-1098

[IBM QE-Z] Verify Bug OADP-1039 - Image push to openshift internal image registry fails after restoring the application on new cluster using datamover

XMLWordPrintable

    • Icon: Sub-task Sub-task
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • None
    • None
    • None
    • Quality / Stability / Reliability
    • 4
    • False
    • Hide

      None

      Show
      None
    • False
    • ToDo
    • 0
    • Very Likely
    • 0
    • None
    • Unset
    • Unknown

      Description of problem:

      Image push to the openshift internal image registry fails after restoring the application on the new cluster using datamover 

       

      Version-Release number of selected component (if applicable):

       

      How reproducible:

      Always

      Steps to Reproduce:
      1.Deploy Openshift and ODF
      2. Deploy a stateful application , which pushes its image to the internal openshift image registry; for example mongo-persistent

      [root@m4205001 ~]# oc get all,pvc -n mongo-persistent
      NAME                            READY   STATUS      RESTARTS   AGE
      pod/mongo-666589749c-7vdhm      1/1     Running     0          2m39s
      pod/todolist-1-deploy           0/1     Completed   0          98s
      pod/todolist-1-drlzk            1/1     Running     0          95s
      pod/todolist-mongo-go-1-build   0/1     Completed   0          2m39s
      NAME                               DESIRED   CURRENT   READY   AGE
      replicationcontroller/todolist-1   1         1         1       98s
      NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
      service/mongo      ClusterIP   172.30.98.230   <none>        27017/TCP   2m39s
      service/todolist   ClusterIP   172.30.7.247    <none>        8000/TCP    2m39s
      NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
      deployment.apps/mongo   1/1     1            1           2m39s
      NAME                               DESIRED   CURRENT   READY   AGE
      replicaset.apps/mongo-666589749c   1         1         1       2m39s
      NAME                                          REVISION   DESIRED   CURRENT   TRIGGERED BY
      deploymentconfig.apps.openshift.io/todolist   1          1         1         config,image(todolist-mongo-go:latest)
      NAME                                               TYPE     FROM         LATEST
      buildconfig.build.openshift.io/todolist-mongo-go   Docker   Git@master   1
      NAME                                           TYPE     FROM          STATUS     STARTED         DURATION
      build.build.openshift.io/todolist-mongo-go-1   Docker   Git@8c32a90   Complete   2 minutes ago   1m4s
      NAME                                               IMAGE REPOSITORY                                                                      TAGS     UPDATED
      imagestream.image.openshift.io/todolist-mongo-go   image-registry.openshift-image-registry.svc:5000/mongo-persistent/todolist-mongo-go   latest   About a minute ago
      NAME                                      HOST/PORT                                                      PATH   SERVICES   PORT    TERMINATION   WILDCARD
      route.route.openshift.io/todolist-route   todolist-route-mongo-persistent.apps.ocsm4205001.lnxero1.boe   /      todolist   <all>                 None
      NAME                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                AGE
      persistentvolumeclaim/mongo   Bound    pvc-4a37fb4e-d5a2-439d-8ae8-7e2cf847a267   1Gi        RWX            ocs-storagecluster-cephfs   2m39s

      3. Make sure you have an appropriate StorageClass and VolumeShapshotClass and there is only one default of each.

        - Include the label velero.io/csi-volumesnapshot-class: 'true' in your VolumeSnapshotClass to let Velero know which to use.
        - deletionPolicy must be set to Retain for the VolumeSnapshotClass.

      4. Install the OADP operator from the  Operator Hub

      5. Install the VolSync operator from the  Operator Hub

      6. 

      Configure a restic repository secret

       

      cat << EOF > ./restic-secret.yaml
      apiVersion: v1
      kind: Secret
      metadata:
        name: <secret-name>
      type: Opaque
      stringData:
        # The repository encryption key
        RESTIC_PASSWORD: my-secure-restic-password
      EOF

      7. 

       

       Create a credentials-velero file with the AWS access key
         

      #  cat << EOF > ./credentials-velero
      > [default]
      > aws_access_key_id=<AWS_ACCESS_KEY_ID>
      > aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
      > EOF

       

      8. Create a default Secret

      # oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero

      9. Create a DPA instance from the opesnhft-console by enabling CSI
         Operators -> Installed Operators -> OADP Operator -> Create DPA 

      kind: DataProtectionApplication
      apiVersion: oadp.openshift.io/v1alpha1
      metadata:
        name: velero-sample
        namespace: openshift-adp
      spec:
        features:
          dataMover: 
            enable: true
            credentialName: restic-repo-secret
        backupLocations:
          - velero:
              config:
                profile: default
                region: eu-central-1
              credential:
                key: cloud
                name: cloud-credentials
              default: true
              objectStorage:
                bucket: myoadptestbucket
                prefix: velero
              provider: aws
        configuration:
          restic:
            enable: false
          velero:
            defaultPlugins:
              - openshift
              - aws
              - kubevirt
              - csi
          featureFlags:
            - EnableCSI
        snapshotLocations:
          - velero:
              config:
                profile: default
                region: eu-central-1
              provider: aws

      10.  Created a backup of the application 

      # oc get backup -n openshift-adp mongo-backup-withcsi1-nov3 -ojsonpath='{.status.phase}'
      Complete

       

      # oc get vsb -n mongo-persistent
      NAME        AGE
      vsb-xzkbj   119m

      11. Restore the application on a new cluster

      [root@m4202001 ~]# oc get all,pvc -n mongo-persistent
      NAME                            READY   STATUS      RESTARTS   AGE
      pod/mongo-666589749c-4cs7n      1/1     Running     0          37m
      pod/todolist-1-deploy           0/1     Completed   0          38m
      pod/todolist-1-qw974            1/1     Running     0          37m
      pod/todolist-mongo-go-1-build   0/1     Error       0          38m
      NAME                               DESIRED   CURRENT   READY   AGE
      replicationcontroller/todolist-1   1         1         1       38m
      NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
      service/mongo      ClusterIP   172.30.2.86      <none>        27017/TCP   38m
      service/todolist   ClusterIP   172.30.225.191   <none>        8000/TCP    38m
      NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
      deployment.apps/mongo   1/1     1            1           38m
      NAME                               DESIRED   CURRENT   READY   AGE
      replicaset.apps/mongo-666589749c   1         1         1       38m
      NAME                                          REVISION   DESIRED   CURRENT   TRIGGERED BY
      deploymentconfig.apps.openshift.io/todolist   1          1         1         config,image(todolist-mongo-go:latest)
      NAME                                               TYPE     FROM         LATEST
      buildconfig.build.openshift.io/todolist-mongo-go   Docker   Git@master   1
      NAME                                           TYPE     FROM          STATUS                               STARTED          DURATION
      build.build.openshift.io/todolist-mongo-go-1   Docker   Git@8c32a90   Failed (PushImageToRegistryFailed)   38 minutes ago   2m7s
      NAME                                               IMAGE REPOSITORY                                                                      TAGS     UPDATED
      imagestream.image.openshift.io/todolist-mongo-go   image-registry.openshift-image-registry.svc:5000/mongo-persistent/todolist-mongo-go   latest   38 minutes ago
      NAME                                      HOST/PORT                                                      PATH   SERVICES   PORT    TERMINATION   WILDCARD
      route.route.openshift.io/todolist-route   todolist-route-mongo-persistent.apps.ocpm4202001.lnxero1.boe   /      todolist   <all>                 None
      NAME                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                AGE
      persistentvolumeclaim/mongo   Bound    pvc-ed480d38-5250-4873-8fd3-c6fe82ef43cd   1Gi        RWX            ocs-storagecluster-cephfs   38m
      
      #  oc get restore -n openshift-adp restore-mongo-withcsi1-nov3 -ojsonpath={.status.phase}
      Completed

       

      [root@m4202001 ~]# oc get vsr -n mongo-persistent 
      NAME        AGE 
      vsr-fvsvj   40m

      Actual results:

      Application is restored successfully on the new cluster , however the build pod goes into Error state as the image push to the internal registry fails on the restore cluster.

       

      [root@m4202001 ~]# oc logs pod/todolist-mongo-go-1-build -n mongo-persistent Defaulted container "docker-build" out of: docker-build, git-clone (init), manage-dockerfile (init) time="2022-11-03T10:47:18Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERL AY_FS_REDIRECT_DIR enabled" I1103 10:47:18.106388       1 defaults.go:112] Defaulting to storage driver "overlay" with options [mountopt=metacopy=on]. Caching blobs under "/var/cache/blobs". Pulling image golang:1.17.8-alpine ... Resolving "golang" using unqualified-search registries (/var/run/configs/openshift.io/build-system/registries.conf) Trying to pull registry.redhat.io/golang:1.17.8-alpine... time="2022-11-03T10:47:19Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://registry.redhat.io/golang:1.17.8-alpine: r eading manifest 1.17.8-alpine in registry.redhat.io/golang: unknown: Not Found" time="2022-11-03T10:47:21Z" level=warning msg="Failed, retrying in 2s ... (2/3). Error: initializing source docker://registry.redhat.io/golang:1.17.8-alpine: r eading manifest 1.17.8-alpine in registry.redhat.io/golang: unknown: Not Found" time="2022-11-03T10:47:23Z" level=warning msg="Failed, retrying in 4s ... (3/3). Error: initializing source docker://registry.redhat.io/golang:1.17.8-alpine: r eading manifest 1.17.8-alpine in registry.redhat.io/golang: unknown: Not Found" Trying to pull registry.access.redhat.com/golang:1.17.8-alpine... Trying to pull quay.io/golang:1.17.8-alpine... Trying to pull docker.io/library/golang:1.17.8-alpine... Getting image source signatures Copying blob sha256:86765c73189fadce4ca276af5f7d9440213d5986a9687b648a51d45978e3a8c4 Copying blob sha256:a27b630f446c3da376a30cf610e4bfa6847f8b87c83702c29e72b986f4e52d28 Copying blob sha256:72593bb761a2ce336603e9e84973d1e284f39e0188e758ad8bc22c45cd9b6395 Copying blob sha256:aaf480933488ffcb2c9c9269292e2e3d643bea6eeb04cf948790ca50666a61c8 Copying blob sha256:a2b3e51e0eff0c3e6118a6448425ada0b8e1725d738559d5b5e29ef58c570e8d Copying config sha256:77ddef1bc87c75f112a8c2ce8c1da837e2ff061b80fa32080e62d3952a04dc8f Writing manifest to image destination Storing signatures Adding transient rw bind mount for /run/secrets/rhsm [1/2] STEP 1/9: FROM golang:1.17.8-alpine AS build-env [1/2] STEP 2/9: RUN mkdir /build --> 842cfe48fb0 [1/2] STEP 3/9: WORKDIR /build --> 1da041ac644 [1/2] STEP 4/9: COPY apps/todolist-mongo-go/*.go . --> 066a5320cdf [1/2] STEP 5/9: COPY apps/todolist-mongo-go/go.mod . --> 5e4790ac3f6 [1/2] STEP 6/9: COPY apps/todolist-mongo-go/go.sum . --> 3834dc8fcf4 [1/2] STEP 7/9: RUN go mod download --> 2329ebb3698 [1/2] STEP 8/9: RUN go mod tidy --> 6802fe855cb [1/2] STEP 9/9: RUN CGO_ENABLED=0 GOOS=linux GOARCH=s390x go build -v -a -installsuffix cgo -o app internal/unsafeheader internal/itoa encoding internal/race unicode internal/cpu internal/goexperiment crypto/internal/subtle container/list unicode/utf16 math/bits runtime/internal/sys unicode/utf8 internal/abi sync/atomic runtime/internal/atomic crypto/subtle vendor/golang.org/x/crypto/cryptobyte/asn1 internal/nettrace vendor/golang.org/x/crypto/internal/subtle go.mongodb.org/mongo-driver/bson/bsonoptions runtime/internal/math go.mongodb.org/mongo-driver/bson/bsontype go.mongodb.org/mongo-driver/version internal/bytealg math runtime github.com/klauspost/compress internal/reflectlite sync internal/testlog internal/singleflight math/rand errors sort internal/oserror io path vendor/golang.org/x/net/dns/dnsmessage crypto/elliptic/internal/fiat strconv syscall bytes strings hash crypto/internal/randutil hash/crc32 crypto/hmac hash/adler32 crypto reflect crypto/rc4 vendor/golang.org/x/crypto/hkdf golang.org/x/crypto/pbkdf2 vendor/golang.org/x/text/transform golang.org/x/text/transform bufio net/http/internal/ascii regexp/syntax internal/syscall/execenv internal/syscall/unix time regexp context internal/poll io/fs encoding/binary internal/fmtsort golang.org/x/sync/errgroup os encoding/base64 crypto/ed25519/internal/edwards25519/field crypto/md5 crypto/sha512 crypto/cipher crypto/sha1 crypto/sha256 github.com/golang/snappy golang.org/x/sys/unix github.com/klauspost/compress/internal/snapref github.com/klauspost/compress/zstd/internal/xxhash github.com/xdg-go/pbkdf2 encoding/pem crypto/aes crypto/des crypto/ed25519/internal/edwards25519 fmt net path/filepath runtime/debug io/ioutil vendor/golang.org/x/sys/cpu os/exec encoding/json compress/flate encoding/hex math/big net/url vendor/golang.org/x/crypto/curve25519 log vendor/golang.org/x/net/http2/hpack mime mime/quotedprintable net/http/internal vendor/golang.org/x/text/unicode/norm github.com/go-stack/stack vendor/golang.org/x/crypto/chacha20 vendor/golang.org/x/crypto/poly1305 github.com/pkg/errors go.mongodb.org/mongo-driver/tag github.com/klauspost/compress/fse vendor/golang.org/x/text/unicode/bidi go.mongodb.org/mongo-driver/mongo/readpref golang.org/x/text/unicode/norm github.com/klauspost/compress/huff0 compress/gzip compress/zlib vendor/golang.org/x/crypto/chacha20poly1305 net/textproto go.mongodb.org/mongo-driver/mongo/address go.mongodb.org/mongo-driver/x/mongo/driver/dns github.com/sirupsen/logrus github.com/xdg-go/stringprep vendor/golang.org/x/text/secure/bidirule vendor/golang.org/x/net/idna crypto/dsa crypto/rand crypto/elliptic encoding/asn1 crypto/ed25519 crypto/rsa mime/multipart go.mongodb.org/mongo-driver/bson/primitive github.com/klauspost/compress/zstd go.mongodb.org/mongo-driver/internal/randutil vendor/golang.org/x/net/http/httpguts vendor/golang.org/x/net/http/httpproxy go.mongodb.org/mongo-driver/x/mongo/driver/uuid vendor/golang.org/x/crypto/cryptobyte vendor/golang.org/x/net/http/httpproxy go.mongodb.org/mongo-driver/x/mongo/driver/uuid vendor/golang.org/x/crypto/cryptobyte crypto/x509/pkix go.mongodb.org/mongo-driver/x/bsonx/bsoncore crypto/ecdsa go.mongodb.org/mongo-driver/bson/bsonrw go.mongodb.org/mongo-driver/mongo/readconcern go.mongodb.org/mongo-driver/x/mongo/driver/mongocrypt/options go.mongodb.org/mongo-driver/x/mongo/driver/wiremessage crypto/x509 go.mongodb.org/mongo-driver/x/mongo/driver/mongocrypt github.com/xdg-go/scram go.mongodb.org/mongo-driver/bson/bsoncodec crypto/tls github.com/youmark/pkcs8 golang.org/x/crypto/ocsp go.mongodb.org/mongo-driver/bson net/http/httptrace go.mongodb.org/mongo-driver/internal go.mongodb.org/mongo-driver/mongo/writeconcern go.mongodb.org/mongo-driver/x/bsonx net/http go.mongodb.org/mongo-driver/x/mongo/driver/connstring go.mongodb.org/mongo-driver/mongo/description go.mongodb.org/mongo-driver/event go.mongodb.org/mongo-driver/x/mongo/driver/session go.mongodb.org/mongo-driver/x/mongo/driver go.mongodb.org/mongo-driver/x/mongo/driver/operation go.mongodb.org/mongo-driver/mongo/options github.com/gorilla/mux github.com/rs/cors go.mongodb.org/mongo-driver/x/mongo/driver/ocsp go.mongodb.org/mongo-driver/x/mongo/driver/auth/internal/awsv4 go.mongodb.org/mongo-driver/x/mongo/driver/auth go.mongodb.org/mongo-driver/x/mongo/driver/topology go.mongodb.org/mongo-driver/mongo github.com/weshayutin/todolist-mongo-go --> 26272181930 [2/2] STEP 1/8: FROM scratch [2/2] STEP 2/8: COPY --from=build-env /build/app /app --> 18dcf9b5c3e [2/2] STEP 3/8: COPY apps/todolist-mongo-go/resources/ /resources/ --> 0b07d306e60 [2/2] STEP 4/8: COPY apps/todolist-mongo-go/index.html . --> 18c49192117 [2/2] STEP 5/8: EXPOSE 8000 --> fefac7ce9b7 [2/2] STEP 6/8: CMD ["./app"] --> d9fa56089de [2/2] STEP 7/8: ENV "OPENSHIFT_BUILD_NAME"="todolist-mongo-go-1" "OPENSHIFT_BUILD_NAMESPACE"="mongo-persistent" "OPENSHIFT_BUILD_SOURCE"="https://github.com/Sr avikaz/mig-demo-apps" "OPENSHIFT_BUILD_REFERENCE"="master" "OPENSHIFT_BUILD_COMMIT"="8c32a90cf30573955dc2f894be8f4d4df0a6205e" --> 3af6febb1f8 [2/2] STEP 8/8: LABEL "io.openshift.build.commit.author"="Sravika Balusu <sravika.balusu@ibm.com>" "io.openshift.build.commit.date"="Fri Oct 28 17:40:26 2022 + 0200" "io.openshift.build.commit.id"="8c32a90cf30573955dc2f894be8f4d4df0a6205e" "io.openshift.build.commit.message"="Updating the arch to s390x" "io.openshift. build.commit.ref"="master" "io.openshift.build.name"="todolist-mongo-go-1" "io.openshift.build.namespace"="mongo-persistent" "io.openshift.build.source-locatio n"="https://github.com/Sravikaz/mig-demo-apps" [2/2] COMMIT temp.builder.openshift.io/mongo-persistent/todolist-mongo-go-1:32e73353 --> be5574a49b2 Successfully tagged temp.builder.openshift.io/mongo-persistent/todolist-mongo-go-1:32e73353 be5574a49b29be836245f158a84eb6cc8cfc614c370c2d2a770a5086915d034e Pushing image image-registry.openshift-image-registry.svc:5000/mongo-persistent/todolist-mongo-go:latest ... Getting image source signatures Copying blob sha256:db0a92fdac4d51b347d2030e099b48b63beeb4cb88d24abeda93727cd9aaecd6 Copying blob sha256:bbaf53dba39788594f229611d3aa1fbcbff2f666761a4288ee3be1d6e21e8c1d Copying blob sha256:44aa4088990452a22552f03a8d8564a82902877cbbd88461a6c3721fdd21aca3 Warning: Push failed, retrying in 5s ... Getting image source signatures Copying blob sha256:db0a92fdac4d51b347d2030e099b48b63beeb4cb88d24abeda93727cd9aaecd6 Copying blob sha256:44aa4088990452a22552f03a8d8564a82902877cbbd88461a6c3721fdd21aca3 Copying blob sha256:bbaf53dba39788594f229611d3aa1fbcbff2f666761a4288ee3be1d6e21e8c1d Warning: Push failed, retrying in 5s ... Getting image source signatures Copying blob sha256:db0a92fdac4d51b347d2030e099b48b63beeb4cb88d24abeda93727cd9aaecd6 Copying blob sha256:bbaf53dba39788594f229611d3aa1fbcbff2f666761a4288ee3be1d6e21e8c1d Copying blob sha256:44aa4088990452a22552f03a8d8564a82902877cbbd88461a6c3721fdd21aca3 Warning: Push failed, retrying in 5s ... Registry server Address: Registry server User Name: serviceaccount Registry server Email: serviceaccount@example.org Registry server Password: <<non-empty>> error: build error: Failed to push image: trying to reuse blob sha256:bbaf53dba39788594f229611d3aa1fbcbff2f666761a4288ee3be1d6e21e8c1d at destination: pinging container registry image-registry.openshift-image-registry.svc:5000: Get "https://image-registry.openshift-image-registry.svc:5000/v2/": x509: certificate sign ed by unknown authority

       

      Expected results:

      Image push should not fail and the build pod should be in running state

      Additional info:

              rh-ee-rkedia Rishika Kedia
              mperetz@redhat.com Maya Peretz
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: