-
Bug
-
Resolution: Won't Do
-
Minor
-
None
-
Quality / Stability / Reliability
-
1
-
False
-
-
False
-
QE - Ack
-
ToDo
-
-
-
0
-
Very Likely
-
0
-
None
-
Unset
-
Unknown
-
No
Description of problem:
Image push to the openshift internal image registry fails after restoring the application on the new cluster using datamover
Version-Release number of selected component (if applicable):
How reproducible:
Always
Steps to Reproduce:
1.Deploy Openshift and ODF
2. Deploy a stateful application , which pushes its image to the internal openshift image registry; for example mongo-persistent
[root@m4205001 ~]# oc get all,pvc -n mongo-persistent NAME READY STATUS RESTARTS AGE pod/mongo-666589749c-7vdhm 1/1 Running 0 2m39s pod/todolist-1-deploy 0/1 Completed 0 98s pod/todolist-1-drlzk 1/1 Running 0 95s pod/todolist-mongo-go-1-build 0/1 Completed 0 2m39s NAME DESIRED CURRENT READY AGE replicationcontroller/todolist-1 1 1 1 98s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mongo ClusterIP 172.30.98.230 <none> 27017/TCP 2m39s service/todolist ClusterIP 172.30.7.247 <none> 8000/TCP 2m39s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/mongo 1/1 1 1 2m39s NAME DESIRED CURRENT READY AGE replicaset.apps/mongo-666589749c 1 1 1 2m39s NAME REVISION DESIRED CURRENT TRIGGERED BY deploymentconfig.apps.openshift.io/todolist 1 1 1 config,image(todolist-mongo-go:latest) NAME TYPE FROM LATEST buildconfig.build.openshift.io/todolist-mongo-go Docker Git@master 1 NAME TYPE FROM STATUS STARTED DURATION build.build.openshift.io/todolist-mongo-go-1 Docker Git@8c32a90 Complete 2 minutes ago 1m4s NAME IMAGE REPOSITORY TAGS UPDATED imagestream.image.openshift.io/todolist-mongo-go image-registry.openshift-image-registry.svc:5000/mongo-persistent/todolist-mongo-go latest About a minute ago NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD route.route.openshift.io/todolist-route todolist-route-mongo-persistent.apps.ocsm4205001.lnxero1.boe / todolist <all> None NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/mongo Bound pvc-4a37fb4e-d5a2-439d-8ae8-7e2cf847a267 1Gi RWX ocs-storagecluster-cephfs 2m39s
3. Make sure you have an appropriate StorageClass and VolumeShapshotClass and there is only one default of each.
- Include the label velero.io/csi-volumesnapshot-class: 'true' in your VolumeSnapshotClass to let Velero know which to use.
- deletionPolicy must be set to Retain for the VolumeSnapshotClass.
4. Install the OADP operator from the Operator Hub
5. Install the VolSync operator from the Operator Hub
6.
Configure a restic repository secret
cat << EOF > ./restic-secret.yaml apiVersion: v1 kind: Secret metadata: name: <secret-name> type: Opaque stringData: # The repository encryption key RESTIC_PASSWORD: my-secure-restic-password EOF
7.
Create a credentials-velero file with the AWS access key
# cat << EOF > ./credentials-velero > [default] > aws_access_key_id=<AWS_ACCESS_KEY_ID> > aws_secret_access_key=<AWS_SECRET_ACCESS_KEY> > EOF
8. Create a default Secret
# oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
9. Create a DPA instance from the opesnhft-console by enabling CSI
Operators -> Installed Operators -> OADP Operator -> Create DPA
kind: DataProtectionApplication apiVersion: oadp.openshift.io/v1alpha1 metadata: name: velero-sample namespace: openshift-adp spec: features: dataMover: enable: true credentialName: restic-repo-secret backupLocations: - velero: config: profile: default region: eu-central-1 credential: key: cloud name: cloud-credentials default: true objectStorage: bucket: myoadptestbucket prefix: velero provider: aws configuration: restic: enable: false velero: defaultPlugins: - openshift - aws - kubevirt - csi featureFlags: - EnableCSI snapshotLocations: - velero: config: profile: default region: eu-central-1 provider: aws
10. Created a backup of the application
# oc get backup -n openshift-adp mongo-backup-withcsi1-nov3 -ojsonpath='{.status.phase}' Complete
# oc get vsb -n mongo-persistent NAME AGE vsb-xzkbj 119m
11. Restore the application on a new cluster
[root@m4202001 ~]# oc get all,pvc -n mongo-persistent NAME READY STATUS RESTARTS AGE pod/mongo-666589749c-4cs7n 1/1 Running 0 37m pod/todolist-1-deploy 0/1 Completed 0 38m pod/todolist-1-qw974 1/1 Running 0 37m pod/todolist-mongo-go-1-build 0/1 Error 0 38m NAME DESIRED CURRENT READY AGE replicationcontroller/todolist-1 1 1 1 38m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mongo ClusterIP 172.30.2.86 <none> 27017/TCP 38m service/todolist ClusterIP 172.30.225.191 <none> 8000/TCP 38m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/mongo 1/1 1 1 38m NAME DESIRED CURRENT READY AGE replicaset.apps/mongo-666589749c 1 1 1 38m NAME REVISION DESIRED CURRENT TRIGGERED BY deploymentconfig.apps.openshift.io/todolist 1 1 1 config,image(todolist-mongo-go:latest) NAME TYPE FROM LATEST buildconfig.build.openshift.io/todolist-mongo-go Docker Git@master 1 NAME TYPE FROM STATUS STARTED DURATION build.build.openshift.io/todolist-mongo-go-1 Docker Git@8c32a90 Failed (PushImageToRegistryFailed) 38 minutes ago 2m7s NAME IMAGE REPOSITORY TAGS UPDATED imagestream.image.openshift.io/todolist-mongo-go image-registry.openshift-image-registry.svc:5000/mongo-persistent/todolist-mongo-go latest 38 minutes ago NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD route.route.openshift.io/todolist-route todolist-route-mongo-persistent.apps.ocpm4202001.lnxero1.boe / todolist <all> None NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/mongo Bound pvc-ed480d38-5250-4873-8fd3-c6fe82ef43cd 1Gi RWX ocs-storagecluster-cephfs 38m
# oc get restore -n openshift-adp restore-mongo-withcsi1-nov3 -ojsonpath={.status.phase} Completed
[root@m4202001 ~]# oc get vsr -n mongo-persistent NAME AGE vsr-fvsvj 40m
Actual results:
Application is restored successfully on the new cluster , however the build pod goes into Error state as the image push to the internal registry fails on the restore cluster.
[root@m4202001 ~]# oc logs pod/todolist-mongo-go-1-build -n mongo-persistent Defaulted container "docker-build" out of: docker-build, git-clone (init), manage-dockerfile (init) time="2022-11-03T10:47:18Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERL AY_FS_REDIRECT_DIR enabled" I1103 10:47:18.106388 1 defaults.go:112] Defaulting to storage driver "overlay" with options [mountopt=metacopy=on]. Caching blobs under "/var/cache/blobs". Pulling image golang:1.17.8-alpine ... Resolving "golang" using unqualified-search registries (/var/run/configs/openshift.io/build-system/registries.conf) Trying to pull registry.redhat.io/golang:1.17.8-alpine... time="2022-11-03T10:47:19Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://registry.redhat.io/golang:1.17.8-alpine: r eading manifest 1.17.8-alpine in registry.redhat.io/golang: unknown: Not Found" time="2022-11-03T10:47:21Z" level=warning msg="Failed, retrying in 2s ... (2/3). Error: initializing source docker://registry.redhat.io/golang:1.17.8-alpine: r eading manifest 1.17.8-alpine in registry.redhat.io/golang: unknown: Not Found" time="2022-11-03T10:47:23Z" level=warning msg="Failed, retrying in 4s ... (3/3). Error: initializing source docker://registry.redhat.io/golang:1.17.8-alpine: r eading manifest 1.17.8-alpine in registry.redhat.io/golang: unknown: Not Found" Trying to pull registry.access.redhat.com/golang:1.17.8-alpine... Trying to pull quay.io/golang:1.17.8-alpine... Trying to pull docker.io/library/golang:1.17.8-alpine... Getting image source signatures Copying blob sha256:86765c73189fadce4ca276af5f7d9440213d5986a9687b648a51d45978e3a8c4 Copying blob sha256:a27b630f446c3da376a30cf610e4bfa6847f8b87c83702c29e72b986f4e52d28 Copying blob sha256:72593bb761a2ce336603e9e84973d1e284f39e0188e758ad8bc22c45cd9b6395 Copying blob sha256:aaf480933488ffcb2c9c9269292e2e3d643bea6eeb04cf948790ca50666a61c8 Copying blob sha256:a2b3e51e0eff0c3e6118a6448425ada0b8e1725d738559d5b5e29ef58c570e8d Copying config sha256:77ddef1bc87c75f112a8c2ce8c1da837e2ff061b80fa32080e62d3952a04dc8f Writing manifest to image destination Storing signatures Adding transient rw bind mount for /run/secrets/rhsm [1/2] STEP 1/9: FROM golang:1.17.8-alpine AS build-env [1/2] STEP 2/9: RUN mkdir /build --> 842cfe48fb0 [1/2] STEP 3/9: WORKDIR /build --> 1da041ac644 [1/2] STEP 4/9: COPY apps/todolist-mongo-go/*.go . --> 066a5320cdf [1/2] STEP 5/9: COPY apps/todolist-mongo-go/go.mod . --> 5e4790ac3f6 [1/2] STEP 6/9: COPY apps/todolist-mongo-go/go.sum . --> 3834dc8fcf4 [1/2] STEP 7/9: RUN go mod download --> 2329ebb3698 [1/2] STEP 8/9: RUN go mod tidy --> 6802fe855cb [1/2] STEP 9/9: RUN CGO_ENABLED=0 GOOS=linux GOARCH=s390x go build -v -a -installsuffix cgo -o app internal/unsafeheader internal/itoa encoding internal/race unicode internal/cpu internal/goexperiment crypto/internal/subtle container/list unicode/utf16 math/bits runtime/internal/sys unicode/utf8 internal/abi sync/atomic runtime/internal/atomic crypto/subtle vendor/golang.org/x/crypto/cryptobyte/asn1 internal/nettrace vendor/golang.org/x/crypto/internal/subtle go.mongodb.org/mongo-driver/bson/bsonoptions runtime/internal/math go.mongodb.org/mongo-driver/bson/bsontype go.mongodb.org/mongo-driver/version internal/bytealg math runtime github.com/klauspost/compress internal/reflectlite sync internal/testlog internal/singleflight math/rand errors sort internal/oserror io path vendor/golang.org/x/net/dns/dnsmessage crypto/elliptic/internal/fiat strconv syscall bytes strings hash crypto/internal/randutil hash/crc32 crypto/hmac hash/adler32 crypto reflect crypto/rc4 vendor/golang.org/x/crypto/hkdf golang.org/x/crypto/pbkdf2 vendor/golang.org/x/text/transform golang.org/x/text/transform bufio net/http/internal/ascii regexp/syntax internal/syscall/execenv internal/syscall/unix time regexp context internal/poll io/fs encoding/binary internal/fmtsort golang.org/x/sync/errgroup os encoding/base64 crypto/ed25519/internal/edwards25519/field crypto/md5 crypto/sha512 crypto/cipher crypto/sha1 crypto/sha256 github.com/golang/snappy golang.org/x/sys/unix github.com/klauspost/compress/internal/snapref github.com/klauspost/compress/zstd/internal/xxhash github.com/xdg-go/pbkdf2 encoding/pem crypto/aes crypto/des crypto/ed25519/internal/edwards25519 fmt net path/filepath runtime/debug io/ioutil vendor/golang.org/x/sys/cpu os/exec encoding/json compress/flate encoding/hex math/big net/url vendor/golang.org/x/crypto/curve25519 log vendor/golang.org/x/net/http2/hpack mime mime/quotedprintable net/http/internal vendor/golang.org/x/text/unicode/norm github.com/go-stack/stack vendor/golang.org/x/crypto/chacha20 vendor/golang.org/x/crypto/poly1305 github.com/pkg/errors go.mongodb.org/mongo-driver/tag github.com/klauspost/compress/fse vendor/golang.org/x/text/unicode/bidi go.mongodb.org/mongo-driver/mongo/readpref golang.org/x/text/unicode/norm github.com/klauspost/compress/huff0 compress/gzip compress/zlib vendor/golang.org/x/crypto/chacha20poly1305 net/textproto go.mongodb.org/mongo-driver/mongo/address go.mongodb.org/mongo-driver/x/mongo/driver/dns github.com/sirupsen/logrus github.com/xdg-go/stringprep vendor/golang.org/x/text/secure/bidirule vendor/golang.org/x/net/idna crypto/dsa crypto/rand crypto/elliptic encoding/asn1 crypto/ed25519 crypto/rsa mime/multipart go.mongodb.org/mongo-driver/bson/primitive github.com/klauspost/compress/zstd go.mongodb.org/mongo-driver/internal/randutil vendor/golang.org/x/net/http/httpguts vendor/golang.org/x/net/http/httpproxy go.mongodb.org/mongo-driver/x/mongo/driver/uuid vendor/golang.org/x/crypto/cryptobyte vendor/golang.org/x/net/http/httpproxy go.mongodb.org/mongo-driver/x/mongo/driver/uuid vendor/golang.org/x/crypto/cryptobyte crypto/x509/pkix go.mongodb.org/mongo-driver/x/bsonx/bsoncore crypto/ecdsa go.mongodb.org/mongo-driver/bson/bsonrw go.mongodb.org/mongo-driver/mongo/readconcern go.mongodb.org/mongo-driver/x/mongo/driver/mongocrypt/options go.mongodb.org/mongo-driver/x/mongo/driver/wiremessage crypto/x509 go.mongodb.org/mongo-driver/x/mongo/driver/mongocrypt github.com/xdg-go/scram go.mongodb.org/mongo-driver/bson/bsoncodec crypto/tls github.com/youmark/pkcs8 golang.org/x/crypto/ocsp go.mongodb.org/mongo-driver/bson net/http/httptrace go.mongodb.org/mongo-driver/internal go.mongodb.org/mongo-driver/mongo/writeconcern go.mongodb.org/mongo-driver/x/bsonx net/http go.mongodb.org/mongo-driver/x/mongo/driver/connstring go.mongodb.org/mongo-driver/mongo/description go.mongodb.org/mongo-driver/event go.mongodb.org/mongo-driver/x/mongo/driver/session go.mongodb.org/mongo-driver/x/mongo/driver go.mongodb.org/mongo-driver/x/mongo/driver/operation go.mongodb.org/mongo-driver/mongo/options github.com/gorilla/mux github.com/rs/cors go.mongodb.org/mongo-driver/x/mongo/driver/ocsp go.mongodb.org/mongo-driver/x/mongo/driver/auth/internal/awsv4 go.mongodb.org/mongo-driver/x/mongo/driver/auth go.mongodb.org/mongo-driver/x/mongo/driver/topology go.mongodb.org/mongo-driver/mongo github.com/weshayutin/todolist-mongo-go --> 26272181930 [2/2] STEP 1/8: FROM scratch [2/2] STEP 2/8: COPY --from=build-env /build/app /app --> 18dcf9b5c3e [2/2] STEP 3/8: COPY apps/todolist-mongo-go/resources/ /resources/ --> 0b07d306e60 [2/2] STEP 4/8: COPY apps/todolist-mongo-go/index.html . --> 18c49192117 [2/2] STEP 5/8: EXPOSE 8000 --> fefac7ce9b7 [2/2] STEP 6/8: CMD ["./app"] --> d9fa56089de [2/2] STEP 7/8: ENV "OPENSHIFT_BUILD_NAME"="todolist-mongo-go-1" "OPENSHIFT_BUILD_NAMESPACE"="mongo-persistent" "OPENSHIFT_BUILD_SOURCE"="https://github.com/Sr avikaz/mig-demo-apps" "OPENSHIFT_BUILD_REFERENCE"="master" "OPENSHIFT_BUILD_COMMIT"="8c32a90cf30573955dc2f894be8f4d4df0a6205e" --> 3af6febb1f8 [2/2] STEP 8/8: LABEL "io.openshift.build.commit.author"="Sravika Balusu <sravika.balusu@ibm.com>" "io.openshift.build.commit.date"="Fri Oct 28 17:40:26 2022 + 0200" "io.openshift.build.commit.id"="8c32a90cf30573955dc2f894be8f4d4df0a6205e" "io.openshift.build.commit.message"="Updating the arch to s390x" "io.openshift. build.commit.ref"="master" "io.openshift.build.name"="todolist-mongo-go-1" "io.openshift.build.namespace"="mongo-persistent" "io.openshift.build.source-locatio n"="https://github.com/Sravikaz/mig-demo-apps" [2/2] COMMIT temp.builder.openshift.io/mongo-persistent/todolist-mongo-go-1:32e73353 --> be5574a49b2 Successfully tagged temp.builder.openshift.io/mongo-persistent/todolist-mongo-go-1:32e73353 be5574a49b29be836245f158a84eb6cc8cfc614c370c2d2a770a5086915d034e Pushing image image-registry.openshift-image-registry.svc:5000/mongo-persistent/todolist-mongo-go:latest ... Getting image source signatures Copying blob sha256:db0a92fdac4d51b347d2030e099b48b63beeb4cb88d24abeda93727cd9aaecd6 Copying blob sha256:bbaf53dba39788594f229611d3aa1fbcbff2f666761a4288ee3be1d6e21e8c1d Copying blob sha256:44aa4088990452a22552f03a8d8564a82902877cbbd88461a6c3721fdd21aca3 Warning: Push failed, retrying in 5s ... Getting image source signatures Copying blob sha256:db0a92fdac4d51b347d2030e099b48b63beeb4cb88d24abeda93727cd9aaecd6 Copying blob sha256:44aa4088990452a22552f03a8d8564a82902877cbbd88461a6c3721fdd21aca3 Copying blob sha256:bbaf53dba39788594f229611d3aa1fbcbff2f666761a4288ee3be1d6e21e8c1d Warning: Push failed, retrying in 5s ... Getting image source signatures Copying blob sha256:db0a92fdac4d51b347d2030e099b48b63beeb4cb88d24abeda93727cd9aaecd6 Copying blob sha256:bbaf53dba39788594f229611d3aa1fbcbff2f666761a4288ee3be1d6e21e8c1d Copying blob sha256:44aa4088990452a22552f03a8d8564a82902877cbbd88461a6c3721fdd21aca3 Warning: Push failed, retrying in 5s ... Registry server Address: Registry server User Name: serviceaccount Registry server Email: serviceaccount@example.org Registry server Password: <<non-empty>> error: build error: Failed to push image: trying to reuse blob sha256:bbaf53dba39788594f229611d3aa1fbcbff2f666761a4288ee3be1d6e21e8c1d at destination: pinging container registry image-registry.openshift-image-registry.svc:5000: Get "https://image-registry.openshift-image-registry.svc:5000/v2/": x509: certificate sign ed by unknown authority
Expected results:
Image push should not fail and the build pod should be in running state
Additional info: