-
Bug
-
Resolution: Unresolved
-
Normal
-
None
-
4.15.0
-
None
-
Yes
-
5
-
OTA 254
-
1
-
Rejected
-
False
-
Description of problem:
While failing to verify a release(not the 1st time to fail verifying release) during upgrade, no RetrievePayloadFailed event found in cvo ns, but cvo has logged it as RetrievePayloadFailed. there is only RetrievePayload event rotate per 5min.
//no RetrievePayloadFailed event to indicate the verify failure
# ./oc -n openshift-cluster-version get events|grep b2becd2a761cd4a582c7e6587a253bfbcf091b7e014dee82e5ed7bd7c768ea83
3m58s Normal RetrievePayload clusterversion/version Retrieving and verifying payload version="" image="registry.ci.openshift.org/ocp/release@sha256:b2becd2a761cd4a582c7e6587a253bfbcf091b7e014dee82e5ed7bd7c768ea83"
//cvo has logged it.
# ./oc -n openshift-cluster-version logs cluster-version-operator-6fffdf5979-kv7b9|grep b2becd2a761cd4a582c7e6587a253bfbcf091b7e014dee82e5ed7bd7c768ea83|grep RetrievePayloadFailed|tail -n1
I0112 07:26:08.193579 1 event.go:298] Event(v1.ObjectReference{Kind:"ClusterVersion", Namespace:"openshift-cluster-version", Name:"version", UID:"", APIVersion:"config.openshift.io/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RetrievePayloadFailed' Retrieving payload failed version="" image="registry.ci.openshift.org/ocp/release@sha256:b2becd2a761cd4a582c7e6587a253bfbcf091b7e014dee82e5ed7bd7c768ea83" failure=The update cannot be verified: unable to verify sha256:b2becd2a761cd4a582c7e6587a253bfbcf091b7e014dee82e5ed7bd7c768ea83 against keyrings: verifier-public-key-redhat // [2024-01-12T07:26:08Z: prefix sha256-b2becd2a761cd4a582c7e6587a253bfbcf091b7e014dee82e5ed7bd7c768ea83 in config map signatures-managed: no more signatures to check, 2024-01-12T07:26:08Z: unable to retrieve signature from https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=b2becd2a761cd4a582c7e6587a253bfbcf091b7e014dee82e5ed7bd7c768ea83/signature-1: no more signatures to check, 2024-01-12T07:26:08Z: unable to retrieve signature from https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/sha256=b2becd2a761cd4a582c7e6587a253bfbcf091b7e014dee82e5ed7bd7c768ea83/signature-1: no more signatures to check, 2024-01-12T07:26:08Z: parallel signature store wrapping containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release, containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release: no more signatures to check, 2024-01-12T07:26:08Z: serial signature store wrapping ClusterVersion signatureStores unset, falling back to default stores, parallel signature store wrapping containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release, containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release: no more signatures to check, 2024-01-12T07:26:08Z: serial signature store wrapping config maps in openshift-config-managed with label "release.openshift.io/verification-signatures", serial signature store wrapping ClusterVersion signatureStores unset, falling back to default stores, parallel signature store wrapping containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release, containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release: no more signatures to check]
//there is only the old RetrievePayloadFailed event when failing to verify the release payload for the 1st time.
# ./oc -n openshift-cluster-version get events|grep "RetrievePayload"| tail -n2
15m Warning RetrievePayloadFailed clusterversion/version (combined from similar events): Retrieving payload failed version="" image="registry.ci.openshift.org/ocp/release@sha256:bcdf0531023624a83f0713e465d1fd3ebc71b0dea898ec722dba12bdc68ee658" failure=The update cannot be verified: unable to verify sha256:bcdf0531023624a83f0713e465d1fd3ebc71b0dea898ec722dba12bdc68ee658 against keyrings: verifier-public-key-redhat // [2024-01-12T06:25:23Z: prefix sha256-bcdf0531023624a83f0713e465d1fd3ebc71b0dea898ec722dba12bdc68ee658 in config map signatures-managed: no more signatures to check, 2024-01-12T06:25:23Z: unable to retrieve signature from https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=bcdf0531023624a83f0713e465d1fd3ebc71b0dea898ec722dba12bdc68ee658/signature-1: no more signatures to check, 2024-01-12T06:25:23Z: unable to retrieve signature from https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/sha256=bcdf0531023624a83f0713e465d1fd3ebc71b0dea898ec722dba12bdc68ee658/signature-1: no more signatures to check, 2024-01-12T06:25:23Z: parallel signature store wrapping containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release, containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release: no more signatures to check, 2024-01-12T06:25:23Z: serial signature store wrapping ClusterVersion signatureStores unset, falling back to default stores, parallel signature store wrapping containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release, containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release: no more signatures to check, 2024-01-12T06:25:23Z: serial signature store wrapping config maps in openshift-config-managed with label "release.openshift.io/verification-signatures", serial signature store wrapping ClusterVersion signatureStores unset, falling back to default stores, parallel signature store wrapping containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release, containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release: no more signatures to check]
117s Normal RetrievePayload clusterversion/version Retrieving and verifying payload version="" image="registry.ci.openshift.org/ocp/release@sha256:b2becd2a761cd4a582c7e6587a253bfbcf091b7e014dee82e5ed7bd7c768ea83"
Version-Release number of selected component (if applicable):
4.15.0-0.nightly-2024-01-10-101042
How reproducible:
always
Steps to Reproduce:
1. trigger upgrade to a nightly build A without signature(registry.ci.openshift.org/ocp/release@sha256:bcdf0531023624a83f0713e465d1fd3ebc71b0dea898ec722dba12bdc68ee658)
2. check there is RetrievePayloadFailed event of A(expected)
3. clear above unstarted upgrade
4. trigger upgrade to a nightly build B without signature(registry.ci.openshift.org/ocp/release@sha256:b2becd2a761cd4a582c7e6587a253bfbcf091b7e014dee82e5ed7bd7c768ea83)
Actual results:
no RetrievePayloadFailed event while failing to verify target payload B
Expected results:
RetrievePayloadFailed event about B should be return
Additional info:
no such issue in 4.14, should be regression of https://issues.redhat.com/browse/OCPBUGS-25055