Major Description of problem:
when I verify https://issues.redhat.com/browse/OCPBUGS-18305, I got https://github.com/operator-framework/operator-lifecycle-manager/issues/3010.
and then I got notice form storage qe team on they are failing to install lso operator with error, like
09-11 22:23:48.750 Message: constraints not satisfiable: @existing/e2e-test-storage-lso-2fgs5//local-storage-operator.v4.14.0-202308281027 and qe-app-registry/openshift-marketplace/stable/local-storage-operator.v4.14.0-202308281027 originate from package local-storage-operator, subscription lso-sub-2i71zf1q requires qe-app-registry/openshift-marketplace/stable/local-storage-operator.v4.14.0-202308281027, subscription lso-sub-2i71zf1q exists, clusterserviceversion local-storage-operator.v4.14.0-202308281027 exists and is not referenced by a subscription
and then I think it is different with https://github.com/operator-framework/operator-lifecycle-manager/issues/3010 because it install operator in different ns and with differnt sub name.
so, I create this bug to track the issue.
Version-Release number of selected component (if applicable):
[root@preserve-olm-env2 bin]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.14.0-0.nightly-2023-09-12-195514 True False 5h42m Cluster version is 4.14.0-0.nightly-2023-09-12-195514
How reproducible:
not always, but happen some times.
Steps to Reproduce:
1. install the operator in different ns and with different sub name at same time 2. delete them 3. reinstall them
here is the script
#!/bin/bash
function createCatsrc(){
oc create -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: test-app-registry
namespace: openshift-marketplace
spec:
displayName: test Operators
image: quay.io/openshift-qe-optional-operators/aosqe-index:v4.14
publisher: OpenShift TEST
sourceType: grpc
updateStrategy:
registryPoll:
interval: 15m
EOF
}
function createNs(){
local rid=$1
oc create ns "testns$rid"
}
function createSub(){
local rid=$1
oc create -f - <<EOF
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: local-operator-group
namespace: "testns$rid"
spec:
targetNamespaces:
- "testns$rid"
EOF
oc create -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: "lso-$rid"
namespace: "testns$rid"
spec:
channel: stable
installPlanApproval: Automatic
name: local-storage-operator
source: test-app-registry
sourceNamespace: openshift-marketplace
EOF
}
function deleteNs(){
local rid=$1
oc delete ns "testns$rid"
}
function watchCsvStatus(){
local rid=$1
local subName=lso-$rid
subcommand="oc get sub -n testns$rid $subName -o jsonpath={.status.installedCSV}"
declare -i count=0
while true; do
count+=1
csv=$($subcommand 2> /dev/null)
if [[ $csv != "" ]]; then
echo "can $csv is found for $subName"
break
fi
echo "can not get csv of $subName, try next"
# If we're stuck in this loop then it's likely the issue has been reproduced
sleep 2
if [ $count -gt 150 ]; then
echo "the csv can not installed"
oc get sub -n testns$rid $subName -o jsonpath={.status} | jq .
exit 1
fi
done
echo "Checking status of $csv..."
csvcommand="oc get csv -n testns$rid $csv -o jsonpath={.status.phase}"
while true; do
status=$($csvcommand 2> /dev/null)
printf "Current Status: $status\033[0K\r"
if [[ $status == "Succeeded" || $status == "Failed" ]]; then
watchResult=$status
echo
return 0
fi
# If we're stuck in this loop then it's likely the issue has been reproduced
sleep 2
done
}
createCatsrc
declare -i iterations=0
while true; do
iterations+=1
random_number1=$(( RANDOM % 1000 ))
random_number2=$(( RANDOM % 1000 ))
random_number3=$(( RANDOM % 1000 ))
createNs $random_number1
createNs $random_number2
createNs $random_number3
createSub $random_number1
createSub $random_number2
createSub $random_number3
watchCsvStatus $random_number1
if [[ $watchResult == "Failed" ]]; then
echo "the issue is reproduced"
exit 0
fi
watchCsvStatus $random_number2
if [[ $watchResult == "Failed" ]]; then
echo "the issue is reproduced"
exit 0
fi
watchCsvStatus $random_number3
if [[ $watchResult == "Failed" ]]; then
echo "the issue is reproduced"
exit 0
fi
deleteNs $random_number1
deleteNs $random_number2
deleteNs $random_number3
echo "Iterations: $iterations"
if [ $iterations -gt 5 ]; then
echo "we test it for $iterations times"
exit 0
fi
# sleep 60
done
Actual results:
the sub can not install.
here is the output:
[root@preserve-olm-env2 OCPBUGS-18511]# ./reproduce.sh catalogsource.operators.coreos.com/test-app-registry created namespace/testns393 created namespace/testns288 created namespace/testns988 created operatorgroup.operators.coreos.com/local-operator-group created subscription.operators.coreos.com/lso-393 created operatorgroup.operators.coreos.com/local-operator-group created subscription.operators.coreos.com/lso-288 created operatorgroup.operators.coreos.com/local-operator-group created subscription.operators.coreos.com/lso-988 created can not get csv of lso-393, try next can not get csv of lso-393, try next can not get csv of lso-393, try next can not get csv of lso-393, try next can not get csv of lso-393, try next can not get csv of lso-393, try next can not get csv of lso-393, try next can not get csv of lso-393, try next can not get csv of lso-393, try next can not get csv of lso-393, try next can not get csv of lso-393, try next can not get csv of lso-393, try next can not get csv of lso-393, try next can not get csv of lso-393, try next can not get csv of lso-393, try next can local-storage-operator.v4.14.0-202309012142 is found for lso-393 Checking status of local-storage-operator.v4.14.0-202309012142... Current Status: Succeeded can local-storage-operator.v4.14.0-202309012142 is found for lso-288 Checking status of local-storage-operator.v4.14.0-202309012142... Current Status: Succeeded can not get csv of lso-988, try next can not get csv of lso-988, try next can not get csv of lso-988, try next ... the csv can not installed { "catalogHealth": [ { "catalogSourceRef": { "apiVersion": "operators.coreos.com/v1alpha1", "kind": "CatalogSource", "name": "certified-operators", "namespace": "openshift-marketplace", "resourceVersion": "184546", "uid": "c59efaa1-d361-4e24-be81-5bb0db5fe1a6" }, "healthy": true, "lastUpdated": "2023-09-15T06:21:44Z" }, { "catalogSourceRef": { "apiVersion": "operators.coreos.com/v1alpha1", "kind": "CatalogSource", "name": "community-operators", "namespace": "openshift-marketplace", "resourceVersion": "183278", "uid": "5b35d161-603b-4556-9453-a69776651c86" }, "healthy": true, "lastUpdated": "2023-09-15T06:21:44Z" }, { "catalogSourceRef": { "apiVersion": "operators.coreos.com/v1alpha1", "kind": "CatalogSource", "name": "qe-app-registry", "namespace": "openshift-marketplace", "resourceVersion": "178012", "uid": "2a53cd7d-fab8-4eea-a597-d0d8c68e8401" }, "healthy": true, "lastUpdated": "2023-09-15T06:21:44Z" }, { "catalogSourceRef": { "apiVersion": "operators.coreos.com/v1alpha1", "kind": "CatalogSource", "name": "redhat-marketplace", "namespace": "openshift-marketplace", "resourceVersion": "184556", "uid": "6c2ace10-6ceb-49b4-9d4e-2d20247697a7" }, "healthy": true, "lastUpdated": "2023-09-15T06:21:44Z" }, { "catalogSourceRef": { "apiVersion": "operators.coreos.com/v1alpha1", "kind": "CatalogSource", "name": "redhat-operators", "namespace": "openshift-marketplace", "resourceVersion": "183923", "uid": "fb4f2e5f-0f54-4356-9b97-f5cdebd7f171" }, "healthy": true, "lastUpdated": "2023-09-15T06:21:44Z" }, { "catalogSourceRef": { "apiVersion": "operators.coreos.com/v1alpha1", "kind": "CatalogSource", "name": "test-app-registry", "namespace": "openshift-marketplace", "resourceVersion": "185236", "uid": "ac90c5e8-bf01-455a-a294-1d8be28d4c28" }, "healthy": true, "lastUpdated": "2023-09-15T06:21:44Z" } ], "conditions": [ { "message": "constraints not satisfiable: @existing/testns988//local-storage-operator.v4.14.0-202309012142 and test-app-registry/openshift-marketplace/stable/local-storage-operator.v4.14.0-202309012142 originate from package local-storage-operator, subscription lso-988 exists, subscription lso-988 requires test-app-registry/openshift-marketplace/stable/local-storage-operator.v4.14.0-202309012142, clusterserviceversion local-storage-operator.v4.14.0-202309012142 exists and is not referenced by a subscription", "reason": "ConstraintsNotSatisfiable", "status": "True", "type": "ResolutionFailed" }, { "lastTransitionTime": "2023-09-15T06:21:44Z", "message": "all available catalogsources are healthy", "reason": "AllCatalogSourcesHealthy", "status": "False", "type": "CatalogSourcesUnhealthy" } ], "lastUpdated": "2023-09-15T06:22:46Z" }
actually there is already csv in ns testns988
[root@preserve-olm-env2 cases]# oc get csv NAME DISPLAY VERSION REPLACES PHASE elasticsearch-operator.v5.8.0 OpenShift Elasticsearch Operator 5.8.0 Succeeded local-storage-operator.v4.14.0-202309012142 Local Storage 4.14.0-202309012142 Succeeded loki-operator.v5.8.0 Loki Operator 5.8.0 Succeeded
the output of sub
[root@preserve-olm-env2 cases]# oc get sub lso-988 -o yaml apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: creationTimestamp: "2023-09-15T06:21:21Z" generation: 1 labels: operators.coreos.com/local-storage-operator.testns988: "" name: lso-988 namespace: testns988 resourceVersion: "186250" uid: 51ec09fa-b6e1-4483-b0d6-9562c2b8304b spec: channel: stable installPlanApproval: Automatic name: local-storage-operator source: test-app-registry sourceNamespace: openshift-marketplace status: catalogHealth: - catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: certified-operators namespace: openshift-marketplace resourceVersion: "184546" uid: c59efaa1-d361-4e24-be81-5bb0db5fe1a6 healthy: true lastUpdated: "2023-09-15T06:21:44Z" - catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: community-operators namespace: openshift-marketplace resourceVersion: "183278" uid: 5b35d161-603b-4556-9453-a69776651c86 healthy: true lastUpdated: "2023-09-15T06:21:44Z" - catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: qe-app-registry namespace: openshift-marketplace resourceVersion: "178012" uid: 2a53cd7d-fab8-4eea-a597-d0d8c68e8401 healthy: true lastUpdated: "2023-09-15T06:21:44Z" - catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: redhat-marketplace namespace: openshift-marketplace resourceVersion: "184556" uid: 6c2ace10-6ceb-49b4-9d4e-2d20247697a7 healthy: true lastUpdated: "2023-09-15T06:21:44Z" - catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: redhat-operators namespace: openshift-marketplace resourceVersion: "183923" uid: fb4f2e5f-0f54-4356-9b97-f5cdebd7f171 healthy: true lastUpdated: "2023-09-15T06:21:44Z" - catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: test-app-registry namespace: openshift-marketplace resourceVersion: "185236" uid: ac90c5e8-bf01-455a-a294-1d8be28d4c28 healthy: true lastUpdated: "2023-09-15T06:21:44Z" conditions: - message: 'constraints not satisfiable: @existing/testns988//local-storage-operator.v4.14.0-202309012142 and test-app-registry/openshift-marketplace/stable/local-storage-operator.v4.14.0-202309012142 originate from package local-storage-operator, subscription lso-988 exists, subscription lso-988 requires test-app-registry/openshift-marketplace/stable/local-storage-operator.v4.14.0-202309012142, clusterserviceversion local-storage-operator.v4.14.0-202309012142 exists and is not referenced by a subscription' reason: ConstraintsNotSatisfiable status: "True" type: ResolutionFailed - lastTransitionTime: "2023-09-15T06:21:44Z" message: all available catalogsources are healthy reason: AllCatalogSourcesHealthy status: "False" type: CatalogSourcesUnhealthy lastUpdated: "2023-09-15T06:22:46Z"
Expected results:
can install it successfully each time
Additional info:
please run the script to reproduce it.
