Loading...

Type: Bug
Resolution: Unresolved
Priority: Undefined
Fix Version/s: None
Affects Version/s: 4.21, 4.22
Component/s: HyperShift
Labels:
None

Activity Type:
None
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Story Points:
None
Severity:
Moderate
Regression:
None

Target Backport Versions:
None
Target Version:
None
Release Blocker:
None
Sprint:
None

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

PX Impact Score:

Release Note Status:
None
Release Note Type:
None
Release Note Text:
None

Escape Reason:
None
Escape Impact:
None
Corrective Measures:
None
SDLC stage when should've been found:
None

Description of problem

CI is flaky because of test failures such as the following:

{Failed  === RUN   TestCreateCluster/Main/Check_if_the_config.json_is_correct_in_all_of_the_nodes
    globalps.go:209: Creating kubelet config verifier DaemonSet
    globalps.go:214: Waiting for OVN, GlobalPullSecret, Konnectivity and kubelet config verifier DaemonSets to be ready
    util.go:2095: Waiting for ovnkube-node DaemonSet to be ready with 1 nodes
    util.go:2137: DaemonSet ovnkube-node ready: 1/1 pods
    util.go:2145: ✓ ovnkube-node DaemonSet is ready
    util.go:2095: Waiting for global-pull-secret-syncer DaemonSet to be ready with 1 nodes
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2133: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready
    util.go:2102: Failed to get DaemonSet global-pull-secret-syncer: context deadline exceeded
    globalps.go:226: 
        failed to wait for DaemonSets to be ready
        Unexpected error:
            <*fmt.wrapError | 0xc002202000>: 
            failed to wait for DaemonSet global-pull-secret-syncer to be ready: context deadline exceeded
            {
                msg: "failed to wait for DaemonSet global-pull-secret-syncer to be ready: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            }
        occurred
        --- FAIL: TestCreateCluster/Main/Check_if_the_config.json_is_correct_in_all_of_the_nodes (1200.51s)
}

This particular failure comes from https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-ingress-operator/1329/pull-ci-openshift-cluster-ingress-operator-master-hypershift-e2e-aks/2003561199664369664. Search.ci has other similar failures.

Version-Release number of selected component (if applicable)

I have seen this in 4.22 and 4.21 CI jobs.

How reproducible

Presently, search.ci shows the following stats for the past two days:

pull-ci-openshift-hypershift-main-e2e-aks (all) - 19 runs, 53% failed, 40% of failures match = 21% impact
pull-ci-openshift-cluster-ingress-operator-master-hypershift-e2e-aks (all) - 2 runs, 50% failed, 100% of failures match = 50% impact
periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aks (all) - 32 runs, 53% failed, 12% of failures match = 6% impact
periodic-ci-openshift-hypershift-release-4.22-periodics-e2e-aks (all) - 26 runs, 46% failed, 17% of failures match = 8% impact
periodic-ci-openshift-hypershift-release-4.22-periodics-e2e-aws-external-oidc (all) - 4 runs, 50% failed, 50% of failures match = 25% impact
pull-ci-openshift-hypershift-main-e2e-aks-4-21 (all) - 21 runs, 52% failed, 18% of failures match = 10% impact
periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aws-external-oidc (all) - 4 runs, 75% failed, 33% of failures match = 25% impact
periodic-ci-openshift-aws-karpenter-provider-aws-release-4.21-periodics-e2e-hypershift (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
pull-ci-openshift-csi-operator-main-hypershift-e2e-aks (all) - 6 runs, 67% failed, 25% of failures match = 17% impact

Searching over only "hypershift" jobs in the past 14 days shows the following:

pull-ci-openshift-hypershift-main-e2e-aks (all) - 401 runs, 51% failed, 8% of failures match = 4% impact
pull-ci-openshift-cluster-ingress-operator-master-hypershift-e2e-aks (all) - 13 runs, 23% failed, 67% of failures match = 15% impact
periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aks (all) - 189 runs, 50% failed, 16% of failures match = 8% impact
periodic-ci-openshift-hypershift-release-4.22-periodics-e2e-aks (all) - 179 runs, 36% failed, 14% of failures match = 5% impact
periodic-ci-openshift-hypershift-release-4.22-periodics-e2e-aws-external-oidc (all) - 26 runs, 54% failed, 43% of failures match = 23% impact
pull-ci-openshift-hypershift-main-e2e-aks-4-21 (all) - 388 runs, 50% failed, 7% of failures match = 3% impact
periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aws-external-oidc (all) - 25 runs, 64% failed, 56% of failures match = 36% impact
periodic-ci-openshift-aws-karpenter-provider-aws-release-4.21-periodics-e2e-hypershift (all) - 2 runs, 100% failed, 50% of failures match = 50% impact
pull-ci-openshift-csi-operator-main-hypershift-e2e-aks (all) - 24 runs, 33% failed, 38% of failures match = 13% impact
periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aks-multi-x-ax (all) - 57 runs, 54% failed, 10% of failures match = 5% impact
periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aks-multi-x-ax (all) - 48 runs, 56% failed, 11% of failures match = 6% impact
periodic-ci-openshift-hypershift-release-4.22-periodics-e2e-aks-multi-x-ax (all) - 67 runs, 49% failed, 15% of failures match = 7% impact
periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aks (all) - 66 runs, 36% failed, 13% of failures match = 5% impact
pull-ci-openshift-cluster-network-operator-master-hypershift-e2e-aks (all) - 35 runs, 54% failed, 11% of failures match = 6% impact
periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-aks-multi-x-ax (all) - 28 runs, 39% failed, 9% of failures match = 4% impact
pull-ci-openshift-hypershift-release-4.21-e2e-aks (all) - 8 runs, 50% failed, 25% of failures match = 13% impact
pull-ci-openshift-cloud-network-config-controller-main-hypershift-e2e-aks (all) - 10 runs, 40% failed, 25% of failures match = 10% impact
pull-ci-openshift-hypershift-main-e2e-aks-4-20 (all) - 151 runs, 53% failed, 3% of failures match = 1% impact
pull-ci-openshift-hypershift-release-4.20-e2e-aks (all) - 8 runs, 25% failed, 100% of failures match = 25% impact

Steps to Reproduce

1. Post a PR and have bad luck.
2. Check search.ci: https://search.dptools.openshift.org/?search=Failed+to+get+DaemonSet+global-pull-secret-syncer%3A+context+deadline+exceeded&maxAge=48h&context=1&type=bug%2Bissue%2Bjunit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

Actual results

CI fails.

Expected results

CI passes, or fails on some other test failure.

Additional info

In the search.ci results, the failures all appear in AWS or AKS jobs, but that might only be because most of our hypershift CI jobs are on AWS and AKS.

Details

Description

Description of problem

Version-Release number of selected component (if applicable)

How reproducible

Steps to Reproduce

Actual results

Expected results

Additional info

Attachments

Easy Agile Planning Poker

Activity

People

Dates