Loading...

Type: Bug
Resolution: Unresolved
Priority: Major
Fix Version/s: None
Affects Version/s: 4.21
Component/s: HyperShift
Labels:
- triaged

Activity Type:
Quality / Stability / Reliability
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Story Points:
None
Severity:
None
Regression:
None

Target Backport Versions:
None
Target Version:

4.21.0
Release Blocker:
Rejected
Sprint:
None

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

PX Impact Score:

Release Note Status:
In Progress
Release Note Type:
Release Note Not Required
Release Note Text:
None

Escape Reason:
None
Escape Impact:
None
Corrective Measures:
None
SDLC stage when should've been found:
None

https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_hypershift/6745/pull-ci-openshift-hypershift-main-e2e-aws/1983843735657517056

https://search.dptools.openshift.org/?search=subtest+may+have+called+FailNow+on+a+parent+test&maxAge=336h&context=1&type=bug%2Bissue%2Bjunit&name=.*openshift-hypershift.*&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

: TestCreateClusterCustomConfig/EnsureHostedCluster/EnsureGlobalPullSecret/Wait_GlobalPullSecret_and_Konnectivity_DaemonSets_to_be_ready_-_first_check#01	5m0s
{Failed  === RUN   TestCreateClusterCustomConfig/EnsureHostedCluster/EnsureGlobalPullSecret/Wait_GlobalPullSecret_and_Konnectivity_DaemonSets_to_be_ready_-_first_check#01
    testing.go:1679: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test
            --- FAIL: TestCreateClusterCustomConfig/EnsureHostedCluster/EnsureGlobalPullSecret/Wait_GlobalPullSecret_and_Konnectivity_DaemonSets_to_be_ready_-_first_check#01 (300.00s)
}

: TestCreateClusterCustomConfig/EnsureHostedCluster/EnsureGlobalPullSecret	5m30s
{Failed  === RUN   TestCreateClusterCustomConfig/EnsureHostedCluster/EnsureGlobalPullSecret
    util.go:280: Successfully waited for kubeconfig to be published for HostedCluster e2e-clusters-wfbb9/custom-config-4fwrb in 0s
    util.go:297: Successfully waited for kubeconfig secret to have data in 0s
    util.go:351: Successfully waited for a successful connection to the guest API server in 0s
    util.go:1865: NodePool replicas: 2, Available nodes: 2
    util.go:1972: 
        Timed out after 300.001s.
        DaemonSets are not ready
        Expected success, but got an error:
            <*errors.errorString | 0xc004ec1a30>: 
            GlobalPullSecret DaemonSet is not ready: 1/2 pods ready
            {
                s: "GlobalPullSecret DaemonSet is not ready: 1/2 pods ready",
            }
        --- FAIL: TestCreateClusterCustomConfig/EnsureHostedCluster/EnsureGlobalPullSecret (330.20s)
}

I've also seen this fail like this - https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_hypershift/7187/pull-ci-openshift-hypershift-main-e2e-aks/1993669876236947456

{Failed  === RUN   TestCreateCluster/EnsureHostedCluster/EnsureGlobalPullSecret/Wait_for_pull_secret_synchronization_to_stabilize_across_all_nodes
    util.go:2014: Waiting for GlobalPullSecretDaemonSet to process the deletion and stabilize all nodes
    util.go:2068: Waiting for global-pull-secret-syncer DaemonSet to be ready with ≤1 available nodes
    util.go:2097: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready (based on available nodes)
    util.go:2097: DaemonSet global-pull-secret-syncer not ready: 2/1 pods ready (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    util.go:2091: DaemonSet global-pull-secret-syncer update in flight: 2/1 pods updated (based on available nodes)
    testing.go:1679: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test
            --- FAIL: TestCreateCluster/EnsureHostedCluster/EnsureGlobalPullSecret/Wait_for_pull_secret_synchronization_to_stabilize_across_all_nodes (1200.15s)

links to

openshift/hypershift#7294: OCPBUGS-63717: Fix flaky EnsureGlobalPullSecret test race condition

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates