Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-54339

Component Readiness: new kubectl port forwarding test struggling on bare metal

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • 4.19.0
    • oc
    • None
    • Approved
    • False
    • Hide

      None

      Show
      None

      Component Readiness has found a potential problem in the following test:

      [sig-cli] Kubectl Port forwarding Shutdown client connection while the remote stream is writing data to the port-forward connection port-forward should keep working after detect broken connection [Suite:openshift/conformance/parallel] [Suite:k8s]

      Test has a 94.55% pass rate, but 95.00% is required.

      Sample (being evaluated) Release: 4.19
      Start Time: 2025-03-21T00:00:00Z
      End Time: 2025-03-28T08:00:00Z
      Success Rate: 94.55%
      Successes: 52
      Failures: 3
      Flakes: 0

      View the test details report for additional context.

      This appears to be a new test and is held to a 95% pass rate on all variant combos in the main dashboard. One in particular seems to be showing an issue, specifically on bare metal this test doesn't seem to be doing so well, everywhere else it's 100%.

      The three job runs in question in the report above are all failing on just this one test, with the same error message.

      Looking at the global job runs where this fails and ignoring the libvirt ones (not terribly concerned about that, but maybe it's the same issue), it always seems to be a metal -bm job where this struggles.

      Also interesting, the test gets retried, and the error messages are very different. Looks like the first failure may cause a problem for the retry?

      https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-metal-ipi-ovn-bm/1905170012562264064

      Run #0: Failed expand_less 	32s
      {  fail [k8s.io/kubernetes/test/e2e/kubectl/portforward.go:668]: couldn't get http response from port-forward: Get "http://127.0.0.1:40295/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)}
      
      Run #1: Failed expand_less 	1s
      {  I0327 10:39:32.078766  128571 factory.go:193] Registered Plugin "containerd"
      openshift-tests 4.19.0-202503270540.p0.g3313032.assembly.stream.el9-3313032
      time="2025-03-27T10:39:32Z" level=info msg="Decoding provider" clusterState="<nil>" discover=false dryRun=false func=DecodeProvider providerType="{\"type\":\"skeleton\",\"ProjectID\":\"\",\"Region\":\"\",\"Zone\":\"\",\"NumNodes\":3,\"MultiMaster\":true,\"MultiZone\":false,\"Zones\":[],\"ConfigFile\":\"\",\"Disconnected\":false,\"SingleReplicaTopology\":false,\"NetworkPlugin\":\"OVNKubernetes\",\"HasIPv4\":true,\"HasIPv6\":false,\"IPFamily\":\"ipv4\",\"HasSCTP\":false,\"IsProxied\":true,\"IsIBMROKS\":false,\"HasNoOptionalCapabilities\":false}"
      time="2025-03-27T10:39:32Z" level=warning msg="config was nil" func=DecodeProvider
        I0327 10:39:32.115065  128571 test_setup.go:94] Extended test version 4.19.0-202503270540.p0.g3313032.assembly.stream.el9-3313032
        I0327 10:39:32.115086  128571 test_context.go:558] Tolerating taints "node-role.kubernetes.io/control-plane" when considering if nodes are ready
        I0327 10:39:32.206706 128571 framework.go:2310] microshift-version configmap not found
        I0327 10:39:32.206752  128571 command.go:57] Loaded test configuration: &framework.TestContextType{KubeConfig:"/tmp/secret/kubeconfig", KubeContext:"", KubeAPIContentType:"application/vnd.kubernetes.protobuf", KubeletRootDir:"/var/lib/kubelet", KubeletConfigDropinDir:"", CertDir:"", Host:"https://api.cluster9.ocpci.eng.rdu2.redhat.com:6443", BearerToken:"SNIP", RepoRoot:"../../", ListImages:false, listTests:false, listLabels:false, ListConformanceTests:false, Provider:"skeleton", Tooling:"", timeouts:framework.TimeoutContext{Poll:2000000000, PodStart:300000000000, PodStartShort:120000000000, PodStartSlow:900000000000, PodDelete:300000000000, ClaimProvision:300000000000, DataSourceProvision:300000000000, ClaimProvisionShort:60000000000, ClaimBound:180000000000, PVReclaim:180000000000, PVBound:180000000000, PVCreate:180000000000, PVDelete:300000000000, PVDeleteSlow:1200000000000, SnapshotCreate:300000000000, SnapshotDelete:300000000000, SnapshotControllerMetrics:300000000000, SystemPodsStartup:600000000000, NodeSchedulable:1800000000000, SystemDaemonsetStartup:300000000000, NodeNotReady:180000000000}, CloudConfig:framework.CloudConfig{APIEndpoint:"", ProjectID:"", Zone:"", Zones:[]string{}, Region:"", MultiZone:false, MultiMaster:true, Cluster:"", MasterName:"", NodeInstanceGroup:"", NumNodes:3, ClusterIPRange:"", ClusterTag:"", Network:"", ConfigFile:"", NodeTag:"", MasterTag:"", Provider:framework.NullProvider{}}, KubectlPath:"kubectl", OutputDir:"/tmp", ReportDir:"", ReportPrefix:"", ReportCompleteGinkgo:false, ReportCompleteJUnit:false, Prefix:"e2e", MinStartupPods:-1, EtcdUpgradeStorage:"", EtcdUpgradeVersion:"", GCEUpgradeScript:"", ContainerRuntimeEndpoint:"unix:///run/containerd/containerd.sock", ContainerRuntimeProcessName:"containerd", ContainerRuntimePidFile:"/run/containerd/containerd.pid", SystemdServices:"containerd*", DumpSystemdJournal:false, ImageServiceEndpoint:"", MasterOSDistro:"custom", NodeOSDistro:"custom", NodeOSArch:"amd64", VerifyServiceAccount:true, DeleteNamespace:true, DeleteNamespaceOnFailure:true, AllowedNotReadyNodes:-1, CleanStart:false, GatherKubeSystemResourceUsageData:"false", GatherLogsSizes:false, GatherMetricsAfterTest:"false", GatherSuiteMetricsAfterTest:false, MaxNodesToGather:0, IncludeClusterAutoscalerMetrics:false, OutputPrintType:"json", CreateTestingNS:(framework.CreateTestingNSFn)(0x5c2e1a0), DumpLogsOnFailure:true, DisableLogDump:false, LogexporterGCSPath:"", NodeTestContextType:framework.NodeTestContextType{NodeE2E:false, NodeName:"", NodeConformance:false, PrepullImages:false, ImageDescription:"", RuntimeConfig:map[string]string(nil), SystemSpecName:"", RestartKubelet:false, ExtraEnvs:map[string]string(nil), StandaloneMode:false, CriProxyEnabled:false}, ClusterDNSDomain:"cluster.local", NodeKiller:framework.NodeKillerConfig{Enabled:false, FailureRatio:0.01, Interval:60000000000, JitterFactor:60, SimulatedDowntime:600000000000, NodeKillerStopCtx:context.Context(nil), NodeKillerStop:(func())(nil)}, IPFamily:"ipv4", NonblockingTaints:"node-role.kubernetes.io/control-plane", ProgressReportURL:"", SriovdpConfigMapFile:"", SpecSummaryOutput:"", DockerConfigFile:"", E2EDockerConfigFile:"", KubeTestRepoList:"", SnapshotControllerPodName:"", SnapshotControllerHTTPPort:0, RequireDevices:false, EnabledVolumeDrivers:[]string(nil)}
      error: no test exists with that name: [sig-cli] Kubectl Port forwarding Shutdown client connection while the remote stream is writing data to the port-forward connection port-forward should keep working after detect broken connection [Suite:openshift/conformance/parallel] [Suite:k8s]}
      

              aos-workloads-staff Workloads Team Bot Account
              rhn-engineering-dgoodwin Devan Goodwin
              Ying Zhou Ying Zhou
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: