Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-18777

Observed "Node process segfaulted" error on worker node in 4.14 PowerVS IPI cluster

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Minor Minor
    • None
    • 4.14
    • apiserver-auth
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • No
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      Observed a segfault error in one of the worker nodes as shown below
      
      Sep 11 02:28:25.869698 rdr-multiarch-mon01-tbngs-worker-jhl2x crio[2717]: time="2023-09-11 02:28:25.855478957Z" level=info msg="Stopping container: 7dc9afe509536227503cf27da7b612d233b8d479d5a7110ba646c0779e7dede5 (timeout: 30s)" id=17d3decb-ff20-477b-86d2-deae2af746ea name=/runtime.v1.RuntimeService/StopContainer
      Sep 11 02:28:25.880266 rdr-multiarch-mon01-tbngs-worker-jhl2x kernel: slapd[367598]: segfault (11) at 7fff88f200c8 nip 12ea981c8 lr 12ea981b8 code 1 in slapd[12e970000+260000]
      Sep 11 02:28:25.880498 rdr-multiarch-mon01-tbngs-worker-jhl2x kernel: slapd[367598]: code: 7c0802a6 fbc1fff0 fbe1fff8 f8010010 f821ffd1 7c7f1b78 ebc3000a 4befe44d 
      Sep 11 02:28:25.881320 rdr-multiarch-mon01-tbngs-worker-jhl2x kernel: slapd[367598]: code: e8410018 7c1e1800 4082000c 39200000 <913f0008> 38210030 e8010010 ebc1fff0 
      Sep 11 02:28:25.881461 rdr-multiarch-mon01-tbngs-worker-jhl2x kubenswrapper[2800]: E0911 02:28:25.880632    2800 event.go:280] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"openldap-server-66cb67bc95-72lwl.1783b730fd1fedb0", GenerateName:"", Namespace:"e2e-test-oauth-ldap-idp-9w6t2", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Pod", Namespace:"e2e-test-oauth-ldap-idp-9w6t2", Name:"openldap-server-66cb67bc95-72lwl", UID:"fa597a4f-0b41-4b8f-ae7b-a47a538ea1b2", APIVersion:"v1", ResourceVersion:"213112", FieldPath:"spec.containers{openldap-server}"}, Reason:"Killing", Message:"Stopping container openldap-server", Source:v1.EventSource{Component:"kubelet", Host:"rdr-multiarch-mon01-tbngs-worker-jhl2x"}, FirstTimestamp:time.Date(2023, time.September, 11, 2, 28, 25, 854479792, time.Local), LastTimestamp:time.Date(2023, time.September, 11, 2, 28, 25, 854479792, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "openldap-server-66cb67bc95-72lwl.1783b730fd1fedb0" is forbidden: unable to create new content in namespace e2e-test-oauth-ldap-idp-9w6t2 because it is being terminated' (will not retry!)
      Sep 11 02:28:26.125375 rdr-multiarch-mon01-tbngs-worker-jhl2x ovs-vswitchd[1651]: ovs|04461|connmgr|INFO|br-ex<->unix#6507: 2 flow_mods in the last 0 s (2 adds)
      Sep 11 02:28:26.249161 rdr-multiarch-mon01-tbngs-worker-jhl2x systemd[1]: Created slice Slice /system/systemd-coredump.
      
      
      

      Prow Job Link- https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-ovn-ppc64le-powervs/1701022936401448960

              kostrows@redhat.com Krzysztof Ostrowski
              shgokul Shilpa Gokul (Inactive)
              None
              None
              Gaoyun Pei Gaoyun Pei
              None
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: