Uploaded image for project: 'Network Observability'
  1. Network Observability
  2. NETOBSERV-2074

Network Observability Operator doesn't work on S390x

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • netobserv-1.8
    • netobserv-1.7, netobserv-1.8-candidate
    • eBPF
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • NetObserv - Sprint 265, NetObserv - Sprint 266
    • None
    • None
    • None

      Description of problem:

      I was trying to setup NetObserv v1.7, but getting error on console but pods and lokistack looks good:

          [root@ocpz-m42lp36-bastion jitendra]# oc get co
      NAME                    VERSION  AVAILABLE  PROGRESSING  DEGRADED  SINCE  MESSAGE
      authentication               4.16.2  True    False     False   7h55m
      baremetal                 4.16.2  True    False     False   31d
      cloud-controller-manager          4.16.2  True    False     False   31d
      cloud-credential              4.16.2  True    False     False   31d
      cluster-autoscaler             4.16.2  True    False     False   31d
      config-operator              4.16.2  True    False     False   31d
      console                  4.16.2  True    False     False   9d
      control-plane-machine-set         4.16.2  True    False     False   31d
      csi-snapshot-controller          4.16.2  True    False     False   11d
      dns                    4.16.2  True    False     False   7d14h
      etcd                    4.16.2  True    False     False   23d
      image-registry               4.16.2  True    False     False   31d
      ingress                  4.16.2  True    False     False   31d
      insights                  4.16.2  True    False     False   31d
      kube-apiserver               4.16.2  True    False     False   31d
      kube-controller-manager          4.16.2  True    False     False   31d
      kube-scheduler               4.16.2  True    False     False   31d
      kube-storage-version-migrator       4.16.2  True    False     False   2d18h
      machine-api                4.16.2  True    False     False   31d
      machine-approver              4.16.2  True    False     False   31d
      machine-config               4.16.2  True    False     False   31d
      marketplace                4.16.2  True    False     False   31d
      monitoring                 4.16.2  True    False     False   15h
      network                  4.16.2  True    False     False   31d
      node-tuning                4.16.2  True    False     False   31d
      openshift-apiserver            4.16.2  True    False     False   12h
      openshift-controller-manager        4.16.2  True    False     False   9d
      openshift-samples             4.16.2  True    False     False   31d
      operator-lifecycle-manager         4.16.2  True    False     False   31d
      operator-lifecycle-manager-catalog     4.16.2  True    False     False   31d
      operator-lifecycle-manager-packageserver  4.16.2  True    False     False   27h
      service-ca                 4.16.2  True    False     False   31d
      storage                  4.16.2  True    False     False   31d
      [root@ocpz-m42lp36-bastion jitendra]# oc get csv
      ocNAME                        DISPLAY             VERSION          REPLACES                PHASE
      cluster-logging.v5.9.10              Red Hat OpenShift Logging    5.9.10           cluster-logging.v5.9.9         Succeeded
      cluster-observability-operator.v1.0.0       Cluster Observability Operator  1.0.0                               Succeeded
      loki-operator.v5.9.10               Loki Operator          5.9.10           loki-operator.v5.9.9          Succeeded
      network-observability-operator.v1.7.0       Network Observability      1.7.0           network-observability-operator.v1.6.2  Succeeded
      opentelemetry-operator.v0.107.0-4-0.1727168137.p  Red Hat build of OpenTelemetry  0.107.0-4+0.1727168137.p  opentelemetry-operator.v0.102.0-3    Succeeded
      [root@ocpz-m42lp36-bastion jitendra]# oc get pods -n netobserv
      NAME                 READY  STATUS  RESTARTS  AGE
      flowlogs-pipeline-989n4        1/1   Running  0     3h8m
      flowlogs-pipeline-9c46t        1/1   Running  0     3h8m
      flowlogs-pipeline-9tln2        1/1   Running  0     3h8m
      flowlogs-pipeline-bfqh6        1/1   Running  0     3h8m
      flowlogs-pipeline-k5jzm        1/1   Running  0     3h8m
      flowlogs-pipeline-knwrr        1/1   Running  0     3h8m
      loki-compactor-0           1/1   Running  0     43m
      loki-distributor-5b9cc4998f-v9b7z   1/1   Running  0     43m
      loki-gateway-6f9468b96-spz6w     2/2   Running  0     42m
      loki-gateway-6f9468b96-xg2t5     2/2   Running  0     42m
      loki-index-gateway-0         1/1   Running  0     42m
      loki-ingester-0            1/1   Running  0     43m
      loki-querier-5c9848f6c-jwxsl     1/1   Running  0     43m
      loki-query-frontend-b9c5ccf5d-rkrrm  1/1   Running  0     43m
      netobserv-plugin-66779c9486-jdhb8   1/1   Running  0     3h8m
      [root@ocpz-m42lp36-bastion jitendra]# oc logs flowlogs-pipeline-989n4 -n netobserv
      Starting flowlogs-pipeline:
      =====
      Build Version: v1.7.0
      Build Date: 2024-10-14 16:14
      Using configuration:
      {
        "PipeLine": "[{\"name\":\"grpc\"},{\"follows\":\"grpc\",\"name\":\"enrich\"},{\"follows\":\"enrich\",\"name\":\"loki\"},{\"follows\":\"enrich\",\"name\":\"prometheus\"}]",
        "Parameters": "[{\"ingest\":{\"grpc\":{\"port\":2055},\"type\":\"grpc\"},\"name\":\"grpc\"},{\"name\":\"enrich\",\"transform\":{\"network\":{\"directioninfo\":{\"dsthostfield\":\"DstK8S_HostIP\",\"flowdirectionfield\":\"FlowDirection\",\"reporteripfield\":\"AgentIP\",\"srchostfield\":\"SrcK8S_HostIP\"},\"kubeconfig\":{},\"rules\":[{\"kubernetes\":{\"interfacesfield\":\"Interfaces\",\"ipfield\":\"SrcAddr\",\"macfield\":\"SrcMac\",\"output\":\"SrcK8S\"},\"type\":\"add_kubernetes\"},{\"kubernetes\":{\"interfacesfield\":\"Interfaces\",\"ipfield\":\"DstAddr\",\"macfield\":\"DstMac\",\"output\":\"DstK8S\"},\"type\":\"add_kubernetes\"},{\"type\":\"reinterpret_direction\"},{\"kubernetes_infra\":{\"infra_prefixes\":[\"netobserv\",\"openshift\"],\"infra_refs\":[{\"name\":\"kubernetes\",\"namespace\":\"default\"},{\"name\":\"openshift\",\"namespace\":\"default\"}],\"namespacenamefields\":[{\"name\":\"SrcK8S_Name\",\"namespace\":\"SrcK8S_Namespace\"},{\"name\":\"DstK8S_Name\",\"namespace\":\"DstK8S_Namespace\"}],\"output\":\"K8S_FlowLayer\"},\"type\":\"add_kubernetes_infra\"},{\"add_subnet_label\":{\"input\":\"SrcAddr\",\"output\":\"SrcSubnetLabel\"},\"type\":\"add_subnet_label\"},{\"add_subnet_label\":{\"input\":\"DstAddr\",\"output\":\"DstSubnetLabel\"},\"type\":\"add_subnet_label\"}],\"subnetlabels\":[{\"cidrs\":[\"10.128.0.0/14\"],\"name\":\"Pods\"},{\"cidrs\":[\"172.30.0.0/16\"],\"name\":\"Services\"},{\"cidrs\":[\"10.0.0.0/16\"],\"name\":\"Machines\"}]},\"type\":\"network\"}},{\"name\":\"loki\",\"write\":{\"loki\":{\"batchsize\":102400,\"batchwait\":\"1s\",\"clientconfig\":{\"authorization\":{\"credentials_file\":\"/var/run/secrets/tokens/flowlogs-pipeline\",\"type\":\"Bearer\"},\"enable_http2\":false,\"follow_redirects\":false,\"proxy_url\":null,\"tls_config\":{\"ca_file\":\"/var/loki-certs-ca/service-ca.crt\",\"insecure_skip_verify\":false}},\"labels\":[\"SrcK8S_Namespace\",\"SrcK8S_OwnerName\",\"SrcK8S_Type\",\"DstK8S_Namespace\",\"DstK8S_OwnerName\",\"DstK8S_Type\",\"K8S_FlowLayer\",\"FlowDirection\"],\"maxbackoff\":\"5s\",\"maxretries\":2,\"minbackoff\":\"1s\",\"staticlabels\":{\"app\":\"netobserv-flowcollector\"},\"tenantid\":\"network\",\"timeout\":\"10s\",\"timestamplabel\":\"TimeFlowEndMs\",\"timestampscale\":\"1ms\",\"url\":\"https://loki-gateway-http.netobserv.svc:8080/api/logs/v1/network/\"},\"type\":\"loki\"}}]",
        "DynamicParameters": "{\"filename\":\"config.json\",\"name\":\"flowlogs-pipeline-config-dynamic\",\"namespace\":\"netobserv\"}",
        "MetricsSettings": "{\"nopanic\":true,\"port\":9401,\"prefix\":\"netobserv_\"}",
        "Health": {
          "Address": "0.0.0.0",
          "Port": "8080"
        },
        "Profile": {
          "Port": 6060
        }
      }
      time=2024-12-11T09:56:39Z level=info msg=StartServerAsync: addr = :9401 component=prometheus
      time=2024-12-11T09:56:40Z level=info msg=connecting stages: grpc --> enrich
      time=2024-12-11T09:56:40Z level=info msg=connecting stages: enrich --> loki
      time=2024-12-11T09:56:40Z level=info msg=connecting stages: enrich --> prometheus
      time=2024-12-11T09:56:40Z level=info msg=starting PProf HTTP listener port=6060
      [root@ocpz-m42lp36-bastion jitendra]#

      It seems like bpf_probe_read function is not available in the kernel !

      Version-Release number of selected component (if applicable):

          OpenShift version : 4.16
      
      It's RHEL 9 only this is right kernel version :[core@ocpz-m42lp36-compute-0 ~]$ uname -a
      Linux ocpz-m42lp36-compute-0 5.14.0-427.24.1.el9_4.s390x #1 SMP Sun Jun 23 11:54:29 EDT 2024 s390x s390x s390x GNU/Linux
      [core@ocpz-m42lp36-compute-0 ~]${code}
       
      
      How reproducible:
      {code:none}
          Just try to install Network Observability Operator from Operatorhub

      Steps to Reproduce:

          1. Just try to install Network Observability Operator from Operatorhub.       2. Now, Go to Observe tab to check the status of Network traffic.
          3.
          

      Actual results:

          Netobserv flowlogs-pipeline is not receving an y flow, this is either a connection issue with agent.

      Expected results:

          We must be able to the data in dashboard

      Additional info:

          

        1. screenshot-1.png
          679 kB
          Julien Pinsonneau
        2. image (6).png
          393 kB
          Jitendra Singh
        3. image (7).png
          351 kB
          Jitendra Singh

              ocp-docs-bot OCP DocsBot
              jitensin Jitendra Singh (Inactive)
              None
              None
              None
              Jitendra Singh Jitendra Singh (Inactive)
              None
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: