Uploaded image for project: 'Network Observability'
  1. Network Observability
  2. NETOBSERV-1430

Some flows are not seen as metrics (both prom and loki)

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • netobserv-1.5
    • netobserv-1.4
    • Kafka
    • None
    • False
    • None
    • False
    • Hide
      Previously, with some specific cluster configuration, it happened that the eBPF Agent wasn't able to figure out on which node it was running, resulting in cascading consequences that ended up in failing to provide some of the traffic metrics.
      Now the agent's node IP is safely provided by the Operator, inferred from the pod status, which restores those missing metrics.
      Show
      Previously, with some specific cluster configuration, it happened that the eBPF Agent wasn't able to figure out on which node it was running, resulting in cascading consequences that ended up in failing to provide some of the traffic metrics. Now the agent's node IP is safely provided by the Operator, inferred from the pod status, which restores those missing metrics.
    • NetObserv - Sprint 246, NetObserv - Sprint 247

      Description of problem:

      I have setup a sample app [1] that involves microservices communicating via Kafka. One of the service receives events from Kafka. In netobserv, in the flow logs table I'm seeing this in and out traffic. But in the topology, overview and even in the prometheus metrics, I'm only seeing traffic from the service to kafka and not the other way around.

      Steps to Reproduce:

      Not sure yet how reproducible this is, however this was seen using this sample app: https://github.com/ia3andy/one-two-three-quarkus/
      

      [EDIT after more investigation]

      The bug is actually tied to a specific cluster configuration which I got by using a demo cluster on http://demo.redhat.com
      I don't know how to reproduce a similar configuration on a more typical cluster (cluster-bot etc.). But I noticed a bunch of differences in netboserv when using that cluster compared to a standard one - for instance, traffic from routes where being seen as coming from nodes, without involving ingress router.
      Note that the CNI was still OVN.
      Anyway: with that config, the "AgentIP" in our flows is like "192.168.12.2" instead of using the machine network 10.10.10.0/24, which is the root cause of the bug seen. 

       

      Actual results:

      cf screen captures

      Expected results:

      in/out Traffic seen in topology & overview & dashboards

      PS : I haven't checked if it also affects 1.4

       

      See also: two samples of flow json that are visible in flow table but not as metrics: sample.txt

        1. Capture d’écran du 2023-12-08 14-29-00.png
          175 kB
          Joel Takvorian
        2. Capture d’écran du 2023-12-08 14-30-05.png
          83 kB
          Joel Takvorian
        3. Capture d’écran du 2023-12-08 14-30-09.png
          67 kB
          Joel Takvorian
        4. Capture d’écran du 2023-12-08 14-38-26.png
          120 kB
          Joel Takvorian
        5. image.png
          126 kB
          Joel Takvorian
        6. image-2024-01-11-10-42-17-984.png
          146 kB
          Nathan Weinberg
        7. image-2024-01-11-10-43-06-284.png
          146 kB
          Nathan Weinberg
        8. sample.txt
          2 kB
          Joel Takvorian

              jtakvori Joel Takvorian
              jtakvori Joel Takvorian
              Nathan Weinberg Nathan Weinberg
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: