Uploaded image for project: 'Network Observability'
  1. Network Observability
  2. NETOBSERV-1304

DNS Requests for SRV records are not being traced by agent

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Undefined Undefined
    • netobserv-1.5
    • netobserv-1.4-candidate
    • Prometheus
    • None
    • False
    • None
    • False
    • OCPSTRAT-965 - DNS tracking improvements
    • Hide
      1.5: n/a (work done as part of the DNS tracking 1.5 epic)

      (old - 1.4)
      Currently, larger DNS packets require the eBPF agent to look for DNS header outside of the 1st socket buffer "SKB" segment. A new ebpf agent helper function is needed. Currently, there is no workaround for this issue.
      Show
      1.5: n/a (work done as part of the DNS tracking 1.5 epic) (old - 1.4) Currently, larger DNS packets require the eBPF agent to look for DNS header outside of the 1st socket buffer "SKB" segment. A new ebpf agent helper function is needed. Currently, there is no workaround for this issue.
    • NetObserv - Sprint 242, NetObserv - Sprint 243, NetObserv - Sprint 244
    • Important

      Steps to reproduce:

      1. Create a sleep pod which includes dig tool 
        $ cat ~/workspaces/cluster_bot/sleep_pod.yaml
        apiVersion: v1
        kind: Pod
        metadata:
          name: sleeppod
        spec:
          securityContext:
            runAsNonRoot: true
            seccompProfile:
              type: RuntimeDefault
          dnsConfig:
            options:
              - name: "use-vc"
          containers:
            - name: utils
              image: quay.io/openshifttest/hello-openshift:multiarch
              command: ["/bin/sh", "-ec", "sleep 3600"]
              restartPolicy: Never
              securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                  drop: ["ALL"] 
      1. oc rsh into pod to do below dig request
        ~ $ dig _grpc._tcp.prometheus-operated.openshift-monitoring.svc.cluster.local. SRV; <<>> DiG 9.16.20 <<>> _grpc._tcp.prometheus-operated.openshift-monitoring.svc.cluster.local. SRV
        ;; global options: +cmd
        ;; Got answer:
        ;; WARNING: .local is reserved for Multicast DNS
        ;; You are currently testing what happens when an mDNS query is leaked to DNS
        ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36791
        ;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 3
        ;; WARNING: recursion requested but not available;; OPT PSEUDOSECTION:
        ; EDNS: version: 0, flags:; udp: 1232
        ; COOKIE: 9d53552d9ba7132b (echoed)
        ;; QUESTION SECTION:
        ;_grpc._tcp.prometheus-operated.openshift-monitoring.svc.cluster.local. IN SRV;; ANSWER SECTION:
        _grpc._tcp.prometheus-operated.openshift-monitoring.svc.cluster.local. 5 IN SRV    0 50 10901 prometheus-k8s-1.prometheus-operated.openshift-monitoring.svc.cluster.local.
        _grpc._tcp.prometheus-operated.openshift-monitoring.svc.cluster.local. 5 IN SRV    0 50 10901 prometheus-k8s-0.prometheus-operated.openshift-monitoring.svc.cluster.local.;; ADDITIONAL SECTION:
        prometheus-k8s-1.prometheus-operated.openshift-monitoring.svc.cluster.local. 5 IN A 10.128.2.19
        prometheus-k8s-0.prometheus-operated.openshift-monitoring.svc.cluster.local. 5 IN A 10.129.2.15;; Query time: 1 msec
        ;; SERVER: 172.30.0.10#53(172.30.0.10)
        ;; WHEN: Tue Sep 12 22:36:26 UTC 2023
        ;; MSG SIZE  rcvd: 620 

        3. In UI flows captured by tc hook are seen but not DNS flows:


      4. Raw flowlogs 

      {
        "AgentIP": "10.0.28.190",
        "Bytes": 662,
        "DstAddr": "10.130.2.41",
        "DstK8S_HostIP": "10.0.28.190",
        "DstK8S_HostName": "ip-10-0-28-190.us-east-2.compute.internal",
        "DstK8S_Name": "sleeppod",
        "DstK8S_Namespace": "dns-traffic",
        "DstK8S_OwnerName": "sleeppod",
        "DstK8S_OwnerType": "Pod",
        "DstK8S_Type": "Pod",
        "DstMac": "0A:58:0A:82:02:29",
        "DstPort": 48765,
        "Duplicate": false,
        "Etype": 2048,
        "FlowDirection": "0",
        "IfDirection": 0,
        "Interface": "eth0",
        "K8S_ClusterName": "73a968c4-7ba3-4877-8667-5fc0db72e926",
        "Packets": 1,
        "Proto": 17,
        "SrcAddr": "172.30.0.10",
        "SrcK8S_Name": "dns-default",
        "SrcK8S_Namespace": "openshift-dns",
        "SrcK8S_OwnerName": "dns-default",
        "SrcK8S_OwnerType": "Service",
        "SrcK8S_Type": "Service",
        "SrcMac": "0A:58:0A:82:02:05",
        "SrcPort": 53,
        "TimeFlowEndMs": 1694558186991,
        "TimeFlowStartMs": 1694558186991,
        "TimeReceived": 1694558189,
        "app": "netobserv-flowcollector"
      } 

      Expected Results

      SRV Records should be captured as applications uses it to discover port and IP both to establish connection

        1. image-2023-09-12-18-42-33-101.png
          251 kB
          Mehul Modi
        2. Screenshot 2023-10-25 at 12.53.08 PM.png
          473 kB
          Amogh Rameshappa Devapura
        3. Screenshot 2023-10-25 at 12.53.18 PM.png
          384 kB
          Amogh Rameshappa Devapura

              mmahmoud@redhat.com Mohamed Mahmoud
              rhn-support-memodi Mehul Modi
              Amogh Rameshappa Devapura Amogh Rameshappa Devapura
              Sara Thomas Sara Thomas
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: