Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-8277

fails to access APIServer service IP assigned on lo device

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Critical Critical
    • 4.13.0, 4.14.0
    • 4.13.0
    • MicroShift
    • None
    • Critical
    • No
    • uShift Sprint 233
    • 1
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      *Cause*: To fix a certificate issue we used the internal apiserver IP configured in loopback device.
      *Consequence*: ovnk picks up this virtual ip as apiserver backend, making it unreachable.
      *Fix*: Use a different virtual IP to configure loopback device.
      *Result*: Bug doesn’t present anymore.
      Show
      *Cause*: To fix a certificate issue we used the internal apiserver IP configured in loopback device. *Consequence*: ovnk picks up this virtual ip as apiserver backend, making it unreachable. *Fix*: Use a different virtual IP to configure loopback device. *Result*: Bug doesn’t present anymore.
    • Bug Fix

      Description of problem:

      In order to solve the certificate issue: https://issues.redhat.com/browse/OCPBUGS-7442, we added the first kubernetes service IP (for APIServer service) to the lo device:

$ ip a s lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.43.0.1/32 scope global lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

This has side effect that ovnk may pick up this "node" IP from lo device as the backend of APIServer service, which in turn results in failure accessing the actual API pod (which runs in host network and uses the actual node IP).

      Version-Release number of selected component (if applicable):

      4.12, 4.13

      How reproducible:

      not always

      Steps to Reproduce:

      1. install microshift
2. restart node
3.

      Actual results:

      [redhat@dhcp-1-235-104 ~]$ ip a s lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.43.0.1/32 scope global lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

[redhat@dhcp-1-235-104 ~]$ oc -n openshift-ovn-kubernetes exec -it ovnkube-master-tqtk9 -c northd -- bash

[root@dhcp-1-235-104 ~]# ovn-nbctl ls-list
85747199-e6b4-4523-ad6a-e24c4c2c9a7b (dhcp-1-235-104.arm.eng.rdu2.redhat.com)
85bd04db-0df3-4fe0-b9f5-a322452db706 (ext_dhcp-1-235-104.arm.eng.rdu2.redhat.com)
0ce5719f-7ac4-4670-9161-88cbd25fa236 (join)

[root@dhcp-1-235-104 ~]# ovn-nbctl ls-lb-list dhcp-1-235-104.arm.eng.rdu2.redhat.com
UUID                                    LB                  PROTO      VIP                   IPs
c6eaad35-a683-463a-8abd-a4f17f493e37    Service_default/    tcp        10.43.0.1:443         10.43.0.1:6443



Note:
10.43.0.1:443 is the k8s service IP for APIServer
10.43.0.1:6443 is the APIServer pod IP that serves the requests to APIServer service

      Expected results:

       

      Additional info:

       

            pacevedo@redhat.com Pablo Acevedo Montserrat
            zshi@redhat.com Zenghui Shi
            Rahul Gangwar Rahul Gangwar
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: