Uploaded image for project: 'Maistra'
  1. Maistra
  2. MAISTRA-2683

Federation watch server go brrrrr

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • None
    • maistra-2.1.0
    • federation
    • None
    • Sprint 10, Sprint 11

      The discovery server in federation does not shut down properly. It doesn't close incoming connections and logs a million debug messages (one every ~30 microseconds).

      2021-09-23T12:23:41.363358Z     info    federation      Removing registry for Federation cluster mesh1  component=federation-discovery-controller
      2021-09-23T12:23:41.363364Z     info    federation      deleting discovery resources for Federation cluster mesh1       component=federation-discovery-controller
      2021-09-23T12:23:41.363393Z     info    federation      ServiceMeshPeer deleted component=federation-discovery-controller
      2021-09-23T12:23:41.363398Z     debug   federation      Completed reconciliation of ServiceMeshPeer mesh2-system/mesh1  component=federation-discovery-controller
      2021-09-23T12:23:41.363564Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363601Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363613Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363633Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363644Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363654Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363663Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363672Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363681Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363691Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363700Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363710Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363719Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363730Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363758Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363767Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363776Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363768Z     info    federation      Federation Controller terminated        component=federation-registry
      2021-09-23T12:23:41.363785Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363795Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363804Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363812Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.363822Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364034Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364072Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364121Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364132Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364152Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364162Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364172Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364181Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364190Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364199Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364208Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364217Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364228Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364237Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364247Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364250Z     debug   federation      mesh2-system/mesh1.WatchTerminated(200 OK)      component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364257Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364277Z     debug   federation      could not locate ServiceMeshPeer mesh2-system/mesh1 for status update   component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364284Z     info    federation      watch cancelled component=federation-registry
      2021-09-23T12:23:41.364288Z     info    federation      watch stopped   component=federation-registry
      2021-09-23T12:23:41.364304Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364314Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364324Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364334Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364343Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364353Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364361Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364371Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364380Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364394Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364423Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364428Z     info    ior     Ignoring Gateway mesh2-system/federation-discovery-mesh1-ingress as it is not managed by Istiod
      2021-09-23T12:23:41.364434Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364457Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364469Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364479Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364498Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364512Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364522Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364531Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364552Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364562Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364571Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364581Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364590Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364600Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364609Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364618Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364627Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364636Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364659Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364672Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364682Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364691Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364701Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364725Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364735Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364761Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364773Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364786Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364796Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364806Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364815Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364824Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364863Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364877Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      2021-09-23T12:23:41.364898Z     debug   federation      mesh2-system/mesh1.WatchEventSent(10.131.0.197) component=federation-status peer=mesh2-system/mesh1
      

            mluksa@redhat.com Marko Luksa
            mluksa@redhat.com Marko Luksa
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: