Uploaded image for project: 'Maistra'
  1. Maistra
  2. MAISTRA-1005

Having too many namespaces in SMMR causes istio-operator to repeatedly reconsile ServiceMeshMemberRoll, forcing pilot into recreating namespace watches every few seconds

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Critical
    • maistra-1.0.2
    • maistra-1.0.1
    • None
    • None
    • MAISTRA 1.0.2

    Description

      On OCP 4.1.18 on AWS, with Servicemesh 1.0.1 and Serverless 1.1.0 (pre-release build) installed,

      Having tens of namespaces in SMMR, the istio-operator seems to be stuck in a reconsiliation loop, causing the pilot to recreate namespace watches every few seconds, which breaks Serverless functionality

      2019-10-15T20:08:26.790597Z     info    ServiceMeshMemberRoll default updated, namespaces now ["knative-serving" "myproject" "foo2" "foo8" "serving-tests" "foo5" "foo6" "foo7" "foo1" "foo4" "serving-tests-alt" "foo3" "foo9" "istio-system"]
      2019-10-15T20:08:26.790557Z     info    ServiceMeshMemberRoll default updated, namespaces now ["knative-serving" "myproject" "foo2" "foo8" "serving-tests" "foo5" "foo6" "foo7" "foo1" "foo4" "serving-tests-alt" "foo3" "foo9" "istio-system"]
      2019-10-15T20:08:26.790569Z     info    ServiceMeshMemberRoll default updated, namespaces now ["knative-serving" "myproject" "foo2" "foo8" "serving-tests" "foo5" "foo6" "foo7" "foo1" "foo4" "serving-tests-alt" "foo3" "foo9" "istio-system"]
      2019-10-15T20:08:26.791157Z     warn    istio.io/istio/pilot/pkg/serviceregistry/kube/controller.go:352: watch of *v1.Service ended with: Namespaces Updated
      2019-10-15T20:08:26.791281Z     warn    istio.io/istio/pilot/pkg/serviceregistry/kube/controller.go:360: watch of *v1.Endpoints ended with: Namespaces Updated
      2019-10-15T20:08:26.791355Z     warn    istio.io/istio/pilot/pkg/serviceregistry/kube/controller.go:353: watch of *v1.Pod ended with: Namespaces Updated
      2019-10-15T20:08:35.390649Z     info    ServiceMeshMemberRoll default updated, namespaces now ["foo2" "foo5" "foo6" "knative-serving" "serving-tests-alt" "foo3" "foo7" "myproject" "serving-tests" "foo4" "foo1" "foo8" "foo9" "istio-system"]
      2019-10-15T20:08:35.390669Z     info    ServiceMeshMemberRoll default updated, namespaces now ["foo2" "foo5" "foo6" "knative-serving" "serving-tests-alt" "foo3" "foo7" "myproject" "serving-tests" "foo4" "foo1" "foo8" "foo9" "istio-system"]
      2019-10-15T20:08:35.390686Z     info    ServiceMeshMemberRoll default updated, namespaces now ["foo2" "foo5" "foo6" "knative-serving" "serving-tests-alt" "foo3" "foo7" "myproject" "serving-tests" "foo4" "foo1" "foo8" "foo9" "istio-system"]
      2019-10-15T20:08:35.391171Z     warn    istio.io/istio/pilot/pkg/serviceregistry/kube/controller.go:352: watch of *v1.Service ended with: Namespaces Updated
      2019-10-15T20:08:35.391330Z     warn    istio.io/istio/pilot/pkg/serviceregistry/kube/controller.go:360: watch of *v1.Endpoints ended with: Namespaces Updated
      2019-10-15T20:08:35.391387Z     warn    istio.io/istio/pilot/pkg/serviceregistry/kube/controller.go:353: watch of *v1.Pod ended with: Namespaces Updated
      2019-10-15T20:08:43.989619Z     info    ServiceMeshMemberRoll default updated, namespaces now ["foo2" "foo3" "foo4" "foo6" "myproject" "serving-tests" "serving-tests-alt" "knative-serving" "foo7" "foo8" "foo9" "foo1" "foo5" "istio-system"]
      2019-10-15T20:08:43.990095Z     warn    istio.io/istio/pilot/pkg/serviceregistry/kube/controller.go:353: watch of *v1.Pod ended with: Namespaces Updated
      2019-10-15T20:08:43.990143Z     info    ServiceMeshMemberRoll default updated, namespaces now ["foo2" "foo3" "foo4" "foo6" "myproject" "serving-tests" "serving-tests-alt" "knative-serving" "foo7" "foo8" "foo9" "foo1" "foo5" "istio-system"]
      2019-10-15T20:08:43.990274Z     warn    istio.io/istio/pilot/pkg/serviceregistry/kube/controller.go:352: watch of *v1.Service ended with: Namespaces Updated
      2019-10-15T20:08:43.990308Z     info    ServiceMeshMemberRoll default updated, namespaces now ["foo2" "foo3" "foo4" "foo6" "myproject" "serving-tests" "serving-tests-alt" "knative-serving" "foo7" "foo8" "foo9" "foo1" "foo5" "istio-system"]
      2019-10-15T20:08:43.990433Z     warn    istio.io/istio/pilot/pkg/serviceregistry/kube/controller.go:360: watch of *v1.Endpoints ended with: Namespaces Updated
      

      Attachments

        Activity

          People

            dgrimm@redhat.com Daniel Grimm
            maschmid@redhat.com Marek Schmidt
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: