Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-9316

[Multiple namespaces, FBI] Unable to write event - Permissions issues on drainer to inform operator about successful drain/message migration

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • AMQ 7.12.1.OPR.1.GA
    • operator
    • None

      FileBased installation,
      manually updated operator file to WATCH_NAMESPACES (* or selected few only - same scenario)
      4 prod brokers scaled down to 2. (they have related DR mirror brokers in mirror-dr-tests namespace)

      ---
      apiVersion: "rbac.authorization.k8s.io/v1"
      kind: "ClusterRoleBinding"
      metadata:
        name: "amq-broker-operator-rolebinding"
      roleRef:
        kind: "ClusterRole"
        apiGroup: "rbac.authorization.k8s.io"
        name: "amq-broker-operator-role"
      subjects:
      - kind: "ServiceAccount"
        name: "amq-broker-controller-manager"
        namespace: "mirror-prod-tests"
      

      operator update

      ...
      - name: "WATCH_NAMESPACE"
                value: "*"
              - name: "ENABLE_WEBHOOKS"  // or "mirror-prod-tests,mirror-dr-tests"
                value: "false"
              image: "registry.redhat.io/amq7/amq-broker-rhel8-operator@sha256:19816bd0f3a9b35ec342aae78dc9fabf9054e60057b3aaaac23e30cd2b144388"
      ...
      
      2024-07-29T11:11:59.058Z	INFO	setup	Starting workers	{"controller": "activemqartemisaddress", "controllerGroup": "broker.amq.io", "controllerKind": "ActiveMQArtemisAddress", "worker count": 1}
      2024-07-29T11:14:22.100Z	INFO	ActiveMQArtemisReconciler	updated Failed	{"kind ": "v1.StatefulSet", "named ": "dr-broker-ss", "error ": "failed to update *v1.StatefulSet, failed to update *v1.StatefulSet due to conflict"}
      2024-07-29T11:14:22.100Z	ERROR	ActiveMQArtemisReconciler	error processing resources	{"error": "[\"failed to update *v1.StatefulSet, failed to update *v1.StatefulSet due to conflict\"]"}
      github.com/go-logr/logr.Logger.Error
      	/opt/app-root/src/vendor/github.com/go-logr/logr/logr.go:299
      github.com/artemiscloud/activemq-artemis-operator/controllers.(*ActiveMQArtemisReconcilerImpl).Process
      	/opt/app-root/src/controllers/activemqartemis_reconciler.go:182
      github.com/artemiscloud/activemq-artemis-operator/controllers.(*ActiveMQArtemisReconciler).Reconcile
      	/opt/app-root/src/controllers/activemqartemis_controller.go:171
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
      	/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
      	/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
      	/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
      	/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
      2024-07-29T11:14:22.423Z	INFO	setup	Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes reqeueuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler	{"controller": "activemqartemis", "controllerGroup": "broker.amq.io", "controllerKind": "ActiveMQArtemis", "ActiveMQArtemis": {"name":"dr-broker","namespace":"mirror-dr-tests"}, "namespace": "mirror-dr-tests", "name": "dr-broker", "reconcileID": "f18d662d-fafd-4def-8e87-215431ac3a25"}
      2024-07-29T11:14:22.423Z	ERROR	setup	Reconciler error	{"controller": "activemqartemis", "controllerGroup": "broker.amq.io", "controllerKind": "ActiveMQArtemis", "ActiveMQArtemis": {"name":"dr-broker","namespace":"mirror-dr-tests"}, "namespace": "mirror-dr-tests", "name": "dr-broker", "reconcileID": "f18d662d-fafd-4def-8e87-215431ac3a25", "error": "[\"failed to update *v1.StatefulSet, failed to update *v1.StatefulSet due to conflict\"]"}
      github.com/go-logr/logr.Logger.Error
      	/opt/app-root/src/vendor/github.com/go-logr/logr/logr.go:299
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
      	/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
      	/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
      	/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
      E0729 11:19:16.701399       1 event.go:289] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"prod-broker-ss.17e6ab079ce649b5", GenerateName:"", Namespace:"mirror-prod-tests", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"StatefulSet", Namespace:"mirror-prod-tests", Name:"prod-broker-ss", UID:"3214c793-83dd-459d-a278-9dd6f36e9044", APIVersion:"apps/v1", ResourceVersion:"30188535", FieldPath:""}, Reason:"SuccessfulCreate", Message:"create Drain Pod prod-broker-ss-3 in StatefulSet prod-broker-ss successful", Source:v1.EventSource{Component:"statefulset-drain-controller", Host:""}, FirstTimestamp:time.Date(2024, time.July, 29, 11, 19, 16, 701252021, time.Local), LastTimestamp:time.Date(2024, time.July, 29, 11, 19, 16, 701252021, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"statefulset-drain-controller", ReportingInstance:""}': 'can't create an event with namespace 'mirror-prod-tests' in namespace 'mirror-dr-tests''(may retry after sleeping)
      E0729 11:19:25.643341       1 event.go:289] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"prod-broker-ss.17e6ab079ce649b5", GenerateName:"", Namespace:"mirror-prod-tests", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"StatefulSet", Namespace:"mirror-prod-tests", Name:"prod-broker-ss", UID:"3214c793-83dd-459d-a278-9dd6f36e9044", APIVersion:"apps/v1", ResourceVersion:"30188535", FieldPath:""}, Reason:"SuccessfulCreate", Message:"create Drain Pod prod-broker-ss-3 in StatefulSet prod-broker-ss successful", Source:v1.EventSource{Component:"statefulset-drain-controller", Host:""}, FirstTimestamp:time.Date(2024, time.July, 29, 11, 19, 16, 701252021, time.Local), LastTimestamp:time.Date(2024, time.July, 29, 11, 19, 16, 701252021, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"statefulset-drain-controller", ReportingInstance:""}': 'can't create an event with namespace 'mirror-prod-tests' in namespace 'mirror-dr-tests''(may retry after sleeping)
      E0729 11:19:35.643822       1 event.go:289] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"prod-broker-ss.17e6ab079ce649b5", GenerateName:"", Namespace:"mirror-prod-tests", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"StatefulSet", Namespace:"mirror-prod-tests", Name:"prod-broker-ss", UID:"3214c793-83dd-459d-a278-9dd6f36e9044", APIVersion:"apps/v1", ResourceVersion:"30188535", FieldPath:""}, Reason:"SuccessfulCreate", Message:"create Drain Pod prod-broker-ss-3 in StatefulSet prod-broker-ss successful", Source:v1.EventSource{Component:"statefulset-drain-controller", Host:""}, FirstTimestamp:time.Date(2024, time.July, 29, 11, 19, 16, 701252021, time.Local), LastTimestamp:time.Date(2024, time.July, 29, 11, 19, 16, 701252021, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"statefulset-drain-controller", ReportingInstance:""}': 'can't create an event with namespace 'mirror-prod-tests' in namespace 'mirror-dr-tests''(may retry after sleeping)
      

              Unassigned Unassigned
              mtoth@redhat.com Michal Toth
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: