Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-4149

ImageStreamChange triggers using annotations does not work for daemonset



    • Bug
    • Resolution: Won't Do
    • Major
    • None
    • 4.11.z
    • ImageStreams
    • Rejected
    • False
    • Hide




      Description of problem:

          Imagestream trigger annonation doesn't work for daemonset

      Version-Release number of selected component (if applicable):

          Server Version: 4.11.9
          Kubernetes Version: v1.24.0+dc5a2fd

      How reproducible:


      Steps to Reproduce:

          1.Create a imagestream to v1 image 
            oc import-image trigger-example:v1 --from quay.io/mwasher/trigger-example:v1  --confirm
            oc import-image trigger-example:v2 --from quay.io/mwasher/trigger-example:v2  --confirm
            oc tag trigger-example:v1 trigger-example:latest
          2. Apply the DaemonSet and Deployment Configuration
          $ oc apply -f deployment.yaml
          $ oc apply -f daemonset.yaml
          $ sleep 10 
          3. Retag the `trigger-example:latest` ImageStream
          oc tag trigger-example:v2 trigger-example:latest    
          4. The Deployment starts rolling out, but the DaemonSet does not

      Actual results:
      Pods are not rolled out

      Expected results:
      The DaemonSet image is updated and Pods rolledout

      Additional info:

          Confirmed that this issue is seen in `Server Version: 4.11.9`. 
          Reviewing the ControllerManager code, DaemonSet, Deployment, StatefulSet are all part of the update triggers.
          Debugging the failures in cluster, increase the OpenShiftControllerManager logLevel to TraceAll: 
          ~~~ |⇒ kl  -n openshift-controller-manager controller-manager-bw7z5 | grep -ie "DaemonSet" -e "daemonset-example" | less
              I1128 05:48:32.540957       1 image_trigger_controller.go:332] Error syncing resource daemonsets.apps/trigger-example/daemonset-example: daemonsets.apps "daemonset-example" is forbidden: User "system:serviceaccoun
              t:openshift-infra:image-trigger-controller" cannot update resource "daemonsets" in API group "apps" in the namespace "trigger-example"
              I1128 05:48:33.541988       1 image_trigger_controller.go:375] Started syncing resource "daemonsets.apps/trigger-example/daemonset-example"
              I1128 05:48:33.542080       1 annotations.go:126] *v1.DaemonSet/daemonset-example has triggers: []trigger.ObjectFieldTrigger{trigger.ObjectFieldTrigger{From:trigger.ObjectReference{Kind:"ImageStreamTag", Name:"tri
              gger-example:latest", Namespace:"", APIVersion:""}, FieldPath:"spec.template.spec.containers[?(@.name==\"daemonset-example\")].image", Paused:false}}
              I1128 05:48:33.542139       1 annotations.go:164] *v1.DaemonSet/daemonset-example detected change on spec.template.spec.containers[?(@.name=="daemonset-example")].image = quay.io/mwasher/trigger-example@sha256:a5f
              00000010  09 44 61 65 6d 6f 6e 53  65 74 12 86 18 0a b2 14  |.DaemonSet......|
              00000020  0a 11 64 61 65 6d 6f 6e  73 65 74 2d 65 78 61 6d  |..daemonset-exam|
              I1128 05:48:33.542421       1 round_trippers.go:463] PUT
              I1128 05:48:33.544316       1 image_trigger_controller.go:377] Finished syncing resource "daemonsets.apps/trigger-example/daemonset-example" (2.346928ms)
          Reviewing the `system:serviceaccount:openshift-infra:image-trigger-controller` ClusterRoleBindings+ClusterRole for the SA, it looks like the RBAC policies are in place for all the required types but I believe the APIGroup is wrong.
          The ClusterRole appears to be managed by the OpenShift API and not the Operator, but this looks like it might be in the process of changing.[0]
          I believe the bug is here: [1]
                      rbacv1helpers.NewRule("get", "update").Groups(extensionsGroup).Resources("daemonsets").RuleOrDie(),
          I believe that it should read as follows, adding the appsGroup, otherwise the resulting RBAC doesn't include the right ApiGroups
                      rbacv1helpers.NewRule("get", "update").Groups(extensionsGroup, appsGroup).Resources("daemonsets").RuleOrDie(),
          A workaround for fixing the issue is manually adding the permissions to the clusterrole and re-tagging the Image to trigger a rollout.
      [0] https://github.com/openshift/openshift-apiserver/blob/master/pkg/bootstrappolicy/controller_policy.go/#L216-L217
      [1] https://github.com/openshift/openshift-apiserver/blame/0f9edf3f0e2eca1998ae15fca71e0844aae8641c/pkg/bootstrappolicy/controller_policy.go#L212-L234




            fmissi Flavian Missi
            rhn-support-mwasher Michael Washer
            0 Vote for this issue
            3 Start watching this issue