Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-4149

ImageStreamChange triggers using annotations does not work for daemonset

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Major Major
    • None
    • 4.11.z
    • ImageStreams
    • None
    • Rejected
    • False
    • Hide

      None

      Show
      None

      Description of problem:

          Imagestream trigger annonation doesn't work for daemonset

      Version-Release number of selected component (if applicable):

          Server Version: 4.11.9
          Kubernetes Version: v1.24.0+dc5a2fd
          

      How reproducible:

          Always

      Steps to Reproduce:

          1.Create a imagestream to v1 image 
            oc import-image trigger-example:v1 --from quay.io/mwasher/trigger-example:v1  --confirm
            oc import-image trigger-example:v2 --from quay.io/mwasher/trigger-example:v2  --confirm
            oc tag trigger-example:v1 trigger-example:latest
      
          2. Apply the DaemonSet and Deployment Configuration
          $ oc apply -f deployment.yaml
          $ oc apply -f daemonset.yaml
          $ sleep 10 
          
          3. Retag the `trigger-example:latest` ImageStream
          oc tag trigger-example:v2 trigger-example:latest    
      
          4. The Deployment starts rolling out, but the DaemonSet does not
          

      Actual results:
      Pods are not rolled out

      Expected results:
      The DaemonSet image is updated and Pods rolledout

      Additional info:

          Confirmed that this issue is seen in `Server Version: 4.11.9`. 
      
          Reviewing the ControllerManager code, DaemonSet, Deployment, StatefulSet are all part of the update triggers.
      
          Debugging the failures in cluster, increase the OpenShiftControllerManager logLevel to TraceAll: 
          ~~~ |⇒ kl  -n openshift-controller-manager controller-manager-bw7z5 | grep -ie "DaemonSet" -e "daemonset-example" | less
              I1128 05:48:32.540957       1 image_trigger_controller.go:332] Error syncing resource daemonsets.apps/trigger-example/daemonset-example: daemonsets.apps "daemonset-example" is forbidden: User "system:serviceaccoun
              t:openshift-infra:image-trigger-controller" cannot update resource "daemonsets" in API group "apps" in the namespace "trigger-example"
              I1128 05:48:33.541988       1 image_trigger_controller.go:375] Started syncing resource "daemonsets.apps/trigger-example/daemonset-example"
              I1128 05:48:33.542080       1 annotations.go:126] *v1.DaemonSet/daemonset-example has triggers: []trigger.ObjectFieldTrigger{trigger.ObjectFieldTrigger{From:trigger.ObjectReference{Kind:"ImageStreamTag", Name:"tri
              gger-example:latest", Namespace:"", APIVersion:""}, FieldPath:"spec.template.spec.containers[?(@.name==\"daemonset-example\")].image", Paused:false}}
              I1128 05:48:33.542139       1 annotations.go:164] *v1.DaemonSet/daemonset-example detected change on spec.template.spec.containers[?(@.name=="daemonset-example")].image = quay.io/mwasher/trigger-example@sha256:a5f
              9005dc27a5da42d051d3f1724fcd87598f8a888943ad9b4375b7e8a047a62
              00000010  09 44 61 65 6d 6f 6e 53  65 74 12 86 18 0a b2 14  |.DaemonSet......|
              00000020  0a 11 64 61 65 6d 6f 6e  73 65 74 2d 65 78 61 6d  |..daemonset-exam|
              I1128 05:48:33.542421       1 round_trippers.go:463] PUT https://172.30.0.1:443/apis/apps/v1/namespaces/trigger-example/daemonsets/daemonset-example
              I1128 05:48:33.544316       1 image_trigger_controller.go:377] Finished syncing resource "daemonsets.apps/trigger-example/daemonset-example" (2.346928ms)
          ~~~
      
          Reviewing the `system:serviceaccount:openshift-infra:image-trigger-controller` ClusterRoleBindings+ClusterRole for the SA, it looks like the RBAC policies are in place for all the required types but I believe the APIGroup is wrong.
      
          The ClusterRole appears to be managed by the OpenShift API and not the Operator, but this looks like it might be in the process of changing.[0]
      
          I believe the bug is here: [1]
          ~~~
                      rbacv1helpers.NewRule("get", "update").Groups(extensionsGroup).Resources("daemonsets").RuleOrDie(),
          ~~~
      
          I believe that it should read as follows, adding the appsGroup, otherwise the resulting RBAC doesn't include the right ApiGroups
          ~~~
                      rbacv1helpers.NewRule("get", "update").Groups(extensionsGroup, appsGroup).Resources("daemonsets").RuleOrDie(),
          ~~~
      
          A workaround for fixing the issue is manually adding the permissions to the clusterrole and re-tagging the Image to trigger a rollout.
      
      Resources:
      [0] https://github.com/openshift/openshift-apiserver/blob/master/pkg/bootstrappolicy/controller_policy.go/#L216-L217
      [1] https://github.com/openshift/openshift-apiserver/blame/0f9edf3f0e2eca1998ae15fca71e0844aae8641c/pkg/bootstrappolicy/controller_policy.go#L212-L234
      
      

              fmissi Flavian Missi
              rhn-support-mwasher Michael Washer (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: