Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-8821

Improvement of log error messages in builds

XMLWordPrintable

    • Moderate
    • All
    • Hide
      Cause: error handling considers "access denied" error only as "authentication required".

      Consequence: bug causes incorrect error log, that may lead to confusions.

      Fix: through docker distribution error handling changed error output from "authentication required" to "access denied"

      Result: "access denied" error gives more precise error logs
      Show
      Cause: error handling considers "access denied" error only as "authentication required". Consequence: bug causes incorrect error log, that may lead to confusions. Fix: through docker distribution error handling changed error output from "authentication required" to "access denied" Result: "access denied" error gives more precise error logs
    • Bug Fix

      Description of problem:

      Example 1: S2I build in project A from image in project B

      This requires a RoleBinding in project B on system:image-puller to allow Group system:serviceaccounts:A. If this is missing or incorrect you get the following error when building:

      Cloning "ssh://git@git.internal/example.git" ...
      Commit: a42782afc295e8ab019728ccc998bf0c3f4a2e74 (Test)
      Author: Dave <dave@example.com>
      Date: Fri Oct 16 10:58:31 2020 +0100
      Caching blobs under "/var/cache/blobs".
      Warning: Pull failed, retrying in 5s ...
      Warning: Pull failed, retrying in 5s ...
      Warning: Pull failed, retrying in 5s ...
      error: build error: After retrying 2 times, Pull image still failed due to error: unauthorized: authentication required

      This issue is nothing about authentication - the serviceaccount builder in project A has successfully authenticated to the registry, but it not authorised to pull the S2I builder image. I'd suggest a much more useful error would be "system:serviceaccounts:A:builder does not have permission to pull image-mage-registry.openshift-image-registry.svc:5000/B/s2i-builder-xyz"

      For reproducer:

      apiVersion: v1
      kind: Template
      metadata:
      name: test-a
      parameters:
      objects:

      • apiVersion: rbac.authorization.k8s.io/v1
        kind: RoleBinding
        metadata:
        name: "system:image-pullers"
        roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: "system:image-puller"
        subjects:
      • apiGroup: rbac.authorization.k8s.io
        kind: Group
        name: "system:serviceaccounts:test-b"
      • apiVersion: image.openshift.io/v1
        kind: ImageStream
        metadata:
        name: python
      • apiVersion: build.openshift.io/v1
        kind: BuildConfig
        metadata:
        name: python-36
        spec:
        failedBuildsHistoryLimit: 1
        successfulBuildsHistoryLimit: 1
        output:
        to:
        kind: ImageStreamTag
        name: python:3.6
        runPolicy: Serial
        source:
        type: Dockerfile
        dockerfile: |
        FROM ignored
        RUN echo hello > /tmp/test
        strategy:
        dockerStrategy:
        from:
        kind: ImageStreamTag
        namespace: openshift
        name: python:3.6

      Example 2: DeploymentConfig in project A references image in project B

      eg. spec.template.spec.containers[].image: image-registry.openshift-image-registry.svc:5000/B/xyz:latest

      Pod status ends up in ImagePullBackOff / ErrImagePull with an error in the Events tab:

      Failed to pull image "image-registry.openshift-image-registry.svc:5000/B/xyz:latest": rpc error: code = Unknown desc = Error reading manifest latest in image-registry.openshift-image-registry.svc:5000/B/xyz: unauthorized: authentication required

      For reproducer.
      apiVersion: v1
      kind: Template
      metadata:
      name: test-b
      parameters:
      objects:

      1. ----------- IMAGE STREAM --------------------
      • apiVersion: image.openshift.io/v1
        kind: ImageStream
        metadata:
        name: django-ex
      1. ----------- BUILD --------------------
      • apiVersion: build.openshift.io/v1
        kind: BuildConfig
        metadata:
        name: django-ex
        spec:
        failedBuildsHistoryLimit: 1
        successfulBuildsHistoryLimit: 1
        output:
        to:
        kind: ImageStreamTag
        name: "django-ex:latest"
        resources:
        requests:
        cpu: 100m
        memory: 100Mi
        limits:
        cpu: 2
        memory: 2Gi
        source:
        git:
        uri: "https://github.com/sclorg/django-ex.git"
        type: Git
        strategy:
        sourceStrategy:
        from:
        kind: ImageStreamTag
        name: "python:3.6"
        namespace: test-a
      • apiVersion: v1
        kind: DeploymentConfig
        metadata:
        name: python-example
        spec:
        replicas: 1
        selector:
        app: python-example
        deploymentconfig: python-example
        template:
        metadata:
        labels:
        app: python-example
        deploymentconfig: python-example
        spec:
        containers:
      • image: image-registry.openshift-image-registry.svc:5000/test-a/python:3.6
        imagePullPolicy: Always
        name: python-example
        command: [ "/bin/bash", "-c", "sleep infinity" ]
        resources:
        requests:
        cpu: 100m
        memory: 100Mi
        limits:

      Attached two templates, one for a project called test-a and another called test-b. If you apply them, you'll be able to run both builds and roll out the deploymentconfig. However if you then remove the system:image-puller rolebinding in test-a allowing test-b, then try rebuilding the build in test-b and rolling out the deploymentconfig in test-b you'll get the error messages discussed above.

      Actual results: Incorrect log message errors or not very accurate for troubleshooting.

      Expected results: Better logs.

            fmissi Flavian Missi
            rhn-support-dahernan David Hernandez Fernandez
            Wen Wang Wen Wang
            Red Hat Employee
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: