Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-13362

Regular "timeout or abort" messages reported by kube-apiserver when talking to aggregated openshift-apiserver

    XMLWordPrintable

Details

    • Low
    • No
    • False
    • Hide

      None

      Show
      None
    • Customer Escalated
    • 8/8: pending triage by kube-apiserver team, need m-g/logs; not related to OCPBUGS-2474

    Description

      Description of problem:

      
In various OpenShift Container Platform 4.12 clusters (also earlier version) we see regularly the below messages reported even though the OpenShift Container Platform 4 - Control-Plane Node(s) look healthy and have enough resources. Further, also openshift-apiserver looks healthy and no sign of problem. Yet there appears to be problems when talking to openshift-apiserver.

2023-04-25T09:26:05.456861999Z E0425 09:26:05.456748      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/route.openshift.io/v1/namespaces/foo/routes?allowWatchBookmarks=true&resourceVersion=3049544273&watch=true" audit-ID="17df2f86-5504-4894-88fe-8f83e7830c65"
2023-04-25T09:26:05.754898161Z E0425 09:26:05.754789      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/template.openshift.io/v1/namespaces/bar/templates?allowWatchBookmarks=true&resourceVersion=3049544290&watch=true" audit-ID="f9142e12-8dcd-47ba-8330-d9d372674546"
2023-04-25T09:26:05.796985286Z E0425 09:26:05.796868      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/route.openshift.io/v1/namespaces/abc/routes?allowWatchBookmarks=true&resourceVersion=3049544292&watch=true" audit-ID="fc7f9c6b-af53-41fb-af6f-d723880d7c0f"
2023-04-25T09:26:05.952677485Z E0425 09:26:05.952580      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/template.openshift.io/v1/namespaces/xyz/templates?allowWatchBookmarks=true&resourceVersion=3049544298&watch=true" audit-ID="4baaa4cf-3542-4ba0-b4ac-6c9cd930f960"
2023-04-25T09:26:05.976663210Z E0425 09:26:05.976553      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/route.openshift.io/v1/namespaces/def/routes?allowWatchBookmarks=true&resourceVersion=3049544302&watch=true" audit-ID="31948a9b-61fb-4f68-bac0-549b552a6f38"
2023-04-25T09:26:06.013610251Z E0425 09:26:06.013488      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/image.openshift.io/v1/namespaces/foo/imagestreams?allowWatchBookmarks=true&resourceVersion=3049544309&watch=true" audit-ID="87d0f4c3-0174-4104-ab64-fbde7ca4171b"
2023-04-25T09:26:06.329287187Z E0425 09:26:06.327531      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/apps.openshift.io/v1/namespaces/bar/deploymentconfigs?allowWatchBookmarks=true&resourceVersion=3049544335&watch=true" audit-ID="66f8a57f-8866-4607-b038-fc29ff0868fb"
2023-04-25T09:26:06.380052191Z E0425 09:26:06.377449      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/image.openshift.io/v1/namespaces/def/imagestreams?allowWatchBookmarks=true&resourceVersion=3049544338&watch=true" audit-ID="eed22620-e44a-4745-84b3-ddd5d224852f"
2023-04-25T09:26:06.394277976Z E0425 09:26:06.394151      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/template.openshift.io/v1/namespaces/abc/templates?allowWatchBookmarks=true&resourceVersion=3049544340&watch=true" audit-ID="966f3946-35a0-48a1-a83e-cde576cabf84"
2023-04-25T09:26:06.453989174Z E0425 09:26:06.453875      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/route.openshift.io/v1/namespaces/xyz/routes?allowWatchBookmarks=true&resourceVersion=3049544342&watch=true" audit-ID="61a85cca-bb6c-4c56-89cf-e23b2ac24431"
2023-04-25T09:26:06.705008639Z E0425 09:26:06.704889      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/apps.openshift.io/v1/namespaces/abc/deploymentconfigs?allowWatchBookmarks=true&resourceVersion=3049544351&watch=true" audit-ID="6bd31193-c2fd-4b25-9d98-807a83974019"
2023-04-25T09:26:06.798081380Z E0425 09:26:06.797951      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/image.openshift.io/v1/namespaces/lmn/imagestreams?allowWatchBookmarks=true&resourceVersion=3049544359&watch=true" audit-ID="12d13a73-bf51-4de5-82b2-8da07d97e389"
2023-04-25T09:26:06.858225902Z E0425 09:26:06.857918      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/build.openshift.io/v1/namespaces/bar/buildconfigs?allowWatchBookmarks=true&resourceVersion=3049544365&watch=true" audit-ID="943f1337-3ad5-4cf7-ae8a-543883657338"
2023-04-25T09:26:07.118518079Z E0425 09:26:07.118416      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/apps.openshift.io/v1/namespaces/xyz/deploymentconfigs?allowWatchBookmarks=true&resourceVersion=3049544384&watch=true" audit-ID="a9b95c61-9773-41e7-87f9-813e40d02459"
2023-04-25T09:26:07.596106858Z E0425 09:26:07.596004      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/template.openshift.io/v1/namespaces/hij/templates?allowWatchBookmarks=true&resourceVersion=3049544423&watch=true" audit-ID="f88a123f-d946-4eb2-b525-db0a35da9702"
2023-04-25T09:26:07.615173411Z E0425 09:26:07.615036      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/route.openshift.io/v1/namespaces/bar/routes?allowWatchBookmarks=true&resourceVersion=3049544432&watch=true" audit-ID="1f8146c7-d441-43c4-a220-5da39f858f30"
2023-04-25T09:26:07.804898790Z E0425 09:26:07.804785      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/image.openshift.io/v1/namespaces/xyz/imagestreams?allowWatchBookmarks=true&resourceVersion=3049544451&watch=true" audit-ID="a7974474-48ca-47ea-80ad-7207e5d2864c"

Currently there does not appear to be an impact on the overall OpenShift Container Platform 4 - Cluster but it would be good to understand the root cause for them so we can document it and also highlight whether it's actually a problem or not.

Specifically in Red Hat Technical Support such errors are a problem as it's never clear whether those are expected and can be ignored or if they expose a problematic state and therefore need to be resolved.

      Version-Release number of selected component (if applicable):

      OpenShift Container Platform 4.10, 4.11 and 4.12

      How reproducible:

      Always

      Steps to Reproduce:

      1. Steps are not clear as they just show up and it's currently not clear what is triggering them

      Actual results:

      The below logs are reported by kube-apiserver when it tries to reach the aggregated openshift-apiserver

2023-04-25T09:26:05.456861999Z E0425 09:26:05.456748      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/route.openshift.io/v1/namespaces/foo/routes?allowWatchBookmarks=true&resourceVersion=3049544273&watch=true" audit-ID="17df2f86-5504-4894-88fe-8f83e7830c65"
2023-04-25T09:26:05.754898161Z E0425 09:26:05.754789      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/template.openshift.io/v1/namespaces/bar/templates?allowWatchBookmarks=true&resourceVersion=3049544290&watch=true" audit-ID="f9142e12-8dcd-47ba-8330-d9d372674546"
2023-04-25T09:26:05.796985286Z E0425 09:26:05.796868      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/route.openshift.io/v1/namespaces/abc/routes?allowWatchBookmarks=true&resourceVersion=3049544292&watch=true" audit-ID="fc7f9c6b-af53-41fb-af6f-d723880d7c0f"
2023-04-25T09:26:05.952677485Z E0425 09:26:05.952580      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/template.openshift.io/v1/namespaces/xyz/templates?allowWatchBookmarks=true&resourceVersion=3049544298&watch=true" audit-ID="4baaa4cf-3542-4ba0-b4ac-6c9cd930f960"
2023-04-25T09:26:05.976663210Z E0425 09:26:05.976553      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/route.openshift.io/v1/namespaces/def/routes?allowWatchBookmarks=true&resourceVersion=3049544302&watch=true" audit-ID="31948a9b-61fb-4f68-bac0-549b552a6f38"
2023-04-25T09:26:06.013610251Z E0425 09:26:06.013488      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/image.openshift.io/v1/namespaces/foo/imagestreams?allowWatchBookmarks=true&resourceVersion=3049544309&watch=true" audit-ID="87d0f4c3-0174-4104-ab64-fbde7ca4171b"
2023-04-25T09:26:06.329287187Z E0425 09:26:06.327531      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/apps.openshift.io/v1/namespaces/bar/deploymentconfigs?allowWatchBookmarks=true&resourceVersion=3049544335&watch=true" audit-ID="66f8a57f-8866-4607-b038-fc29ff0868fb"
2023-04-25T09:26:06.380052191Z E0425 09:26:06.377449      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/image.openshift.io/v1/namespaces/def/imagestreams?allowWatchBookmarks=true&resourceVersion=3049544338&watch=true" audit-ID="eed22620-e44a-4745-84b3-ddd5d224852f"
2023-04-25T09:26:06.394277976Z E0425 09:26:06.394151      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/template.openshift.io/v1/namespaces/abc/templates?allowWatchBookmarks=true&resourceVersion=3049544340&watch=true" audit-ID="966f3946-35a0-48a1-a83e-cde576cabf84"
2023-04-25T09:26:06.453989174Z E0425 09:26:06.453875      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/route.openshift.io/v1/namespaces/xyz/routes?allowWatchBookmarks=true&resourceVersion=3049544342&watch=true" audit-ID="61a85cca-bb6c-4c56-89cf-e23b2ac24431"
2023-04-25T09:26:06.705008639Z E0425 09:26:06.704889      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/apps.openshift.io/v1/namespaces/abc/deploymentconfigs?allowWatchBookmarks=true&resourceVersion=3049544351&watch=true" audit-ID="6bd31193-c2fd-4b25-9d98-807a83974019"
2023-04-25T09:26:06.798081380Z E0425 09:26:06.797951      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/image.openshift.io/v1/namespaces/lmn/imagestreams?allowWatchBookmarks=true&resourceVersion=3049544359&watch=true" audit-ID="12d13a73-bf51-4de5-82b2-8da07d97e389"
2023-04-25T09:26:06.858225902Z E0425 09:26:06.857918      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/build.openshift.io/v1/namespaces/bar/buildconfigs?allowWatchBookmarks=true&resourceVersion=3049544365&watch=true" audit-ID="943f1337-3ad5-4cf7-ae8a-543883657338"
2023-04-25T09:26:07.118518079Z E0425 09:26:07.118416      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/apps.openshift.io/v1/namespaces/xyz/deploymentconfigs?allowWatchBookmarks=true&resourceVersion=3049544384&watch=true" audit-ID="a9b95c61-9773-41e7-87f9-813e40d02459"
2023-04-25T09:26:07.596106858Z E0425 09:26:07.596004      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/template.openshift.io/v1/namespaces/hij/templates?allowWatchBookmarks=true&resourceVersion=3049544423&watch=true" audit-ID="f88a123f-d946-4eb2-b525-db0a35da9702"
2023-04-25T09:26:07.615173411Z E0425 09:26:07.615036      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/route.openshift.io/v1/namespaces/bar/routes?allowWatchBookmarks=true&resourceVersion=3049544432&watch=true" audit-ID="1f8146c7-d441-43c4-a220-5da39f858f30"
2023-04-25T09:26:07.804898790Z E0425 09:26:07.804785      18 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/image.openshift.io/v1/namespaces/xyz/imagestreams?allowWatchBookmarks=true&resourceVersion=3049544451&watch=true" audit-ID="a7974474-48ca-47ea-80ad-7207e5d2864c"

      Expected results:

      No timeout or aboard message reported by kube-apiserver when talking to the aggregated openshift-apiserver or at least have understanding whether the messages reported are indicating a problem or not.

      Additional info:

      Attachments

        Activity

          People

            akashem@redhat.com Abu H Kashem
            rhn-support-sreber Simon Reber
            Rahul Gangwar Rahul Gangwar
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated: