Uploaded image for project: 'Knative Serving'
  1. Knative Serving
  2. SRVKS-512

knative-serving controller and autoscaler-hpa restarts several times every few hours

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • 1.7.0
    • 1.7.0
    • None

      On OCP 4.3.9 on AWS, Serverless 1.7.0, installed from serverles-operator master (./hack/catalog.sh | oc apply -n openshift-marketplace -f - )

      After 8h of cluster runtime, the restart count of knative-serving controller container is suspiciously high.

      [maschmid@archie ocf-qe-testsuite]$ oc get pods -n knative-serving
      NAME                              READY   STATUS    RESTARTS   AGE
      activator-785fdf59dd-mqqsz        1/1     Running   0          5h18m
      autoscaler-75f56584bf-dqpdr       1/1     Running   0          5h18m
      autoscaler-hpa-848b6bdfd4-8prdw   1/1     Running   16         5h17m
      autoscaler-hpa-848b6bdfd4-jk9tr   1/1     Running   20         5h17m
      controller-868c5f8d68-hjb7b       1/1     Running   20         5h18m
      controller-868c5f8d68-qbcpn       1/1     Running   19         5h18m
      webhook-67fb8b566d-lf7fq          1/1     Running   0          5h18m
      

      Logs from one of the controller runs including the end

      $ oc logs -n knative-serving controller-868c5f8d68-hjb7b -p
      2020/04/08 16:58:02 Registering 4 clients
      2020/04/08 16:58:02 Registering 3 informer factories
      2020/04/08 16:58:02 Registering 13 informers
      2020/04/08 16:58:02 Registering 7 controllers
      {"level":"info","ts":"2020-04-08T16:58:02.346Z","caller":"logging/config.go:108","msg":"Successfully created the logger."}
      {"level":"info","ts":"2020-04-08T16:58:02.346Z","caller":"logging/config.go:109","msg":"Logging level set to info"}
      {"level":"info","ts":"2020-04-08T16:58:02.346Z","caller":"logging/config.go:76","msg":"Fetch GitHub commit ID from kodata failed: \"KO_DATA_PATH\" does not exist or is empty"}
      {"level":"info","ts":"2020-04-08T16:58:02.346Z","logger":"controller","caller":"profiling/server.go:59","msg":"Profiling enabled: false"}
      {"level":"info","ts":"2020-04-08T16:58:02.352Z","logger":"controller","caller":"sharedmain/main.go:423","msg":"controller will run in leader-elected mode with id controller-868c5f8d68-hjb7b_c336708a-24f3-4c57-9632-9ef87230f238"}
      I0408 16:58:02.352573       1 leaderelection.go:241] attempting to acquire leader lease  knative-serving/controller...
      I0408 16:58:19.787998       1 leaderelection.go:251] successfully acquired lease knative-serving/controller
      {"level":"info","ts":"2020-04-08T16:58:19.788Z","logger":"controller.event-broadcaster","caller":"record/event.go:274","msg":"Event(v1.ObjectReference{Kind:\"Lease\", Namespace:\"knative-serving\", Name:\"controller\", UID:\"0f3f1f3d-d16f-427d-bf17-8fe41e4a72a7\", APIVersion:\"coordination.k8s.io/v1\", ResourceVersion:\"371235\", FieldPath:\"\"}): type: 'Normal' reason: 'LeaderElection' controller-868c5f8d68-hjb7b_c336708a-24f3-4c57-9632-9ef87230f238 became leader"}
      {"level":"info","ts":"2020-04-08T16:58:19.788Z","logger":"controller.configuration-controller","caller":"configuration/controller.go:52","msg":"Setting up event handlers","knative.dev/controller":"configuration-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.788Z","logger":"controller.labeler-controller","caller":"labeler/controller.go:60","msg":"Setting up event handlers","knative.dev/controller":"labeler-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.804Z","logger":"controller.revision-controller","caller":"revision/controller.go:115","msg":"Setting up event handlers","knative.dev/controller":"revision-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.804Z","logger":"controller.route-controller","caller":"route/controller.go:97","msg":"Setting up event handlers","knative.dev/controller":"route-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.804Z","logger":"controller.serverlessservice-controller","caller":"serverlessservice/controller.go:64","msg":"Setting up event handlers","knative.dev/controller":"serverlessservice-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.804Z","logger":"controller.service-controller","caller":"service/controller.go:60","msg":"Setting up event handlers","knative.dev/controller":"service-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.804Z","logger":"controller.revision-gc-controller","caller":"gc/controller.go:53","msg":"Setting up event handlers","knative.dev/controller":"revision-gc-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.804Z","logger":"controller.revision-gc-controller","caller":"gc/controller.go:66","msg":"Setting up ConfigMap receivers with resync func","knative.dev/controller":"revision-gc-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.804Z","logger":"controller.revision-gc-controller","caller":"gc/controller.go:75","msg":"Setting up ConfigMap receivers","knative.dev/controller":"revision-gc-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.811Z","logger":"controller","caller":"sharedmain/main.go:172","msg":"Starting configuration manager..."}
      {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.revision-controller.config-store","caller":"configmap/store.go:157","msg":"revision config \"config-logging\" config was added or updated: &logging.Config{LoggingConfig:\"{\\n  \\\"level\\\": \\\"info\\\",\\n  \\\"development\\\": false,\\n  \\\"outputPaths\\\": [\\\"stdout\\\"],\\n  \\\"errorOutputPaths\\\": [\\\"stderr\\\"],\\n  \\\"encoding\\\": \\\"json\\\",\\n  \\\"encoderConfig\\\": {\\n    \\\"timeKey\\\": \\\"ts\\\",\\n    \\\"levelKey\\\": \\\"level\\\",\\n    \\\"nameKey\\\": \\\"logger\\\",\\n    \\\"callerKey\\\": \\\"caller\\\",\\n    \\\"messageKey\\\": \\\"msg\\\",\\n    \\\"stacktraceKey\\\": \\\"stacktrace\\\",\\n    \\\"lineEnding\\\": \\\"\\\",\\n    \\\"levelEncoder\\\": \\\"\\\",\\n    \\\"timeEncoder\\\": \\\"iso8601\\\",\\n    \\\"durationEncoder\\\": \\\"\\\",\\n    \\\"callerEncoder\\\": \\\"\\\"\\n  }\\n}\", LoggingLevel:map[string]zapcore.Level{}}","knative.dev/controller":"revision-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.revision-controller.config-store","caller":"configmap/store.go:157","msg":"revision config \"config-observability\" config was added or updated: &metrics.ObservabilityConfig{EnableVarLogCollection:false, LoggingURLTemplate:\"http://localhost:8001/api/v1/namespaces/knative-monitoring/services/kibana-logging/proxy/app/kibana#/discover?_a=(query:(match:(kubernetes.labels.knative-dev%2FrevisionUID:(query:'${REVISION_UID}',type:phrase))))\", RequestLogTemplate:\"\", EnableProbeRequestLog:false, RequestMetricsBackend:\"prometheus\", EnableProfiling:false}","knative.dev/controller":"revision-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller","caller":"metrics/exporter.go:126","msg":"Flushing the existing exporter before setting up the new exporter."}
      {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller","caller":"metrics/prometheus_exporter.go:37","msg":"Created Opencensus Prometheus exporter with config: &{knative.dev/internal/serving controller prometheus 5000000000 <nil>  false 9090 false   {   false}}. Start the server for Prometheus exporter."}
      {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller","caller":"metrics/exporter.go:135","msg":"Successfully updated the metrics exporter; old config: <nil>; new config &{knative.dev/internal/serving controller prometheus 5000000000 <nil>  false 9090 false   {   false}}"}
      {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.route-controller.config-store","caller":"configmap/store.go:157","msg":"route config \"config-gc\" config was added or updated: &gc.Config{StaleRevisionCreateDelay:172800000000000, StaleRevisionTimeout:54000000000000, StaleRevisionMinimumGenerations:20, StaleRevisionLastpinnedDebounce:18000000000000}","knative.dev/controller":"route-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.revision-gc-controller.config-store","caller":"configmap/store.go:157","msg":"configuration config \"config-gc\" config was added or updated: &gc.Config{StaleRevisionCreateDelay:172800000000000, StaleRevisionTimeout:54000000000000, StaleRevisionMinimumGenerations:20, StaleRevisionLastpinnedDebounce:18000000000000}","knative.dev/controller":"revision-gc-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.revision-controller.config-store","caller":"configmap/store.go:157","msg":"revision config \"config-deployment\" config was added or updated: &deployment.Config{QueueSidecarImage:\"registry.svc.ci.openshift.org/openshift/knative-v0.13.2:knative-serving-queue\", RegistriesSkippingTagResolving:sets.String{\"dev.local\":sets.Empty{}, \"ko.local\":sets.Empty{}}}","knative.dev/controller":"revision-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.revision-controller.config-store","caller":"configmap/store.go:157","msg":"revision config \"config-tracing\" config was added or updated: &config.Config{Backend:\"none\", ZipkinEndpoint:\"\", StackdriverProjectID:\"\", Debug:false, SampleRate:0.1}","knative.dev/controller":"revision-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.revision-controller.config-store","caller":"configmap/store.go:157","msg":"revision config \"config-autoscaler\" config was added or updated: &config.Config{EnableScaleToZero:true, EnableGracefulScaledown:false, ContainerConcurrencyTargetFraction:0.7, ContainerConcurrencyTargetDefault:100, TargetUtilization:0.7, RPSTargetDefault:200, TargetBurstCapacity:200, MaxScaleUpRate:1000, MaxScaleDownRate:2, StableWindow:60000000000, PanicWindowPercentage:10, PanicThresholdPercentage:200, TickInterval:2000000000, ScaleToZeroGracePeriod:30000000000, PodAutoscalerClass:\"kpa.autoscaling.knative.dev\"}","knative.dev/controller":"revision-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.route-controller.config-store","caller":"configmap/store.go:157","msg":"route config \"config-domain\" config was added or updated: &config.Domain{Domains:map[string]*config.LabelSelector{\"apps.maschmid1.openshift-aws.rhocf-dev.net\":(*config.LabelSelector)(0xc000010ae0)}}","knative.dev/controller":"route-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.revision-controller.config-store","caller":"configmap/store.go:157","msg":"revision config \"config-defaults\" config was added or updated: &config.Defaults{RevisionTimeoutSeconds:300, MaxRevisionTimeoutSeconds:600, UserContainerNameTemplate:\"user-container\", ContainerConcurrency:0, RevisionCPURequest:(*resource.Quantity)(nil), RevisionCPULimit:(*resource.Quantity)(nil), RevisionMemoryRequest:(*resource.Quantity)(nil), RevisionMemoryLimit:(*resource.Quantity)(nil)}","knative.dev/controller":"revision-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.revision-controller.config-store","caller":"configmap/store.go:157","msg":"revision config \"config-network\" config was added or updated: &network.Config{DefaultIngressClass:\"kourier.ingress.networking.knative.dev\", DomainTemplate:\"{{.Name}}.{{.Namespace}}.{{.Domain}}\", TagTemplate:\"{{.Tag}}-{{.Name}}\", AutoTLS:false, HTTPProtocol:\"enabled\", DefaultCertificateClass:\"cert-manager.certificate.networking.knative.dev\"}","knative.dev/controller":"revision-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.route-controller.config-store","caller":"configmap/store.go:157","msg":"route config \"config-network\" config was added or updated: &network.Config{DefaultIngressClass:\"kourier.ingress.networking.knative.dev\", DomainTemplate:\"{{.Name}}.{{.Namespace}}.{{.Domain}}\", TagTemplate:\"{{.Tag}}-{{.Name}}\", AutoTLS:false, HTTPProtocol:\"enabled\", DefaultCertificateClass:\"cert-manager.certificate.networking.knative.dev\"}","knative.dev/controller":"route-controller"}
      {"level":"info","ts":"2020-04-08T16:58:19.911Z","logger":"controller","caller":"sharedmain/main.go:176","msg":"Starting informers..."}
      {"level":"info","ts":"2020-04-08T16:58:19.938Z","logger":"controller.serverlessservice-controller","caller":"serverlessservice/controller.go:85","msg":"Doing a global resync due to activator endpoint changes","knative.dev/controller":"serverlessservice-controller"}
      {"level":"info","ts":"2020-04-08T16:58:20.324Z","logger":"controller","caller":"sharedmain/main.go:180","msg":"Starting controllers..."}
      {"level":"info","ts":"2020-04-08T16:58:20.324Z","logger":"controller.revision-gc-controller","caller":"controller/controller.go:338","msg":"Starting controller and workers","knative.dev/controller":"revision-gc-controller"}
      {"level":"info","ts":"2020-04-08T16:58:20.324Z","logger":"controller.revision-gc-controller","caller":"controller/controller.go:348","msg":"Started workers","knative.dev/controller":"revision-gc-controller"}
      {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.revision-controller","caller":"controller/controller.go:338","msg":"Starting controller and workers","knative.dev/controller":"revision-controller"}
      {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.revision-controller","caller":"controller/controller.go:348","msg":"Started workers","knative.dev/controller":"revision-controller"}
      {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.revision-gc-controller","caller":"controller/controller.go:403","msg":"Reconcile succeeded. Time taken: 132.258µs.","knative.dev/controller":"revision-gc-controller","knative.dev/traceid":"30d08c91-02e7-4503-84e7-d6291683f379","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.route-controller","caller":"controller/controller.go:338","msg":"Starting controller and workers","knative.dev/controller":"route-controller"}
      {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.route-controller","caller":"controller/controller.go:348","msg":"Started workers","knative.dev/controller":"route-controller"}
      {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.route-controller","caller":"route/route.go:112","msg":"Reconciling route: &v1.Route{TypeMeta:v1.TypeMeta{Kind:\"Route\", APIVersion:\"serving.knative.dev/v1\"}, ObjectMeta:v1.ObjectMeta{Name:\"prime-generator\", GenerateName:\"\", Namespace:\"knativetutorial\", SelfLink:\"/apis/serving.knative.dev/v1/namespaces/knativetutorial/routes/prime-generator\", UID:\"0ebdc4e1-edfc-4a95-9b5d-65bad4708f33\", ResourceVersion:\"357237\", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63721960698, loc:(*time.Location)(0x2c03b40)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"serving.knative.dev/service\":\"prime-generator\"}, Annotations:map[string]string{\"serving.knative.dev/creator\":\"system:admin\", \"serving.knative.dev/lastModifier\":\"system:admin\"}, OwnerReferences:[]v1.OwnerReference{v1.OwnerReference{APIVersion:\"serving.knative.dev/v1\", Kind:\"Service\", Name:\"prime-generator\", UID:\"cc57787e-594e-48e9-a7dd-08584c1d2b6d\", Controller:(*bool)(0xc000761680), BlockOwnerDeletion:(*bool)(0xc000761681)}}, Finalizers:[]string{\"routes.serving.knative.dev\"}, ClusterName:\"\", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.RouteSpec{Traffic:[]v1.TrafficTarget{v1.TrafficTarget{Tag:\"\", RevisionName:\"\", ConfigurationName:\"prime-generator\", LatestRevision:(*bool)(0xc000761682), Percent:(*int64)(0xc000761688), URL:(*apis.URL)(nil)}}}, Status:v1.RouteStatus{Status:v1.Status{ObservedGeneration:1, Conditions:v1.Conditions{apis.Condition{Type:\"AllTrafficAssigned\", Status:\"True\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0x0, ext:63721960708, loc:(*time.Location)(0x2c03b40)}}}, Reason:\"\", Message:\"\"}, apis.Condition{Type:\"IngressReady\", Status:\"True\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0x0, ext:63721960708, loc:(*time.Location)(0x2c03b40)}}}, Reason:\"\", Message:\"\"}, apis.Condition{Type:\"Ready\", Status:\"True\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0x0, ext:63721960708, loc:(*time.Location)(0x2c03b40)}}}, Reason:\"\", Message:\"\"}}}, RouteStatusFields:v1.RouteStatusFields{URL:(*apis.URL)(0xc000f86300), Address:(*v1.Addressable)(0xc000fb80f0), Traffic:[]v1.TrafficTarget{v1.TrafficTarget{Tag:\"\", RevisionName:\"prime-generator-9rvz7\", ConfigurationName:\"\", LatestRevision:(*bool)(0xc000761690), Percent:(*int64)(0xc000761698), URL:(*apis.URL)(nil)}}}}}","knative.dev/controller":"route-controller","knative.dev/traceid":"693daa8d-d948-4f58-9a07-9ecd1c1a6d20","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.route-controller","caller":"route/route.go:308","msg":"Revision knativetutorial/prime-generator-9rvz7 is inactive","knative.dev/controller":"route-controller","knative.dev/traceid":"693daa8d-d948-4f58-9a07-9ecd1c1a6d20","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.route-controller","caller":"route/route.go:330","msg":"All referred targets are routable, marking AllTrafficAssigned with traffic information.","knative.dev/controller":"route-controller","knative.dev/traceid":"693daa8d-d948-4f58-9a07-9ecd1c1a6d20","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.route-controller","caller":"route/route.go:125","msg":"Updating targeted revisions.","knative.dev/controller":"route-controller","knative.dev/traceid":"693daa8d-d948-4f58-9a07-9ecd1c1a6d20","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.route-controller","caller":"route/route.go:139","msg":"Creating placeholder k8s services","knative.dev/controller":"route-controller","knative.dev/traceid":"693daa8d-d948-4f58-9a07-9ecd1c1a6d20","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.serverlessservice-controller","caller":"controller/controller.go:338","msg":"Starting controller and workers","knative.dev/controller":"serverlessservice-controller"}
      {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.serverlessservice-controller","caller":"controller/controller.go:348","msg":"Started workers","knative.dev/controller":"serverlessservice-controller"}
      {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.service-controller","caller":"controller/controller.go:338","msg":"Starting controller and workers","knative.dev/controller":"service-controller"}
      {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.service-controller","caller":"controller/controller.go:348","msg":"Started workers","knative.dev/controller":"service-controller"}
      {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.configuration-controller","caller":"controller/controller.go:338","msg":"Starting controller and workers","knative.dev/controller":"configuration-controller"}
      {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.configuration-controller","caller":"controller/controller.go:348","msg":"Started workers","knative.dev/controller":"configuration-controller"}
      {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.configuration-controller","caller":"configuration/configuration.go:104","msg":"Revision \"prime-generator-9rvz7\" of configuration is ready","knative.dev/controller":"configuration-controller","knative.dev/traceid":"bc93be80-d90f-4663-b1aa-74495cd58629","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.configuration-controller","caller":"controller/controller.go:403","msg":"Reconcile succeeded. Time taken: 108.706µs.","knative.dev/controller":"configuration-controller","knative.dev/traceid":"bc93be80-d90f-4663-b1aa-74495cd58629","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.labeler-controller","caller":"controller/controller.go:338","msg":"Starting controller and workers","knative.dev/controller":"labeler-controller"}
      {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.labeler-controller","caller":"controller/controller.go:348","msg":"Started workers","knative.dev/controller":"labeler-controller"}
      {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.service-controller","caller":"service/service.go:214","msg":"Reconciling configuration diff (-desired, +observed):\n","knative.dev/controller":"service-controller","knative.dev/traceid":"36b252d9-447a-4fe3-9a10-cce4f644ae4d","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.labeler-controller","caller":"controller/controller.go:403","msg":"Reconcile succeeded. Time taken: 116.944µs.","knative.dev/controller":"labeler-controller","knative.dev/traceid":"c5b85abb-1c9b-4572-9cb1-c8a5e4e19043","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.service-controller","caller":"service/service.go:263","msg":"Reconciling route diff (-desired, +observed):\n","knative.dev/controller":"service-controller","knative.dev/traceid":"36b252d9-447a-4fe3-9a10-cce4f644ae4d","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.service-controller","caller":"controller/controller.go:403","msg":"Reconcile succeeded. Time taken: 652.322µs.","knative.dev/controller":"service-controller","knative.dev/traceid":"36b252d9-447a-4fe3-9a10-cce4f644ae4d","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.343Z","logger":"controller.revision-controller","caller":"revision/reconcile_resources.go:135","msg":"Reconciling PA: prime-generator-9rvz7","knative.dev/controller":"revision-controller","knative.dev/traceid":"51bd7828-617b-4eae-bf6a-80379c79c74d","knative.dev/key":"knativetutorial/prime-generator-9rvz7"}
      {"level":"info","ts":"2020-04-08T16:58:20.343Z","logger":"controller.revision-controller","caller":"controller/controller.go:403","msg":"Reconcile succeeded. Time taken: 18.852296ms.","knative.dev/controller":"revision-controller","knative.dev/traceid":"51bd7828-617b-4eae-bf6a-80379c79c74d","knative.dev/key":"knativetutorial/prime-generator-9rvz7"}
      {"level":"info","ts":"2020-04-08T16:58:20.347Z","logger":"controller.route-controller","caller":"route/route.go:163","msg":"Updating placeholder k8s services with ingress information","knative.dev/controller":"route-controller","knative.dev/traceid":"693daa8d-d948-4f58-9a07-9ecd1c1a6d20","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.347Z","logger":"controller.route-controller","caller":"route/route.go:169","msg":"Route successfully synced","knative.dev/controller":"route-controller","knative.dev/traceid":"693daa8d-d948-4f58-9a07-9ecd1c1a6d20","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.347Z","logger":"controller.route-controller","caller":"controller/controller.go:403","msg":"Reconcile succeeded. Time taken: 21.937045ms.","knative.dev/controller":"route-controller","knative.dev/traceid":"693daa8d-d948-4f58-9a07-9ecd1c1a6d20","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.347Z","logger":"controller.route-controller","caller":"route/route.go:112","msg":"Reconciling route: &v1.Route{TypeMeta:v1.TypeMeta{Kind:\"Route\", APIVersion:\"serving.knative.dev/v1\"}, ObjectMeta:v1.ObjectMeta{Name:\"prime-generator\", GenerateName:\"\", Namespace:\"knativetutorial\", SelfLink:\"/apis/serving.knative.dev/v1/namespaces/knativetutorial/routes/prime-generator\", UID:\"0ebdc4e1-edfc-4a95-9b5d-65bad4708f33\", ResourceVersion:\"357237\", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63721960698, loc:(*time.Location)(0x2c03b40)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"serving.knative.dev/service\":\"prime-generator\"}, Annotations:map[string]string{\"serving.knative.dev/creator\":\"system:admin\", \"serving.knative.dev/lastModifier\":\"system:admin\"}, OwnerReferences:[]v1.OwnerReference{v1.OwnerReference{APIVersion:\"serving.knative.dev/v1\", Kind:\"Service\", Name:\"prime-generator\", UID:\"cc57787e-594e-48e9-a7dd-08584c1d2b6d\", Controller:(*bool)(0xc0009ae4d0), BlockOwnerDeletion:(*bool)(0xc0009ae4d1)}}, Finalizers:[]string{\"routes.serving.knative.dev\"}, ClusterName:\"\", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.RouteSpec{Traffic:[]v1.TrafficTarget{v1.TrafficTarget{Tag:\"\", RevisionName:\"\", ConfigurationName:\"prime-generator\", LatestRevision:(*bool)(0xc0009ae4d2), Percent:(*int64)(0xc0009ae4d8), URL:(*apis.URL)(nil)}}}, Status:v1.RouteStatus{Status:v1.Status{ObservedGeneration:1, Conditions:v1.Conditions{apis.Condition{Type:\"AllTrafficAssigned\", Status:\"True\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0x0, ext:63721960708, loc:(*time.Location)(0x2c03b40)}}}, Reason:\"\", Message:\"\"}, apis.Condition{Type:\"IngressReady\", Status:\"True\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0x0, ext:63721960708, loc:(*time.Location)(0x2c03b40)}}}, Reason:\"\", Message:\"\"}, apis.Condition{Type:\"Ready\", Status:\"True\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0x0, ext:63721960708, loc:(*time.Location)(0x2c03b40)}}}, Reason:\"\", Message:\"\"}}}, RouteStatusFields:v1.RouteStatusFields{URL:(*apis.URL)(0xc00085f680), Address:(*v1.Addressable)(0xc0000110e8), Traffic:[]v1.TrafficTarget{v1.TrafficTarget{Tag:\"\", RevisionName:\"prime-generator-9rvz7\", ConfigurationName:\"\", LatestRevision:(*bool)(0xc0009ae570), Percent:(*int64)(0xc0009ae578), URL:(*apis.URL)(nil)}}}}}","knative.dev/controller":"route-controller","knative.dev/traceid":"d1412786-f476-4f52-b493-c5aeb1ed4210","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.347Z","logger":"controller.route-controller","caller":"route/route.go:308","msg":"Revision knativetutorial/prime-generator-9rvz7 is inactive","knative.dev/controller":"route-controller","knative.dev/traceid":"d1412786-f476-4f52-b493-c5aeb1ed4210","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.347Z","logger":"controller.route-controller","caller":"route/route.go:330","msg":"All referred targets are routable, marking AllTrafficAssigned with traffic information.","knative.dev/controller":"route-controller","knative.dev/traceid":"d1412786-f476-4f52-b493-c5aeb1ed4210","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.347Z","logger":"controller.route-controller","caller":"route/route.go:125","msg":"Updating targeted revisions.","knative.dev/controller":"route-controller","knative.dev/traceid":"d1412786-f476-4f52-b493-c5aeb1ed4210","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.347Z","logger":"controller.route-controller","caller":"route/route.go:139","msg":"Creating placeholder k8s services","knative.dev/controller":"route-controller","knative.dev/traceid":"d1412786-f476-4f52-b493-c5aeb1ed4210","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.379Z","logger":"controller.route-controller","caller":"route/route.go:163","msg":"Updating placeholder k8s services with ingress information","knative.dev/controller":"route-controller","knative.dev/traceid":"d1412786-f476-4f52-b493-c5aeb1ed4210","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.380Z","logger":"controller.route-controller","caller":"route/route.go:169","msg":"Route successfully synced","knative.dev/controller":"route-controller","knative.dev/traceid":"d1412786-f476-4f52-b493-c5aeb1ed4210","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.380Z","logger":"controller.route-controller","caller":"controller/controller.go:403","msg":"Reconcile succeeded. Time taken: 32.869249ms.","knative.dev/controller":"route-controller","knative.dev/traceid":"d1412786-f476-4f52-b493-c5aeb1ed4210","knative.dev/key":"knativetutorial/prime-generator"}
      {"level":"info","ts":"2020-04-08T16:58:20.508Z","logger":"controller.serverlessservice-controller","caller":"serverlessservice/serverlessservice.go:209","msg":"Endpoints prime-generator-9rvz7 has no ready endpoints","knative.dev/controller":"serverlessservice-controller","knative.dev/traceid":"210aa2ec-6c51-421e-b467-4d0971c3d675","knative.dev/key":"knativetutorial/prime-generator-9rvz7"}
      {"level":"info","ts":"2020-04-08T16:58:20.508Z","logger":"controller.serverlessservice-controller","caller":"controller/controller.go:403","msg":"Reconcile succeeded. Time taken: 183.060201ms.","knative.dev/controller":"serverlessservice-controller","knative.dev/traceid":"210aa2ec-6c51-421e-b467-4d0971c3d675","knative.dev/key":"knativetutorial/prime-generator-9rvz7"}
      {"level":"info","ts":"2020-04-08T16:58:21.752Z","logger":"controller.serverlessservice-controller","caller":"serverlessservice/serverlessservice.go:209","msg":"Endpoints prime-generator-9rvz7 has no ready endpoints","knative.dev/controller":"serverlessservice-controller","knative.dev/traceid":"f2b8e646-dd96-4c97-8009-a29bcf817cf6","knative.dev/key":"knativetutorial/prime-generator-9rvz7"}
      {"level":"info","ts":"2020-04-08T16:58:21.752Z","logger":"controller.serverlessservice-controller","caller":"controller/controller.go:403","msg":"Reconcile succeeded. Time taken: 335.757µs.","knative.dev/controller":"serverlessservice-controller","knative.dev/traceid":"f2b8e646-dd96-4c97-8009-a29bcf817cf6","knative.dev/key":"knativetutorial/prime-generator-9rvz7"}
      I0408 17:00:53.735214       1 leaderelection.go:287] failed to renew lease knative-serving/controller: failed to tryAcquireOrRenew context deadline exceeded
      {"level":"info","ts":"2020-04-08T17:00:53.735Z","logger":"controller.event-broadcaster","caller":"record/event.go:274","msg":"Event(v1.ObjectReference{Kind:\"Lease\", Namespace:\"knative-serving\", Name:\"controller\", UID:\"0f3f1f3d-d16f-427d-bf17-8fe41e4a72a7\", APIVersion:\"coordination.k8s.io/v1\", ResourceVersion:\"373414\", FieldPath:\"\"}): type: 'Normal' reason: 'LeaderElection' controller-868c5f8d68-hjb7b_c336708a-24f3-4c57-9632-9ef87230f238 stopped leading"}
      {"level":"fatal","ts":"2020-04-08T17:00:53.735Z","logger":"controller","caller":"sharedmain/main.go:448","msg":"leaderelection lost","stacktrace":"knative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain.RunLeaderElected.func2\n\t/go/src/knative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain/main.go:448\nknative.dev/serving/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run.func1\n\t/go/src/knative.dev/serving/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:199\nknative.dev/serving/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run\n\t/go/src/knative.dev/serving/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:208\nknative.dev/serving/vendor/k8s.io/client-go/tools/leaderelection.RunOrDie\n\t/go/src/knative.dev/serving/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:220\nknative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain.RunLeaderElected\n\t/go/src/knative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain/main.go:440\nknative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain.MainWithConfig\n\t/go/src/knative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain/main.go:197\nknative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain.MainWithContext\n\t/go/src/knative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain/main.go:128\nknative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain.Main\n\t/go/src/knative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain/main.go:122\nmain.main\n\t/go/src/knative.dev/serving/cmd/controller/main.go:34\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"}
      {"level":"info","ts":"2020-04-08T17:00:53.735Z","logger":"controller.labeler-controller","caller":"controller/controller.go:350","msg":"Shutting down workers","knative.dev/controller":"labeler-controller"}
      {"level":"info","ts":"2020-04-08T17:00:53.735Z","logger":"controller.configuration-controller","caller":"controller/controller.go:350","msg":"Shutting down workers","knative.dev/controller":"configuration-controller"}
      

              Unassigned Unassigned
              maschmid@redhat.com Marek Schmidt
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: