-
Bug
-
Resolution: Done
-
Undefined
-
1.7.0
-
None
On OCP 4.3.9 on AWS, Serverless 1.7.0, installed from serverles-operator master (./hack/catalog.sh | oc apply -n openshift-marketplace -f - )
After 8h of cluster runtime, the restart count of knative-serving controller container is suspiciously high.
[maschmid@archie ocf-qe-testsuite]$ oc get pods -n knative-serving NAME READY STATUS RESTARTS AGE activator-785fdf59dd-mqqsz 1/1 Running 0 5h18m autoscaler-75f56584bf-dqpdr 1/1 Running 0 5h18m autoscaler-hpa-848b6bdfd4-8prdw 1/1 Running 16 5h17m autoscaler-hpa-848b6bdfd4-jk9tr 1/1 Running 20 5h17m controller-868c5f8d68-hjb7b 1/1 Running 20 5h18m controller-868c5f8d68-qbcpn 1/1 Running 19 5h18m webhook-67fb8b566d-lf7fq 1/1 Running 0 5h18m
Logs from one of the controller runs including the end
$ oc logs -n knative-serving controller-868c5f8d68-hjb7b -p 2020/04/08 16:58:02 Registering 4 clients 2020/04/08 16:58:02 Registering 3 informer factories 2020/04/08 16:58:02 Registering 13 informers 2020/04/08 16:58:02 Registering 7 controllers {"level":"info","ts":"2020-04-08T16:58:02.346Z","caller":"logging/config.go:108","msg":"Successfully created the logger."} {"level":"info","ts":"2020-04-08T16:58:02.346Z","caller":"logging/config.go:109","msg":"Logging level set to info"} {"level":"info","ts":"2020-04-08T16:58:02.346Z","caller":"logging/config.go:76","msg":"Fetch GitHub commit ID from kodata failed: \"KO_DATA_PATH\" does not exist or is empty"} {"level":"info","ts":"2020-04-08T16:58:02.346Z","logger":"controller","caller":"profiling/server.go:59","msg":"Profiling enabled: false"} {"level":"info","ts":"2020-04-08T16:58:02.352Z","logger":"controller","caller":"sharedmain/main.go:423","msg":"controller will run in leader-elected mode with id controller-868c5f8d68-hjb7b_c336708a-24f3-4c57-9632-9ef87230f238"} I0408 16:58:02.352573 1 leaderelection.go:241] attempting to acquire leader lease knative-serving/controller... I0408 16:58:19.787998 1 leaderelection.go:251] successfully acquired lease knative-serving/controller {"level":"info","ts":"2020-04-08T16:58:19.788Z","logger":"controller.event-broadcaster","caller":"record/event.go:274","msg":"Event(v1.ObjectReference{Kind:\"Lease\", Namespace:\"knative-serving\", Name:\"controller\", UID:\"0f3f1f3d-d16f-427d-bf17-8fe41e4a72a7\", APIVersion:\"coordination.k8s.io/v1\", ResourceVersion:\"371235\", FieldPath:\"\"}): type: 'Normal' reason: 'LeaderElection' controller-868c5f8d68-hjb7b_c336708a-24f3-4c57-9632-9ef87230f238 became leader"} {"level":"info","ts":"2020-04-08T16:58:19.788Z","logger":"controller.configuration-controller","caller":"configuration/controller.go:52","msg":"Setting up event handlers","knative.dev/controller":"configuration-controller"} {"level":"info","ts":"2020-04-08T16:58:19.788Z","logger":"controller.labeler-controller","caller":"labeler/controller.go:60","msg":"Setting up event handlers","knative.dev/controller":"labeler-controller"} {"level":"info","ts":"2020-04-08T16:58:19.804Z","logger":"controller.revision-controller","caller":"revision/controller.go:115","msg":"Setting up event handlers","knative.dev/controller":"revision-controller"} {"level":"info","ts":"2020-04-08T16:58:19.804Z","logger":"controller.route-controller","caller":"route/controller.go:97","msg":"Setting up event handlers","knative.dev/controller":"route-controller"} {"level":"info","ts":"2020-04-08T16:58:19.804Z","logger":"controller.serverlessservice-controller","caller":"serverlessservice/controller.go:64","msg":"Setting up event handlers","knative.dev/controller":"serverlessservice-controller"} {"level":"info","ts":"2020-04-08T16:58:19.804Z","logger":"controller.service-controller","caller":"service/controller.go:60","msg":"Setting up event handlers","knative.dev/controller":"service-controller"} {"level":"info","ts":"2020-04-08T16:58:19.804Z","logger":"controller.revision-gc-controller","caller":"gc/controller.go:53","msg":"Setting up event handlers","knative.dev/controller":"revision-gc-controller"} {"level":"info","ts":"2020-04-08T16:58:19.804Z","logger":"controller.revision-gc-controller","caller":"gc/controller.go:66","msg":"Setting up ConfigMap receivers with resync func","knative.dev/controller":"revision-gc-controller"} {"level":"info","ts":"2020-04-08T16:58:19.804Z","logger":"controller.revision-gc-controller","caller":"gc/controller.go:75","msg":"Setting up ConfigMap receivers","knative.dev/controller":"revision-gc-controller"} {"level":"info","ts":"2020-04-08T16:58:19.811Z","logger":"controller","caller":"sharedmain/main.go:172","msg":"Starting configuration manager..."} {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.revision-controller.config-store","caller":"configmap/store.go:157","msg":"revision config \"config-logging\" config was added or updated: &logging.Config{LoggingConfig:\"{\\n \\\"level\\\": \\\"info\\\",\\n \\\"development\\\": false,\\n \\\"outputPaths\\\": [\\\"stdout\\\"],\\n \\\"errorOutputPaths\\\": [\\\"stderr\\\"],\\n \\\"encoding\\\": \\\"json\\\",\\n \\\"encoderConfig\\\": {\\n \\\"timeKey\\\": \\\"ts\\\",\\n \\\"levelKey\\\": \\\"level\\\",\\n \\\"nameKey\\\": \\\"logger\\\",\\n \\\"callerKey\\\": \\\"caller\\\",\\n \\\"messageKey\\\": \\\"msg\\\",\\n \\\"stacktraceKey\\\": \\\"stacktrace\\\",\\n \\\"lineEnding\\\": \\\"\\\",\\n \\\"levelEncoder\\\": \\\"\\\",\\n \\\"timeEncoder\\\": \\\"iso8601\\\",\\n \\\"durationEncoder\\\": \\\"\\\",\\n \\\"callerEncoder\\\": \\\"\\\"\\n }\\n}\", LoggingLevel:map[string]zapcore.Level{}}","knative.dev/controller":"revision-controller"} {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.revision-controller.config-store","caller":"configmap/store.go:157","msg":"revision config \"config-observability\" config was added or updated: &metrics.ObservabilityConfig{EnableVarLogCollection:false, LoggingURLTemplate:\"http://localhost:8001/api/v1/namespaces/knative-monitoring/services/kibana-logging/proxy/app/kibana#/discover?_a=(query:(match:(kubernetes.labels.knative-dev%2FrevisionUID:(query:'${REVISION_UID}',type:phrase))))\", RequestLogTemplate:\"\", EnableProbeRequestLog:false, RequestMetricsBackend:\"prometheus\", EnableProfiling:false}","knative.dev/controller":"revision-controller"} {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller","caller":"metrics/exporter.go:126","msg":"Flushing the existing exporter before setting up the new exporter."} {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller","caller":"metrics/prometheus_exporter.go:37","msg":"Created Opencensus Prometheus exporter with config: &{knative.dev/internal/serving controller prometheus 5000000000 <nil> false 9090 false { false}}. Start the server for Prometheus exporter."} {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller","caller":"metrics/exporter.go:135","msg":"Successfully updated the metrics exporter; old config: <nil>; new config &{knative.dev/internal/serving controller prometheus 5000000000 <nil> false 9090 false { false}}"} {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.route-controller.config-store","caller":"configmap/store.go:157","msg":"route config \"config-gc\" config was added or updated: &gc.Config{StaleRevisionCreateDelay:172800000000000, StaleRevisionTimeout:54000000000000, StaleRevisionMinimumGenerations:20, StaleRevisionLastpinnedDebounce:18000000000000}","knative.dev/controller":"route-controller"} {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.revision-gc-controller.config-store","caller":"configmap/store.go:157","msg":"configuration config \"config-gc\" config was added or updated: &gc.Config{StaleRevisionCreateDelay:172800000000000, StaleRevisionTimeout:54000000000000, StaleRevisionMinimumGenerations:20, StaleRevisionLastpinnedDebounce:18000000000000}","knative.dev/controller":"revision-gc-controller"} {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.revision-controller.config-store","caller":"configmap/store.go:157","msg":"revision config \"config-deployment\" config was added or updated: &deployment.Config{QueueSidecarImage:\"registry.svc.ci.openshift.org/openshift/knative-v0.13.2:knative-serving-queue\", RegistriesSkippingTagResolving:sets.String{\"dev.local\":sets.Empty{}, \"ko.local\":sets.Empty{}}}","knative.dev/controller":"revision-controller"} {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.revision-controller.config-store","caller":"configmap/store.go:157","msg":"revision config \"config-tracing\" config was added or updated: &config.Config{Backend:\"none\", ZipkinEndpoint:\"\", StackdriverProjectID:\"\", Debug:false, SampleRate:0.1}","knative.dev/controller":"revision-controller"} {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.revision-controller.config-store","caller":"configmap/store.go:157","msg":"revision config \"config-autoscaler\" config was added or updated: &config.Config{EnableScaleToZero:true, EnableGracefulScaledown:false, ContainerConcurrencyTargetFraction:0.7, ContainerConcurrencyTargetDefault:100, TargetUtilization:0.7, RPSTargetDefault:200, TargetBurstCapacity:200, MaxScaleUpRate:1000, MaxScaleDownRate:2, StableWindow:60000000000, PanicWindowPercentage:10, PanicThresholdPercentage:200, TickInterval:2000000000, ScaleToZeroGracePeriod:30000000000, PodAutoscalerClass:\"kpa.autoscaling.knative.dev\"}","knative.dev/controller":"revision-controller"} {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.route-controller.config-store","caller":"configmap/store.go:157","msg":"route config \"config-domain\" config was added or updated: &config.Domain{Domains:map[string]*config.LabelSelector{\"apps.maschmid1.openshift-aws.rhocf-dev.net\":(*config.LabelSelector)(0xc000010ae0)}}","knative.dev/controller":"route-controller"} {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.revision-controller.config-store","caller":"configmap/store.go:157","msg":"revision config \"config-defaults\" config was added or updated: &config.Defaults{RevisionTimeoutSeconds:300, MaxRevisionTimeoutSeconds:600, UserContainerNameTemplate:\"user-container\", ContainerConcurrency:0, RevisionCPURequest:(*resource.Quantity)(nil), RevisionCPULimit:(*resource.Quantity)(nil), RevisionMemoryRequest:(*resource.Quantity)(nil), RevisionMemoryLimit:(*resource.Quantity)(nil)}","knative.dev/controller":"revision-controller"} {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.revision-controller.config-store","caller":"configmap/store.go:157","msg":"revision config \"config-network\" config was added or updated: &network.Config{DefaultIngressClass:\"kourier.ingress.networking.knative.dev\", DomainTemplate:\"{{.Name}}.{{.Namespace}}.{{.Domain}}\", TagTemplate:\"{{.Tag}}-{{.Name}}\", AutoTLS:false, HTTPProtocol:\"enabled\", DefaultCertificateClass:\"cert-manager.certificate.networking.knative.dev\"}","knative.dev/controller":"revision-controller"} {"level":"info","ts":"2020-04-08T16:58:19.837Z","logger":"controller.route-controller.config-store","caller":"configmap/store.go:157","msg":"route config \"config-network\" config was added or updated: &network.Config{DefaultIngressClass:\"kourier.ingress.networking.knative.dev\", DomainTemplate:\"{{.Name}}.{{.Namespace}}.{{.Domain}}\", TagTemplate:\"{{.Tag}}-{{.Name}}\", AutoTLS:false, HTTPProtocol:\"enabled\", DefaultCertificateClass:\"cert-manager.certificate.networking.knative.dev\"}","knative.dev/controller":"route-controller"} {"level":"info","ts":"2020-04-08T16:58:19.911Z","logger":"controller","caller":"sharedmain/main.go:176","msg":"Starting informers..."} {"level":"info","ts":"2020-04-08T16:58:19.938Z","logger":"controller.serverlessservice-controller","caller":"serverlessservice/controller.go:85","msg":"Doing a global resync due to activator endpoint changes","knative.dev/controller":"serverlessservice-controller"} {"level":"info","ts":"2020-04-08T16:58:20.324Z","logger":"controller","caller":"sharedmain/main.go:180","msg":"Starting controllers..."} {"level":"info","ts":"2020-04-08T16:58:20.324Z","logger":"controller.revision-gc-controller","caller":"controller/controller.go:338","msg":"Starting controller and workers","knative.dev/controller":"revision-gc-controller"} {"level":"info","ts":"2020-04-08T16:58:20.324Z","logger":"controller.revision-gc-controller","caller":"controller/controller.go:348","msg":"Started workers","knative.dev/controller":"revision-gc-controller"} {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.revision-controller","caller":"controller/controller.go:338","msg":"Starting controller and workers","knative.dev/controller":"revision-controller"} {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.revision-controller","caller":"controller/controller.go:348","msg":"Started workers","knative.dev/controller":"revision-controller"} {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.revision-gc-controller","caller":"controller/controller.go:403","msg":"Reconcile succeeded. Time taken: 132.258µs.","knative.dev/controller":"revision-gc-controller","knative.dev/traceid":"30d08c91-02e7-4503-84e7-d6291683f379","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.route-controller","caller":"controller/controller.go:338","msg":"Starting controller and workers","knative.dev/controller":"route-controller"} {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.route-controller","caller":"controller/controller.go:348","msg":"Started workers","knative.dev/controller":"route-controller"} {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.route-controller","caller":"route/route.go:112","msg":"Reconciling route: &v1.Route{TypeMeta:v1.TypeMeta{Kind:\"Route\", APIVersion:\"serving.knative.dev/v1\"}, ObjectMeta:v1.ObjectMeta{Name:\"prime-generator\", GenerateName:\"\", Namespace:\"knativetutorial\", SelfLink:\"/apis/serving.knative.dev/v1/namespaces/knativetutorial/routes/prime-generator\", UID:\"0ebdc4e1-edfc-4a95-9b5d-65bad4708f33\", ResourceVersion:\"357237\", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63721960698, loc:(*time.Location)(0x2c03b40)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"serving.knative.dev/service\":\"prime-generator\"}, Annotations:map[string]string{\"serving.knative.dev/creator\":\"system:admin\", \"serving.knative.dev/lastModifier\":\"system:admin\"}, OwnerReferences:[]v1.OwnerReference{v1.OwnerReference{APIVersion:\"serving.knative.dev/v1\", Kind:\"Service\", Name:\"prime-generator\", UID:\"cc57787e-594e-48e9-a7dd-08584c1d2b6d\", Controller:(*bool)(0xc000761680), BlockOwnerDeletion:(*bool)(0xc000761681)}}, Finalizers:[]string{\"routes.serving.knative.dev\"}, ClusterName:\"\", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.RouteSpec{Traffic:[]v1.TrafficTarget{v1.TrafficTarget{Tag:\"\", RevisionName:\"\", ConfigurationName:\"prime-generator\", LatestRevision:(*bool)(0xc000761682), Percent:(*int64)(0xc000761688), URL:(*apis.URL)(nil)}}}, Status:v1.RouteStatus{Status:v1.Status{ObservedGeneration:1, Conditions:v1.Conditions{apis.Condition{Type:\"AllTrafficAssigned\", Status:\"True\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0x0, ext:63721960708, loc:(*time.Location)(0x2c03b40)}}}, Reason:\"\", Message:\"\"}, apis.Condition{Type:\"IngressReady\", Status:\"True\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0x0, ext:63721960708, loc:(*time.Location)(0x2c03b40)}}}, Reason:\"\", Message:\"\"}, apis.Condition{Type:\"Ready\", Status:\"True\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0x0, ext:63721960708, loc:(*time.Location)(0x2c03b40)}}}, Reason:\"\", Message:\"\"}}}, RouteStatusFields:v1.RouteStatusFields{URL:(*apis.URL)(0xc000f86300), Address:(*v1.Addressable)(0xc000fb80f0), Traffic:[]v1.TrafficTarget{v1.TrafficTarget{Tag:\"\", RevisionName:\"prime-generator-9rvz7\", ConfigurationName:\"\", LatestRevision:(*bool)(0xc000761690), Percent:(*int64)(0xc000761698), URL:(*apis.URL)(nil)}}}}}","knative.dev/controller":"route-controller","knative.dev/traceid":"693daa8d-d948-4f58-9a07-9ecd1c1a6d20","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.route-controller","caller":"route/route.go:308","msg":"Revision knativetutorial/prime-generator-9rvz7 is inactive","knative.dev/controller":"route-controller","knative.dev/traceid":"693daa8d-d948-4f58-9a07-9ecd1c1a6d20","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.route-controller","caller":"route/route.go:330","msg":"All referred targets are routable, marking AllTrafficAssigned with traffic information.","knative.dev/controller":"route-controller","knative.dev/traceid":"693daa8d-d948-4f58-9a07-9ecd1c1a6d20","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.route-controller","caller":"route/route.go:125","msg":"Updating targeted revisions.","knative.dev/controller":"route-controller","knative.dev/traceid":"693daa8d-d948-4f58-9a07-9ecd1c1a6d20","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.route-controller","caller":"route/route.go:139","msg":"Creating placeholder k8s services","knative.dev/controller":"route-controller","knative.dev/traceid":"693daa8d-d948-4f58-9a07-9ecd1c1a6d20","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.serverlessservice-controller","caller":"controller/controller.go:338","msg":"Starting controller and workers","knative.dev/controller":"serverlessservice-controller"} {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.serverlessservice-controller","caller":"controller/controller.go:348","msg":"Started workers","knative.dev/controller":"serverlessservice-controller"} {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.service-controller","caller":"controller/controller.go:338","msg":"Starting controller and workers","knative.dev/controller":"service-controller"} {"level":"info","ts":"2020-04-08T16:58:20.325Z","logger":"controller.service-controller","caller":"controller/controller.go:348","msg":"Started workers","knative.dev/controller":"service-controller"} {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.configuration-controller","caller":"controller/controller.go:338","msg":"Starting controller and workers","knative.dev/controller":"configuration-controller"} {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.configuration-controller","caller":"controller/controller.go:348","msg":"Started workers","knative.dev/controller":"configuration-controller"} {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.configuration-controller","caller":"configuration/configuration.go:104","msg":"Revision \"prime-generator-9rvz7\" of configuration is ready","knative.dev/controller":"configuration-controller","knative.dev/traceid":"bc93be80-d90f-4663-b1aa-74495cd58629","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.configuration-controller","caller":"controller/controller.go:403","msg":"Reconcile succeeded. Time taken: 108.706µs.","knative.dev/controller":"configuration-controller","knative.dev/traceid":"bc93be80-d90f-4663-b1aa-74495cd58629","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.labeler-controller","caller":"controller/controller.go:338","msg":"Starting controller and workers","knative.dev/controller":"labeler-controller"} {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.labeler-controller","caller":"controller/controller.go:348","msg":"Started workers","knative.dev/controller":"labeler-controller"} {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.service-controller","caller":"service/service.go:214","msg":"Reconciling configuration diff (-desired, +observed):\n","knative.dev/controller":"service-controller","knative.dev/traceid":"36b252d9-447a-4fe3-9a10-cce4f644ae4d","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.labeler-controller","caller":"controller/controller.go:403","msg":"Reconcile succeeded. Time taken: 116.944µs.","knative.dev/controller":"labeler-controller","knative.dev/traceid":"c5b85abb-1c9b-4572-9cb1-c8a5e4e19043","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.service-controller","caller":"service/service.go:263","msg":"Reconciling route diff (-desired, +observed):\n","knative.dev/controller":"service-controller","knative.dev/traceid":"36b252d9-447a-4fe3-9a10-cce4f644ae4d","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.326Z","logger":"controller.service-controller","caller":"controller/controller.go:403","msg":"Reconcile succeeded. Time taken: 652.322µs.","knative.dev/controller":"service-controller","knative.dev/traceid":"36b252d9-447a-4fe3-9a10-cce4f644ae4d","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.343Z","logger":"controller.revision-controller","caller":"revision/reconcile_resources.go:135","msg":"Reconciling PA: prime-generator-9rvz7","knative.dev/controller":"revision-controller","knative.dev/traceid":"51bd7828-617b-4eae-bf6a-80379c79c74d","knative.dev/key":"knativetutorial/prime-generator-9rvz7"} {"level":"info","ts":"2020-04-08T16:58:20.343Z","logger":"controller.revision-controller","caller":"controller/controller.go:403","msg":"Reconcile succeeded. Time taken: 18.852296ms.","knative.dev/controller":"revision-controller","knative.dev/traceid":"51bd7828-617b-4eae-bf6a-80379c79c74d","knative.dev/key":"knativetutorial/prime-generator-9rvz7"} {"level":"info","ts":"2020-04-08T16:58:20.347Z","logger":"controller.route-controller","caller":"route/route.go:163","msg":"Updating placeholder k8s services with ingress information","knative.dev/controller":"route-controller","knative.dev/traceid":"693daa8d-d948-4f58-9a07-9ecd1c1a6d20","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.347Z","logger":"controller.route-controller","caller":"route/route.go:169","msg":"Route successfully synced","knative.dev/controller":"route-controller","knative.dev/traceid":"693daa8d-d948-4f58-9a07-9ecd1c1a6d20","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.347Z","logger":"controller.route-controller","caller":"controller/controller.go:403","msg":"Reconcile succeeded. Time taken: 21.937045ms.","knative.dev/controller":"route-controller","knative.dev/traceid":"693daa8d-d948-4f58-9a07-9ecd1c1a6d20","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.347Z","logger":"controller.route-controller","caller":"route/route.go:112","msg":"Reconciling route: &v1.Route{TypeMeta:v1.TypeMeta{Kind:\"Route\", APIVersion:\"serving.knative.dev/v1\"}, ObjectMeta:v1.ObjectMeta{Name:\"prime-generator\", GenerateName:\"\", Namespace:\"knativetutorial\", SelfLink:\"/apis/serving.knative.dev/v1/namespaces/knativetutorial/routes/prime-generator\", UID:\"0ebdc4e1-edfc-4a95-9b5d-65bad4708f33\", ResourceVersion:\"357237\", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63721960698, loc:(*time.Location)(0x2c03b40)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"serving.knative.dev/service\":\"prime-generator\"}, Annotations:map[string]string{\"serving.knative.dev/creator\":\"system:admin\", \"serving.knative.dev/lastModifier\":\"system:admin\"}, OwnerReferences:[]v1.OwnerReference{v1.OwnerReference{APIVersion:\"serving.knative.dev/v1\", Kind:\"Service\", Name:\"prime-generator\", UID:\"cc57787e-594e-48e9-a7dd-08584c1d2b6d\", Controller:(*bool)(0xc0009ae4d0), BlockOwnerDeletion:(*bool)(0xc0009ae4d1)}}, Finalizers:[]string{\"routes.serving.knative.dev\"}, ClusterName:\"\", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.RouteSpec{Traffic:[]v1.TrafficTarget{v1.TrafficTarget{Tag:\"\", RevisionName:\"\", ConfigurationName:\"prime-generator\", LatestRevision:(*bool)(0xc0009ae4d2), Percent:(*int64)(0xc0009ae4d8), URL:(*apis.URL)(nil)}}}, Status:v1.RouteStatus{Status:v1.Status{ObservedGeneration:1, Conditions:v1.Conditions{apis.Condition{Type:\"AllTrafficAssigned\", Status:\"True\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0x0, ext:63721960708, loc:(*time.Location)(0x2c03b40)}}}, Reason:\"\", Message:\"\"}, apis.Condition{Type:\"IngressReady\", Status:\"True\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0x0, ext:63721960708, loc:(*time.Location)(0x2c03b40)}}}, Reason:\"\", Message:\"\"}, apis.Condition{Type:\"Ready\", Status:\"True\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0x0, ext:63721960708, loc:(*time.Location)(0x2c03b40)}}}, Reason:\"\", Message:\"\"}}}, RouteStatusFields:v1.RouteStatusFields{URL:(*apis.URL)(0xc00085f680), Address:(*v1.Addressable)(0xc0000110e8), Traffic:[]v1.TrafficTarget{v1.TrafficTarget{Tag:\"\", RevisionName:\"prime-generator-9rvz7\", ConfigurationName:\"\", LatestRevision:(*bool)(0xc0009ae570), Percent:(*int64)(0xc0009ae578), URL:(*apis.URL)(nil)}}}}}","knative.dev/controller":"route-controller","knative.dev/traceid":"d1412786-f476-4f52-b493-c5aeb1ed4210","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.347Z","logger":"controller.route-controller","caller":"route/route.go:308","msg":"Revision knativetutorial/prime-generator-9rvz7 is inactive","knative.dev/controller":"route-controller","knative.dev/traceid":"d1412786-f476-4f52-b493-c5aeb1ed4210","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.347Z","logger":"controller.route-controller","caller":"route/route.go:330","msg":"All referred targets are routable, marking AllTrafficAssigned with traffic information.","knative.dev/controller":"route-controller","knative.dev/traceid":"d1412786-f476-4f52-b493-c5aeb1ed4210","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.347Z","logger":"controller.route-controller","caller":"route/route.go:125","msg":"Updating targeted revisions.","knative.dev/controller":"route-controller","knative.dev/traceid":"d1412786-f476-4f52-b493-c5aeb1ed4210","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.347Z","logger":"controller.route-controller","caller":"route/route.go:139","msg":"Creating placeholder k8s services","knative.dev/controller":"route-controller","knative.dev/traceid":"d1412786-f476-4f52-b493-c5aeb1ed4210","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.379Z","logger":"controller.route-controller","caller":"route/route.go:163","msg":"Updating placeholder k8s services with ingress information","knative.dev/controller":"route-controller","knative.dev/traceid":"d1412786-f476-4f52-b493-c5aeb1ed4210","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.380Z","logger":"controller.route-controller","caller":"route/route.go:169","msg":"Route successfully synced","knative.dev/controller":"route-controller","knative.dev/traceid":"d1412786-f476-4f52-b493-c5aeb1ed4210","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.380Z","logger":"controller.route-controller","caller":"controller/controller.go:403","msg":"Reconcile succeeded. Time taken: 32.869249ms.","knative.dev/controller":"route-controller","knative.dev/traceid":"d1412786-f476-4f52-b493-c5aeb1ed4210","knative.dev/key":"knativetutorial/prime-generator"} {"level":"info","ts":"2020-04-08T16:58:20.508Z","logger":"controller.serverlessservice-controller","caller":"serverlessservice/serverlessservice.go:209","msg":"Endpoints prime-generator-9rvz7 has no ready endpoints","knative.dev/controller":"serverlessservice-controller","knative.dev/traceid":"210aa2ec-6c51-421e-b467-4d0971c3d675","knative.dev/key":"knativetutorial/prime-generator-9rvz7"} {"level":"info","ts":"2020-04-08T16:58:20.508Z","logger":"controller.serverlessservice-controller","caller":"controller/controller.go:403","msg":"Reconcile succeeded. Time taken: 183.060201ms.","knative.dev/controller":"serverlessservice-controller","knative.dev/traceid":"210aa2ec-6c51-421e-b467-4d0971c3d675","knative.dev/key":"knativetutorial/prime-generator-9rvz7"} {"level":"info","ts":"2020-04-08T16:58:21.752Z","logger":"controller.serverlessservice-controller","caller":"serverlessservice/serverlessservice.go:209","msg":"Endpoints prime-generator-9rvz7 has no ready endpoints","knative.dev/controller":"serverlessservice-controller","knative.dev/traceid":"f2b8e646-dd96-4c97-8009-a29bcf817cf6","knative.dev/key":"knativetutorial/prime-generator-9rvz7"} {"level":"info","ts":"2020-04-08T16:58:21.752Z","logger":"controller.serverlessservice-controller","caller":"controller/controller.go:403","msg":"Reconcile succeeded. Time taken: 335.757µs.","knative.dev/controller":"serverlessservice-controller","knative.dev/traceid":"f2b8e646-dd96-4c97-8009-a29bcf817cf6","knative.dev/key":"knativetutorial/prime-generator-9rvz7"} I0408 17:00:53.735214 1 leaderelection.go:287] failed to renew lease knative-serving/controller: failed to tryAcquireOrRenew context deadline exceeded {"level":"info","ts":"2020-04-08T17:00:53.735Z","logger":"controller.event-broadcaster","caller":"record/event.go:274","msg":"Event(v1.ObjectReference{Kind:\"Lease\", Namespace:\"knative-serving\", Name:\"controller\", UID:\"0f3f1f3d-d16f-427d-bf17-8fe41e4a72a7\", APIVersion:\"coordination.k8s.io/v1\", ResourceVersion:\"373414\", FieldPath:\"\"}): type: 'Normal' reason: 'LeaderElection' controller-868c5f8d68-hjb7b_c336708a-24f3-4c57-9632-9ef87230f238 stopped leading"} {"level":"fatal","ts":"2020-04-08T17:00:53.735Z","logger":"controller","caller":"sharedmain/main.go:448","msg":"leaderelection lost","stacktrace":"knative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain.RunLeaderElected.func2\n\t/go/src/knative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain/main.go:448\nknative.dev/serving/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run.func1\n\t/go/src/knative.dev/serving/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:199\nknative.dev/serving/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run\n\t/go/src/knative.dev/serving/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:208\nknative.dev/serving/vendor/k8s.io/client-go/tools/leaderelection.RunOrDie\n\t/go/src/knative.dev/serving/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:220\nknative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain.RunLeaderElected\n\t/go/src/knative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain/main.go:440\nknative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain.MainWithConfig\n\t/go/src/knative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain/main.go:197\nknative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain.MainWithContext\n\t/go/src/knative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain/main.go:128\nknative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain.Main\n\t/go/src/knative.dev/serving/vendor/knative.dev/pkg/injection/sharedmain/main.go:122\nmain.main\n\t/go/src/knative.dev/serving/cmd/controller/main.go:34\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"} {"level":"info","ts":"2020-04-08T17:00:53.735Z","logger":"controller.labeler-controller","caller":"controller/controller.go:350","msg":"Shutting down workers","knative.dev/controller":"labeler-controller"} {"level":"info","ts":"2020-04-08T17:00:53.735Z","logger":"controller.configuration-controller","caller":"controller/controller.go:350","msg":"Shutting down workers","knative.dev/controller":"configuration-controller"}
- relates to
-
SRVKS-513 knative-serving webhook CPU throttled, asked for AdmissionReview for knative-serving/config-service-ca 10 times a second
- Closed