W0404 13:26:45.808595 1 cmd.go:245] Using insecure, self-signed certificates I0404 13:26:46.368393 1 leaderelection.go:122] The leader election gives 4 retries and allows for 30s of clock skew. The kube-apiserver downtime tolerance is 78s. Worst non-graceful lease acquisition is 2m43s. Worst graceful lease acquisition is {26s}. I0404 13:26:46.369103 1 observer_polling.go:159] Starting file observer I0404 13:26:46.405021 1 builder.go:299] submariner-agent version v0.0.0-unknown-- I0404 13:26:47.043683 1 secure_serving.go:57] Forcing use of http/1.1 only W0404 13:26:47.043714 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0404 13:26:47.043720 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. I0404 13:26:47.049055 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0404 13:26:47.049100 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0404 13:26:47.049185 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController I0404 13:26:47.049240 1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController I0404 13:26:47.049426 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0404 13:26:47.049712 1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/serving-cert-3395436570/tls.crt::/tmp/serving-cert-3395436570/tls.key" I0404 13:26:47.049719 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0404 13:26:47.050623 1 secure_serving.go:213] Serving securely on [::]:8443 I0404 13:26:47.050675 1 tlsconfig.go:240] "Starting DynamicServingCertificateController" I0404 13:26:47.051693 1 leaderelection.go:250] attempting to acquire leader lease submariner-operator/submariner-agent-lock... I0404 13:26:47.077179 1 leaderelection.go:260] successfully acquired lease submariner-operator/submariner-agent-lock I0404 13:26:47.077477 1 event.go:364] Event(v1.ObjectReference{Kind:"Lease", Namespace:"submariner-operator", Name:"submariner-agent-lock", UID:"c0ce82d7-8320-45c4-adc4-6df5389f1926", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"931123", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' submariner-addon-7686fd7759-ws9n2_576b6c44-b0f2-492c-b618-11a385051297 became leader I0404 13:26:47.096681 1 base_controller.go:67] Waiting for caches to sync for GatewaysStatusController I0404 13:26:47.097525 1 base_controller.go:67] Waiting for caches to sync for SubmarinerConfigController I0404 13:26:47.097577 1 base_controller.go:67] Waiting for caches to sync for ConnectionsStatusController I0404 13:26:47.098919 1 base_controller.go:67] Waiting for caches to sync for DeploymentStatusController W0404 13:26:47.100854 1 reflector.go:539] k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list submariner.io/v1alpha1, Resource=submariners: the server could not find the requested resource E0404 13:26:47.100942 1 reflector.go:147] k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch submariner.io/v1alpha1, Resource=submariners: failed to list submariner.io/v1alpha1, Resource=submariners: the server could not find the requested resource I0404 13:26:47.150002 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0404 13:26:47.150155 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0404 13:26:47.150159 1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController I0404 13:26:47.197399 1 base_controller.go:73] Caches are synced for GatewaysStatusController I0404 13:26:47.197452 1 base_controller.go:110] Starting #1 worker of GatewaysStatusController controller ... I0404 13:26:47.197650 1 base_controller.go:73] Caches are synced for SubmarinerConfigController I0404 13:26:47.197691 1 base_controller.go:110] Starting #1 worker of SubmarinerConfigController controller ... 2024-04-04T13:26:47.247Z INF ..eways_controller.go:100 ..ysStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerGatewayNodesLabeled", "status": "False", "lastTransitionTime": null, "reason": "SubmarinerGatewayNodesUnlabeled", "message": "There are no nodes with label \"submariner.io/gateway\"" } I0404 13:26:47.247616 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"Unknown", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 45, 0, time.Local), Reason:"ManagedClusterAddOnLeaseNotFound", Message:"The status of submariner add-on is unknown."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 47, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}} 2024-04-04T13:26:47.302Z INF ..onfig_controller.go:654 ..erConfigController SubmarinerConfig network type was set to "OVNKubernetes" for managed cluster "rdr-c2" I0404 13:26:47.302493 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SubmarinerConfigNetworkTypeSet' SubmarinerConfig network type was set to "OVNKubernetes" for managed cluster "rdr-c2" I0404 13:26:47.423866 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic W0404 13:26:48.284136 1 reflector.go:539] k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list submariner.io/v1alpha1, Resource=submariners: the server could not find the requested resource E0404 13:26:48.284179 1 reflector.go:147] k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch submariner.io/v1alpha1, Resource=submariners: failed to list submariner.io/v1alpha1, Resource=submariners: the server could not find the requested resource W0404 13:26:50.609444 1 reflector.go:539] k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list submariner.io/v1alpha1, Resource=submariners: the server could not find the requested resource E0404 13:26:50.609508 1 reflector.go:147] k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch submariner.io/v1alpha1, Resource=submariners: failed to list submariner.io/v1alpha1, Resource=submariners: the server could not find the requested resource I0404 13:26:52.596801 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:26:53.898526 1 base_controller.go:73] Caches are synced for ConnectionsStatusController I0404 13:26:53.898577 1 base_controller.go:110] Starting #1 worker of ConnectionsStatusController controller ... I0404 13:26:53.899742 1 base_controller.go:73] Caches are synced for DeploymentStatusController I0404 13:26:53.899778 1 base_controller.go:110] Starting #1 worker of DeploymentStatusController controller ... 2024-04-04T13:26:53.945Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "NoOperatorDeployment,NoLighthouseAgentDeployment,NoLighthouseCoreDNSDeployment,NoGatewayDaemonSet,NoRouteAgentDaemonSet,NoMetricsProxyDaemonSet", "message": "The submariner operator deployment does not exist\nThe submariner lighthouse agent deployment does not exist\nThe submariner lighthouse coredns deployment does not exist\nThe gateway daemon set does not exist\nThe route agents are not found\nThe metrics proxy daemon set does not exist" } I0404 13:26:53.946275 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"Unknown", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 45, 0, time.Local), Reason:"ManagedClusterAddOnLeaseNotFound", Message:"The status of submariner add-on is unknown."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 47, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"NoOperatorDeployment,NoLighthouseAgentDeployment,NoLighthouseCoreDNSDeployment,NoGatewayDaemonSet,NoRouteAgentDaemonSet,NoMetricsProxyDaemonSet", Message:"The submariner operator deployment does not exist\nThe submariner lighthouse agent deployment does not exist\nThe submariner lighthouse coredns deployment does not exist\nThe gateway daemon set does not exist\nThe route agents are not found\nThe metrics proxy daemon set does not exist"}} 2024-04-04T13:26:55.515Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "NoOperatorAvailable,NoLighthouseAgentDeployment,NoLighthouseCoreDNSDeployment,NoGatewayDaemonSet,NoRouteAgentDaemonSet,NoMetricsProxyDaemonSet", "message": "There are no submariner operator replica available\nThe submariner lighthouse agent deployment does not exist\nThe submariner lighthouse coredns deployment does not exist\nThe gateway daemon set does not exist\nThe route agents are not found\nThe metrics proxy daemon set does not exist" } I0404 13:26:55.515618 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"Unknown", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 45, 0, time.Local), Reason:"ManagedClusterAddOnLeaseNotFound", Message:"The status of submariner add-on is unknown."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 47, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"NoOperatorAvailable,NoLighthouseAgentDeployment,NoLighthouseCoreDNSDeployment,NoGatewayDaemonSet,NoRouteAgentDaemonSet,NoMetricsProxyDaemonSet", Message:"There are no submariner operator replica available\nThe submariner lighthouse agent deployment does not exist\nThe submariner lighthouse coredns deployment does not exist\nThe gateway daemon set does not exist\nThe route agents are not found\nThe metrics proxy daemon set does not exist"}} I0404 13:26:55.734039 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:26:55.753004 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:26:55.791329 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:26:55.791657 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SubmarinerClusterEnvPrepared' submariner cluster environment was prepared for managed cluster rdr-c2 2024-04-04T13:26:55.843Z INF ..onfig_controller.go:654 ..erConfigController SubmarinerConfig network type was set to "OVNKubernetes" for managed cluster "rdr-c2" I0404 13:26:55.844697 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SubmarinerConfigNetworkTypeSet' SubmarinerConfig network type was set to "OVNKubernetes" for managed cluster "rdr-c2" I0404 13:26:55.883873 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:26:58.835442 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes 2024-04-04T13:27:00.687Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "NoLighthouseAgentDeployment,NoLighthouseCoreDNSDeployment,NoGatewayDaemonSet,NoRouteAgentDaemonSet,NoMetricsProxyDaemonSet", "message": "The submariner lighthouse agent deployment does not exist\nThe submariner lighthouse coredns deployment does not exist\nThe gateway daemon set does not exist\nThe route agents are not found\nThe metrics proxy daemon set does not exist" } I0404 13:27:00.687803 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"Unknown", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 45, 0, time.Local), Reason:"ManagedClusterAddOnLeaseNotFound", Message:"The status of submariner add-on is unknown."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 47, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"NoLighthouseAgentDeployment,NoLighthouseCoreDNSDeployment,NoGatewayDaemonSet,NoRouteAgentDaemonSet,NoMetricsProxyDaemonSet", Message:"The submariner lighthouse agent deployment does not exist\nThe submariner lighthouse coredns deployment does not exist\nThe gateway daemon set does not exist\nThe route agents are not found\nThe metrics proxy daemon set does not exist"}} I0404 13:27:01.195674 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:27:01.203262 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:27:01.211825 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" 2024-04-04T13:27:01.260Z INF ..onfig_controller.go:654 ..erConfigController SubmarinerConfig network type was set to "OVNKubernetes" for managed cluster "rdr-c2" I0404 13:27:01.260259 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SubmarinerConfigNetworkTypeSet' SubmarinerConfig network type was set to "OVNKubernetes" for managed cluster "rdr-c2" I0404 13:27:01.316394 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:27:03.899472 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes 2024-04-04T13:27:04.115Z INF ..ctions_controller.go:90 ..nsStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerConnectionDegraded", "status": "True", "lastTransitionTime": null, "reason": "ConnectionsNotEstablished", "message": "There are no connections on gateways" } I0404 13:27:04.115453 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"Unknown", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 45, 0, time.Local), Reason:"ManagedClusterAddOnLeaseNotFound", Message:"The status of submariner add-on is unknown."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 47, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"NoLighthouseAgentDeployment,NoLighthouseCoreDNSDeployment,NoGatewayDaemonSet,NoRouteAgentDaemonSet,NoMetricsProxyDaemonSet", Message:"The submariner lighthouse agent deployment does not exist\nThe submariner lighthouse coredns deployment does not exist\nThe gateway daemon set does not exist\nThe route agents are not found\nThe metrics proxy daemon set does not exist"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:27:04.219Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "NoLighthouseAgentDeployment,NoLighthouseCoreDNSDeployment,NoGatewayDaemonSet,NoRouteAgentDaemonSet,NoMetricsProxyDaemonSet,NoGlobalnetDaemonSet", "message": "The submariner lighthouse agent deployment does not exist\nThe submariner lighthouse coredns deployment does not exist\nThe gateway daemon set does not exist\nThe route agents are not found\nThe metrics proxy daemon set does not exist\nThe globalnet daemon set does not exist" } I0404 13:27:04.220210 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"Unknown", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 45, 0, time.Local), Reason:"ManagedClusterAddOnLeaseNotFound", Message:"The status of submariner add-on is unknown."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 47, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"NoLighthouseAgentDeployment,NoLighthouseCoreDNSDeployment,NoGatewayDaemonSet,NoRouteAgentDaemonSet,NoMetricsProxyDaemonSet,NoGlobalnetDaemonSet", Message:"The submariner lighthouse agent deployment does not exist\nThe submariner lighthouse coredns deployment does not exist\nThe gateway daemon set does not exist\nThe route agents are not found\nThe metrics proxy daemon set does not exist\nThe globalnet daemon set does not exist"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:27:04.264Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "NoLighthouseAgentDeployment,NoLighthouseCoreDNSDeployment,NoScheduledGateways,NoRouteAgentDaemonSet,NoMetricsProxyDaemonSet,NoGlobalnetDaemonSet", "message": "The submariner lighthouse agent deployment does not exist\nThe submariner lighthouse coredns deployment does not exist\nThere are no nodes to run the gateways\nThe route agents are not found\nThe metrics proxy daemon set does not exist\nThe globalnet daemon set does not exist" } I0404 13:27:04.264890 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"Unknown", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 45, 0, time.Local), Reason:"ManagedClusterAddOnLeaseNotFound", Message:"The status of submariner add-on is unknown."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 47, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"NoLighthouseAgentDeployment,NoLighthouseCoreDNSDeployment,NoScheduledGateways,NoRouteAgentDaemonSet,NoMetricsProxyDaemonSet,NoGlobalnetDaemonSet", Message:"The submariner lighthouse agent deployment does not exist\nThe submariner lighthouse coredns deployment does not exist\nThere are no nodes to run the gateways\nThe route agents are not found\nThe metrics proxy daemon set does not exist\nThe globalnet daemon set does not exist"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:27:04.696Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "NoLighthouseAgentDeployment,NoLighthouseCoreDNSDeployment,NoScheduledGateways,NoMetricsProxyDaemonSet,NoGlobalnetDaemonSet", "message": "The submariner lighthouse agent deployment does not exist\nThe submariner lighthouse coredns deployment does not exist\nThere are no nodes to run the gateways\nThe metrics proxy daemon set does not exist\nThe globalnet daemon set does not exist" } I0404 13:27:04.696838 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"Unknown", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 45, 0, time.Local), Reason:"ManagedClusterAddOnLeaseNotFound", Message:"The status of submariner add-on is unknown."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 47, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"NoLighthouseAgentDeployment,NoLighthouseCoreDNSDeployment,NoScheduledGateways,NoMetricsProxyDaemonSet,NoGlobalnetDaemonSet", Message:"The submariner lighthouse agent deployment does not exist\nThe submariner lighthouse coredns deployment does not exist\nThere are no nodes to run the gateways\nThe metrics proxy daemon set does not exist\nThe globalnet daemon set does not exist"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:27:05.092Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "NoLighthouseAgentDeployment,NoLighthouseCoreDNSDeployment,NoScheduledGateways,NoScheduledMetricsProxy,NoScheduledGlobalnet", "message": "The submariner lighthouse agent deployment does not exist\nThe submariner lighthouse coredns deployment does not exist\nThere are no nodes to run the gateways\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods" } I0404 13:27:05.092410 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"Unknown", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 45, 0, time.Local), Reason:"ManagedClusterAddOnLeaseNotFound", Message:"The status of submariner add-on is unknown."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 47, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"NoLighthouseAgentDeployment,NoLighthouseCoreDNSDeployment,NoScheduledGateways,NoScheduledMetricsProxy,NoScheduledGlobalnet", Message:"The submariner lighthouse agent deployment does not exist\nThe submariner lighthouse coredns deployment does not exist\nThere are no nodes to run the gateways\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:27:05.891Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "NoLighthouseAgentAvailable,NoLighthouseCoreDNSDeployment,NoScheduledGateways,RouteAgentsUnavailable,NoScheduledMetricsProxy,NoScheduledGlobalnet", "message": "There are no submariner lighthouse agent replica available\nThe submariner lighthouse coredns deployment does not exist\nThere are no nodes to run the gateways\nThere are 6 unavailable route agents\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods" } I0404 13:27:05.892276 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"Unknown", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 45, 0, time.Local), Reason:"ManagedClusterAddOnLeaseNotFound", Message:"The status of submariner add-on is unknown."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 47, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"NoLighthouseAgentAvailable,NoLighthouseCoreDNSDeployment,NoScheduledGateways,RouteAgentsUnavailable,NoScheduledMetricsProxy,NoScheduledGlobalnet", Message:"There are no submariner lighthouse agent replica available\nThe submariner lighthouse coredns deployment does not exist\nThere are no nodes to run the gateways\nThere are 6 unavailable route agents\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:27:06.496Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "NoLighthouseAgentAvailable,NoLighthouseCoreDNSAvailable,NoScheduledGateways,RouteAgentsUnavailable,NoScheduledMetricsProxy,NoScheduledGlobalnet", "message": "There are no submariner lighthouse agent replica available\nThere are no submariner lighthouse coredns replica available\nThere are no nodes to run the gateways\nThere are 6 unavailable route agents\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods" } I0404 13:27:06.496631 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"Unknown", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 45, 0, time.Local), Reason:"ManagedClusterAddOnLeaseNotFound", Message:"The status of submariner add-on is unknown."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 47, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"NoLighthouseAgentAvailable,NoLighthouseCoreDNSAvailable,NoScheduledGateways,RouteAgentsUnavailable,NoScheduledMetricsProxy,NoScheduledGlobalnet", Message:"There are no submariner lighthouse agent replica available\nThere are no submariner lighthouse coredns replica available\nThere are no nodes to run the gateways\nThere are 6 unavailable route agents\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} I0404 13:27:06.900867 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:27:06.927101 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:27:06.942719 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:27:06.996028 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:27:09.382405 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:27:12.222320 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:27:12.260981 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:27:12.271329 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:27:12.343622 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic 2024-04-04T13:27:12.349Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "NoLighthouseAgentAvailable,NoLighthouseCoreDNSAvailable,NoScheduledGateways,RouteAgentsUnavailable,NoScheduledMetricsProxy,NoScheduledGlobalnet", "message": "There are no submariner lighthouse agent replica available\nThere are no submariner lighthouse coredns replica available\nThere are no nodes to run the gateways\nThere are 5 unavailable route agents\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods" } I0404 13:27:12.350191 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"Unknown", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 45, 0, time.Local), Reason:"ManagedClusterAddOnLeaseNotFound", Message:"The status of submariner add-on is unknown."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 47, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"NoLighthouseAgentAvailable,NoLighthouseCoreDNSAvailable,NoScheduledGateways,RouteAgentsUnavailable,NoScheduledMetricsProxy,NoScheduledGlobalnet", Message:"There are no submariner lighthouse agent replica available\nThere are no submariner lighthouse coredns replica available\nThere are no nodes to run the gateways\nThere are 5 unavailable route agents\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:27:13.482Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "NoLighthouseCoreDNSAvailable,NoScheduledGateways,RouteAgentsUnavailable,NoScheduledMetricsProxy,NoScheduledGlobalnet", "message": "There are no submariner lighthouse coredns replica available\nThere are no nodes to run the gateways\nThere are 5 unavailable route agents\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods" } I0404 13:27:13.482461 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"Unknown", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 45, 0, time.Local), Reason:"ManagedClusterAddOnLeaseNotFound", Message:"The status of submariner add-on is unknown."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 47, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"NoLighthouseCoreDNSAvailable,NoScheduledGateways,RouteAgentsUnavailable,NoScheduledMetricsProxy,NoScheduledGlobalnet", Message:"There are no submariner lighthouse coredns replica available\nThere are no nodes to run the gateways\nThere are 5 unavailable route agents\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:27:13.697Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "NoLighthouseCoreDNSAvailable,NoScheduledGateways,RouteAgentsUnavailable,NoScheduledMetricsProxy,NoScheduledGlobalnet", "message": "There are no submariner lighthouse coredns replica available\nThere are no nodes to run the gateways\nThere are 4 unavailable route agents\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods" } I0404 13:27:13.697979 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"Unknown", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 45, 0, time.Local), Reason:"ManagedClusterAddOnLeaseNotFound", Message:"The status of submariner add-on is unknown."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 47, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"NoLighthouseCoreDNSAvailable,NoScheduledGateways,RouteAgentsUnavailable,NoScheduledMetricsProxy,NoScheduledGlobalnet", Message:"There are no submariner lighthouse coredns replica available\nThere are no nodes to run the gateways\nThere are 4 unavailable route agents\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:27:14.659Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "NoScheduledGateways,RouteAgentsUnavailable,NoScheduledMetricsProxy,NoScheduledGlobalnet", "message": "There are no nodes to run the gateways\nThere are 4 unavailable route agents\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods" } I0404 13:27:14.659306 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"Unknown", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 45, 0, time.Local), Reason:"ManagedClusterAddOnLeaseNotFound", Message:"The status of submariner add-on is unknown."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 47, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"NoScheduledGateways,RouteAgentsUnavailable,NoScheduledMetricsProxy,NoScheduledGlobalnet", Message:"There are no nodes to run the gateways\nThere are 4 unavailable route agents\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} I0404 13:27:15.809404 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes 2024-04-04T13:27:16.482Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "NoScheduledGateways,RouteAgentsUnavailable,NoScheduledMetricsProxy,NoScheduledGlobalnet", "message": "There are no nodes to run the gateways\nThere are 3 unavailable route agents\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods" } I0404 13:27:16.483312 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 47, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"NoScheduledGateways,RouteAgentsUnavailable,NoScheduledMetricsProxy,NoScheduledGlobalnet", Message:"There are no nodes to run the gateways\nThere are 3 unavailable route agents\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:27:16.541Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "NoScheduledGateways,RouteAgentsUnavailable,NoScheduledMetricsProxy,NoScheduledGlobalnet", "message": "There are no nodes to run the gateways\nThere are 2 unavailable route agents\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods" } I0404 13:27:16.542173 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 47, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"NoScheduledGateways,RouteAgentsUnavailable,NoScheduledMetricsProxy,NoScheduledGlobalnet", Message:"There are no nodes to run the gateways\nThere are 2 unavailable route agents\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:27:16.897Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "NoScheduledGateways,RouteAgentsUnavailable,NoScheduledMetricsProxy,NoScheduledGlobalnet", "message": "There are no nodes to run the gateways\nThere are 1 unavailable route agents\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods" } I0404 13:27:16.897297 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 47, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"NoScheduledGateways,RouteAgentsUnavailable,NoScheduledMetricsProxy,NoScheduledGlobalnet", Message:"There are no nodes to run the gateways\nThere are 1 unavailable route agents\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} I0404 13:27:18.024032 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:27:18.500512 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:27:18.519000 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:27:18.582633 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic 2024-04-04T13:27:20.816Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "NoScheduledGateways,NoScheduledMetricsProxy,NoScheduledGlobalnet", "message": "There are no nodes to run the gateways\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods" } I0404 13:27:20.816782 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 47, 0, time.Local), Reason:"SubmarinerGatewayNodesUnlabeled", Message:"There are no nodes with label \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"NoScheduledGateways,NoScheduledMetricsProxy,NoScheduledGlobalnet", Message:"There are no nodes to run the gateways\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} I0404 13:27:21.527623 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:27:26.693797 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:27:26.699704 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:27:26.716765 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:27:26.829116 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:27:29.116061 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:27:31.228042 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:27:31.240267 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:27:31.265680 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:27:31.318607 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:27:34.425141 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:27:36.827072 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:27:36.833401 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:27:36.845098 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:27:36.908265 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:27:39.202095 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:27:41.953150 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:27:42.137801 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:27:42.153694 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:27:42.198098 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:27:45.529953 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:27:48.475038 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:27:48.482263 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:27:48.495412 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:27:48.544121 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:28:00.356529 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:28:02.720071 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:28:02.729319 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:28:02.740121 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:28:02.815666 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:28:05.393614 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:28:08.279416 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:28:08.289545 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:28:08.300919 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:28:08.348156 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:28:11.289598 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:28:14.122840 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:28:14.132719 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:28:14.155928 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:28:18.631022 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:28:20.973830 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:28:23.924889 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:28:23.934073 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:28:23.941393 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:28:32.073849 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:28:35.089179 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:28:46.384247 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:28:46.393574 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:28:46.401448 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:28:50.404381 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:28:52.903970 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:28:55.198208 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:28:55.218354 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:28:55.227754 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:29:01.284658 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:29:04.530950 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:29:07.706777 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:29:07.713969 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:29:07.723341 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:29:07.778172 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:29:10.858501 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:29:13.847736 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:29:13.858149 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:29:13.867397 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:29:48.344200 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:29:51.855340 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:29:54.371340 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:29:54.379651 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:29:54.389406 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:29:56.494719 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:29:59.496065 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:30:01.514438 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:30:01.586977 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:30:01.625665 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:30:21.529655 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:30:24.115000 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:30:26.457798 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:30:26.463762 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:30:26.488120 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:30:28.756792 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:30:31.175452 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:30:34.127738 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:30:34.166204 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:30:34.205562 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:30:40.140879 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:30:43.115734 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:30:46.281414 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:30:46.293960 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:30:46.304059 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:30:50.317896 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:30:53.175377 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:30:55.582162 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:30:55.588860 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:30:55.597251 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" 2024-04-04T13:30:59.822Z INF ..eways_controller.go:100 ..ysStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerGatewayNodesLabeled", "status": "True", "lastTransitionTime": null, "reason": "SubmarinerGatewayNodesLabeled", "message": "The nodes \"rdr-c2-svvgg-worker-0-jkxcs\" are labeled with \"submariner.io/gateway\"" } I0404 13:30:59.822878 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"NoScheduledGateways,NoScheduledMetricsProxy,NoScheduledGlobalnet", Message:"There are no nodes to run the gateways\nThere are no nodes to run the metrics proxy\nThere are no nodes to run the globalnet pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} I0404 13:30:59.864710 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic 2024-04-04T13:31:00.002Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "GatewaysUnavailable,NoScheduledMetricsProxy,GlobalnetUnavailable", "message": "There are 1 unavailable gateways\nThere are no nodes to run the metrics proxy\nThere are 1 unavailable globalnet pods" } I0404 13:31:00.002921 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"GatewaysUnavailable,NoScheduledMetricsProxy,GlobalnetUnavailable", Message:"There are 1 unavailable gateways\nThere are no nodes to run the metrics proxy\nThere are 1 unavailable globalnet pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:31:00.045Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "GatewaysUnavailable,MetricsProxyUnavailable,GlobalnetUnavailable", "message": "There are 1 unavailable gateways\nThere are 1 unavailable metrics proxy pods\nThere are 1 unavailable globalnet pods" } I0404 13:31:00.046416 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"GatewaysUnavailable,MetricsProxyUnavailable,GlobalnetUnavailable", Message:"There are 1 unavailable gateways\nThere are 1 unavailable metrics proxy pods\nThere are 1 unavailable globalnet pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:31:04.526Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "GatewaysUnavailable,MetricsProxyUnavailable", "message": "There are 1 unavailable gateways\nThere are 1 unavailable metrics proxy pods" } I0404 13:31:04.526787 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"GatewaysUnavailable,MetricsProxyUnavailable", Message:"There are 1 unavailable gateways\nThere are 1 unavailable metrics proxy pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} I0404 13:31:04.896948 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes 2024-04-04T13:31:06.509Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "GatewaysUnavailable", "message": "There are 1 unavailable gateways" } I0404 13:31:06.509975 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 53, 0, time.Local), Reason:"GatewaysUnavailable", Message:"There are 1 unavailable gateways"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} I0404 13:31:07.822675 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:31:07.829763 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:31:07.846362 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:31:07.921557 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic 2024-04-04T13:31:09.583Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "False", "lastTransitionTime": null, "reason": "SubmarinerAgentDeployed", "message": "Submariner (submariner.v0.17.0) is deployed on managed cluster." } I0404 13:31:09.584007 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 31, 9, 0, time.Local), Reason:"SubmarinerAgentDeployed", Message:"Submariner (submariner.v0.17.0) is deployed on managed cluster."}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} I0404 13:31:11.346282 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:31:14.307722 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:31:14.323949 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:31:14.334832 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:31:14.420667 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic 2024-04-04T13:31:17.171Z INF ..eways_controller.go:100 ..ysStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerGatewayNodesLabeled", "status": "True", "lastTransitionTime": null, "reason": "SubmarinerGatewayNodesLabeled", "message": "The nodes \"rdr-c2-svvgg-worker-0-jkxcs,rdr-c2-svvgg-worker-0-qtb8r\" are labeled with \"submariner.io/gateway\"" } I0404 13:31:17.172172 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs,rdr-c2-svvgg-worker-0-qtb8r\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 31, 9, 0, time.Local), Reason:"SubmarinerAgentDeployed", Message:"Submariner (submariner.v0.17.0) is deployed on managed cluster."}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:31:17.293Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "GatewaysUnavailable", "message": "There are 1 unavailable gateways" } I0404 13:31:17.293831 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs,rdr-c2-svvgg-worker-0-qtb8r\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 31, 17, 0, time.Local), Reason:"GatewaysUnavailable", Message:"There are 1 unavailable gateways"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:31:17.382Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "GatewaysUnavailable,MetricsProxyUnavailable", "message": "There are 1 unavailable gateways\nThere are 1 unavailable metrics proxy pods" } I0404 13:31:17.385417 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs,rdr-c2-svvgg-worker-0-qtb8r\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 31, 17, 0, time.Local), Reason:"GatewaysUnavailable,MetricsProxyUnavailable", Message:"There are 1 unavailable gateways\nThere are 1 unavailable metrics proxy pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:31:17.436Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "GatewaysUnavailable,MetricsProxyUnavailable,GlobalnetUnavailable", "message": "There are 1 unavailable gateways\nThere are 1 unavailable metrics proxy pods\nThere are 1 unavailable globalnet pods" } I0404 13:31:17.437325 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs,rdr-c2-svvgg-worker-0-qtb8r\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 31, 17, 0, time.Local), Reason:"GatewaysUnavailable,MetricsProxyUnavailable,GlobalnetUnavailable", Message:"There are 1 unavailable gateways\nThere are 1 unavailable metrics proxy pods\nThere are 1 unavailable globalnet pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} I0404 13:31:19.599393 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:31:22.750503 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:31:22.769075 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:31:22.785549 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:31:22.902109 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic 2024-04-04T13:31:23.208Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "GatewaysUnavailable,GlobalnetUnavailable", "message": "There are 1 unavailable gateways\nThere are 1 unavailable globalnet pods" } I0404 13:31:23.208965 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs,rdr-c2-svvgg-worker-0-qtb8r\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 31, 17, 0, time.Local), Reason:"GatewaysUnavailable,GlobalnetUnavailable", Message:"There are 1 unavailable gateways\nThere are 1 unavailable globalnet pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:31:23.290Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "GatewaysUnavailable", "message": "There are 1 unavailable gateways" } I0404 13:31:23.291287 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs,rdr-c2-svvgg-worker-0-qtb8r\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 31, 17, 0, time.Local), Reason:"GatewaysUnavailable", Message:"There are 1 unavailable gateways"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:31:26.238Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "False", "lastTransitionTime": null, "reason": "SubmarinerAgentDeployed", "message": "Submariner (submariner.v0.17.0) is deployed on managed cluster." } I0404 13:31:26.238622 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs,rdr-c2-svvgg-worker-0-qtb8r\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 31, 26, 0, time.Local), Reason:"SubmarinerAgentDeployed", Message:"Submariner (submariner.v0.17.0) is deployed on managed cluster."}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} I0404 13:31:26.810107 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:31:29.632702 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:31:29.639925 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:31:29.660899 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" I0404 13:31:29.704215 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic I0404 13:31:33.177630 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:31:35.586696 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Deploying dedicated Submariner gateway node rdr-c2-svvgg-submariner-gw0 I0404 13:31:35.593014 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RHOSCloudProvider' Unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" E0404 13:31:35.606564 1 base_controller.go:268] SubmarinerConfigController reconciliation failed: unable to deploy gateway: error getting the worker image: could not retrieve the image of one of the worker nodes from the infra "rdr-c2-svvgg" 2024-04-04T13:31:39.359Z INF ..eways_controller.go:100 ..ysStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerGatewayNodesLabeled", "status": "True", "lastTransitionTime": null, "reason": "SubmarinerGatewayNodesLabeled", "message": "The nodes \"rdr-c2-svvgg-worker-0-jkxcs,rdr-c2-svvgg-worker-0-qtb8r,rdr-c2-svvgg-worker-0-wbcdz\" are labeled with \"submariner.io/gateway\"" } I0404 13:31:39.359835 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs,rdr-c2-svvgg-worker-0-qtb8r,rdr-c2-svvgg-worker-0-wbcdz\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 31, 26, 0, time.Local), Reason:"SubmarinerAgentDeployed", Message:"Submariner (submariner.v0.17.0) is deployed on managed cluster."}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} I0404 13:31:39.367230 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Configuring the required firewall rules for inter-cluster traffic 2024-04-04T13:31:39.435Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "GatewaysUnavailable", "message": "There are 1 unavailable gateways" } I0404 13:31:39.435514 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs,rdr-c2-svvgg-worker-0-qtb8r,rdr-c2-svvgg-worker-0-wbcdz\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 31, 39, 0, time.Local), Reason:"GatewaysUnavailable", Message:"There are 1 unavailable gateways"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:31:39.472Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "GatewaysUnavailable,MetricsProxyUnavailable,GlobalnetUnavailable", "message": "There are 1 unavailable gateways\nThere are 1 unavailable metrics proxy pods\nThere are 1 unavailable globalnet pods" } I0404 13:31:39.473218 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs,rdr-c2-svvgg-worker-0-qtb8r,rdr-c2-svvgg-worker-0-wbcdz\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 31, 39, 0, time.Local), Reason:"GatewaysUnavailable,MetricsProxyUnavailable,GlobalnetUnavailable", Message:"There are 1 unavailable gateways\nThere are 1 unavailable metrics proxy pods\nThere are 1 unavailable globalnet pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} I0404 13:31:44.469334 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Opened external ports "4500/udp, 4900/udp, 0/esp, 0/ah" in security group "rdr-c2-svvgg-submariner-gw-sg" on RHOS for existing g/w nodes I0404 13:31:44.469389 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' Current Submariner gateways match the required number of Submariner gateways I0404 13:31:44.469413 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RHOSCloudProvider' The Submariner cluster environment has been set up on RHOS 2024-04-04T13:31:44.507Z INF ..onfig_controller.go:283 ..erConfigController Submariner environment was prepared for cluster "rdr-c2": v1alpha1.ManagedClusterInfo{ClusterName:"rdr-c2", Vendor:"OpenShift", Platform:"OpenStack", Region:"", InfraID:"rdr-c2-svvgg", VendorVersion:"4.15.6", NetworkType:"OVNKubernetes"} I0404 13:31:44.507152 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SubmarinerClusterEnvPrepared' submariner cluster environment was prepared for managed cluster rdr-c2 2024-04-04T13:31:44.563Z INF ..onfig_controller.go:326 ..erConfigController Updated SubmarinerConfig status condition: { "type": "SubmarinerGatewaysLabeled", "status": "True", "lastTransitionTime": null, "reason": "Success", "message": "3 node(s) (\"rdr-c2-svvgg-worker-0-jkxcs,rdr-c2-svvgg-worker-0-qtb8r,rdr-c2-svvgg-worker-0-wbcdz\") are labeled as gateways" } I0404 13:31:44.563459 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SubmarinerConfigStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"SubmarinerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"SubmarinerConfigApplied", Message:"SubmarinerConfig was applied"}, v1.Condition{Type:"SubmarinerClusterEnvironmentPrepared", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 31, 44, 0, time.Local), Reason:"SubmarinerClusterEnvPrepared", Message:"Submariner cluster environment was prepared"}, v1.Condition{Type:"SubmarinerGatewaysLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 31, 44, 0, time.Local), Reason:"Success", Message:"3 node(s) (\"rdr-c2-svvgg-worker-0-jkxcs,rdr-c2-svvgg-worker-0-qtb8r,rdr-c2-svvgg-worker-0-wbcdz\") are labeled as gateways"}} 2024-04-04T13:31:46.151Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "GatewaysUnavailable,MetricsProxyUnavailable", "message": "There are 1 unavailable gateways\nThere are 1 unavailable metrics proxy pods" } I0404 13:31:46.151990 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs,rdr-c2-svvgg-worker-0-qtb8r,rdr-c2-svvgg-worker-0-wbcdz\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 31, 39, 0, time.Local), Reason:"GatewaysUnavailable,MetricsProxyUnavailable", Message:"There are 1 unavailable gateways\nThere are 1 unavailable metrics proxy pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:31:46.247Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "True", "lastTransitionTime": null, "reason": "MetricsProxyUnavailable", "message": "There are 1 unavailable metrics proxy pods" } I0404 13:31:46.248599 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs,rdr-c2-svvgg-worker-0-qtb8r,rdr-c2-svvgg-worker-0-wbcdz\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 31, 39, 0, time.Local), Reason:"MetricsProxyUnavailable", Message:"There are 1 unavailable metrics proxy pods"}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:31:47.458Z INF ..yment_controller.go:151 ..ntStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerAgentDegraded", "status": "False", "lastTransitionTime": null, "reason": "SubmarinerAgentDeployed", "message": "Submariner (submariner.v0.17.0) is deployed on managed cluster." } I0404 13:31:47.460434 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs,rdr-c2-svvgg-worker-0-qtb8r,rdr-c2-svvgg-worker-0-wbcdz\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 31, 47, 0, time.Local), Reason:"SubmarinerAgentDeployed", Message:"Submariner (submariner.v0.17.0) is deployed on managed cluster."}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsNotEstablished", Message:"There are no connections on gateways"}} 2024-04-04T13:33:40.872Z INF ..ctions_controller.go:90 ..nsStatusController Updated submariner ManagedClusterAddOn status condition: { "type": "SubmarinerConnectionDegraded", "status": "True", "lastTransitionTime": null, "reason": "ConnectionsDegraded", "message": "The connection between clusters \"rdr-c2\" and \"rdr-c1\" is not established (status=connecting)" } I0404 13:33:40.872801 1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"9ab0a025-8c3a-4fb9-b386-f695c330ba2f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 34, 0, time.Local), Reason:"SetPermissionApplied", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2024-04-04 13:21:34 +0000 UTC to 2024-04-21 05:03:18 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 35, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 26, 36, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 15, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 30, 59, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"rdr-c2-svvgg-worker-0-jkxcs,rdr-c2-svvgg-worker-0-qtb8r,rdr-c2-svvgg-worker-0-wbcdz\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"SubmarinerAgentDegraded", Status:"False", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 31, 47, 0, time.Local), Reason:"SubmarinerAgentDeployed", Message:"Submariner (submariner.v0.17.0) is deployed on managed cluster."}, v1.Condition{Type:"SubmarinerConnectionDegraded", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2024, time.April, 4, 13, 27, 4, 0, time.Local), Reason:"ConnectionsDegraded", Message:"The connection between clusters \"rdr-c2\" and \"rdr-c1\" is not established (status=connecting)"}} E0406 21:40:08.271451 1 lease_controller.go:127] Failed to update lease submariner/submariner-operator: Operation cannot be fulfilled on leases.coordination.k8s.io "submariner": the object has been modified; please apply your changes to the latest version and try again on managed cluster