W0327 05:56:47.715668 1 cmd.go:245] Using insecure, self-signed certificates I0327 05:56:48.706674 1 leaderelection.go:122] The leader election gives 4 retries and allows for 30s of clock skew. The kube-apiserver downtime tolerance is 78s. Worst non-graceful lease acquisition is 2m43s. Worst graceful lease acquisition is {26s}. I0327 05:56:48.709918 1 observer_polling.go:159] Starting file observer W0327 05:56:48.754897 1 builder.go:267] unable to get owner reference (falling back to namespace): Get "https://172.30.0.1:443/api/v1/namespaces/open-cluster-management/pods/submariner-addon-76cbb78c5b-qf7p4": dial tcp 172.30.0.1:443: connect: connection refused I0327 05:56:48.755249 1 builder.go:299] submariner-controller version v0.0.0-unknown-- I0327 05:57:11.309829 1 secure_serving.go:57] Forcing use of http/1.1 only W0327 05:57:11.309881 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0327 05:57:11.309891 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. I0327 05:57:11.387504 1 leaderelection.go:250] attempting to acquire leader lease open-cluster-management/submariner-controller-lock... I0327 05:57:11.436656 1 secure_serving.go:213] Serving securely on [::]:8443 I0327 05:57:11.436839 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController I0327 05:57:11.436881 1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController I0327 05:57:11.437168 1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/serving-cert-3728385415/tls.crt::/tmp/serving-cert-3728385415/tls.key" I0327 05:57:11.437462 1 tlsconfig.go:240] "Starting DynamicServingCertificateController" I0327 05:57:11.442849 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0327 05:57:11.442893 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0327 05:57:11.443904 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0327 05:57:11.444218 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0327 05:57:11.557186 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0327 05:57:11.557256 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0327 05:57:11.637580 1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController I0327 06:00:05.905682 1 leaderelection.go:260] successfully acquired lease open-cluster-management/submariner-controller-lock I0327 06:00:05.909442 1 event.go:364] Event(v1.ObjectReference{Kind:"Lease", Namespace:"open-cluster-management", Name:"submariner-controller-lock", UID:"4f653524-25f3-4a11-ba02-b435ea03277f", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"6218126", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' submariner-addon-76cbb78c5b-qf7p4_1115e4be-27e1-4d31-b10a-86f117068635 became leader I0327 06:00:06.230670 1 manager.go:229] Checking if ClusterRole "access-to-brokers-submariner-crd" exists I0327 06:00:06.793714 1 base_controller.go:67] Waiting for caches to sync for SubmarinerBrokerCRDsController I0327 06:00:06.793801 1 base_controller.go:67] Waiting for caches to sync for SubmarinerBrokerController I0327 06:00:06.793822 1 base_controller.go:67] Waiting for caches to sync for SubmarinerAgentController I0327 06:00:07.017836 1 base_controller.go:73] Caches are synced for SubmarinerBrokerController I0327 06:00:07.017865 1 base_controller.go:110] Starting #1 worker of SubmarinerBrokerController controller ... I0327 06:00:07.294159 1 base_controller.go:73] Caches are synced for SubmarinerAgentController I0327 06:00:07.294357 1 base_controller.go:110] Starting #1 worker of SubmarinerAgentController controller ... I0327 06:00:07.325290 1 base_controller.go:34] Waiting for caches to sync for addon-deploy-controller I0327 06:00:07.325333 1 base_controller.go:34] Waiting for caches to sync for addon-registration-controller I0327 06:00:07.325350 1 base_controller.go:34] Waiting for caches to sync for addon-install-controller I0327 06:00:07.325378 1 base_controller.go:34] Waiting for caches to sync for management-addon-controller I0327 06:00:07.325405 1 base_controller.go:34] Waiting for caches to sync for addon-owner-controller I0327 06:00:07.325420 1 base_controller.go:34] Waiting for caches to sync for addon-config-controller I0327 06:00:07.325434 1 base_controller.go:34] Waiting for caches to sync for management-addon-config-controller I0327 06:00:07.325449 1 base_controller.go:34] Waiting for caches to sync for addon-configuration-controller I0327 06:00:07.325475 1 base_controller.go:34] Waiting for caches to sync for CSRApprovingController I0327 06:00:07.325495 1 base_controller.go:34] Waiting for caches to sync for CSRApprovingController I0327 06:00:07.439161 1 base_controller.go:40] Caches are synced for CSRApprovingController I0327 06:00:07.439210 1 base_controller.go:78] Starting #1 worker of CSRApprovingController controller ... I0327 06:00:07.439423 1 base_controller.go:40] Caches are synced for CSRApprovingController I0327 06:00:07.439441 1 base_controller.go:78] Starting #1 worker of CSRApprovingController controller ... I0327 06:00:07.680183 1 base_controller.go:40] Caches are synced for management-addon-controller I0327 06:00:07.680355 1 base_controller.go:78] Starting #1 worker of management-addon-controller controller ... I0327 06:00:07.680222 1 base_controller.go:40] Caches are synced for management-addon-config-controller I0327 06:00:07.680494 1 base_controller.go:78] Starting #1 worker of management-addon-config-controller controller ... I0327 06:00:07.727479 1 base_controller.go:40] Caches are synced for addon-configuration-controller I0327 06:00:07.727520 1 base_controller.go:78] Starting #1 worker of addon-configuration-controller controller ... I0327 06:00:07.727833 1 base_controller.go:40] Caches are synced for addon-deploy-controller I0327 06:00:07.727848 1 base_controller.go:78] Starting #1 worker of addon-deploy-controller controller ... I0327 06:00:07.732377 1 base_controller.go:40] Caches are synced for addon-registration-controller I0327 06:00:07.732403 1 base_controller.go:78] Starting #1 worker of addon-registration-controller controller ... I0327 06:00:07.733251 1 base_controller.go:40] Caches are synced for addon-install-controller I0327 06:00:07.733262 1 base_controller.go:78] Starting #1 worker of addon-install-controller controller ... I0327 06:00:07.733316 1 base_controller.go:40] Caches are synced for addon-owner-controller I0327 06:00:07.733322 1 base_controller.go:78] Starting #1 worker of addon-owner-controller controller ... I0327 06:00:07.733807 1 base_controller.go:40] Caches are synced for addon-config-controller I0327 06:00:07.733814 1 base_controller.go:78] Starting #1 worker of addon-config-controller controller ... I0327 06:00:10.895184 1 base_controller.go:73] Caches are synced for SubmarinerBrokerCRDsController I0327 06:00:10.895230 1 base_controller.go:110] Starting #1 worker of SubmarinerBrokerCRDsController controller ... E0327 09:03:40.341674 1 leaderelection.go:369] Failed to update lock: Operation cannot be fulfilled on leases.coordination.k8s.io "submariner-controller-lock": the object has been modified; please apply your changes to the latest version and try again E0327 09:04:21.365013 1 leaderelection.go:369] Failed to update lock: Operation cannot be fulfilled on leases.coordination.k8s.io "submariner-controller-lock": the object has been modified; please apply your changes to the latest version and try again E0329 11:02:24.315807 1 leaderelection.go:332] error retrieving resource lock open-cluster-management/submariner-controller-lock: Get "https://172.30.0.1:443/apis/coordination.k8s.io/v1/namespaces/open-cluster-management/leases/submariner-controller-lock?timeout=1m47s": dial tcp 172.30.0.1:443: connect: connection refused E0329 11:02:50.336352 1 leaderelection.go:332] error retrieving resource lock open-cluster-management/submariner-controller-lock: Get "https://172.30.0.1:443/apis/coordination.k8s.io/v1/namespaces/open-cluster-management/leases/submariner-controller-lock?timeout=1m47s": dial tcp 172.30.0.1:443: connect: connection refused W0404 13:26:34.327358 1 controller.go:99] OwnerReferences is not set for "rdr-c1/submariner" W0404 13:26:34.336638 1 controller.go:99] OwnerReferences is not set for "rdr-c2/submariner" 2024-04-04T13:26:34.486Z INF ..agent/controller.go:323 ..nerAgentController Added finalizer to ManagedClusterAddOn "submariner" in cluster "rdr-c1" I0404 13:26:34.510895 1 event.go:364] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management", Name:"open-cluster-management", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RoleCreated' Created Role.rbac.authorization.k8s.io/open-cluster-management:submariner-addon:agent -n rdr-c1 because it was missing 2024-04-04T13:26:34.537Z INF ..agent/controller.go:323 ..nerAgentController Added finalizer to ManagedClusterAddOn "submariner" in cluster "rdr-c2" I0404 13:26:34.559802 1 event.go:364] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management", Name:"open-cluster-management", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RoleBindingCreated' Created RoleBinding.rbac.authorization.k8s.io/open-cluster-management:submariner-addon:agent -n rdr-c1 because it was missing E0404 13:26:34.579238 1 base_controller.go:159] "addon-registration-controller" controller failed to sync "rdr-c1/submariner", err: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io "submariner": the object has been modified; please apply your changes to the latest version and try again I0404 13:26:34.597916 1 event.go:364] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management", Name:"open-cluster-management", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RoleCreated' Created Role.rbac.authorization.k8s.io/open-cluster-management:submariner-addon:agent -n rdr-c2 because it was missing I0404 13:26:34.631173 1 event.go:364] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management", Name:"open-cluster-management", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'BrokerConfigApplied' Configuration applied from brokers.submariner.io object in "rdr-cs-broker" namespace I0404 13:26:34.643566 1 event.go:364] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management", Name:"open-cluster-management", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RoleBindingCreated' Created RoleBinding.rbac.authorization.k8s.io/open-cluster-management:submariner-addon:agent -n rdr-c2 because it was missing I0404 13:26:34.744415 1 event.go:364] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management", Name:"open-cluster-management", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SubmarinerConfigApplied' SubmarinerConfig "submariner" was applied for managed cluster "rdr-c1" I0404 13:26:34.785632 1 event.go:364] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management", Name:"open-cluster-management", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManifestWorkApplied' manifestwork rdr-c1/submariner-operator was created 2024-04-04T13:26:34.785Z INF ..anifestwork/apply.go:33 ManifestWork Created ManifestWork "rdr-c1/submariner-operator": { "apiVersion": "rbac.authorization.k8s.io/v1", "kind": "ClusterRole", "metadata": { "labels": { "rbac.authorization.k8s.io/aggregate-to-admin": "true" }, "name": "open-cluster-management:submariner-addon-operatorgroups-aggregate-clusterrole" }, "rules": [ { "apiGroups": [ "operators.coreos.com" ], "resources": [ "operatorgroups" ], "verbs": [ "get", "create", "delete", "update" ] } ] } { "apiVersion": "rbac.authorization.k8s.io/v1", "kind": "ClusterRole", "metadata": { "labels": { "rbac.authorization.k8s.io/aggregate-to-admin": "true" }, "name": "open-cluster-management:submariner-scc-admin-aggregate-clusterrole" }, "rules": [ { "apiGroups": [ "security.openshift.io" ], "resources": [ "securitycontextconstraints" ], "verbs": [ "get", "create", "delete" ] } ] } { "apiVersion": "operators.coreos.com/v1", "kind": "OperatorGroup", "metadata": { "name": "submariner-operator", "namespace": "submariner-operator" }, "spec": { "targetNamespaces": [ "submariner-operator" ] } } { "apiVersion": "operators.coreos.com/v1alpha1", "kind": "Subscription", "metadata": { "name": "submariner", "namespace": "submariner-operator" }, "spec": { "channel": "stable-0.17", "installPlanApproval": "Automatic", "name": "submariner", "source": "redhat-operators", "sourceNamespace": "openshift-marketplace" } } I0404 13:26:34.844134 1 event.go:364] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management", Name:"open-cluster-management", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManifestWorkApplied' manifestwork rdr-c1/submariner-resource was created 2024-04-04T13:26:34.845Z INF ..anifestwork/apply.go:33 ManifestWork Created ManifestWork "rdr-c1/submariner-resource": { "apiVersion": "submariner.io/v1alpha1", "kind": "Submariner", "metadata": { "name": "submariner", "namespace": "submariner-operator" }, "spec": { "airGappedDeployment": false, "broker": "k8s", "brokerK8sApiServer": "##redacted##", "brokerK8sApiServerToken": "##redacted##", "brokerK8sCA": "##redacted##", "brokerK8sInsecure": false, "brokerK8sRemoteNamespace": "rdr-cs-broker", "cableDriver": "libreswan", "ceIPSecDebug": false, "ceIPSecNATTPort": 4500, "ceIPSecPSK": "##redacted##", "clusterCIDR": "", "clusterID": "rdr-c1", "colorCodes": "blue", "debug": false, "globalCIDR": "242.0.0.0/16", "haltOnCertificateError": true, "loadBalancerEnabled": false, "namespace": "submariner-operator", "natEnabled": true, "repository": "registry.redhat.io/rhacm2", "serviceCIDR": "", "serviceDiscoveryEnabled": true } } I0404 13:26:35.254655 1 event.go:364] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management", Name:"open-cluster-management", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'BrokerConfigApplied' Configuration applied from brokers.submariner.io object in "rdr-cs-broker" namespace I0404 13:26:35.426778 1 event.go:364] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management", Name:"open-cluster-management", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SubmarinerConfigApplied' SubmarinerConfig "submariner" was applied for managed cluster "rdr-c2" I0404 13:26:35.467359 1 event.go:364] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management", Name:"open-cluster-management", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManifestWorkApplied' manifestwork rdr-c2/submariner-operator was created 2024-04-04T13:26:35.467Z INF ..anifestwork/apply.go:33 ManifestWork Created ManifestWork "rdr-c2/submariner-operator": { "apiVersion": "rbac.authorization.k8s.io/v1", "kind": "ClusterRole", "metadata": { "labels": { "rbac.authorization.k8s.io/aggregate-to-admin": "true" }, "name": "open-cluster-management:submariner-addon-operatorgroups-aggregate-clusterrole" }, "rules": [ { "apiGroups": [ "operators.coreos.com" ], "resources": [ "operatorgroups" ], "verbs": [ "get", "create", "delete", "update" ] } ] } { "apiVersion": "rbac.authorization.k8s.io/v1", "kind": "ClusterRole", "metadata": { "labels": { "rbac.authorization.k8s.io/aggregate-to-admin": "true" }, "name": "open-cluster-management:submariner-scc-admin-aggregate-clusterrole" }, "rules": [ { "apiGroups": [ "security.openshift.io" ], "resources": [ "securitycontextconstraints" ], "verbs": [ "get", "create", "delete" ] } ] } { "apiVersion": "operators.coreos.com/v1", "kind": "OperatorGroup", "metadata": { "name": "submariner-operator", "namespace": "submariner-operator" }, "spec": { "targetNamespaces": [ "submariner-operator" ] } } { "apiVersion": "operators.coreos.com/v1alpha1", "kind": "Subscription", "metadata": { "name": "submariner", "namespace": "submariner-operator" }, "spec": { "channel": "stable-0.17", "installPlanApproval": "Automatic", "name": "submariner", "source": "redhat-operators", "sourceNamespace": "openshift-marketplace" } } I0404 13:26:35.572214 1 event.go:364] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management", Name:"open-cluster-management", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManifestWorkApplied' manifestwork rdr-c2/submariner-resource was created 2024-04-04T13:26:35.572Z INF ..anifestwork/apply.go:33 ManifestWork Created ManifestWork "rdr-c2/submariner-resource": { "apiVersion": "submariner.io/v1alpha1", "kind": "Submariner", "metadata": { "name": "submariner", "namespace": "submariner-operator" }, "spec": { "airGappedDeployment": false, "broker": "k8s", "brokerK8sApiServer": "##redacted##", "brokerK8sApiServerToken": "##redacted##", "brokerK8sCA": "##redacted##", "brokerK8sInsecure": false, "brokerK8sRemoteNamespace": "rdr-cs-broker", "cableDriver": "libreswan", "ceIPSecDebug": false, "ceIPSecNATTPort": 4500, "ceIPSecPSK": "##redacted##", "clusterCIDR": "", "clusterID": "rdr-c2", "colorCodes": "blue", "debug": false, "globalCIDR": "242.1.0.0/16", "haltOnCertificateError": true, "loadBalancerEnabled": false, "namespace": "submariner-operator", "natEnabled": true, "repository": "registry.redhat.io/rhacm2", "serviceCIDR": "", "serviceDiscoveryEnabled": true } } I0404 13:26:46.612972 1 event.go:364] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management", Name:"open-cluster-management", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SubmarinerConfigApplied' SubmarinerConfig "submariner" was applied for managed cluster "rdr-c1" I0404 13:26:47.434224 1 event.go:364] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management", Name:"open-cluster-management", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SubmarinerConfigApplied' SubmarinerConfig "submariner" was applied for managed cluster "rdr-c2" I0404 13:26:53.835164 1 event.go:364] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management", Name:"open-cluster-management", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SubmarinerConfigApplied' SubmarinerConfig "submariner" was applied for managed cluster "rdr-c1" I0404 13:26:55.951382 1 event.go:364] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management", Name:"open-cluster-management", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SubmarinerConfigApplied' SubmarinerConfig "submariner" was applied for managed cluster "rdr-c2"