[kni@cert-rhosp-02 ~]$ oc get csv NAME DISPLAY VERSION REPLACES PHASE fence-agents-remediation.v0.6.0 Fence Agents Remediation Operator 0.6.0 fence-agents-remediation.v0.5.1 Succeeded node-healthcheck-operator.v0.10.0 Node Health Check Operator 0.10.0 node-healthcheck-operator.v0.9.1 Succeeded ========================================================================================================================================================================================== [kni@cert-rhosp-02 ~]$ oc get node/worker-0-2 NAME STATUS ROLES AGE VERSION worker-0-2 Ready worker 10h v1.32.7 ========================================================================================================================================================================================== [kni@cert-rhosp-02 ~]$ oc get far No resources found in openshift-workload-availability namespace. ========================================================================================================================================================================================== [kni@cert-rhosp-02 ~]$ oc get secret/test-far-shared -o yaml apiVersion: v1 data: --password: cGFzc3dvcmQ= --username: YWRtaW4= kind: Secret metadata: creationTimestamp: "2025-09-03T18:56:49Z" name: test-far-shared namespace: openshift-workload-availability resourceVersion: "228417" uid: ee92ed97-3fac-4864-8520-4663f80255a9 type: Opaque ========================================================================================================================================================================================== [kni@cert-rhosp-02 ~]$ oc get far No resources found in openshift-workload-availability namespace. ========================================================================================================================================================================================== [kni@cert-rhosp-02 ~]$ oc get node/worker-0-2 NAME STATUS ROLES AGE VERSION worker-0-2 NotReady worker 10h v1.32.7 ========================================================================================================================================================================================== [kni@cert-rhosp-02 ~]$ oc get far NAME AGE worker-0-2 5s ========================================================================================================================================================================================== [kni@cert-rhosp-02 ~]$ oc get far/worker-0-2 -o yaml apiVersion: fence-agents-remediation.medik8s.io/v1alpha1 kind: FenceAgentsRemediation metadata: creationTimestamp: "2025-09-03T19:03:22Z" finalizers: - fence-agents-remediation.medik8s.io/far-finalizer generation: 2 name: worker-0-2 namespace: openshift-workload-availability resourceVersion: "230837" uid: acb96e5c-2aa3-4ac1-9524-3bd616afb8dc spec: agent: fence_ipmilan nodeparameters: --ipport: master-0-0: "6230" master-0-1: "6231" master-0-2: "6232" worker-0-0: "6233" worker-0-1: "6234" worker-0-2: "6235" remediationStrategy: ResourceDeletion retrycount: 5 retryinterval: 10s sharedSecretName: test-far-shared sharedparameters: --action: reboot --ip: 192.168.123.1 --lanplus: "" timeout: 5m0s status: conditions: - lastTransitionTime: "2025-09-03T19:03:30Z" message: The unhealthy node was fully remediated (it was tainted, fenced using the fence agent and all the node resources have been deleted) reason: RemediationFinishedSuccessfully status: "False" type: Processing - lastTransitionTime: "2025-09-03T19:03:27Z" message: FAR taint was added and the fence agent command has been created and executed successfully reason: FenceAgentSucceeded status: "True" type: FenceAgentActionSucceeded - lastTransitionTime: "2025-09-03T19:03:30Z" message: The unhealthy node was fully remediated (it was tainted, fenced using the fence agent and all the node resources have been deleted) reason: RemediationFinishedSuccessfully status: "True" type: Succeeded lastUpdateTime: "2025-09-03T19:03:30Z" Far Logs ========================================================================================================================================================================================== 2025-09-03T19:00:41.386395544Z INFO controllers.FenceAgentsRemediation FenceAgentsRemediation CR was not found {"CR Name": "worker-0-2", "CR Namespace": "openshift-workload-availability"} 2025-09-03T19:00:41.386403215Z INFO controllers.FenceAgentsRemediation Finish FenceAgentsRemediation Reconcile 2025-09-03T19:00:41.386411505Z DEBUG events [remediation] Finalizer was removed {"type": "Normal", "object": {"kind":"FenceAgentsRemediation","namespace":"openshift-workload-availability","name":"worker-0-2","uid":"2195a06f-fdb5-410e-8ce7-aa62b05e32ca","apiVersion":"fence-agents-remediation.medik8s.io/v1alpha1","resourceVersion":"229693"}, "reason": "RemoveFinalizer"} 2025-09-03T19:03:22.892253867Z INFO fenceagentsremediation-resource validate create {"name": "worker-0-2"} 2025-09-03T19:03:22.895193327Z INFO controllers.FenceAgentsRemediation Begin FenceAgentsRemediation Reconcile 2025-09-03T19:03:22.895232667Z INFO controllers.FenceAgentsRemediation Check FAR CR's name 2025-09-03T19:03:22.898182254Z INFO fenceagentsremediation-resource validate update {"name": "worker-0-2"} 2025-09-03T19:03:22.901440461Z INFO controllers.FenceAgentsRemediation Finalizer was added {"CR Name": "worker-0-2"} 2025-09-03T19:03:22.901491734Z INFO controllers.FenceAgentsRemediation Updating Status Condition {"processingConditionStatus": "True", "fenceAgentActionSucceededConditionStatus": "Unknown", "succeededConditionStatus": "Unknown", "reason": "RemediationStarted", "LastUpdateTime": "2025-09-03T19:03:22.901490568Z"} 2025-09-03T19:03:22.901512965Z DEBUG events [remediation] Remediation started {"type": "Normal", "object": {"kind":"FenceAgentsRemediation","namespace":"openshift-workload-availability","name":"worker-0-2","uid":"acb96e5c-2aa3-4ac1-9524-3bd616afb8dc","apiVersion":"fence-agents-remediation.medik8s.io/v1alpha1","resourceVersion":"230656"}, "reason": "RemediationStarted"} 2025-09-03T19:03:22.901792916Z DEBUG events [remediation] Finalizer was added {"type": "Normal", "object": {"kind":"FenceAgentsRemediation","namespace":"openshift-workload-availability","name":"worker-0-2","uid":"acb96e5c-2aa3-4ac1-9524-3bd616afb8dc","apiVersion":"fence-agents-remediation.medik8s.io/v1alpha1","resourceVersion":"230656"}, "reason": "AddFinalizer"} 2025-09-03T19:03:22.906458469Z INFO controllers.FenceAgentsRemediation Finish FenceAgentsRemediation Reconcile 2025-09-03T19:03:22.906572158Z INFO controllers.FenceAgentsRemediation Begin FenceAgentsRemediation Reconcile 2025-09-03T19:03:22.906600178Z INFO controllers.FenceAgentsRemediation Check FAR CR's name 2025-09-03T19:03:22.91362655Z INFO taints Taint was added {"taint effect": "NoExecute", "taint list": [{"key":"node.kubernetes.io/unreachable","effect":"NoSchedule","timeAdded":"2025-09-03T19:02:49Z"},{"key":"node.kubernetes.io/unreachable","effect":"NoExecute","timeAdded":"2025-09-03T19:02:55Z"},{"key":"medik8s.io/fence-agents-remediation","effect":"NoExecute","timeAdded":"2025-09-03T19:03:22Z"}]} 2025-09-03T19:03:22.913669387Z INFO controllers.FenceAgentsRemediation FAR remediation taint was added {"Node Name": "worker-0-2"} 2025-09-03T19:03:22.913700417Z INFO controllers.FenceAgentsRemediation Build fence agent command line {"Fence Agent": "fence_ipmilan", "Node Name": "worker-0-2"} 2025-09-03T19:03:22.913722029Z INFO controllers.FenceAgentsRemediation found a value from secret {"secret name": "test-far-shared", "parameter name": "--password"} 2025-09-03T19:03:22.91372465Z INFO controllers.FenceAgentsRemediation found a value from secret {"secret name": "test-far-shared", "parameter name": "--username"} 2025-09-03T19:03:22.913742224Z DEBUG events [remediation] Remediation taint was added {"type": "Normal", "object": {"kind":"Node","name":"worker-0-2","uid":"28702782-f67d-4bc9-9792-a97600872477","apiVersion":"v1","resourceVersion":"230527"}, "reason": "AddRemediationTaint"} 2025-09-03T19:03:22.913753389Z INFO controllers.FenceAgentsRemediation Execute the fence agent {"Fence Agent": "fence_ipmilan", "Node Name": "worker-0-2", "FAR uid": "acb96e5c-2aa3-4ac1-9524-3bd616afb8dc", "ParametersError": "json: unsupported type: iter.Seq[github.com/medik8s/fence-agents-remediation/api/v1alpha1.ParameterName]"} 2025-09-03T19:03:22.913822028Z INFO executer fence agent start {"uid": "acb96e5c-2aa3-4ac1-9524-3bd616afb8dc", "fence_agent": "fence_ipmilan", "retryCount": 5, "retryInterval": "10s", "timeout": "5m0s"} 2025-09-03T19:03:22.913928407Z DEBUG events [remediation] Fence agent was executed {"type": "Normal", "object": {"kind":"FenceAgentsRemediation","namespace":"openshift-workload-availability","name":"worker-0-2","uid":"acb96e5c-2aa3-4ac1-9524-3bd616afb8dc","apiVersion":"fence-agents-remediation.medik8s.io/v1alpha1","resourceVersion":"230657"}, "reason": "FenceAgentExecuted"} 2025-09-03T19:03:22.924024461Z INFO controllers.FenceAgentsRemediation Finish FenceAgentsRemediation Reconcile 2025-09-03T19:03:22.92408917Z INFO controllers.FenceAgentsRemediation Begin FenceAgentsRemediation Reconcile 2025-09-03T19:03:22.924108318Z INFO controllers.FenceAgentsRemediation Check FAR CR's name 2025-09-03T19:03:22.924171241Z INFO controllers.FenceAgentsRemediation A Fence Agent is already running {"Fence Agent": "fence_ipmilan", "Node Name": "worker-0-2", "FAR uid": "acb96e5c-2aa3-4ac1-9524-3bd616afb8dc"} 2025-09-03T19:03:22.932030768Z INFO controllers.FenceAgentsRemediation Finish FenceAgentsRemediation Reconcile 2025-09-03T19:03:27.913296448Z INFO executer command completed {"uid": "acb96e5c-2aa3-4ac1-9524-3bd616afb8dc", "response": "Success: Rebooted\n", "errMessage": "", "err": null} 2025-09-03T19:03:27.913332149Z INFO executer fence agent done {"uid": "acb96e5c-2aa3-4ac1-9524-3bd616afb8dc", "fence_agent": "fence_ipmilan", "stdout": "Success: Rebooted\n", "stderr": "", "err": null} 2025-09-03T19:03:27.913337395Z INFO executer updating status {"FAR uid": "acb96e5c-2aa3-4ac1-9524-3bd616afb8dc"} 2025-09-03T19:03:27.913415502Z INFO executer Updating Status Condition {"processingConditionStatus": "", "fenceAgentActionSucceededConditionStatus": "True", "succeededConditionStatus": "", "reason": "FenceAgentSucceeded", "LastUpdateTime": "2025-09-03T19:03:27.913414055Z"} 2025-09-03T19:03:27.913485773Z DEBUG events [remediation] Fence agent was succeeded {"type": "Normal", "object": {"kind":"FenceAgentsRemediation","namespace":"openshift-workload-availability","name":"worker-0-2","uid":"acb96e5c-2aa3-4ac1-9524-3bd616afb8dc","apiVersion":"fence-agents-remediation.medik8s.io/v1alpha1","resourceVersion":"230657"}, "reason": "FenceAgentSucceeded"} 2025-09-03T19:03:27.918773446Z INFO executer status updated {"FAR uid": "acb96e5c-2aa3-4ac1-9524-3bd616afb8dc"} 2025-09-03T19:03:27.918981376Z INFO controllers.FenceAgentsRemediation Begin FenceAgentsRemediation Reconcile 2025-09-03T19:03:27.919030993Z INFO controllers.FenceAgentsRemediation Check FAR CR's name 2025-09-03T19:03:27.91909143Z INFO controllers.FenceAgentsRemediation Remediation strategy is ResourceDeletion which explicitly deletes resources - manually deleting workload {"Node Name": "worker-0-2"} 2025-09-03T19:03:27.919329742Z DEBUG events [remediation] Manually delete pods from the unhealthy node {"type": "Normal", "object": {"kind":"Node","name":"worker-0-2","uid":"28702782-f67d-4bc9-9792-a97600872477","apiVersion":"v1","resourceVersion":"230660"}, "reason": "DeleteResources"} 2025-09-03T19:03:27.91964607Z INFO commons-resource starting to delete pods {"node name": "worker-0-2"} 2025-09-03T19:03:30.376360784Z INFO commons-resource done deleting pods {"node name": "worker-0-2"} 2025-09-03T19:03:30.37639436Z INFO controllers.FenceAgentsRemediation Updating Status Condition {"processingConditionStatus": "False", "fenceAgentActionSucceededConditionStatus": "", "succeededConditionStatus": "True", "reason": "RemediationFinishedSuccessfully", "LastUpdateTime": "2025-09-03T19:03:30.376392963Z"} 2025-09-03T19:03:30.376411314Z INFO executer cancelling fence agent routine {"uid": "acb96e5c-2aa3-4ac1-9524-3bd616afb8dc"} 2025-09-03T19:03:30.376419896Z INFO controllers.FenceAgentsRemediation FenceAgentsRemediation CR has completed to remediate the node {"Node Name": "worker-0-2"} 2025-09-03T19:03:30.376549157Z DEBUG events [remediation] Unhealthy node remediation was completed {"type": "Normal", "object": {"kind":"Node","name":"worker-0-2","uid":"28702782-f67d-4bc9-9792-a97600872477","apiVersion":"v1","resourceVersion":"230660"}, "reason": "NodeRemediationCompleted"} 2025-09-03T19:03:30.376584509Z DEBUG events [remediation] Remediation finished {"type": "Normal", "object": {"kind":"FenceAgentsRemediation","namespace":"openshift-workload-availability","name":"worker-0-2","uid":"acb96e5c-2aa3-4ac1-9524-3bd616afb8dc","apiVersion":"fence-agents-remediation.medik8s.io/v1alpha1","resourceVersion":"230701"}, "reason": "RemediationFinished"} 2025-09-03T19:03:30.381526649Z INFO controllers.FenceAgentsRemediation Finish FenceAgentsRemediation Reconcile 2025-09-03T19:03:30.381596109Z INFO controllers.FenceAgentsRemediation Begin FenceAgentsRemediation Reconcile 2025-09-03T19:03:30.381604904Z INFO controllers.FenceAgentsRemediation Check FAR CR's name 2025-09-03T19:03:30.386259186Z INFO controllers.FenceAgentsRemediation Finish FenceAgentsRemediation Reconcile