Loading...

Type: Bug
Resolution: Unresolved
Priority: Undefined
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:
- oap-plan

Story Points:
3
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Intelligence Requested:
Market:

Sprint:
OAPE Sprint 277, OAPE Sprint 278
sprint_count:
2

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

PX Impact Score:

When deploying ESO via gitops sometimes the operator does not reconcile the externalsecrets.openshift.operator.io object

This is a bit timing dependent, but I can reproduce this relatively often (50%
of the times or so, so far). I believe the sequence of events is:
1. Apply subscription+operatorgroup+externalsecrets.openshift.operator.io/cluster+clustersecretstore in a single ArgoCD application
2. Everything gets applied eventually applied correctly
3. The clustersecretstore (pointing to a vault installation), never becomes available

The logs of the ESO operator pod in the external-secrets-operator ns are:

I0908 14:40:16.455870       1 main.go:158] "starting the controller manager" logger="setup"
I0908 14:40:16.456396       1 server.go:83] "starting server" name="health probe" addr="[::]:8081"
I0908 14:40:16.456686       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-manager" controllerGroup="operator.openshift.io" controllerKind="ExternalSecretsManager" source="kind source: *v1alpha1.ExternalSecrets"
I0908 14:40:16.457146       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1alpha1.ExternalSecretsManager"
I0908 14:40:16.457326       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-manager" controllerGroup="operator.openshift.io" controllerKind="ExternalSecretsManager" source="kind source: *v1alpha1.ExternalSecretsManager"
I0908 14:40:16.457442       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1.RoleBinding"
I0908 14:40:16.457580       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1.Secret"
I0908 14:40:16.457607       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1alpha1.ExternalSecrets"
I0908 14:40:16.457633       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1.ClusterRole"
I0908 14:40:16.457669       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1.Role"
I0908 14:40:16.457711       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1.Deployment"
I0908 14:40:16.457807       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1.ClusterRoleBinding"
I0908 14:40:16.457854       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1.Service"
I0908 14:40:16.457861       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1.ServiceAccount"
I0908 14:40:16.457943       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1.ValidatingWebhookConfiguration"
I0908 14:40:16.928739       1 controller.go:233] "Starting Controller" logger="operator-manager" controller="external-secrets-manager" controllerGroup="operator.openshift.io" controllerKind="ExternalSecretsManager"
I0908 14:40:16.928815       1 controller.go:242] "Starting workers" logger="operator-manager" controller="external-secrets-manager" controllerGroup="operator.openshift.io" controllerKind="ExternalSecretsManager" worker count=1
I0908 14:40:16.928915       1 controller.go:108] "reconciling" logger="external-secrets-manager" request="/cluster"
I0908 14:40:16.932400       1 controller.go:233] "Starting Controller" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets"
I0908 14:40:16.932451       1 controller.go:242] "Starting workers" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" worker count=1
I0908 14:40:16.941028       1 controller.go:148] "externalsecrets.openshift.operator.io object not found, skipping reconciliation" logger="external-secrets-manager" key="/cluster"
I0908 14:40:16.941103       1 controller.go:108] "reconciling" logger="external-secrets-manager" request="/cluster"
I0908 14:40:16.941141       1 controller.go:148] "externalsecrets.openshift.operator.io object not found, skipping reconciliation" logger="external-secrets-manager" key="/cluster"
I0908 14:41:55.046639       1 controller.go:330] "reconciling" logger="external-secrets-controller" request="/cluster"
I0908 14:41:55.046712       1 controller.go:339] "externalsecrets.openshift.operator.io object not found, skipping reconciliation" logger="external-secrets-controller" request="/cluster"
I0908 14:41:55.046982       1 controller.go:108] "reconciling" logger="external-secrets-manager" request="/cluster"

The ExternalSecrets.operator.openshift.io/cluster object is the one below and has been applied at 14:41:55 (likely after the last log line above):

apiVersion: operator.openshift.io/v1alpha1
kind: ExternalSecrets
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"operator.openshift.io/v1alpha1","kind":"ExternalSecrets","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"external-secrets-operator","argocd.argoproj.io/instance":"golang-external-secrets"},"name":"cluster"},"spec":{}}
  creationTimestamp: '2025-09-08T14:41:55Z'
  generation: 1
  labels:
    app.kubernetes.io/name: external-secrets-operator
    argocd.argoproj.io/instance: golang-external-secrets
  managedFields:
    - apiVersion: operator.openshift.io/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:kubectl.kubernetes.io/last-applied-configuration': {}
          'f:labels':
            .: {}
            'f:app.kubernetes.io/name': {}
            'f:argocd.argoproj.io/instance': {}
        'f:spec': {}
      manager: argocd-controller
      operation: Update
      time: '2025-09-08T14:41:55Z'
  name: cluster
  resourceVersion: '28266'
  uid: f4b7bfa6-c5a2-439e-8111-3effd26e0352
spec: {}

I waited for >30 minutes and nothing happened. Then I just deleted the pod and everything worked. After the operator pod removal these were the logs:

I0908 15:15:38.283519       1 main.go:158] "starting the controller manager" logger="setup"
I0908 15:15:38.284465       1 server.go:83] "starting server" name="health probe" addr="[::]:8081"
I0908 15:15:38.285126       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-manager" controllerGroup="operator.openshift.io" controllerKind="ExternalSecretsManager" source="kind source: *v1alpha1.ExternalSecrets"
I0908 15:15:38.285283       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-manager" controllerGroup="operator.openshift.io" controllerKind="ExternalSecretsManager" source="kind source: *v1alpha1.ExternalSecretsManager"
I0908 15:15:38.285319       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1alpha1.ExternalSecretsManager"
I0908 15:15:38.285385       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1.RoleBinding"
I0908 15:15:38.285419       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1alpha1.ExternalSecrets"
I0908 15:15:38.285468       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1.Secret"
I0908 15:15:38.285500       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1.ClusterRoleBinding"
I0908 15:15:38.285449       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1.ClusterRole"
I0908 15:15:38.285545       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1.Deployment"
I0908 15:15:38.285546       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1.Service"
I0908 15:15:38.285582       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1.ServiceAccount"
I0908 15:15:38.285588       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1.Role"
I0908 15:15:38.285653       1 controller.go:198] "Starting EventSource" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" source="kind source: *v1.ValidatingWebhookConfiguration"
I0908 15:15:38.734261       1 controller.go:233] "Starting Controller" logger="operator-manager" controller="external-secrets-manager" controllerGroup="operator.openshift.io" controllerKind="ExternalSecretsManager"
I0908 15:15:38.734327       1 controller.go:242] "Starting workers" logger="operator-manager" controller="external-secrets-manager" controllerGroup="operator.openshift.io" controllerKind="ExternalSecretsManager" worker count=1
I0908 15:15:38.734407       1 controller.go:108] "reconciling" logger="external-secrets-manager" request="/cluster"
I0908 15:15:38.734552       1 controller.go:233] "Starting Controller" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets"
I0908 15:15:38.734562       1 controller.go:242] "Starting workers" logger="operator-manager" controller="external-secrets-controller" controllerGroup="operator.openshift.io" controllerKind="ExternalSecrets" worker count=1
I0908 15:15:38.734591       1 controller.go:330] "reconciling" logger="external-secrets-controller" request="/cluster"
I0908 15:15:38.753115       1 controller.go:387] "starting reconciliation of newly created externalsecrets.openshift.operator.io" logger="external-secrets-controller" namespace="" name="cluster"
I0908 15:15:38.789694       1 recorder.go:104] "Created serviceaccount external-secrets/external-secrets" logger="operator-manager.events" type="Normal" object={"kind":"ExternalSecrets","name":"cluster","uid":"f4b7bfa6-c5a2-439e-8111-3effd26e0352","apiVersion":"operator.openshift.io/v1alpha1","resourceVersion":"37745"} reason="Reconciled"
I0908 15:15:38.814773       1 recorder.go:104] "Created serviceaccount external-secrets/external-secrets-webhook" logger="operator-manager.events" type="Normal" object={"kind":"ExternalSecrets","name":"cluster","uid":"f4b7bfa6-c5a2-439e-8111-3effd26e0352","apiVersion":"operator.openshift.io/v1alpha1","resourceVersion":"37745"} reason="Reconciled"
I0908 15:15:38.824884       1 recorder.go:104] "Created serviceaccount external-secrets/external-secrets-cert-controller" logger="operator-manager.events" type="Normal" object={"kind":"ExternalSecrets","name":"cluster","uid":"f4b7bfa6-c5a2-439e-8111-3effd26e0352","apiVersion":"operator.openshift.io/v1alpha1","resourceVersion":"37745"} reason="Reconciled"
I0908 15:15:38.829177       1 recorder.go:104] "secret resource external-secrets/external-secrets-webhook created" logger="operator-manager.events" type="Normal" object={"kind":"ExternalSecrets","name":"cluster","uid":"f4b7bfa6-c5a2-439e-8111-3effd26e0352","apiVersion":"operator.openshift.io/v1alpha1","resourceVersion":"37745"} reason="Reconciled"
I0908 15:15:38.933858       1 recorder.go:104] "clusterrole resource external-secrets-controller created" logger="operator-manager.events" type="Normal" object={"kind":"ExternalSecrets","name":"cluster","uid":"f4b7bfa6-c5a2-439e-8111-3effd26e0352","apiVersion":"operator.openshift.io/v1alpha1","resourceVersion":"37745"} reason="Reconciled"
I0908 15:15:39.039215       1 recorder.go:104] "clusterrole resource external-secrets-edit created" logger="operator-manager.events" type="Normal" object={"kind":"ExternalSecrets","name":"cluster","uid":"f4b7bfa6-c5a2-439e-8111-3effd26e0352","apiVersion":"operator.openshift.io/v1alpha1","resourceVersion":"37745"} reason="Reconciled"
I0908 15:15:39.119391       1 recorder.go:104] "clusterrole resource external-secrets-servicebindings created" logger="operator-manager.events" type="Normal" object={"kind":"ExternalSecrets","name":"cluster","uid":"f4b7bfa6-c5a2-439e-8111-3effd26e0352","apiVersion":"operator.openshift.io/v1alpha1","resourceVersion":"37745"} reason="Reconciled"
I0908 15:15:39.232909       1 recorder.go:104] "clusterrole resource external-secrets-view created" logger="operator-manager.events" type="Normal" object={"kind":"ExternalSecrets","name":"cluster","uid":"f4b7bfa6-c5a2-439e-8111-3effd26e0352","apiVersion":"operator.openshift.io/v1alpha1","resourceVersion":"37745"} reason="Reconciled"
I0908 15:15:39.377424       1 recorder.go:104] "clusterrolebinding resource external-secrets-controller created" logger="operator-manager.events" type="Normal" object={"kind":"ExternalSecrets","name":"cluster","uid":"f4b7bfa6-c5a2-439e-8111-3effd26e0352","apiVersion":"operator.openshift.io/v1alpha1","resourceVersion":"37745"} reason="Reconciled"
I0908 15:15:39.477760       1 recorder.go:104] "role resource external-secrets/external-secrets-leaderelection created" logger="operator-manager.events" type="Normal" object={"kind":"ExternalSecrets","name":"cluster","uid":"f4b7bfa6-c5a2-439e-8111-3effd26e0352","apiVersion":"operator.openshift.io/v1alpha1","resourceVersion":"37745"} reason="Reconciled"
I0908 15:15:39.577469       1 recorder.go:104] "rolebinding resource external-secrets/external-secrets-leaderelection created" logger="operator-manager.events" type="Normal" object={"kind":"ExternalSecrets","name":"cluster","uid":"f4b7bfa6-c5a2-439e-8111-3effd26e0352","apiVersion":"operator.openshift.io/v1alpha1","resourceVersion":"37745"} reason="Reconciled"
I0908 15:15:39.632741       1 recorder.go:104] "clusterrole resource external-secrets-cert-controller created" logger="operator-manager.events" type="Normal" object={"kind":"ExternalSecrets","name":"cluster","uid":"f4b7bfa6-c5a2-439e-8111-3effd26e0352","apiVersion":"operator.openshift.io/v1alpha1","resourceVersion":"37745"} reason="Reconciled"
I0908 15:15:39.722236       1 recorder.go:104] "clusterrolebinding resource external-secrets-cert-controller created" logger="operator-manager.events" type="Normal" object={"kind":"ExternalSecrets","name":"cluster","uid":"f4b7bfa6-c5a2-439e-8111-3effd26e0352","apiVersion":"operator.openshift.io/v1alpha1","resourceVersion":"37745"} reason="Reconciled"
I0908 15:15:39.743170       1 recorder.go:104] "Service external-secrets/external-secrets-webhook created" logger="operator-manager.events" type="Normal" object={"kind":"ExternalSecrets","name":"cluster","uid":"f4b7bfa6-c5a2-439e-8111-3effd26e0352","apiVersion":"operator.openshift.io/v1alpha1","resourceVersion":"37745"} reason="Reconciled"
I0908 15:15:39.781565       1 recorder.go:104] "deployment resource external-secrets/external-secrets created" logger="operator-manager.events" type="Normal" object={"kind":"ExternalSecrets","name":"cluster","uid":"f4b7bfa6-c5a2-439e-8111-3effd26e0352","apiVersion":"operator.openshift.io/v1alpha1","resourceVersion":"37745"} reason="Reconciled"
I0908 15:15:39.821400       1 recorder.go:104] "deployment resource external-secrets/external-secrets-webhook created" logger="operator-manager.events" type="Normal" object={"kind":"ExternalSecrets","name":"cluster","uid":"f4b7bfa6-c5a2-439e-8111-3effd26e0352","apiVersion":"operator.openshift.io/v1alpha1","resourceVersion":"37745"} reason="Reconciled"
I0908 15:15:39.859281       1 recorder.go:104] "deployment resource external-secrets/external-secrets-cert-controller created" logger="operator-manager.events" type="Normal" object={"kind":"ExternalSecrets","name":"cluster","uid":"f4b7bfa6-c5a2-439e-8111-3effd26e0352","apiVersion":"operator.openshift.io/v1alpha1","resourceVersion":"37745"} reason="Reconciled"
I0908 15:15:39.874361       1 controller.go:108] "reconciling" logger="external-secrets-manager" request="/cluster"
I0908 15:15:39.886908       1 recorder.go:104] "validatingWebhook resource externalsecret-validate created" logger="operator-manager.events" type="Normal" object={"kind":"ExternalSecrets","name":"cluster","uid":"f4b7bfa6-c5a2-439e-8111-3effd26e0352","apiVersion":"operator.openshift.io/v1alpha1","resourceVersion":"37745"} reason="Reconciled"
I0908 15:15:39.911577       1 recorder.go:104] "validatingWebhook resource secretstore-validate created" logger="operator-manager.events" type="Normal" object={"kind":"ExternalSecrets","name":"cluster","uid":"f4b7bfa6-c5a2-439e-8111-3effd26e0352","apiVersion":"operator.openshift.io/v1alpha1","resourceVersion":"37745"} reason="Reconciled"
I0908 15:15:40.005825       1 controller.go:108] "reconciling" logger="external-secrets-manager" request="/cluster"
I0908 15:15:40.006092       1 controller.go:330] "reconciling" logger="external-secrets-controller" request="/cluster"
I0908 15:15:40.013881       1 controller.go:108] "reconciling" logger="external-secrets-manager" request="/cluster"
I0908 15:15:40.030993       1 controller.go:108] "reconciling" logger="external-secrets-manager" request="/cluster"
I0908 15:15:40.051589       1 controller.go:330] "reconciling" logger="external-secrets-controller" request="/cluster"
I0908 15:15:40.051589       1 controller.go:108] "reconciling" logger="external-secrets-manager" request="/cluster"
I0908 15:15:40.069204       1 controller.go:108] "reconciling" logger="external-secrets-manager" request="/cluster"
I0908 15:15:44.741689       1 controller.go:330] "reconciling" logger="external-secrets-controller" request="/cluster"
I0908 15:15:45.238414       1 controller.go:330] "reconciling" logger="external-secrets-controller" request="/cluster"
I0908 15:15:45.665973       1 controller.go:330] "reconciling" logger="external-secrets-controller" request="/cluster"
I0908 15:15:45.681872       1 controller.go:330] "reconciling" logger="external-secrets-controller" request="/cluster"

After which the clustersecretstore started to work correctly and the pods in the `external-secrets` namespace got spawned.

Am also attaching two must-gathers, in case it is useful (one before the operator pod restart and one after).

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

must-gather-after-operator-pod-restart.tgz
18.93 MB
2025/09/08 3:23 PM
must-gather-before-operator-pod-restart.tgz
18.62 MB
2025/09/08 3:23 PM

links to

openshift/enhancements#1835: ESO-101: Revisits and restructures external-secrets-operator APIs for GA

Details

Description

Attachments

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates

PagerDuty

Hide