With the secret/configmap watching that Clowder is going to roll out to stage/prod very soon, care needs to be taken to ensure that apps don't end up in a bad state, as they have in ephemeral. One team referenced a database secret that was created by clowder in their clowdapp deployment. The normal flow (pre-change) would have gone like this

1. ClowdApp reconciled
2. Secret for database created
3. Deployment uses secret to populate env vars
4. App successfully comes up

With the secret/config watching, if a secret is down as being required, it must be present at the time of reconciliation. What was happening with the app in question was this.

1. Clowdapp tries to read secret to generate hash
2. Fails......secret not present, hence no database secret created, hence never completes

The app has now changed how they are grabbing their database credentials and are getting them from Clowders cdappconfig, which is the right way to do it.

To avoid this happening to other apps, an audit should be undertaken to ensure that no other apps are doing the same.