Reopened - please see comments.

Reported on Version: 2.2 on-premises in the support case

When an application is added admin portal, the client id was not registered in RH-SSO.
The system-sidekiq log contain "ZyncWorker:UnprocessableEntityError" in the format shown below.

2018-10-22T02:09:23.691124839Z 2018-10-22T02:09:23.690Z 1 TID-y2t00 WARN: {"context":"Job raised exception","job":{"class":"ZyncWorker","args":["....",error_message":"Expected successful response. Got 422","error_class":"ZyncWorker::UnprocessableEntityError"

The way this is implemented is that System sends events to Zync to synchronize the client with RH-SSO. An application has dependencies for eg on a service, account etc. System sends multiple events for creating the application and dependencies, but these can be out of order. If there is an error in creating because of dependency, this raises the 422 UnprocessableEntityError . System hooks into the error, publishes the dependencies and retries until Zync creates all of them. The maximum no of tries is currently set to 3 . See https://github.com/3scale/system/blob/master/app/workers/zync_worker.rb#L12. However, 3 might be too small a number and so should it be increased ? The workaround is to trigger update of the sso by doing any update to the application on the admin portal or just an empty api call to update the application.

From mnoyabon,
Yesterday Hery and I could reproduce this bug. We weren't trying to reproduce it, we were working on something else but we needed zync and we encountered it. We 'solved' it locally by adding to the Service model has_many :proxies, but we assume that if it is not there is for a reason so I just say this for anybody who checks at it, to don't have to look at all of it from scratch and have the clue the we have.

Please read the descriptions of the linked issues.

Dev notes

My guess (hery) is that the proxy is not updated so it is not created in zync
Probably causing the unprocessable entity because an application needs a Proxy (which might be a wrong assertion)

Other path to investigate, We have some message bus timeout errors in production, they might be related