glance_api and glance_internal_api can lose access to /var/lib/cinder/tmp/image_cache_db_init

The problem the customer encountered was triggered by the following sequence of events:

The glance-api service has been deployed with its image cache enabled (GlanceCacheEnabled: true) AND configured to use cinder for its storage backend. 

Pacemaker restarts the cinder-volume service on a controller where the service has not been run before (i.e. cinder-volume is running on that controller for the first time)

Details:

Event #1 is what initially created glance's /var/lib/cinder/tmp/image_cache_db_init lock file, and it would be owned by glance:glance (42415:42415) with 0640 permission.

The upgrade process (from 17.1.4 to 17.1.8) requires pacemaker to reschedule the cinder-volume service on controllers where the service has never run before.

Event #2 cased the cinder-volume service to recursively modify the ownership of all files in the /var/lib/cinder directory to cinder:kolla (42407:42400).

The glance service is a member of the kolla group (42400), but the lock file's permission (0640) meant that glance could read the file but not write to it.

The glance service would fail to start because it no longer had write access to a lock file.

The problem can be resolved by changing the lock file's ownership back to glance:glance (42415:42415) OR by setting the permissions to 0660 so that members of the kolla group have write access, and then restarting the glance-api and glance-api-internal services.