While provisioning RDS Proxy (APPSRE-10773) it came to my attention that the RDS Aurora Database has been configured with a DB Subnet Group which includes 6 subnets (out of 9) which do not have routes back to the Quay clusters

The reader instance in us-east-1 (instance name ending in -1) currently is set up with an IP address that is on a subnet that should not allow network connectivity back to the quay cluster. For some unknown reason network connectivity does work

The previous MySQL database was configured to use only the 3 subnets named "ProtectedSubnet" which are the 3 subnets we should use on the DB Subnet Group

It is possible that in the future an instance may get moved to a different AZ and the subnet group used may lead to connectivity issues

To be safe, we should update the subnet group to only include the 3 subnets named "ProtectedSubnet". This however require removing the reader instance as one of the subnets we want to remove is currently in use by that instance

We have traffic coming from the quayp05ue1 cluster AWS account through VPC peering pcx-0f2735b0bf4a6668e

From the cluster we are able to connect to RDS instances in account quayio-prod

The instances:

• quayio-aurora-prod-0 10.18.44.55
  quayio-aurora-prod-1 10.18.21.232

Route tables are set on the cluster's AWS account's VPC 

The Aurora DB instances were created using a DB Subnet Group comprised of 9 subnets (by mistake, it should have been the 3 named ProtectedSubnet)

The subnets named ProtectedSubnets are assigned to a route table which have a route to 10.25.0.0/16 to send traffic back through the VPC peering connection

However the other subnets don't have such a route

The situation we are facing is that we are able to connect to both instances from the quayp05ue1 cluster, even though from a networking/routing perspective it appears it should not be possible to connect to the second instance (one ending in -1) due to it's IP address's subnet not having a route entry sending traffic back through the VPC peering connection