-
Task
-
Resolution: Unresolved
-
Major
-
None
-
None
-
True
-
False
-
Not Started
-
Not Started
-
Not Started
-
Not Started
-
Not Started
-
Not Started
-
-
-
RHOAM Sprint 64, RHOAM Sprint 65, RHOAM Sprint 66, RHOAM Sprint 67, RHOAM Sprint 68, RHOAM Sprint 69, RHOAM Sprint 70, RHOAM Sprint 71
We need porta and zync to be able to load TLS details to connect to external DBs. In particular, we need:
- CA certificate
- Client certificate
- Client private key
Those can be mounted wherever the operator decides, but the operator should set the next environment variables to inform zync and porta where the files are:
- DATABASE_SSL_CA for the CA certificate
- DATABASE_SSL_CERT for the client certificate
- DATABASE_SSL_KEY for the client key
The client might want to specify a TLS protection level as well:
- DATABASE_SSL_MODE. Values: Mysql ref., Postgres ref.
The client should create the next resources
- TLS Secret (docs) for the CA certificate
- TLS Secret for the client cert and key
- Would look like this:
apiVersion: v1
kind: Secret
metadata:
name: secret-tls
type: kubernetes.io/tls
data:
# values are base64 encoded, which obscures them but does NOT provide
# any useful level of confidentiality
tls.crt: |
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNVakNDQWJzQ0FnMytNQTBHQ1NxR1NJYjNE
UUVCQlFVQU1JR2JNUXN3Q1FZRFZRUUdFd0pLVURFT01Bd0cKQTFVRUNCTUZWRzlyZVc4eEVEQU9C
Z05WQkFjVEIwTm9kVzh0YTNVeEVUQVBCZ05WQkFvVENFWnlZVzVyTkVSRQpNUmd3RmdZRFZRUUxF
dzlYWldKRFpYSjBJRk4xY0hCdmNuUXhHREFXQmdOVkJBTVREMFp5WVc1ck5FUkVJRmRsCllpQkRR
VEVqTUNFR0NTcUdTSWIzRFFFSkFSWVVjM1Z3Y0c5eWRFQm1jbUZ1YXpSa1pDNWpiMjB3SGhjTk1U
TXcKTVRFeE1EUTFNVE01V2hjTk1UZ3dNVEV3TURRMU1UTTVXakJMTVFzd0NRWURWUVFHREFKS1VE
RVBNQTBHQTFVRQpDQXdHWEZSdmEzbHZNUkV3RHdZRFZRUUtEQWhHY21GdWF6UkVSREVZTUJZR0Ex
VUVBd3dQZDNkM0xtVjRZVzF3CmJHVXVZMjl0TUlHYU1BMEdDU3FHU0liM0RRRUJBUVVBQTRHSUFE
Q0JoQUo5WThFaUhmeHhNL25PbjJTbkkxWHgKRHdPdEJEVDFKRjBReTliMVlKanV2YjdjaTEwZjVN
Vm1UQllqMUZTVWZNOU1vejJDVVFZdW4yRFljV29IcFA4ZQpqSG1BUFVrNVd5cDJRN1ArMjh1bklI
QkphVGZlQ09PekZSUFY2MEdTWWUzNmFScG04L3dVVm16eGFLOGtCOWVaCmhPN3F1TjdtSWQxL2pW
cTNKODhDQXdFQUFUQU5CZ2txaGtpRzl3MEJBUVVGQUFPQmdRQU1meTQzeE15OHh3QTUKVjF2T2NS
OEtyNWNaSXdtbFhCUU8xeFEzazlxSGtyNFlUY1JxTVQ5WjVKTm1rWHYxK2VSaGcwTi9WMW5NUTRZ
RgpnWXcxbnlESnBnOTduZUV4VzQyeXVlMFlHSDYyV1hYUUhyOVNVREgrRlowVnQvRGZsdklVTWRj
UUFEZjM4aU9zCjlQbG1kb3YrcE0vNCs5a1h5aDhSUEkzZXZ6OS9NQT09Ci0tLS0tRU5EIENFUlRJ
RklDQVRFLS0tLS0K
# In this example, the key data is not a real PEM-encoded private key
tls.key: |
RXhhbXBsZSBkYXRhIGZvciB0aGUgVExTIGNydCBmaWVsZA==
Then, we would need some yaml keywords so the user can mount the files easily. This is a suggestion:
apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata:
name: apimanager1
spec:
system:
dbCaCertRef:
name: my-secret-with-ca-cert
dbClientCertRef:
name: my-secret-with-client-cert
dbClientTlsMode:verify-ca
zync:
dbCaCertRef:
name: my-secret-with-ca-cert
dbClientCertRef:
name: my-secret-with-client-cert
dbClientTlsMode:verify-ca
This should work for zync and system:
- Zync: listener and que pods
- System: app and sidekiq pods
- blocks
-
-
- New
-
- is related to
-
THREESCALE-4185 Enable ssl client certificate while connecting to external databases.
-
- To Test (QE)
-
- links to
- mentioned on
[THREESCALE-11156] Add a way for the user to provide DBs TLS certs and keys for porta and zync
GitLab CEE Bot
added a comment - CPaaS Service Account mentioned this issue in merge request !1814 of 3scale / 3scale Operator Midstream on branch 3scale-amp-2_upstream_06693104f0418bb10cccb27cf38e0d4d:
Updated US source to: c63db3f Merge pull request #1026 from austincunningham/THREESCALE-11156
GitLab CEE Bot
added a comment - CPaaS Service Account mentioned this issue in merge request !1814 of 3scale / 3scale Operator Midstream on branch 3scale-amp-2_ upstream _06693104f0418bb10cccb27cf38e0d4d : Updated US source to: c63db3f Merge pull request #1026 from austincunningham/ THREESCALE-11156 Mounting the certs has proven to be an issue as when we mount from a secret the permissions on the files are 777 and SSL/TLS doesn't like this and is looking for 0600 to be set on the cert files.
There are two solutions
1. Use SCC and DefaultMode on the volume
this would be the preferred method if we get it to work.
Investigation here with the zync-que doesn't look like it worked.
issue was that secret volume mounts are owned by root an the pods user didn't have permission to read the cert . SCC with 0640 permissions works for the zync and zync-que ,but system app still has an issue with permissions as adding permission to read to the pod user is too open.
System-app-pre failing on permissions
ActiveRecord::ConnectionNotEstablished: private key file "/tls/tls.key" has group or world access; permissions should be u=rw (0600) or less
Looks like the permissions are set 0640
sh-4.4$ ls -la total 0 drwxrwsrwt. 3 root 1001060000 140 Nov 29 13:47 . dr-xr-xr-x. 1 root root 39 Nov 29 13:47 .. drwxr-sr-x. 2 root 1001060000 100 Nov 29 13:47 ..2024_11_29_13_47_23.3640333220 lrwxrwxrwx. 1 root 1001060000 32 Nov 29 13:47 ..data -> ..2024_11_29_13_47_23.3640333220 lrwxrwxrwx. 1 root 1001060000 13 Nov 29 13:47 ca.crt -> ..data/ca.crt lrwxrwxrwx. 1 root 1001060000 14 Nov 29 13:47 tls.crt -> ..data/tls.crt lrwxrwxrwx. 1 root 1001060000 14 Nov 29 13:47 tls.key -> ..data/tls.key sh-4.4$ cd ..data sh-4.4$ ls -la total 12 drwxr-sr-x. 2 root 1001060000 100 Nov 29 13:47 . drwxrwsrwt. 3 root 1001060000 140 Nov 29 13:47 .. -rw-r-----. 1 root 1001060000 1107 Nov 29 13:47 ca.crt -rw-r-----. 1 root 1001060000 993 Nov 29 13:47 tls.crt -rw-r-----. 1 root 1001060000 1704 Nov 29 13:47 tls.key sh-4.4$
If we take the error for what it says, I don't see a way around this with SCC as root will always own the secret volume mount files and there is no way to give permission to the container user/group to allow the container to be able to read the files.
2. Use initContainers to CHMOD the files
Have successfully managed to get this working although it requires and extra image and volume sample code
InitContainers: []v1.Container{
{
Name: "set-permissions",
Image: ZyncUbiImage, // Minimal image for chmod
Command: []string{
"sh",
"-c",
"cp /tls/* /writable-tls/ && chmod 0600 /writable-tls/*",
},
VolumeMounts: []v1.VolumeMount{
{
Name: "tls-secret",
MountPath: "/tls",
ReadOnly: true,
},
{
Name: "writable-tls",
MountPath: "/writable-tls",
ReadOnly: false, // Writable emptyDir volume
},
},
},
}
and volume and volumeMount
VolumeMounts: []v1.VolumeMount{
{
Name: "writable-tls",
MountPath: "/tls",
ReadOnly: true,
},
},
},
},
Volumes: []v1.Volume{
{
Name: "tls-secret",
VolumeSource: v1.VolumeSource{
Secret: &v1.SecretVolumeSource{
SecretName: ZyncSecretName, // Name of the secret containing the TLS certs
DefaultMode: int32Ptr(0600),
Items: []v1.KeyToPath{
{
Key: "SSL_CA",
Path: "ca.crt", // Map the secret key to the ca.crt file in the container
},
{
Key: "SSL_CERT",
Path: "tls.crt", // Map the secret key to the tls.crt file in the container
},
{
Key: "SSL_KEY",
Path: "tls.key", // Map the secret key to the tls.key file
},
},
},
},
},
{
Name: "writable-tls",
VolumeSource: v1.VolumeSource{
EmptyDir: &v1.EmptyDirVolumeSource{},
},
Must investigate the included images in the operator bundle to see If I can avoid adding another image.
yep there is an oc_cli image which works
- name: RELATED_IMAGE_OC_CLI value: "quay.io/openshift/origin-cli:4.7"
tested existing images and the all work for the init container chmod command .
Austin Cunningham
added a comment - - edited Mounting the certs has proven to be an issue as when we mount from a secret the permissions on the files are 777 and SSL/TLS doesn't like this and is looking for 0600 to be set on the cert files.
There are two solutions
1. Use SCC and DefaultMode on the volume
this would be the preferred method if we get it to work.
Investigation here with the zync-que doesn't look like it worked.
issue was that secret volume mounts are owned by root an the pods user didn't have permission to read the cert . SCC with 0640 permissions works for the zync and zync-que ,but system app still has an issue with permissions as adding permission to read to the pod user is too open.
System-app-pre failing on permissions
ActiveRecord::ConnectionNotEstablished: private key file "/tls/tls.key" has group or world access; permissions should be u=rw (0600) or less
Looks like the permissions are set 0640
sh-4.4$ ls -la
total 0
drwxrwsrwt. 3 root 1001060000 140 Nov 29 13:47 .
dr-xr-xr-x. 1 root root 39 Nov 29 13:47 ..
drwxr-sr-x. 2 root 1001060000 100 Nov 29 13:47 ..2024_11_29_13_47_23.3640333220
lrwxrwxrwx. 1 root 1001060000 32 Nov 29 13:47 ..data -> ..2024_11_29_13_47_23.3640333220
lrwxrwxrwx. 1 root 1001060000 13 Nov 29 13:47 ca.crt -> ..data/ca.crt
lrwxrwxrwx. 1 root 1001060000 14 Nov 29 13:47 tls.crt -> ..data/tls.crt
lrwxrwxrwx. 1 root 1001060000 14 Nov 29 13:47 tls.key -> ..data/tls.key
sh-4.4$ cd ..data
sh-4.4$ ls -la
total 12
drwxr-sr-x. 2 root 1001060000 100 Nov 29 13:47 .
drwxrwsrwt. 3 root 1001060000 140 Nov 29 13:47 ..
-rw-r-----. 1 root 1001060000 1107 Nov 29 13:47 ca.crt
-rw-r-----. 1 root 1001060000 993 Nov 29 13:47 tls.crt
-rw-r-----. 1 root 1001060000 1704 Nov 29 13:47 tls.key
sh-4.4$
If we take the error for what it says, I don't see a way around this with SCC as root will always own the secret volume mount files and there is no way to give permission to the container user/group to allow the container to be able to read the files.
2. Use initContainers to CHMOD the files
Have successfully managed to get this working although it requires and extra image and volume sample code
InitContainers: []v1.Container{
{
Name: "set-permissions" ,
Image: ZyncUbiImage, // Minimal image for chmod
Command: [] string {
"sh" ,
"-c" ,
"cp /tls/* /writable-tls/ && chmod 0600 /writable-tls/*" ,
},
VolumeMounts: []v1.VolumeMount{
{
Name: "tls-secret" ,
MountPath: "/tls" ,
ReadOnly: true,
},
{
Name: "writable-tls" ,
MountPath: "/writable-tls" ,
ReadOnly: false, // Writable emptyDir volume
},
},
},
}
and volume and volumeMount
VolumeMounts: []v1.VolumeMount{
{
Name: "writable-tls" ,
MountPath: "/tls" ,
ReadOnly: true,
},
},
},
},
Volumes: []v1.Volume{
{
Name: "tls-secret" ,
VolumeSource: v1.VolumeSource{
Secret: &v1.SecretVolumeSource{
SecretName: ZyncSecretName, // Name of the secret containing the TLS certs
DefaultMode: int32Ptr(0600),
Items: []v1.KeyToPath{
{
Key: "SSL_CA" ,
Path: "ca.crt" , // Map the secret key to the ca.crt file in the container
},
{
Key: "SSL_CERT" ,
Path: "tls.crt" , // Map the secret key to the tls.crt file in the container
},
{
Key: "SSL_KEY" ,
Path: "tls.key" , // Map the secret key to the tls.key file
},
},
},
},
},
{
Name: "writable-tls" ,
VolumeSource: v1.VolumeSource{
EmptyDir: &v1.EmptyDirVolumeSource{},
},
Must investigate the included images in the operator bundle to see If I can avoid adding another image.
yep there is an oc_cli image which works
- name: RELATED_IMAGE_OC_CLI
value: "quay.io/openshift/origin-cli: 4 . 7 "
tested existing images and the all work for the init container chmod command .
Thanks ckyrillo@redhat.com don't think the tls flags in the apimanager CR add anything other that forcing users to go tls on both redis and postgres/mysql/oracle db at the same time if set. Although this may not be a bad thing.
Austin Cunningham
added a comment - - edited Thanks ckyrillo@redhat.com don't think the tls flags in the apimanager CR add anything other that forcing users to go tls on both redis and postgres/mysql/oracle db at the same time if set. Although this may not be a bad thing. aucunnin@redhat.com See this comment for a summary of changes to the work supporting Redis TLS for porta and backend. We want to use a similar approach for this ticket and for THREESCALE-11021.
Postgres
tried aws rds but it doesn't look like it can be setup with a cert/key pair, SSL/TLS is via the Root CA and sslmode
e.g.
psql "host=5555555.cqmwu31sxk5u.us-east-1.rds.amazonaws.com port=5432 dbname=yourdbname user=youruser password=yourpassword sslmode=verify-full sslrootcert=us-east-1-bundle.pem" psql (15.6, server 13.13) SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, compression: off) Type "help" for help. postgres=>
should be enough to test the ssl connection with the env vars mounted in the deployments.
Austin Cunningham
added a comment - - edited Postgres
tried aws rds but it doesn't look like it can be setup with a cert/key pair, SSL/TLS is via the Root CA and sslmode
e.g.
psql "host=5555555.cqmwu31sxk5u.us-east-1.rds.amazonaws.com port=5432 dbname=yourdbname user=youruser password=yourpassword sslmode=verify-full sslrootcert=us-east-1-bundle.pem"
psql (15.6, server 13.13)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, compression: off)
Type "help" for help.
postgres=>
should be enough to test the ssl connection with the env vars mounted in the deployments. Austin Cunningham
added a comment - I tend to agree to using the existing secrets system-database and zync makes most sense.
Austin Cunningham
added a comment - I tend to agree to using the existing secrets system-database and zync makes most sense. GitLab CEE Bot
added a comment - CPaaS Service Account mentioned this issue in a merge request of 3scale / Zync Midstream on branch 3scale-amp-2_upstream_7ced39789125120c7e23fecf8719e155:
Updated US source to: 5b08c2f THREESCALE-4185: Support TLS protected PostgreSQL (#532)
GitLab CEE Bot
added a comment - CPaaS Service Account mentioned this issue in a merge request of 3scale / Zync Midstream on branch 3scale-amp-2_ upstream _7ced39789125120c7e23fecf8719e155 : Updated US source to: 5b08c2f THREESCALE-4185 : Support TLS protected PostgreSQL (#532) GitLab CEE Bot
added a comment - CPaaS Service Account mentioned this issue in a merge request of 3scale / Porta Midstream on branch 3scale-amp-2_upstream_a5f0e5cb5d62c028ffd39a180996fa40:
Updated US source to: 2c43706 THREESCALE-4185: Support TLS protection for external DBs (#3846)
GitLab CEE Bot
added a comment - CPaaS Service Account mentioned this issue in a merge request of 3scale / Porta Midstream on branch 3scale-amp-2_ upstream _a5f0e5cb5d62c028ffd39a180996fa40 : Updated US source to: 2c43706 THREESCALE-4185 : Support TLS protection for external DBs (#3846) My suggestion is not to use dedicated secrets but add keys to the existing database secrets. Moreover AFAIU the tls secret type requires a crt and a key. But for CA, one only has a CRT and it doesn't have to be kept secret anyway. I think keeping database configuration in one place would be much more obvious and convenient for users.
– just my 2 cents
CPaaS Service Account mentioned this issue in merge request !1826 of 3scale / 3scale Operator Midstream on branch 3scale-mas-rhel-9_upstream_a81c3501e9e9ae7f4fec65e29ccc5ce9: