Description

This Epic links together all stories describing advanced tests that will be built for the Service Binding Operator in release 4.3. The categories of tests will include building:

• Sample operators and using them as tests
• Non-functional tests
• Examples from each scenarios
• New tests from community reported issues and requests

Stories for individual tests will be linked to this Epic.

Test Category:  Sample operators and using them as tests

These tests will verify the operation of the Service Binding Operator with other (backing) operators representing each of the categories of operators available at https://operatorhub.io.

These tests will also:

• Provide us with a set of example operators that can be used to support testing the Service Binding Operator
• Enable us to better understand how operators in each category work - for example, do they dynailcally create secrets or do they rely on pre-existing secrets? Do they support linked to secrets? Do they support the use of secrets stored in external (vault) data sources? What types of attributes do they rely on? (e.g., a monitoring operator might rely on URL or many URLs - both dynamic or static).
• Enable us to identify operator type specific requirements that must be provided by the Service Binding Operator
• Testing to date has focused on using the Service Binding Operator with the PostgreSQL database operator.

Functional and non-functional testing of the Service Binding Operator should be extended to cover backing operators of the types available at https://operatorhub.io - from these categories:

• AI/Machine Learning
• Application Runtime
• Big Data
• Cloud Provider
• Database
• PostgreSQL instance in a Pod: https://github.com/baijum/postgresql-operator 
• RDS instance on AWS https://github.com/operator-backing-service-samples/aws-rds
• Developer Tools
• Integration & Delivery
• Logging & Tracing
• Monitoring
• Networking
• Security
• Storage
• Streaming & Messaging

Test Category: Non-functional tests

To date, our testing with the Service Binding Operator has been functional and single-threaded. Testing will be extended to incldue non-functional testing in these categories:

• Compliance testing
• Documentation testing
• Endurance testing
• Load testing
• Localization testing and Internationalization testing
• Performance testing
• Recovery testing
• Resilience testing
• Security testing
• Scalability testing
• Stress testing
• Usability testing
• Volume testing

Test Category: Examples from each user scenario

For each supported user scenario, create an example in the "examples" directory of the Service Binding Operator github repo.

Suggested Scenarios:

• Backing service has secrets placed in an off-cluster vault service. Service Binding Operator is able to bind the secret. https://github.com/redhat-developer/service-binding-operator/issues/74
• Backing service w/TLS-based auth and User CRD
  https://github.com/redhat-developer/service-binding-operator/issues/86
• Single binding request for binding multiple backing services (resources)
  https://github.com/redhat-developer/service-binding-operator/issues/96
• Binding backing service with CRs link to Secrets/ConfigMaps using CR name-based conventions https://github.com/redhat-developer/service-binding-operator/issues/112

Test Category: New tests from community reported issues and requests

New user scenarios and tests should be created from the issues and feature requests contributed from the community: https://github.com/redhat-developer/service-binding-operator/issues

Suggested test scenarios:

• Permute order of Backing service, Application and Service Binding Request to make sure binding works. https://github.com/redhat-developer/service-binding-operator/issues/69
• In case Service Binding Operator stores the its secrets in a vault service, verify the secret is created there after a binding. https://github.com/redhat-developer/service-binding-operator/issues/75

Acceptance Criteria

• New tests are written, automated, configured into PR process
• New examples and sample are writtem, approved, documented for users
• For issues reported, new automated regression tests should be written
• For feature requests submitted, if approved, new user scenarios should be implemented, documented, supported and have new automated regression tests written.