Uploaded image for project: 'Service Binding'
  1. Service Binding
  2. APPSVC-1292

Define RegisteredServices Healthchecks

XMLWordPrintable

    • Icon: Story Story
    • Resolution: Duplicate
    • Icon: Undefined Undefined
    • Primaza 0.1
    • None
    • Service Binding
    • None
    • 0
    • False
    • None
    • False
    • Hide
      Feature: RegisteredServices Healthchecks

          Scenario: Healthcheck completing successfully

              Given Primaza Cluster "primaza-main" is running
              And On Primaza Cluster "primaza-main", Resource is created
              """
              apiVersion: primaza.io/v1alpha1
              kind: ServiceClass
              metadata:
                  name: test-service-class
                  namespace: primaza-system
              spec:
                  serviceClassIdentity:
                  - type: db_type
                    value: test
                  healthcheck:
                      container:
                          image: bash:latest
                          cmd:
                          - echo
                          - "healthcheck ok"
                  resource:
                      kind: testdb
                      apiVersion: testservices.redhat.io/v1alpha1
                      serviceEndpointDefinitionMapping:
                      - name: admin
                        jsonPath: .spec.admin
              """
              When On Primaza Cluster "primaza-main", Resource is created
              """
              apiVersion: primaza.io/v1alpha1
              kind: RegisteredService
              metadata:
                  name: test-registered-service
                  namespace: primaza-system
              spec:
                  serviceClassIdentity:
                  - type: db_type
                    value: test
                  serviceEndpointDefinition:
                  - admin: sa
              """
              Then On Primaza Cluster "primaza-main", CronJob "test-registered-service-healthcheck" is eventually created
              And On Primaza Cluster "primaza-main", CronJob "test-registered-service-healthcheck" completes correctly
              And On Primaza Cluster "primaza-main", RegisteredService "test-registered-service"'s status is eventually updated to "reachable"


          Scenario: Healthcheck failing

              Given Primaza Cluster "primaza-main" is running
              And On Primaza Cluster "primaza-main", Resource is created
              """
              apiVersion: primaza.io/v1alpha1
              kind: ServiceClass
              metadata:
                  name: test-service-class
                  namespace: primaza-system
              spec:
                  serviceClassIdentity:
                  - type: db_type
                    value: test
                  healthcheck:
                      container:
                          image: bash:latest
                          cmd:
                          - echo
                          - "healthcheck failing"
                          - &&
                          - exit
                          - 1
                  resource:
                      kind: testdb
                      apiVersion: testservices.redhat.io/v1alpha1
                      serviceEndpointDefinitionMapping:
                      - name: admin
                        jsonPath: .spec.admin
              """
              When On Primaza Cluster "primaza-main", Resource is created
              """
              apiVersion: primaza.io/v1alpha1
              kind: RegisteredService
              metadata:
                  name: test-registered-service
                  namespace: primaza-system
              spec:
                  serviceClassIdentity:
                  - type: db_type
                    value: test
                  serviceEndpointDefinition:
                  - admin: sa
              """
              Then On Primaza Cluster "primaza-main", CronJob "test-registered-service-healthcheck" is eventually created
              And On Primaza Cluster "primaza-main", CronJob "test-registered-service-healthcheck" fails
              And On Primaza Cluster "primaza-main", RegisteredService "test-registered-service"'s status is eventually updated to "unreachable"


          Scenario: Healthcheck not runnable

              Given Primaza Cluster "primaza-main" is running
              And On Primaza Cluster "primaza-main", Resource is created
              """
              apiVersion: primaza.io/v1alpha1
              kind: ServiceClass
              metadata:
                  name: test-service-class
                  namespace: primaza-system
              spec:
                  serviceClassIdentity:
                  - type: db_type
                    value: test
                  healthcheck:
                      container:
                          image: quay.io/non-existing-repo/non-existing-image:non-existing-tag
                          cmd:
                          - exit
                          - 1
                  resource:
                      kind: testdb
                      apiVersion: testservices.redhat.io/v1alpha1
                      serviceEndpointDefinitionMapping:
                      - name: admin
                        jsonPath: .spec.admin
              """
              When On Primaza Cluster "primaza-main", Resource is created
              """
              apiVersion: primaza.io/v1alpha1
              kind: RegisteredService
              metadata:
                  name: test-registered-service
                  namespace: primaza-system
              spec:
                  serviceClassIdentity:
                  - type: db_type
                    value: test
                  serviceEndpointDefinition:
                  - admin: sa
              """
              Then On Primaza Cluster "primaza-main", CronJob "test-registered-service-healthcheck" is not created with error "ErrImagePull"
              And On Primaza Cluster "primaza-main", RegisteredService "test-registered-service"'s status is eventually updated to "undetermined"
      Show
      Feature: RegisteredServices Healthchecks     Scenario: Healthcheck completing successfully         Given Primaza Cluster "primaza-main" is running         And On Primaza Cluster "primaza-main", Resource is created         """         apiVersion: primaza.io/v1alpha1         kind: ServiceClass         metadata:             name: test-service-class             namespace: primaza-system         spec:             serviceClassIdentity:             - type: db_type               value: test             healthcheck:                 container:                     image: bash:latest                     cmd:                     - echo                     - "healthcheck ok"             resource:                 kind: testdb                 apiVersion: testservices.redhat.io/v1alpha1                 serviceEndpointDefinitionMapping:                 - name: admin                   jsonPath: .spec.admin         """         When On Primaza Cluster "primaza-main", Resource is created         """         apiVersion: primaza.io/v1alpha1         kind: RegisteredService         metadata:             name: test-registered-service             namespace: primaza-system         spec:             serviceClassIdentity:             - type: db_type               value: test             serviceEndpointDefinition:             - admin: sa         """         Then On Primaza Cluster "primaza-main", CronJob "test-registered-service-healthcheck" is eventually created         And On Primaza Cluster "primaza-main", CronJob "test-registered-service-healthcheck" completes correctly         And On Primaza Cluster "primaza-main", RegisteredService "test-registered-service"'s status is eventually updated to "reachable"     Scenario: Healthcheck failing         Given Primaza Cluster "primaza-main" is running         And On Primaza Cluster "primaza-main", Resource is created         """         apiVersion: primaza.io/v1alpha1         kind: ServiceClass         metadata:             name: test-service-class             namespace: primaza-system         spec:             serviceClassIdentity:             - type: db_type               value: test             healthcheck:                 container:                     image: bash:latest                     cmd:                     - echo                     - "healthcheck failing"                     - &&                     - exit                     - 1             resource:                 kind: testdb                 apiVersion: testservices.redhat.io/v1alpha1                 serviceEndpointDefinitionMapping:                 - name: admin                   jsonPath: .spec.admin         """         When On Primaza Cluster "primaza-main", Resource is created         """         apiVersion: primaza.io/v1alpha1         kind: RegisteredService         metadata:             name: test-registered-service             namespace: primaza-system         spec:             serviceClassIdentity:             - type: db_type               value: test             serviceEndpointDefinition:             - admin: sa         """         Then On Primaza Cluster "primaza-main", CronJob "test-registered-service-healthcheck" is eventually created         And On Primaza Cluster "primaza-main", CronJob "test-registered-service-healthcheck" fails         And On Primaza Cluster "primaza-main", RegisteredService "test-registered-service"'s status is eventually updated to "unreachable"     Scenario: Healthcheck not runnable         Given Primaza Cluster "primaza-main" is running         And On Primaza Cluster "primaza-main", Resource is created         """         apiVersion: primaza.io/v1alpha1         kind: ServiceClass         metadata:             name: test-service-class             namespace: primaza-system         spec:             serviceClassIdentity:             - type: db_type               value: test             healthcheck:                 container:                     image: quay.io/non-existing-repo/non-existing-image:non-existing-tag                     cmd:                     - exit                     - 1             resource:                 kind: testdb                 apiVersion: testservices.redhat.io/v1alpha1                 serviceEndpointDefinitionMapping:                 - name: admin                   jsonPath: .spec.admin         """         When On Primaza Cluster "primaza-main", Resource is created         """         apiVersion: primaza.io/v1alpha1         kind: RegisteredService         metadata:             name: test-registered-service             namespace: primaza-system         spec:             serviceClassIdentity:             - type: db_type               value: test             serviceEndpointDefinition:             - admin: sa         """         Then On Primaza Cluster "primaza-main", CronJob "test-registered-service-healthcheck" is not created with error "ErrImagePull"         And On Primaza Cluster "primaza-main", RegisteredService "test-registered-service"'s status is eventually updated to "undetermined"

      Owner: Architect:

      Francesco Ilario

      Story (Required)

      As a Service Administrator, I would like to have RegisteredService healthchecks so that I know if the service is available and reachable

      Background (Required)

      As defined in the Primaza architecture document, we need to define optional healthchecks for RegisteredServices.

      The Healthcheck is similar to how Pods do healthchecks. We want to take it beyond trusting the info provided in Service Endpoint Definition (SED) is always accurate. This could change overtime, for example a password rotation. So instead of the error going all the way down to the app not able to connect, we want to be more proactive and catch it earlier. It will be a good debugging tool.
      So, healthchecks will provide SED validation and service status checks.

      See epic for arch document link.

      Glossary

      See glossary in architecture document

      Out of scope

      • Related Claims status update
      • Other healthchecks mechanisms than container based one

      In Scope

      • RegisteredService status update
      • Container based healthchecks

      Approach(Required)

      Healthchecks will be optional and executed on Primaza cluster.
      We can rely on CronJob for executing them in a timely fashion.

      Healthchecks are defined in ServiceClass.
      Use the ServiceClassIdentity field to retrieve the ServiceClass specification and, hence, the Healthcheck one.
      Healthcheck containers will have the Service Endpoint Definition secret data projected in their filesystem.

      If Primaza has problems executing the healthchecks, like ErrImagePull, RegisteredService state and conditions should be updated accordingly.

      Healthchecks are successful if their pod terminates without any error. If the pod fails, than the healthcheck is considered failed.

      Demo requirements(Required)

      A great demo will show RegisteredService's healthchecks start failing after an incorrect update of ServiceEndpointDefinition data.

      Dependencies

      NA

      Edge Case

      NA

      BDD Tests

      You can find BDD Test specification for this story in the Story's Field 'Testing Instructions' or in the Github Issue linked to this story.
      Click here for all BDD Tests Issues.

      Acceptance Criteria

      • Development
        Primaza should run healthchecks for all RegisteredServices
        Primaza should stop and recreate running healthcheck pods on related data update, like ServiceClass.Healthcheck or RegisteredService.ServiceEndpointDefinition.
        RegisteredService status and conditions should be updated with respect to healthchecks result
      • QE
        There are test cases for passing healthchecks
        There are test cases for failing healthchecks
        There are test cases for in-error healthchecks, like pod not executable (e.g., ErrImagePull)
      • Docs
        There is a section in RegisteredService's doc dedicated to explaining what an healthcheck is and why it's needed
        Update the architecture document with any changes while implementing

      INVEST Checklist

      Dependencies identified
      Blockers noted and expected delivery timelines set
      Design is implementable
      Acceptance criteria agreed upon
      Story estimated

      Legend

      Unknown
      Verified
      Unsatisfied

              Unassigned Unassigned
              rh-ee-filario Francesco Ilario
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: