Uploaded image for project: 'Distributed Tracing'
  1. Distributed Tracing
  2. TRACING-3357

Fix failing OpenTelemetry tests on Prow CI.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • None
    • None
    • OpenTelemetry, QE
    • None
    • 1
    • False
    • None
    • False
    • Tracing Sprint # 239

      The following test cases are failing on Prow CI, which runs the test cases from a pod.

      --- FAIL: kuttl (505.15s)
    --- FAIL: kuttl/harness (0.00s)
        --- FAIL: kuttl/harness/autoscale (54.29s)
        --- FAIL: kuttl/harness/instrumentation-sdk (178.77s)
        --- FAIL: kuttl/harness/instrumentation-go (163.91s)
        --- FAIL: kuttl/harness/instrumentation-apache-multicontainer (165.27s)
        --- FAIL: kuttl/harness/instrumentation-apache-httpd (167.92s)
        --- FAIL: kuttl/harness/route (160.50s)
        --- FAIL: kuttl/harness/targetallocator-features (165.37s)
        --- FAIL: kuttl/harness/prometheus-config-validation (159.63s)
        --- FAIL: kuttl/harness/smoke-targetallocator (159.80s)
FAIL

      Test run link:

      https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_release/40442/rehearse-40442-periodic-ci-open-telemetry-opentelemetry-operator-main-opentelemetry-product-ocp-14-lp-interop-opentelemetry-interop-aws/1675778951194087424/artifacts/opentelemetry-interop-aws/distributed-tracing-tests-opentelemetry/build-log.txt

      How do I test it from a OpenShift pod to simulate a Prow CI run:

      *Create the tests runner image from the opentelemetry-operator repo.

      docker build -t quay.io/rhn_support_ikanse/otel-tests-runner:latest -f tests/e2e-openshift/Dockerfile .

      *Create a OpenShift deployment using this image.

      *Make sure you install any dependency operator for running the tests like OpenTelemetry, Jaeger, Elasticsearch and AMQ streams. 

      *oc rsh into the running pod.

      • export KUBECONFIG=/path/to/a/kubeconfig/for our test cluster, you can use the same cluster on which the pod is running as a test cluster. 

      *Run the tests using steps from this script. Set the ARTIFACT_DIR var to any writable location like /tmp/artifacts. PARALLEL_TESTS=5 REPORT_NAME can be anything, set TIMEOUT=150. 

      #!/bin/bash
set -o nounset
set -o errexit
set -o pipefail
#Add manifest directory for kuttl
mkdir /tmp/kuttl-manifests
#Copy the opentelemetry-operator repo files to a writable directory by kuttl
cp -R /tmp/opentelemetry-operator /tmp/opentelemetry-tests && cd /tmp/opentelemetry-tests
# Execute OpenTelemetry e2e tests
KUBECONFIG=$KUBECONFIG kuttl test \
  --report=xml \
  --artifacts-dir="$ARTIFACT_DIR" \
  --parallel="$PARALLEL_TESTS" \
  --report-name="$REPORT_NAME" \
  --start-kind=false \
  --timeout="$TIMEOUT" \
  --manifest-dir=$MANIFEST_DIR \
  tests/e2e \
  tests/e2e-autoscale

      Link to the script:
      https://github.com/openshift/release/pull/40442/files#diff-27cf97b7a96c8d5b039cde3ac4e19a7002c4a7e26337e6164915092668b56fca

          1.
          		 Fix the autoscale E2E test for Prow CI Sub-task Closed Undefined Israel Blancas Alvarez
          2.
          		 Fix the route E2E test for Prow CI Sub-task Closed Undefined Israel Blancas Alvarez
          3.
          		 Fix the instrumentation-go for Prow CI Sub-task Closed Undefined Israel Blancas Alvarez
          4.
          		 Build the target allocator image in the Prow CI jobs and set the image in the test files. Sub-task Closed Undefined Ishwar Kanse

              rhn-support-iblancas Israel Blancas Alvarez
              rhn-support-ikanse Ishwar Kanse
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: