Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4275

[release-5.7] Vector pods going into a panic state

    XMLWordPrintable

Details

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Hide
      Prior to this fix, the vector log collector occasionally panicked with the following error message in its log:
      thread 'vector-worker' panicked at 'all branches are disabled and there is no else branch', src/kubernetes/reflector.rs:26:9
      After this fix, the vector log collector no longer does that.
      Show
      Prior to this fix, the vector log collector occasionally panicked with the following error message in its log: thread 'vector-worker' panicked at 'all branches are disabled and there is no else branch', src/kubernetes/reflector.rs:26:9 After this fix, the vector log collector no longer does that.
    • Bug Fix
    • High
    • Log Collection - Sprint 238, Log Collection - Sprint 239
    • Important

    Description

      Description of problem:

      Randomly, vector pods allocated in the worker nodes go into a panic state and become unresponsive.
      After restarting the pods manually, the logs are successfully processed again.

      The log in question that we can see in Vector pods is:

      2023-06-26T03:14:23.503804380Z 2023-06-26T03:14:23.487362Z ERROR vector::topology::builder: msg="Healthcheck: Failed Reason." error=Failed to make HTTP(S) request: error trying to connect: dns error: failed to lookup address information: Name or service not known component_kind="sink" component_type="elasticsearch" component_id=external_elasticsearch_ecp component_name=external_elasticsearch_ecp
2023-06-26T03:14:28.566100553Z 2023-06-26T03:14:28.566022Z ERROR kube_client::client::builder: failed with error error trying to connect: dns error: failed to lookup address information: Name or service not known
2023-06-26T03:14:28.566100553Z thread 'vector-worker' panicked at 'all branches are disabled and there is no else branch', src/kubernetes/reflector.rs:26:9
2023-06-26T03:14:28.566152028Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2023-06-26T03:14:33.615156912Z 2023-06-26T03:14:33.615108Z ERROR kube_client::client::builder: failed with error error trying to connect: dns error: failed to lookup address information: Name or service not known
2023-06-26T03:14:33.615156912Z thread 'vector-worker' panicked at 'all branches are disabled and there is no else branch', src/kubernetes/reflector.rs:26:9
2023-06-26T03:14:33.844423262Z 2023-06-26T03:14:33.844348Z ERROR sink{component_kind="sink" component_id=default component_type=elasticsearch component_name=default}: vector::internal_events::http_client: HTTP error. error=error trying to connect: dns error: failed to lookup address information: Name or service not known error_type="request_failed" stage="processing"
2023-06-26T03:14:33.844477233Z 2023-06-26T03:14:33.844433Z  WARN sink{component_kind="sink" component_id=default component_type=elasticsearch component_name=default}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: dns error: failed to lookup address information: Name or service not known

      Version-Release number of selected component (if applicable):

      RHOL 5.6.7
      RHOL 5.7.2

      How reproducible:

      -

      Actual results:

      Vector pods go into a panic state and losing logs.

      Expected results:

      Vector pods working properly

      Additional info:

      After doing some checks, I have found this bug --> https://github.com/vectordotdev/vector/issues/12245

      And it was solved in Vector release 0.21.0--> https://vector.dev/releases/0.21.0/#known-issues

      RHOL 5.6 and 5.7 uses v0.20.1 release If I am not wrong -->https://github.com/ViaQ/vector/tree/release-5.6 and https://github.com/ViaQ/vector/tree/release-5.7

      For next RHOL 5.8 I can see that the Vector release will be v0.28.1 -->https://github.com/ViaQ/vector/tree/release-5.8

      Attachments

        Issue Links

          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. LOG-4276 [release-5.6] Vector pods going into a panic state

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          links to

          Web Link [KCS] Vector pods going into a panic state in RHOCP 4

          GitHub ViaQ/vector#144: LOG-4275: [release-5.7] Vector pods going into a panic state

          Activity

            People

              syedriko_sub@redhat.com Sergey Yedrikov
              acandelp Adrian Candel
              Qiaoling Tang Qiaoling Tang
              Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: