Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-6034

[release-6.0] Vector not releasing deleted file handles

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • ASSIGNED
    • Hide
      Before this change, the collector relied upon the default behavior to collect audit logs with no provisions to adjust for back pressure introduced by output receivers. This change modifies characteristics of collecting audit sources to improve the use of file handles and reading those logs.
      Show
      Before this change, the collector relied upon the default behavior to collect audit logs with no provisions to adjust for back pressure introduced by output receivers. This change modifies characteristics of collecting audit sources to improve the use of file handles and reading those logs.
    • Bug Fix
    • Hide

      Following KCS  https://access.redhat.com/solutions/7007644–> we find the diagnostic steps

      vector  628099 root   21r      REG    8,4 209678855 788530460 /var/log/kube-apiserver/audit-2024-07-23T09-32-40.994.log (deleted)
      vector  628099 root   62r      REG    8,4 209706973 788530467 /var/log/kube-apiserver/audit-2024-07-23T09-56-23.287.log (deleted)
      vector  628099 root   81r      REG    8,4 209632943 788530435 /var/log/kube-apiserver/audit-2024-07-23T10-46-19.105.log (deleted)
      vector  628099 root  104r      REG    8,4 209585344 788529311 /var/log/kube-apiserver/audit-2024-07-23T09-50-25.644.log (deleted)
      vector  628099 root  146r      REG    8,4 209634765 788530457 /var/log/kube-apiserver/audit-2024-07-23T10-47-55.907.log (deleted)
      vector  628099 root  258r      REG    8,4 209590488 788530433 /var/log/kube-apiserver/audit-2024-07-23T10-37-54.841.log (deleted)
      vector  628099 root  260r      REG    8,4 209696153 788530439 /var/log/kube-apiserver/audit-2024-07-23T10-12-45.634.log (deleted)
      vector  628099 root  261r      REG    8,4 209607732 788530443 /var/log/kube-apiserver/audit-2024-07-23T10-45-02.551.log (deleted)
      vector  628099 root  268r      REG    8,4 209661661 788530472 /var/log/kube-apiserver/audit-2024-07-23T09-09-36.261.log (deleted)
      vector  628099 root  269r      REG    8,4 209710461 788530442 /var/log/kube-apiserver/audit-2024-07-23T10-43-35.740.log (deleted)
      vector  628099 root  273r      REG    8,4 209667901 788530462 /var/log/kube-apiserver/audit-2024-07-23T10-49-16.200.log (deleted)
      vector  628099 root  277r      REG    8,4 209588545 788530449 /var/log/kube-apiserver/audit-2024-07-23T09-03-40.208.log (deleted)
      vector  628099 root  280r      REG    8,4 209642861 788530459 /var/log/kube-apiserver/audit-2024-07-23T09-22-26.203.log (deleted)
      vector  628099 root  281r      REG    8,4 209661617 788530463 /var/log/kube-apiserver/audit-2024-07-23T10-42-07.248.log (deleted)
      vector  628099 root  286r      REG    8,4 209559796 788530444 /var/log/kube-apiserver/audit-2024-07-23T09-08-02.263.log (deleted)
      vector  628099 root  287r      REG    8,4 209578636 788530475 /var/log/kube-apiserver/audit-2024-07-23T10-52-15.650.log (deleted)
      vector  628099 root  288r      REG    8,4 209698215 788529297 /var/log/kube-apiserver/audit-2024-07-23T10-23-02.185.log (deleted)
      vector  628099 root  289r      REG    8,4 209541416 788530440 /var/log/kube-apiserver/audit-2024-07-23T09-25-21.538.log (deleted)
      vector  628099 root  290r      REG    8,4 209714935 788530261 /var/log/kube-apiserver/audit-2024-07-23T10-18-27.990.log (deleted)
      vector  628099 root  292r      REG    8,4 209681744 788530437 /var/log/kube-apiserver/audit-2024-07-23T10-39-23.029.log (deleted)
      vector  628099 root  293r      REG    8,4 209599894 788530445 /var/log/kube-apiserver/audit-2024-07-23T09-39-47.390.log (deleted)
      vector  628099 root  294r      REG    8,4 209703092 788530456 /var/log/kube-apiserver/audit-2024-07-23T09-29-41.296.log (deleted)
      vector  628099 root  296r      REG    8,4 209638865 788530468 /var/log/kube-apiserver/audit-2024-07-23T10-50-57.397.log (deleted)
      vector  628099 root  297r      REG    8,4 209606751 788530458 /var/log/kube-apiserver/audit-2024-07-23T10-09-45.882.log (deleted)
      vector  628099 root  298r      REG    8,4 209670824 788529295 /var/log/kube-apiserver/audit-2024-07-23T10-15-44.322.log (deleted)
      vector  628099 root  299r      REG    8,4 209668153 788530465 /var/log/kube-apiserver/audit-2024-07-23T09-28-17.046.log (deleted)
      vector  628099 root  300r      REG    8,4 209674049 788530441 /var/log/kube-apiserver/audit-2024-07-23T10-36-25.930.log (deleted)
      vector  628099 root  302r      REG    8,4 209674666 788530255 /var/log/kube-apiserver/audit-2024-07-23T09-44-31.492.log (deleted)
      vector  628099 root  303r      REG    8,4 209587189 788530478 /var/log/kube-apiserver/audit-2024-07-23T10-53-54.903.log (deleted)
      vector  628099 root  304r      REG    8,4 209629114 788530461 /var/log/kube-apiserver/audit-2024-07-23T09-59-23.542.log (deleted)
      vector  628099 root  305r      REG    8,4 209543252 788530447 /var/log/kube-apiserver/audit-2024-07-23T10-32-14.133.log (deleted)
      vector  628099 root  306r      REG    8,4 209698203 788530450 /var/log/kube-apiserver/audit-2024-07-23T08-57-38.298.log (deleted)
      vector  628099 root  308r      REG    8,4 209669052 788530471 /var/log/kube-apiserver/audit-2024-07-23T09-57-32.093.log (deleted)
      vector  628099 root  309r      REG    8,4 209660521 788530438 /var/log/kube-apiserver/audit-2024-07-23T10-33-35.722.log (deleted)
      vector  628099 root  310r      REG    8,4 209714130 788530454 /var/log/kube-apiserver/audit-2024-07-23T09-36-53.669.log (deleted)
      vector  628099 root  311r      REG    8,4 209683440 788530481 /var/log/kube-apiserver/audit-2024-07-23T10-55-08.952.log (deleted)
      vector  628099 root  312r      REG    8,4 209692429 788530474 /var/log/kube-apiserver/audit-2024-07-23T09-35-38.644.log (deleted)
      vector  628099 root  313r      REG    8,4 209692171 788529293 /var/log/kube-apiserver/audit-2024-07-23T08-30-57.857.log (deleted)
      vector  628099 root  314r      REG    8,4 209642472 788530252 /var/log/kube-apiserver/audit-2024-07-23T10-28-58.985.log (deleted)
      vector  628099 root  315r      REG    8,4 209613475 788530464 /var/log/kube-apiserver/audit-2024-07-23T10-58-04.902.log (deleted)
      vector  628099 root  317r      REG    8,4 209596566 788530477 /var/log/kube-apiserver/audit-2024-07-23T09-38-35.246.log (deleted)
      vector  628099 root  318r      REG    8,4 209597001 788530264 /var/log/kube-apiserver/audit-2024-07-23T10-56-55.449.log (deleted)
      vector  628099 root  320r      REG    8,4 209714332 788530473 /var/log/kube-apiserver/audit-2024-07-23T10-00-47.174.log (deleted)
      vector  628099 root  321r      REG    8,4 209587189 788530492 /var/log/kube-apiserver/audit-2024-07-23T09-41-34.243.log (deleted)
      vector  628099 root  322r      REG    8,4 209633829 788530448 /var/log/kube-apiserver/audit-2024-07-23T09-54-39.390.log (deleted)
      vector  628099 root  323r      REG    8,4 209656543 788530452 /var/log/kube-apiserver/audit-2024-07-23T09-00-39.799.log (deleted)
      vector  628099 root  324r      REG    8,4 209691588 788530263 /var/log/kube-apiserver/audit-2024-07-23T10-05-20.639.log (deleted)
      vector  628099 root  325r      REG    8,4 209652882 788530250 /var/log/kube-apiserver/audit-2024-07-23T10-02-18.694.log (deleted)
      vector  628099 root  326r      REG    8,4 209663547 788530262 /var/log/kube-apiserver/audit-2024-07-23T10-30-33.229.log (deleted)
      vector  628099 root  327r      REG    8,4 209607979 788530266 /var/log/kube-apiserver/audit-2024-07-23T09-47-26.790.log (deleted)
      vector  628099 root  328r      REG    8,4 209548504 788530303 /var/log/kube-apiserver/audit-2024-07-23T10-14-17.932.log (deleted)
      vector  628099 root  330r      REG    8,4 209677442 788530254 /var/log/kube-apiserver/audit-2024-07-23T10-03-58.082.log (deleted)
      vector  628099 root  331r      REG    8,4 209570375 788530267 /var/log/kube-apiserver/audit-2024-07-23T10-26-00.289.log (deleted)
      vector  628099 root  333r      REG    8,4 209670543 788529307 /var/log/kube-apiserver/audit-2024-07-23T09-15-31.156.log (deleted)
      vector  628099 root  334r      REG    8,4 209625107 788529294 /var/log/kube-apiserver/audit-2024-07-23T10-08-19.231.log (deleted)
      vector  628099 root  336r      REG    8,4 209616791 788529288 /var/log/kube-apiserver/audit-2024-07-23T09-53-22.839.log (deleted)
      vector  628099 root  338r      REG    8,4 209710503 788529290 /var/log/kube-apiserver/audit-2024-07-23T10-24-32.372.log (deleted)
      vector  628099 root  339r      REG    8,4 209574506 788529298 /var/log/kube-apiserver/audit-2024-07-23T10-17-13.133.log (deleted)
      vector  628099 root  340r      REG    8,4 209691870 788529299 /var/log/kube-apiserver/audit-2024-07-23T10-27-37.121.log (deleted)
      vector  628099 root  341r      REG    8,4 209629499 788530302 /var/log/kube-apiserver/audit-2024-07-23T10-06-46.433.log (deleted)
      vector  628099 root  344r      REG    8,4 209605936 788529286 /var/log/kube-apiserver/audit-2024-07-23T10-11-18.230.log (deleted)
      vector  628099 root  345r      REG    8,4 209698122 788529303 /var/log/kube-apiserver/audit-2024-07-23T08-27-59.900.log (deleted)
      vector  628099 root  348r      REG    8,4 209685435 788529306 /var/log/kube-apiserver/audit-2024-07-23T09-06-39.460.log (deleted)
      vector  628099 root  350r      REG    8,4 209600596 788529309 /var/log/kube-apiserver/audit-2024-07-23T09-18-27.956.log (deleted)
      vector  628099 root  352r      REG    8,4 209676441 788529312 /var/log/kube-apiserver/audit-2024-07-23T09-51-40.192.log (deleted)
      vector  628099 root  353r      REG    8,4 209664252 788529300 /var/log/kube-apiserver/audit-2024-07-23T10-20-12.682.log (deleted)
      vector  628099 root  355r      REG    8,4 209712890 788529313 /var/log/kube-apiserver/audit-2024-07-23T10-21-29.226.log (deleted) 
      Show
      Following KCS   https://access.redhat.com/solutions/7007644 –> we find the diagnostic steps vector 628099 root 21r REG 8,4 209678855 788530460 / var /log/kube-apiserver/audit-2024-07-23T09-32-40.994.log (deleted) vector 628099 root 62r REG 8,4 209706973 788530467 / var /log/kube-apiserver/audit-2024-07-23T09-56-23.287.log (deleted) vector 628099 root 81r REG 8,4 209632943 788530435 / var /log/kube-apiserver/audit-2024-07-23T10-46-19.105.log (deleted) vector 628099 root 104r REG 8,4 209585344 788529311 / var /log/kube-apiserver/audit-2024-07-23T09-50-25.644.log (deleted) vector 628099 root 146r REG 8,4 209634765 788530457 / var /log/kube-apiserver/audit-2024-07-23T10-47-55.907.log (deleted) vector 628099 root 258r REG 8,4 209590488 788530433 / var /log/kube-apiserver/audit-2024-07-23T10-37-54.841.log (deleted) vector 628099 root 260r REG 8,4 209696153 788530439 / var /log/kube-apiserver/audit-2024-07-23T10-12-45.634.log (deleted) vector 628099 root 261r REG 8,4 209607732 788530443 / var /log/kube-apiserver/audit-2024-07-23T10-45-02.551.log (deleted) vector 628099 root 268r REG 8,4 209661661 788530472 / var /log/kube-apiserver/audit-2024-07-23T09-09-36.261.log (deleted) vector 628099 root 269r REG 8,4 209710461 788530442 / var /log/kube-apiserver/audit-2024-07-23T10-43-35.740.log (deleted) vector 628099 root 273r REG 8,4 209667901 788530462 / var /log/kube-apiserver/audit-2024-07-23T10-49-16.200.log (deleted) vector 628099 root 277r REG 8,4 209588545 788530449 / var /log/kube-apiserver/audit-2024-07-23T09-03-40.208.log (deleted) vector 628099 root 280r REG 8,4 209642861 788530459 / var /log/kube-apiserver/audit-2024-07-23T09-22-26.203.log (deleted) vector 628099 root 281r REG 8,4 209661617 788530463 / var /log/kube-apiserver/audit-2024-07-23T10-42-07.248.log (deleted) vector 628099 root 286r REG 8,4 209559796 788530444 / var /log/kube-apiserver/audit-2024-07-23T09-08-02.263.log (deleted) vector 628099 root 287r REG 8,4 209578636 788530475 / var /log/kube-apiserver/audit-2024-07-23T10-52-15.650.log (deleted) vector 628099 root 288r REG 8,4 209698215 788529297 / var /log/kube-apiserver/audit-2024-07-23T10-23-02.185.log (deleted) vector 628099 root 289r REG 8,4 209541416 788530440 / var /log/kube-apiserver/audit-2024-07-23T09-25-21.538.log (deleted) vector 628099 root 290r REG 8,4 209714935 788530261 / var /log/kube-apiserver/audit-2024-07-23T10-18-27.990.log (deleted) vector 628099 root 292r REG 8,4 209681744 788530437 / var /log/kube-apiserver/audit-2024-07-23T10-39-23.029.log (deleted) vector 628099 root 293r REG 8,4 209599894 788530445 / var /log/kube-apiserver/audit-2024-07-23T09-39-47.390.log (deleted) vector 628099 root 294r REG 8,4 209703092 788530456 / var /log/kube-apiserver/audit-2024-07-23T09-29-41.296.log (deleted) vector 628099 root 296r REG 8,4 209638865 788530468 / var /log/kube-apiserver/audit-2024-07-23T10-50-57.397.log (deleted) vector 628099 root 297r REG 8,4 209606751 788530458 / var /log/kube-apiserver/audit-2024-07-23T10-09-45.882.log (deleted) vector 628099 root 298r REG 8,4 209670824 788529295 / var /log/kube-apiserver/audit-2024-07-23T10-15-44.322.log (deleted) vector 628099 root 299r REG 8,4 209668153 788530465 / var /log/kube-apiserver/audit-2024-07-23T09-28-17.046.log (deleted) vector 628099 root 300r REG 8,4 209674049 788530441 / var /log/kube-apiserver/audit-2024-07-23T10-36-25.930.log (deleted) vector 628099 root 302r REG 8,4 209674666 788530255 / var /log/kube-apiserver/audit-2024-07-23T09-44-31.492.log (deleted) vector 628099 root 303r REG 8,4 209587189 788530478 / var /log/kube-apiserver/audit-2024-07-23T10-53-54.903.log (deleted) vector 628099 root 304r REG 8,4 209629114 788530461 / var /log/kube-apiserver/audit-2024-07-23T09-59-23.542.log (deleted) vector 628099 root 305r REG 8,4 209543252 788530447 / var /log/kube-apiserver/audit-2024-07-23T10-32-14.133.log (deleted) vector 628099 root 306r REG 8,4 209698203 788530450 / var /log/kube-apiserver/audit-2024-07-23T08-57-38.298.log (deleted) vector 628099 root 308r REG 8,4 209669052 788530471 / var /log/kube-apiserver/audit-2024-07-23T09-57-32.093.log (deleted) vector 628099 root 309r REG 8,4 209660521 788530438 / var /log/kube-apiserver/audit-2024-07-23T10-33-35.722.log (deleted) vector 628099 root 310r REG 8,4 209714130 788530454 / var /log/kube-apiserver/audit-2024-07-23T09-36-53.669.log (deleted) vector 628099 root 311r REG 8,4 209683440 788530481 / var /log/kube-apiserver/audit-2024-07-23T10-55-08.952.log (deleted) vector 628099 root 312r REG 8,4 209692429 788530474 / var /log/kube-apiserver/audit-2024-07-23T09-35-38.644.log (deleted) vector 628099 root 313r REG 8,4 209692171 788529293 / var /log/kube-apiserver/audit-2024-07-23T08-30-57.857.log (deleted) vector 628099 root 314r REG 8,4 209642472 788530252 / var /log/kube-apiserver/audit-2024-07-23T10-28-58.985.log (deleted) vector 628099 root 315r REG 8,4 209613475 788530464 / var /log/kube-apiserver/audit-2024-07-23T10-58-04.902.log (deleted) vector 628099 root 317r REG 8,4 209596566 788530477 / var /log/kube-apiserver/audit-2024-07-23T09-38-35.246.log (deleted) vector 628099 root 318r REG 8,4 209597001 788530264 / var /log/kube-apiserver/audit-2024-07-23T10-56-55.449.log (deleted) vector 628099 root 320r REG 8,4 209714332 788530473 / var /log/kube-apiserver/audit-2024-07-23T10-00-47.174.log (deleted) vector 628099 root 321r REG 8,4 209587189 788530492 / var /log/kube-apiserver/audit-2024-07-23T09-41-34.243.log (deleted) vector 628099 root 322r REG 8,4 209633829 788530448 / var /log/kube-apiserver/audit-2024-07-23T09-54-39.390.log (deleted) vector 628099 root 323r REG 8,4 209656543 788530452 / var /log/kube-apiserver/audit-2024-07-23T09-00-39.799.log (deleted) vector 628099 root 324r REG 8,4 209691588 788530263 / var /log/kube-apiserver/audit-2024-07-23T10-05-20.639.log (deleted) vector 628099 root 325r REG 8,4 209652882 788530250 / var /log/kube-apiserver/audit-2024-07-23T10-02-18.694.log (deleted) vector 628099 root 326r REG 8,4 209663547 788530262 / var /log/kube-apiserver/audit-2024-07-23T10-30-33.229.log (deleted) vector 628099 root 327r REG 8,4 209607979 788530266 / var /log/kube-apiserver/audit-2024-07-23T09-47-26.790.log (deleted) vector 628099 root 328r REG 8,4 209548504 788530303 / var /log/kube-apiserver/audit-2024-07-23T10-14-17.932.log (deleted) vector 628099 root 330r REG 8,4 209677442 788530254 / var /log/kube-apiserver/audit-2024-07-23T10-03-58.082.log (deleted) vector 628099 root 331r REG 8,4 209570375 788530267 / var /log/kube-apiserver/audit-2024-07-23T10-26-00.289.log (deleted) vector 628099 root 333r REG 8,4 209670543 788529307 / var /log/kube-apiserver/audit-2024-07-23T09-15-31.156.log (deleted) vector 628099 root 334r REG 8,4 209625107 788529294 / var /log/kube-apiserver/audit-2024-07-23T10-08-19.231.log (deleted) vector 628099 root 336r REG 8,4 209616791 788529288 / var /log/kube-apiserver/audit-2024-07-23T09-53-22.839.log (deleted) vector 628099 root 338r REG 8,4 209710503 788529290 / var /log/kube-apiserver/audit-2024-07-23T10-24-32.372.log (deleted) vector 628099 root 339r REG 8,4 209574506 788529298 / var /log/kube-apiserver/audit-2024-07-23T10-17-13.133.log (deleted) vector 628099 root 340r REG 8,4 209691870 788529299 / var /log/kube-apiserver/audit-2024-07-23T10-27-37.121.log (deleted) vector 628099 root 341r REG 8,4 209629499 788530302 / var /log/kube-apiserver/audit-2024-07-23T10-06-46.433.log (deleted) vector 628099 root 344r REG 8,4 209605936 788529286 / var /log/kube-apiserver/audit-2024-07-23T10-11-18.230.log (deleted) vector 628099 root 345r REG 8,4 209698122 788529303 / var /log/kube-apiserver/audit-2024-07-23T08-27-59.900.log (deleted) vector 628099 root 348r REG 8,4 209685435 788529306 / var /log/kube-apiserver/audit-2024-07-23T09-06-39.460.log (deleted) vector 628099 root 350r REG 8,4 209600596 788529309 / var /log/kube-apiserver/audit-2024-07-23T09-18-27.956.log (deleted) vector 628099 root 352r REG 8,4 209676441 788529312 / var /log/kube-apiserver/audit-2024-07-23T09-51-40.192.log (deleted) vector 628099 root 353r REG 8,4 209664252 788529300 / var /log/kube-apiserver/audit-2024-07-23T10-20-12.682.log (deleted) vector 628099 root 355r REG 8,4 209712890 788529313 / var /log/kube-apiserver/audit-2024-07-23T10-21-29.226.log (deleted)
    • Log Collection - Sprint 258, Log Collection - Sprint 259, Log Collection - Sprint 260, Log Collection - Sprint 261, Log Collection - Sprint 262
    • Important
    • Customer Facing, Customer Reported

      Description of problem:  

      Disk usage was consistently filling up, lsof showed a large number of deleted files were still being locked by Vector.

      This Bug is similar to LOG-3949 and LOG-4501 but this time is happening in RHOL 5.9.

      Version-Release number of selected component (if applicable):

      Red Hat Openshift Logging 5.9.2

      Expected results:

      Vector should release deleted files.

        1. lsof-scan-result
          30 kB
        2. vector.toml
          29 kB

              jcantril@redhat.com Jeffrey Cantrill
              acandelp Adrian Candel
              Anping Li Anping Li
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: