Uploaded image for project: 'JBoss A-MQ'
  1. JBoss A-MQ
  2. ENTMQ-1328

Using LevelDB for Persistence Leaves Open Deleted File Descriptors

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • JBoss A-MQ 6.3
    • JBoss A-MQ 6.1, JBoss A-MQ 6.2, JBoss A-MQ 6.2.1
    • leveldb
    • None
    • Hide
      • Extract AMQ into the target environment.
      • Edit the {AMQ_HOME}/etc/user.properties and {AMQ_HOME}

        /bin/setenv files as needed to obtain a working configuration.

      - Edit the <persistencAdapter> element of the ${AMQ_HOME}/etc/activemq.xml file to look something like:
      
      <persistenceAdapter>
                  <levelDB directory="/opt/data/apps/leveldb" logSize="10485760" />
      </persistenceAdapter>
      
      

      (A smaller value for the logSize, like the 10mb here, will help to reproduce quicker)

      • Start AMQ and use a simple producer load the broker witha set of messages.
      • After the producer finishes connect a consumer to the queue and consume the messages
      • With the broker still running, open a shell and periodically execute lsof | grep deleted on the root directory of the volume where the leveldb is configured to store its logs. You will see open deleted file descriptors taking space on the volume:
      [root@node0 leveldb]# lsof / | grep deleted
      java      12455      root  189r   REG  253,0  10515424 263967 /opt/data/apps/leveldb/0000000000000000.log (deleted)
      java      12455      root  276r   REG  253,0  10502173 263960 /opt/data/apps/leveldb/0000000000a073e0.log (deleted)
      java      12455      root  303r   REG  253,0  10490856 263973 /opt/data/apps/leveldb/000000000140b3fd.log (deleted)
      java      12455      root  304r   REG  253,0  10490739 263969 /opt/data/apps/leveldb/0000000001e0c7e5.log (deleted)
      java      12455      root  305r   REG  253,0  10490928 263978 /opt/data/apps/leveldb/000000000280db58.log (deleted)
      java      12455      root  306r   REG  253,0  10490784 263970 /opt/data/apps/leveldb/000000000320ef88.log (deleted)
      java      12455      root  307r   REG  253,0  10490883 263980 /opt/data/apps/leveldb/0000000003c10328.log (deleted)
      java      12455      root  308r   REG  253,0  10490838 263971 /opt/data/apps/leveldb/000000000461172b.log (deleted)
      java      12455      root  309r   REG  253,0  10490892 263981 /opt/data/apps/leveldb/0000000005012b01.log (deleted)
      java      12455      root  310r   REG  253,0  10490820 263972 /opt/data/apps/leveldb/0000000005a13f0d.log (deleted)
      java      12455      root  311r   REG  253,0  10490982 263982 /opt/data/apps/leveldb/00000000064152d1.log (deleted)
      
      

      Repeat above and each time the producer and consumer are executed, the list of open descriptors grows.

      • Shut down the broker and rerun the lsof command as above. The file descriptors are cleaned up.
      Show
      Extract AMQ into the target environment. Edit the {AMQ_HOME}/etc/user.properties and {AMQ_HOME} /bin/setenv files as needed to obtain a working configuration. - Edit the <persistencAdapter> element of the ${AMQ_HOME}/etc/activemq.xml file to look something like: <persistenceAdapter> <levelDB directory= "/opt/data/apps/leveldb" logSize= "10485760" /> </persistenceAdapter> (A smaller value for the logSize, like the 10mb here, will help to reproduce quicker) Start AMQ and use a simple producer load the broker witha set of messages. After the producer finishes connect a consumer to the queue and consume the messages With the broker still running, open a shell and periodically execute lsof | grep deleted on the root directory of the volume where the leveldb is configured to store its logs. You will see open deleted file descriptors taking space on the volume: [root@node0 leveldb]# lsof / | grep deleted java 12455 root 189r REG 253,0 10515424 263967 /opt/data/apps/leveldb/0000000000000000.log (deleted) java 12455 root 276r REG 253,0 10502173 263960 /opt/data/apps/leveldb/0000000000a073e0.log (deleted) java 12455 root 303r REG 253,0 10490856 263973 /opt/data/apps/leveldb/000000000140b3fd.log (deleted) java 12455 root 304r REG 253,0 10490739 263969 /opt/data/apps/leveldb/0000000001e0c7e5.log (deleted) java 12455 root 305r REG 253,0 10490928 263978 /opt/data/apps/leveldb/000000000280db58.log (deleted) java 12455 root 306r REG 253,0 10490784 263970 /opt/data/apps/leveldb/000000000320ef88.log (deleted) java 12455 root 307r REG 253,0 10490883 263980 /opt/data/apps/leveldb/0000000003c10328.log (deleted) java 12455 root 308r REG 253,0 10490838 263971 /opt/data/apps/leveldb/000000000461172b.log (deleted) java 12455 root 309r REG 253,0 10490892 263981 /opt/data/apps/leveldb/0000000005012b01.log (deleted) java 12455 root 310r REG 253,0 10490820 263972 /opt/data/apps/leveldb/0000000005a13f0d.log (deleted) java 12455 root 311r REG 253,0 10490982 263982 /opt/data/apps/leveldb/00000000064152d1.log (deleted) Repeat above and each time the producer and consumer are executed, the list of open descriptors grows. Shut down the broker and rerun the lsof command as above. The file descriptors are cleaned up.

      When using leveldb for persistence, AMQ leaves open deleted file descriptors when the persistence logs roll over. Severity seems to vary between file systems. On EXT4 the problem was more severe than on GFS2 in my development environment.6.1 GA (379) and 6.2 GA (133) were affected. 6.2P1 (143) did not seem to be affected by the issue.

              hchirino Hiram Chirino
              rhn-support-dhawkins Duane Hawkins
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: