Loading...

Type: Bug
Resolution: Done
Priority: Critical
Fix Version/s: JBoss A-MQ 6.3
Affects Version/s: JBoss A-MQ 6.1, JBoss A-MQ 6.2, JBoss A-MQ 6.2.1
Component/s: leveldb
Labels:
None

GSS Priority:
Steps to Reproduce:
Hide

Extract AMQ into the target environment.

Edit the {AMQ_HOME}/etc/user.properties and {AMQ_HOME}
/bin/setenv files as needed to obtain a working configuration.

- Edit the <persistencAdapter> element of the ${AMQ_HOME}/etc/activemq.xml file to look something like: <persistenceAdapter> <levelDB directory="/opt/data/apps/leveldb" logSize="10485760" /> </persistenceAdapter>

(A smaller value for the logSize, like the 10mb here, will help to reproduce quicker)

Start AMQ and use a simple producer load the broker witha set of messages.

After the producer finishes connect a consumer to the queue and consume the messages

With the broker still running, open a shell and periodically execute lsof | grep deleted on the root directory of the volume where the leveldb is configured to store its logs. You will see open deleted file descriptors taking space on the volume:

[root@node0 leveldb]# lsof / | grep deleted java 12455 root 189r REG 253,0 10515424 263967 /opt/data/apps/leveldb/0000000000000000.log (deleted) java 12455 root 276r REG 253,0 10502173 263960 /opt/data/apps/leveldb/0000000000a073e0.log (deleted) java 12455 root 303r REG 253,0 10490856 263973 /opt/data/apps/leveldb/000000000140b3fd.log (deleted) java 12455 root 304r REG 253,0 10490739 263969 /opt/data/apps/leveldb/0000000001e0c7e5.log (deleted) java 12455 root 305r REG 253,0 10490928 263978 /opt/data/apps/leveldb/000000000280db58.log (deleted) java 12455 root 306r REG 253,0 10490784 263970 /opt/data/apps/leveldb/000000000320ef88.log (deleted) java 12455 root 307r REG 253,0 10490883 263980 /opt/data/apps/leveldb/0000000003c10328.log (deleted) java 12455 root 308r REG 253,0 10490838 263971 /opt/data/apps/leveldb/000000000461172b.log (deleted) java 12455 root 309r REG 253,0 10490892 263981 /opt/data/apps/leveldb/0000000005012b01.log (deleted) java 12455 root 310r REG 253,0 10490820 263972 /opt/data/apps/leveldb/0000000005a13f0d.log (deleted) java 12455 root 311r REG 253,0 10490982 263982 /opt/data/apps/leveldb/00000000064152d1.log (deleted)

Repeat above and each time the producer and consumer are executed, the list of open descriptors grows.

Shut down the broker and rerun the lsof command as above. The file descriptors are cleaned up.
Show
Extract AMQ into the target environment. Edit the {AMQ_HOME}/etc/user.properties and {AMQ_HOME} /bin/setenv files as needed to obtain a working configuration. - Edit the <persistencAdapter> element of the ${AMQ_HOME}/etc/activemq.xml file to look something like: <persistenceAdapter> <levelDB directory= "/opt/data/apps/leveldb" logSize= "10485760" /> </persistenceAdapter> (A smaller value for the logSize, like the 10mb here, will help to reproduce quicker) Start AMQ and use a simple producer load the broker witha set of messages. After the producer finishes connect a consumer to the queue and consume the messages With the broker still running, open a shell and periodically execute lsof | grep deleted on the root directory of the volume where the leveldb is configured to store its logs. You will see open deleted file descriptors taking space on the volume: [root@node0 leveldb]# lsof / | grep deleted java 12455 root 189r REG 253,0 10515424 263967 /opt/data/apps/leveldb/0000000000000000.log (deleted) java 12455 root 276r REG 253,0 10502173 263960 /opt/data/apps/leveldb/0000000000a073e0.log (deleted) java 12455 root 303r REG 253,0 10490856 263973 /opt/data/apps/leveldb/000000000140b3fd.log (deleted) java 12455 root 304r REG 253,0 10490739 263969 /opt/data/apps/leveldb/0000000001e0c7e5.log (deleted) java 12455 root 305r REG 253,0 10490928 263978 /opt/data/apps/leveldb/000000000280db58.log (deleted) java 12455 root 306r REG 253,0 10490784 263970 /opt/data/apps/leveldb/000000000320ef88.log (deleted) java 12455 root 307r REG 253,0 10490883 263980 /opt/data/apps/leveldb/0000000003c10328.log (deleted) java 12455 root 308r REG 253,0 10490838 263971 /opt/data/apps/leveldb/000000000461172b.log (deleted) java 12455 root 309r REG 253,0 10490892 263981 /opt/data/apps/leveldb/0000000005012b01.log (deleted) java 12455 root 310r REG 253,0 10490820 263972 /opt/data/apps/leveldb/0000000005a13f0d.log (deleted) java 12455 root 311r REG 253,0 10490982 263982 /opt/data/apps/leveldb/00000000064152d1.log (deleted) Repeat above and each time the producer and consumer are executed, the list of open descriptors grows. Shut down the broker and rerun the lsof command as above. The file descriptors are cleaned up.

SFDC Cases Counter:
SFDC Cases Links:

When using leveldb for persistence, AMQ leaves open deleted file descriptors when the persistence logs roll over. Severity seems to vary between file systems. On EXT4 the problem was more severe than on GFS2 in my development environment.6.1 GA (379) and 6.2 GA (133) were affected. 6.2P1 (143) did not seem to be affected by the issue.

Details

Description

Attachments

Activity

People

Dates