Uploaded image for project: 'JBoss A-MQ'
  1. JBoss A-MQ
  2. ENTMQ-1411

A-MQ data corruption in case of file system full

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Critical Critical
    • None
    • JBoss A-MQ 6.2
    • broker, kahadb
    • None
    • Hide

      1. Unzip an original A-MQ 6.2.0 and activate (i.e. uncomment) the admin/admin account in etc/users.properties
      2. Install Patch P2 (v -153) using the patch:add and patch:install commands
      3. Remove the memoryLimit on the queue policy in activemq.xml (to facilitate enqueuing of messages)
      4. Start the broker: bin/start
      5. Take note of your "Store limit" which might depend on the available space in your journal data file system (e.g. look at the tree root "amq" in hawtio GUI, attribute "Store limit")
      6. Fill-in the journal data file system with garbage files, until available space is less than your "Store limit"
      7. Inject "enough" persistent messages i.e. more than what remains available in your journal data file system. E.g. To inject approx. 300MB of messages, use: bin/client "activemq:producer --user admin --password admin --messageSize 100000 --messageCount 3000 --transactionBatchSize 30 --destination testQ"
      -> The injector fails with an exception stating "No space left on device"
      9. Stop the broker: bin/stop
      10. Delete the garbage file(s) you have set at step (6.)
      11. Start the broker again: bin/start
      -> The broker logs show: "Corrupt journal records found...", "Broker amq failed to start" and "Failed to recover data at position:XX:YYY"

      Show
      1. Unzip an original A-MQ 6.2.0 and activate (i.e. uncomment) the admin/admin account in etc/users.properties 2. Install Patch P2 (v -153) using the patch:add and patch:install commands 3. Remove the memoryLimit on the queue policy in activemq.xml (to facilitate enqueuing of messages) 4. Start the broker: bin/start 5. Take note of your "Store limit" which might depend on the available space in your journal data file system (e.g. look at the tree root "amq" in hawtio GUI, attribute "Store limit") 6. Fill-in the journal data file system with garbage files, until available space is less than your "Store limit" 7. Inject "enough" persistent messages i.e. more than what remains available in your journal data file system. E.g. To inject approx. 300MB of messages, use: bin/client "activemq:producer --user admin --password admin --messageSize 100000 --messageCount 3000 --transactionBatchSize 30 --destination testQ" -> The injector fails with an exception stating "No space left on device" 9. Stop the broker: bin/stop 10. Delete the garbage file(s) you have set at step (6.) 11. Start the broker again: bin/start -> The broker logs show: "Corrupt journal records found...", "Broker amq failed to start" and "Failed to recover data at position:XX:YYY"

      When hitting filesystem full on data directory, if amq is shut down, some space freed up and amq started again, the broker data is corrupted (amq.log shows "Corrupt journal records"), so
      *not only*the indexThe broker would typically not even start (which I guess was my example) if those data are live data.

            Unassigned Unassigned
            rhn-support-gbonocor Giuseppe Bonocore (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: