Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-7652

[EAP] Split brain occurs when to disconnect network between master and NFS server temporary

XMLWordPrintable

    • False
    • None
    • False
    • Hide
      1. Start master and backup servers using NFS, as usual.
      2. Disconnect network between master and NFS server using tc commands
      3. Wait for the backup to become live status (for about 90 seconds)
      4. Restore the network between master and NFS server using tc commands
        => Both master and backup will go live with split brain.
        => Messages could be sent to both master and backup.
        => After that, "kernel: NFS: __nfs4_reclaim_open_state: Lock reclaim failed!” is logged in /var/log/messages in master but master doesn’t stop being live status.
      tc command: I used tc command wrapper, tcconfig, on the master server to emulate network disconnect
          ~~~
          # tcset eth0 --loss 100% --network 10.0.1.172 --overwrite --tc-command
          /usr/sbin/tc qdisc del dev eth0 root
          /usr/sbin/tc qdisc del dev eth0 ingress
          /usr/sbin/tc qdisc del dev ifb6682 root
          /usr/sbin/ip link set dev ifb6682 down
          /usr/sbin/ip link delete ifb6682 type ifb
          /usr/sbin/tc qdisc add dev eth0 root handle 1a1a: htb default 1
          /usr/sbin/tc class add dev eth0 parent 1a1a: classid 1a1a:1 htb rate 32000000.0kbit
          /usr/sbin/tc class add dev eth0 parent 1a1a: classid 1a1a:96 htb rate 32000000.0Kbit ceil 32000000.0Kbit
          /usr/sbin/tc qdisc add dev eth0 parent 1a1a:96 handle 2e17: netem loss 100.000000%
          /usr/sbin/tc filter add dev eth0 protocol ip parent 1a1a: prio 5 u32 match ip dst 10.0.1.172/32 match ip src 0.0.0.0/0 flowid 1a1a:96 
      
      Show
      Start master and backup servers using NFS, as usual. Disconnect network between master and NFS server using tc commands Wait for the backup to become live status (for about 90 seconds) Restore the network between master and NFS server using tc commands => Both master and backup will go live with split brain. => Messages could be sent to both master and backup. => After that, "kernel: NFS: __nfs4_reclaim_open_state: Lock reclaim failed!” is logged in /var/log/messages in master but master doesn’t stop being live status. tc command: I used tc command wrapper, tcconfig, on the master server to emulate network disconnect     ~~~     # tcset eth0 --loss 100% --network 10.0.1.172 --overwrite --tc-command     /usr/sbin/tc qdisc del dev eth0 root     /usr/sbin/tc qdisc del dev eth0 ingress     /usr/sbin/tc qdisc del dev ifb6682 root     /usr/sbin/ip link set dev ifb6682 down     /usr/sbin/ip link delete ifb6682 type ifb     /usr/sbin/tc qdisc add dev eth0 root handle 1a1a: htb default 1     /usr/sbin/tc class add dev eth0 parent 1a1a: classid 1a1a:1 htb rate 32000000.0kbit     /usr/sbin/tc class add dev eth0 parent 1a1a: classid 1a1a:96 htb rate 32000000.0Kbit ceil 32000000.0Kbit     /usr/sbin/tc qdisc add dev eth0 parent 1a1a:96 handle 2e17: netem loss 100.000000%     /usr/sbin/tc filter add dev eth0 protocol ip parent 1a1a: prio 5 u32 match ip dst 10.0.1.172/32 match ip src 0.0.0.0/0 flowid 1a1a:96 

      Please backport ENTMQBR-7130 to the 7.8.x branches for inclusion into JBoss EAP 7.4.x

            rhn-support-jbertram Justin Bertram
            rhn-support-tyamashi Tomonari Yamashita
            Roman Vais Roman Vais
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: