[2018-06-13T14:25:35,965][INFO ][o.e.n.Node               ] [elasticsearch-0] initializing ...
[2018-06-13T14:25:36,099][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-0] using [1] data paths, mounts [[/elasticsearch/persistent (/dev/mapper/rhel_b25-var)]], net usable_space [41.6gb], net total_space [199.9gb], spins? [possibly], types [xfs]
[2018-06-13T14:25:36,100][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-0] heap size [247.5mb], compressed ordinary object pointers [true]
[2018-06-13T14:25:36,102][INFO ][o.e.n.Node               ] [elasticsearch-0] node name [elasticsearch-0], node ID [qp-etb2kTqKeSgg4-9TFvw]
[2018-06-13T14:25:36,102][INFO ][o.e.n.Node               ] [elasticsearch-0] version[5.5.2], pid[1], build[b2f0c09/2017-08-14T12:33:14.154Z], OS[Linux/3.10.0-862.2.3.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_171/25.171-b10]
[2018-06-13T14:25:36,102][INFO ][o.e.n.Node               ] [elasticsearch-0] JVM arguments [-XX:+UnlockExperimentalVMOptions, -XX:+UseCGroupMemoryLimitForHeap, -XX:MaxRAMFraction=2, -XX:InitialRAMFraction=2, -XX:MinRAMFraction=2, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-06-13T14:25:38,030][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [aggs-matrix-stats]
[2018-06-13T14:25:38,030][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [ingest-common]
[2018-06-13T14:25:38,030][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [lang-expression]
[2018-06-13T14:25:38,030][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [lang-groovy]
[2018-06-13T14:25:38,031][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [lang-mustache]
[2018-06-13T14:25:38,031][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [lang-painless]
[2018-06-13T14:25:38,031][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [parent-join]
[2018-06-13T14:25:38,031][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [percolator]
[2018-06-13T14:25:38,031][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [reindex]
[2018-06-13T14:25:38,031][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [transport-netty3]
[2018-06-13T14:25:38,032][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [transport-netty4]
[2018-06-13T14:25:38,032][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] no plugins loaded
[2018-06-13T14:25:40,769][INFO ][o.e.d.DiscoveryModule    ] [elasticsearch-0] using discovery type [zen]
[2018-06-13T14:25:41,763][INFO ][o.e.n.Node               ] [elasticsearch-0] initialized
[2018-06-13T14:25:41,763][INFO ][o.e.n.Node               ] [elasticsearch-0] starting ...
[2018-06-13T14:25:42,256][INFO ][o.e.t.TransportService   ] [elasticsearch-0] publish_address {10.128.0.122:9300}, bound_addresses {[::]:9300}
[2018-06-13T14:25:42,284][INFO ][o.e.b.BootstrapChecks    ] [elasticsearch-0] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2018-06-13T14:25:45,408][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-0] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-0}{qp-etb2kTqKeSgg4-9TFvw}{nJ1dViSHRFGGucz-A6eGpw}{10.128.0.122}{10.128.0.122:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-06-13T14:25:48,411][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-0] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-0}{qp-etb2kTqKeSgg4-9TFvw}{nJ1dViSHRFGGucz-A6eGpw}{10.128.0.122}{10.128.0.122:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-06-13T14:25:51,414][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-0] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-0}{qp-etb2kTqKeSgg4-9TFvw}{nJ1dViSHRFGGucz-A6eGpw}{10.128.0.122}{10.128.0.122:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-06-13T14:25:54,417][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-0] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-0}{qp-etb2kTqKeSgg4-9TFvw}{nJ1dViSHRFGGucz-A6eGpw}{10.128.0.122}{10.128.0.122:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
ovpn-117-114:standalone kearls$ oc logs -f elasticsearch-0
[2018-06-13T14:26:02,965][INFO ][o.e.n.Node               ] [elasticsearch-0] initializing ...
[2018-06-13T14:26:03,115][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-0] using [1] data paths, mounts [[/elasticsearch/persistent (/dev/mapper/rhel_b25-var)]], net usable_space [41.6gb], net total_space [199.9gb], spins? [possibly], types [xfs]
[2018-06-13T14:26:03,116][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-0] heap size [247.5mb], compressed ordinary object pointers [true]
[2018-06-13T14:26:03,118][INFO ][o.e.n.Node               ] [elasticsearch-0] node name [elasticsearch-0], node ID [qp-etb2kTqKeSgg4-9TFvw]
[2018-06-13T14:26:03,118][INFO ][o.e.n.Node               ] [elasticsearch-0] version[5.5.2], pid[1], build[b2f0c09/2017-08-14T12:33:14.154Z], OS[Linux/3.10.0-862.2.3.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_171/25.171-b10]
[2018-06-13T14:26:03,119][INFO ][o.e.n.Node               ] [elasticsearch-0] JVM arguments [-XX:+UnlockExperimentalVMOptions, -XX:+UseCGroupMemoryLimitForHeap, -XX:MaxRAMFraction=2, -XX:InitialRAMFraction=2, -XX:MinRAMFraction=2, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-06-13T14:26:04,934][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [aggs-matrix-stats]
[2018-06-13T14:26:04,934][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [ingest-common]
[2018-06-13T14:26:04,934][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [lang-expression]
[2018-06-13T14:26:04,934][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [lang-groovy]
[2018-06-13T14:26:04,935][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [lang-mustache]
[2018-06-13T14:26:04,935][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [lang-painless]
[2018-06-13T14:26:04,935][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [parent-join]
[2018-06-13T14:26:04,935][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [percolator]
[2018-06-13T14:26:04,935][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [reindex]
[2018-06-13T14:26:04,935][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [transport-netty3]
[2018-06-13T14:26:04,935][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] loaded module [transport-netty4]
[2018-06-13T14:26:04,936][INFO ][o.e.p.PluginsService     ] [elasticsearch-0] no plugins loaded
[2018-06-13T14:26:07,772][INFO ][o.e.d.DiscoveryModule    ] [elasticsearch-0] using discovery type [zen]
[2018-06-13T14:26:08,711][INFO ][o.e.n.Node               ] [elasticsearch-0] initialized
[2018-06-13T14:26:08,711][INFO ][o.e.n.Node               ] [elasticsearch-0] starting ...
[2018-06-13T14:26:09,033][INFO ][o.e.t.TransportService   ] [elasticsearch-0] publish_address {10.128.0.122:9300}, bound_addresses {[::]:9300}
[2018-06-13T14:26:09,053][INFO ][o.e.b.BootstrapChecks    ] [elasticsearch-0] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2018-06-13T14:26:12,469][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-0] failed to connect to master [{elasticsearch-1}{hDTA2SSUT46mPEWQKDZfJg}{YdEtZhTIRfe_8v-8nIVxow}{10.128.0.124}{10.128.0.124:9300}], retrying...
org.elasticsearch.transport.ConnectTransportException: [elasticsearch-1][10.128.0.124:9300] general node connection failure
	at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:472) ~[elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:332) ~[elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:319) ~[elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.discovery.zen.ZenDiscovery.joinElectedMaster(ZenDiscovery.java:459) [elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:411) [elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.discovery.zen.ZenDiscovery.access$4100(ZenDiscovery.java:83) [elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1188) [elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.5.2.jar:5.5.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.lang.IllegalStateException: handshake failed
	at org.elasticsearch.transport.TcpTransport.executeHandshake(TcpTransport.java:1624) ~[elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:555) ~[elasticsearch-5.5.2.jar:5.5.2]
	... 11 more
Caused by: org.elasticsearch.transport.TransportException: connection reset
	at org.elasticsearch.transport.TcpTransport.onChannelClosed(TcpTransport.java:1666) ~[elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.transport.TcpTransport.lambda$openConnection$1(TcpTransport.java:539) ~[elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.transport.netty4.Netty4Transport.lambda$connectToChannels$2(Netty4Transport.java:354) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) ~[?:?]
	at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) ~[?:?]
	at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1058) ~[?:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:686) ~[?:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:664) ~[?:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:607) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:85) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:142) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
	... 1 more
[2018-06-13T14:26:15,604][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-0] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-0}{qp-etb2kTqKeSgg4-9TFvw}{ZXPhWVQyT0WgW8hIb25_Vw}{10.128.0.122}{10.128.0.122:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-06-13T14:26:18,609][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-0] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-0}{qp-etb2kTqKeSgg4-9TFvw}{ZXPhWVQyT0WgW8hIb25_Vw}{10.128.0.122}{10.128.0.122:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-06-13T14:26:30,373][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-0] failed to connect to master [{elasticsearch-2}{qQGd9DTTRnCip8rb54sjqw}{5z7rCQpETvymtJEHJ3mEhQ}{10.128.0.127}{10.128.0.127:9300}], retrying...
org.elasticsearch.transport.ConnectTransportException: [elasticsearch-2][10.128.0.127:9300] general node connection failure
	at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:484) ~[elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:332) ~[elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:319) ~[elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.discovery.zen.ZenDiscovery.joinElectedMaster(ZenDiscovery.java:459) [elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:411) [elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.discovery.zen.ZenDiscovery.access$4100(ZenDiscovery.java:83) [elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1188) [elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.5.2.jar:5.5.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.lang.IllegalStateException: handshake failed with {elasticsearch-2}{qQGd9DTTRnCip8rb54sjqw}{5z7rCQpETvymtJEHJ3mEhQ}{10.128.0.127}{10.128.0.127:9300}
	at org.elasticsearch.transport.TransportService.handshake(TransportService.java:403) ~[elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.transport.TransportService.lambda$connectToNode$3(TransportService.java:334) ~[elasticsearch-5.5.2.jar:5.5.2]
	at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.5.2.jar:5.5.2]
	... 10 more
Caused by: org.elasticsearch.transport.NodeDisconnectedException: [elasticsearch-2][10.128.0.127:9300][internal:transport/handshake] disconnected
ovpn-117-114:standalone kearls$ oc logs -f elasticsearch-0