Search This Blog

Thursday, May 30, 2013

Accumulo 1.4.3 tablet servers failing to connect with master

--- UPDATE
Apparently a race condition exists in 1.4.3 which can sometimes prevent a tablet server from starting.  This bug has been fixed in version 1.4.4.  Upgrading to 1.4.4 should prevent this race condition from occuring and your tablet servers should start without issue.


I setup Accumulo 1.4.3 with a single hdfs data node and tablet
server.  Added a bit of data to it and once my additional hardware
resources were free up I am now trying to add 3 additional tablet
servers.  I already setup 3 hdfs datanodes, so I wanted to just run the
tserver processes on the same 3 servers:

Node1, Node2, Node3


I keep seeing this error with one or two nodes:

Uncaught exception in TabletServer.main, exiting
        java.lang.RuntimeException: java.lang.RuntimeException: Too many
retries, exiting.
                at
org.apache.accumulo.server.tabletserver.TabletServer.announceExistence(T
abletServer.java:2684)
                at
org.apache.accumulo.server.tabletserver.TabletServer.run(TabletServer.ja
va:2703)
                at
org.apache.accumulo.server.tabletserver.TabletServer.main(TabletServer.j
ava:3168)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
                at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
                at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.accumulo.start.Main$1.run(Main.java:89)
                at java.lang.Thread.run(Thread.java:662)
        Caused by: java.lang.RuntimeException: Too many retries,
exiting.
                at
org.apache.accumulo.server.tabletserver.TabletServer.announceExistence(T
abletServer.java:2681)
                ... 8 more


After verifying there were no issues with my host names, slaves, ips, dns, etc.  I shot an email out to the accumulo dev list.  Someone suggested that I disable IPv6 on the server.  Apparently Accumulo does not perform well with IPv6 enabled.

See the post just below this one for details on how to properly disable IPv6 in Centos 6.

No comments:

Post a Comment