Search This Blog

Thursday, May 30, 2013

Accumulo 1.4.3 tablet servers failing to connect with master

--- UPDATE
Apparently a race condition exists in 1.4.3 which can sometimes prevent a tablet server from starting.  This bug has been fixed in version 1.4.4.  Upgrading to 1.4.4 should prevent this race condition from occuring and your tablet servers should start without issue.


I setup Accumulo 1.4.3 with a single hdfs data node and tablet
server.  Added a bit of data to it and once my additional hardware
resources were free up I am now trying to add 3 additional tablet
servers.  I already setup 3 hdfs datanodes, so I wanted to just run the
tserver processes on the same 3 servers:

Node1, Node2, Node3


I keep seeing this error with one or two nodes:

Uncaught exception in TabletServer.main, exiting
        java.lang.RuntimeException: java.lang.RuntimeException: Too many
retries, exiting.
                at
org.apache.accumulo.server.tabletserver.TabletServer.announceExistence(T
abletServer.java:2684)
                at
org.apache.accumulo.server.tabletserver.TabletServer.run(TabletServer.ja
va:2703)
                at
org.apache.accumulo.server.tabletserver.TabletServer.main(TabletServer.j
ava:3168)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
                at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
                at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.accumulo.start.Main$1.run(Main.java:89)
                at java.lang.Thread.run(Thread.java:662)
        Caused by: java.lang.RuntimeException: Too many retries,
exiting.
                at
org.apache.accumulo.server.tabletserver.TabletServer.announceExistence(T
abletServer.java:2681)
                ... 8 more


After verifying there were no issues with my host names, slaves, ips, dns, etc.  I shot an email out to the accumulo dev list.  Someone suggested that I disable IPv6 on the server.  Apparently Accumulo does not perform well with IPv6 enabled.

See the post just below this one for details on how to properly disable IPv6 in Centos 6.

Disable IPv6 in CentOS 6

#for persistence over reboots:
add the following to /etc/sysctl.conf:
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

#during runtime:
sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1

I did a # service network restart, just for kicks.

Wednesday, May 29, 2013

Export virtual machine from ESX server to OVF file

Previously I have posted an article on how to upload an OVA file to an VMware ESX server.  I recently had a task doing the opposite.  Here is an example using ovftool to download a vm from ESX server and creating portable OVF to deploy on another VMware server.

C:\Program Files\VMware\VMware OVF Tool>ovftool --noImageFiles "vi://root:password@esxHostname/ha-datacenter?ds=[datastore1] JBOSS 5.0.1.GA/JBOSS 5.0.1.GA.vmx" C:\Users\Charles\Desktop\l1-server.ovf


I have underlined above what you will need to change for your environment.  You can determine where your .vmx file is by looking at the disk settings for your VM in vSphere: