Accumulo 1.4.3 tablet servers failing to connect with master

--- UPDATE
Apparently a race condition exists in 1.4.3 which can sometimes prevent a tablet server from starting.  This bug has been fixed in version 1.4.4.  Upgrading to 1.4.4 should prevent this race condition from occuring and your tablet servers should start without issue.


I setup Accumulo 1.4.3 with a single hdfs data node and tablet
server.  Added a bit of data to it and once my additional hardware
resources were free up I am now trying to add 3 additional tablet
servers.  I already setup 3 hdfs datanodes, so I wanted to just run the
tserver processes on the same 3 servers:

Node1, Node2, Node3


I keep seeing this error with one or two nodes:

Uncaught exception in TabletServer.main, exiting
        java.lang.RuntimeException: java.lang.RuntimeException: Too many
retries, exiting.
                at
org.apache.accumulo.server.tabletserver.TabletServer.announceExistence(T
abletServer.java:2684)
                at
org.apache.accumulo.server.tabletserver.TabletServer.run(TabletServer.ja
va:2703)
                at
org.apache.accumulo.server.tabletserver.TabletServer.main(TabletServer.j
ava:3168)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
                at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
                at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.accumulo.start.Main$1.run(Main.java:89)
                at java.lang.Thread.run(Thread.java:662)
        Caused by: java.lang.RuntimeException: Too many retries,
exiting.
                at
org.apache.accumulo.server.tabletserver.TabletServer.announceExistence(T
abletServer.java:2681)
                ... 8 more


After verifying there were no issues with my host names, slaves, ips, dns, etc.  I shot an email out to the accumulo dev list.  Someone suggested that I disable IPv6 on the server.  Apparently Accumulo does not perform well with IPv6 enabled.

See the post just below this one for details on how to properly disable IPv6 in Centos 6.

Disable IPv6 in CentOS 6

#for persistence over reboots:
add the following to /etc/sysctl.conf:
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

#during runtime:
sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1

I did a # service network restart, just for kicks.

Export virtual machine from ESX server to OVF file

Previously I have posted an article on how to upload an OVA file to an VMware ESX server.  I recently had a task doing the opposite.  Here is an example using ovftool to download a vm from ESX server and creating portable OVF to deploy on another VMware server.

C:\Program Files\VMware\VMware OVF Tool>ovftool --noImageFiles "vi://root:password@esxHostname/ha-datacenter?ds=[datastore1] JBOSS 5.0.1.GA/JBOSS 5.0.1.GA.vmx" C:\Users\Charles\Desktop\l1-server.ovf

I have underlined above what you will need to change for your environment.  You can determine where your .vmx file is by looking at the disk settings for your VM in vSphere:

Cloudera CDH3 HDFS Fuse on CentOS 6.2

Goto this site (or the RPM Repo for your version of cloudera) and download 'lib-hdfs' and 'fuse' RPMs.

http://archive.cloudera.com/redhat/6/x86_64/cdh/3u2/RPMS/x86_64/

On your server that you want the mount point, install 'fuse' and 'fuse-libs' from the BaseRepo.

# yum install -y fuse
# yum install -y fuse-libs

Then Install the RPMs you downloaded (in this order)

# rpm -Uvh hadoop-0.20-libhdfs-0.20.2+923.142-1.x86_64.rpm
# rpm -Uvh hadoop-0.20-fuse-0.20.2+923.142-1.x86_64.rpm

# mkdir /mnt/fuse
# hadoop-fuse-dfs dfs://(namenode-hostname):(port) /mnt/fuse

Done!

The default port for a namenode is 8020

Deploy VMware OVA to Remote ESX

Often my customer site provides us with VMs that contain the framework which replicates their environment.  The big issue we have is that with only 8GB of RAM on the older corporate issued development laptops floating around, a VM that needs 6GB to run doesnt leave much space for jboss, eclipse and gwt developer. (And the usual Pandora, Outlook and Instant message client).

A work around I discovered is to deploy these virtual machines to a dedicated server running VMware ESX.

Traditionally I would upload an ISO to the datastore, create a VM configuration.  Mount the ISO, install the OS and configure the network settings manually.

If you already have a VM configured to use DHCP, then its a breeze with the ovftool offered by VMware.

The command is as follows:

ovftool -ds=datastore1 -nw="VM Network" -st=OVA C:\Users\Charles\Downloads\GM_Dev_Local-OWF6.ova vi://root:[password]@[host]

After 100%

That's it!

JBoss (Too Many Files Open)

I was redeploying JBoss on to a new server today.  It was already configured so i didn't think i would have any issues when starting.  However i quickly saw many java.io.Exceptions in the server log.

After a quick bit of research I discovered that the user I created to run JBoss, (webadmin) needed an increase in the maximum number of files that it could have open in the system at one time.

If using Cent OS 6.2  do so by editing the limits.conf file in the /etc/security folder. ( you will need to be root ).

Below is a screenshot of where i increased the limit to 200000 for webadmin.

After logging out and logging back in as webadmin.  the command # ulimit -Hn reported 200000 as the new limit on file descriptors.

After this i started JBoss and it worked like a charm.

Adding Space to Linux LVM

After looking around at many different forums I have figured out how to properly add additional space to your VM.

In my situation I had a RHEL5 Virtual Machine running in VMware ESXi.  The machine had 50GB of space and we quickly ran out.

The first thing I did was go into the Settings for the VM using vSphere and resized the physical disk from 50GB to 256GB.

The next step was to reboot the Virtual Machine so that it would recognize the larger physical disk.  I rebooted the machine and the larger physical disk was apparent when typing the command,

#fdisk -l


Disk /dev/sda: 274.8 GB, 274877906944 bytes
255 heads, 63 sectors/track, 33418 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14        6527    52323705   8e  Linux LVM

Notice that although there are 33418 cylinders in my new 274.8 GB disk, that my second primary partition /dev/sda2 is only using cylinders 14-6527.  Resizing the Partition was not a safe option and I found many links showing users that had tried and failed.

The proper way to add space is to create an additional partition.  This can be done with fdisk.

#fdisk /dev/sda

Once you are in the fdisk utility you will want to create a new partition. 'n'

Specify this as a primary partition (1-4).  I chose 3 , as I already have 2 primary partitions.

You should notice that it will than ask you where to 'start'.  It should default to the 1st cylinder available (6528 in my case).  Press Enter.  Then 'end' should default to the last cylinder on the disk. (33418 in my case).  Press Enter.

Once the partition has been created you will need to change the partition ID to LVM.  The ID for a Linux LVM partition is 8e.  

Remember to press 'w' to write your fdisk changes to disk.  Once this is done reboot your VM.

Once the VM is back up you will want to run #vgdisplay to see your VolumeGroup name.

[root@ssdev ~]# vgdisplay
  /dev/hdc: open failed: No medium found
  --- Volume group ---
  VG Name               VolGroup00...


# pvcreate /dev/sda3
# vgextend VolGroup00 /dev/sda3

Next extend the LV:
[root@ssdev ~]# vgdisplay
  /dev/hdc: open failed: No medium found
  --- Volume group ---
  VG Name               VolGroup00
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  10
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                7
  Open LV               7
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               255.84 GB
  PE Size               32.00 MB
  Total PE              8187
  Alloc PE / Size       1120 / 35.00 GB
  Free  PE / Size       7067 / 220.84 GB
  VG UUID               iF1177-sihW-YEb3-sLIj-AhVl-m6Fd-fMKQjg

[root@ssdev ~]# lvextend -L +220.84G /dev/VolGroup00/RootVol

In the last step i specified RootVol, as that is the volume i wanted resized ( / ).  lvdisplay will list the different volumes you have, and where they are mounted:

[root@ssdev ~]# lvdisplay

  /dev/hdc: open failed: No medium found

  --- Logical volume ---

  LV Name                /dev/VolGroup00/RootVol

  VG Name                VolGroup00

  LV UUID                N7lwvY-uqMg-2PfL-R4Ph-Dnh6-9hpu-a2vWeo

  LV Write Access        read/write

  LV Status              available

  # open                 1

  LV Size                4.00 GB

  Current LE             128

  Segments               1

  Allocation             inherit

  Read ahead sectors     auto

  - currently set to     256

  Block device           253:0

Last step, resize the filesystem on the LV you just extended:

[root@ssdev ~]# resize2fs -p /dev/VolGroup00/RootVol

resize2fs 1.39 (29-May-2006)

Filesystem at /dev/VolGroup00/RootVol is mounted on /; on-line resizing required

Performing an on-line resize of /dev/VolGroup00/RootVol to 58941440 (4k) blocks.

The filesystem on /dev/VolGroup00/RootVol is now 58941440 blocks long.

[root@ssdev ~]# df -m
Filesystem           1M-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-RootVol
                        223030      2322    209198   2% /
/dev/mapper/VolGroup00-VarVol
                          2976       137      2686   5% /var
/dev/mapper/VolGroup00-SrvVol
                         19839       855     17960   5% /srv
/dev/mapper/VolGroup00-VarLogVol
                          3968       182      3582   5% /var/log
/dev/mapper/VolGroup00-VarLogAuditVol
                           992        46       895   5% /var/log/audit
/dev/mapper/VolGroup00-HomeVol
                          1984        68      1815   4% /home
/dev/sda1                   99         9        86   9% /boot
tmpfs                     1510         0      1510   0% /dev/shm
[root@ssdev ~]#

This step should take a while, you can see the disk I/O in vSphere: