Installing Accumulo 1.5.0 on Cloudera Managed CDH4 Cluster

The default Hadoop Prefix for accumulo-env.sh will not work with normal distribution of CDH4 with the Cloudera Manager 4 suite.  Here are my settings for the hadoop prefix and conf directory:

if [ -z "$HADOOP_HOME" ]
then
   test -z "$HADOOP_PREFIX"      && export HADOOP_PREFIX=/opt/cloudera/parcels/CDH/lib/hadoop  

else
   HADOOP_PREFIX="$HADOOP_HOME"
   unset HADOOP_HOME
fi 

test -z "$HADOOP_CONF_DIR"       && export HADOOP_CONF_DIR="$HADOOP_PREFIX/etc/hadoop"

# cdh4
export HADOOP_HDFS_HOME=/opt/cloudera/parcels/CDH/lib/hadoop-hdfs

export HADOOP_MAPREDUCE_HOME=/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce

Then, modify the classpath inside accumulo-site.xml:

  

    general.classpaths
   
      $ACCUMULO_HOME/server/target/classes/,
      $ACCUMULO_HOME/lib/accumulo-server.jar,
      $ACCUMULO_HOME/core/target/classes/,
      $ACCUMULO_HOME/lib/accumulo-core.jar,
      $ACCUMULO_HOME/start/target/classes/,
      $ACCUMULO_HOME/lib/accumulo-start.jar,
      $ACCUMULO_HOME/fate/target/classes/,
      $ACCUMULO_HOME/lib/accumulo-fate.jar,
      $ACCUMULO_HOME/proxy/target/classes/,
      $ACCUMULO_HOME/lib/accumulo-proxy.jar,
      $ACCUMULO_HOME/lib/[^.].*.jar,
      $ZOOKEEPER_HOME/zookeeper[^.].*.jar,
      $HADOOP_CONF_DIR,
      $HADOOP_PREFIX/[^.].*.jar,
      $HADOOP_PREFIX/lib/[^.].*.jar,
      $HADOOP_HDFS_HOME/.*.jar,
      $HADOOP_HDFS_HOME/lib/.*.jar,
      $HADOOP_MAPREDUCE_HOME/.*.jar,
      $HADOOP_MAPREDUCE_HOME/lib/.*.jar
   
    Classpaths that accumulo checks for updates and class files.
      When using the Security Manager, please remove the ".../target/classes/" values.
   
 

Increasing Read Times with Accumulo

A few weeks ago I wanted to increase performance of the Accumulo 1.5 so that scanning through large tables for information would happen a faster rate. I think I was getting around 200,000 entries per second prior to this performance modification.  I was able to increase the speed to what is now almost 3,000,000 entries per second with the following steps.

Stop Accumulo
I am able to stop the entire cluster by running the stop-all.sh script inside the Accumulo home folder's "bin" directory.

Increase JVM Heap Space to accommodate larger Index Cache 
The Tablet server heap space is defined in the file "accumulo-env.sh" located in the Accumulo home folder's "conf" directory.  Inside this folder you can see the settings for tablet server Xmx and Xms at the bottom defined as an environment variable, "$ACCUMULO_TSERVER_OPTS".  Depending on how much memory is available you will want to increase this value to support the increase we will make to the index cache next.  Here is my setting:
ACCUMULO_TSERVER_OPTS="${POLICY} -Xmx1024m -Xms512m "

Increase Index Cache
In the Accumulo home folder's "conf" directory, you should also see a file called "accumulo-site.xml".  Here you can define properties for the Accumulo cluster.  I have set the cache.index.size to 512M:

 
    tserver.cache.index.size
    512M
 

I have not had any issues with tablet server memory yet, so I believe this is a good fix.  Please provide feedback and comments below.

There are various other performance tweaks as well.  Such as NOT using LVM with CentOS/RHEL, and ensuring any virtual machines in the cluster are running with "Independent Disk Mode" so writes are flushed straight to disk. 

kinit: Ticket expired while renewing credentials

Using Kerberos with Hadoop to enable HDFS Security, you should be familiar with the concept of a user being given a 'ticket' that has an expiration date.

The default configuration for kerberos is to set a 'Maximum Renewal Time' of 00:00:00.  This is the ammount of time that is 'added' to the time that the ticket was issued.  So if your ticket was issued on Jan 27 2014 15:25:38, then you will have (Jan 27 2014 15:25:38 + Renewal Time) until you cannot renew the ticket.  This default setting makes it impossible to renew any tickets granted, and instead they need to be kdestoryed and re-issued.

In order to fix this, you need to log into the kerberos principal database using kadmin or kadmin.local on the KDC.  Use the following command for a principal named, 'hdfs@EXAMPLE.COM'

modprinc -maxrenewlife 1week hdfs@EXAMPLE.COM

You will then be able to look at the principal's information:

kadmin.local:  getprinc hdfs@EXAMPLE.COM 

Principal: hdfs@EXAMPLE.COM

Expiration date: [never]

Last password change: Sat Jan 25 21:33:34 EST 2014

Password expiration date: [none]

Maximum ticket life: 1 day 00:00:00

Maximum renewable life: 7 days 00:00:00

Last modified: Tue Jan 28 11:19:59 EST 2014 (root/admin@EXAMPLE.COM)

Last successful authentication: [never]

Last failed authentication: [never]

Failed password attempts: 0

Number of keys: 4

Key: vno 1, aes256-cts-hmac-sha1-96, no salt

Key: vno 1, aes128-cts-hmac-sha1-96, no salt

Key: vno 1, des3-cbc-sha1, no salt

Key: vno 1, arcfour-hmac, no salt

MKey: vno 1

You will still need to use 'kdestroy' and 'kinit' to get a new ticket, since the old one cannot be renewed.  However, your new ticket will be renewable for a week.  It should also be noted that when adding a new principal to the database, the default Renewal time will be obtained from the ticket granting server's Principal (krbtgt/EXAMPLE.COM@EXAMPLE.COM).  If you want all users added in the future to have a week long renewal period, you will need to modify that principal as well.

Increasing Entropy in VM for Kerberos

Today I was trying to setup a kerberos server as a virtual machine.  To my surprise, the kdb5_util was hanging/freezing when running, "kdb5_util create -s" at the step 'Loading random data'.

I did some research and apparently this has to do with the operating system not having enough 'entropy' to create 'quality randomness' for cryptography.   Here is a definition I found helpful:

In computing, entropy is the randomness collected by an operating system or application for use in cryptography or other uses that require random data. This randomness is often collected from hardware sources, either pre-existing ones such as mouse movements or specially provided randomness generators.

Apparently you can see how much 'entropy' your operating system has available by looking at the number contained in the file "/proc/sys/kernel/random/entropy_avail"

When the creation of the kerberos db was hanging, that file was reporting an entropy value of around 75.

So, I found this handy utility called 'haveged' (HArdware Volatile Entropy Gathering and Expansion), available here: http://www.issihosts.com/haveged/downloads.html

More info: http://www.irisa.fr/caps/projects/hipsor/

Of course I had to install a C++ compiler: "yum install -y gcc-c++.x86_64"
Then, $ tar -xvf haveged-x.x.tar.gz
Change into the haveged directory,
$ ./configure
$ make
$ make install
$ haveged -w 1024  #(this starts the daemon)
$ echo "/usr/local/sbin/haveged -w 1024" >> /etc/rc.local  #(this starts the daemon on boot)

Once this was done, my entropy available was showing values between 2000 and 3000.  When I then attempted to create the kerberos DB, it finished within a second! No problems!


...

[Edit]
Looks like I also needed to clear out the mess of 'principal' files in /var/kerberos/krb5kdc/ before re running the create.

"Cloud Manager" for Netbeans

Working on a side project to help automate server maintenance tasks for various open source distributed services.

Zookeeper, Storm, Accumulo, Hadoop, CentOS are the current software packages I want to manage with this tool.  The reason for providing it as a netbeans platform application is for a few reasons:

• Java can run on any platform.
• You don't need to know java to run a netbeans platform application.
• If you already know java, you can contribute to this application through netbeans.

If you want to contribute code or ideas for the project, you can do so through github.

https://github.com/charlescva/cloud-manager

Currently the tool allows you to add some server nodes, create actions for those nodes, and even assign a UI to the action for easier use.  JAXB is used for marshalling xml.  XSDs were generated against the XML on the Accumulo monitor.

SSH code is integrated.  One can easily deploy Storm topologies with the nimbus node action.

Export OVF from VMware Player

Get OVF Tool: https://my.vmware.com/web/vmware/details?downloadGroup=OVFTOOL350&productId=352

From CLI (windows for me, but should be same on other platforms):

C:\Program Files\VMware\VMware OVF Tool\> ovftool "C:\Users\Sam\Documents\Virtual Machines\VmDemo\VmDemo.vmx" C:\Users\Sam\Desktop\VmDemo.ovf

Opening VMX source: C:\Users\Sam\Documents\Virtual Machines\VmDemo\VmDemo.vmx
Opening OVF target: C:\Users\Sam\Desktop\VmDemo.ovf
Writing OVF package: C:\Users\Sam\Desktop\VmDemo.ovf
Transfer Completed
Completed successfully

C:\Program Files\VMware\VMware OVF Tool> _

Java SSH Library

http://www.ganymed.ethz.ch/ssh2/ is the perfect library for making common SSH/SCP calls.  It even support SFTP.  I am using it in my management platform application.  I included the source package with mine, as well as the software license.  I hope that is adequate for anyone seeking to ensure I am not violating any laws in regard to sharing source.  I have also made a good effort to publicize my use of this code, as to not insinuate that it is in any way mine.