kinit: Ticket expired while renewing credentials

Using Kerberos with Hadoop to enable HDFS Security, you should be familiar with the concept of a user being given a 'ticket' that has an expiration date.

The default configuration for kerberos is to set a 'Maximum Renewal Time' of 00:00:00.  This is the ammount of time that is 'added' to the time that the ticket was issued.  So if your ticket was issued on Jan 27 2014 15:25:38, then you will have (Jan 27 2014 15:25:38 + Renewal Time) until you cannot renew the ticket.  This default setting makes it impossible to renew any tickets granted, and instead they need to be kdestoryed and re-issued.

In order to fix this, you need to log into the kerberos principal database using kadmin or kadmin.local on the KDC.  Use the following command for a principal named, 'hdfs@EXAMPLE.COM'

modprinc -maxrenewlife 1week hdfs@EXAMPLE.COM

You will then be able to look at the principal's information:

kadmin.local:  getprinc hdfs@EXAMPLE.COM 

Principal: hdfs@EXAMPLE.COM

Expiration date: [never]

Last password change: Sat Jan 25 21:33:34 EST 2014

Password expiration date: [none]

Maximum ticket life: 1 day 00:00:00

Maximum renewable life: 7 days 00:00:00

Last modified: Tue Jan 28 11:19:59 EST 2014 (root/admin@EXAMPLE.COM)

Last successful authentication: [never]

Last failed authentication: [never]

Failed password attempts: 0

Number of keys: 4

Key: vno 1, aes256-cts-hmac-sha1-96, no salt

Key: vno 1, aes128-cts-hmac-sha1-96, no salt

Key: vno 1, des3-cbc-sha1, no salt

Key: vno 1, arcfour-hmac, no salt

MKey: vno 1

You will still need to use 'kdestroy' and 'kinit' to get a new ticket, since the old one cannot be renewed.  However, your new ticket will be renewable for a week.  It should also be noted that when adding a new principal to the database, the default Renewal time will be obtained from the ticket granting server's Principal (krbtgt/EXAMPLE.COM@EXAMPLE.COM).  If you want all users added in the future to have a week long renewal period, you will need to modify that principal as well.

Increasing Entropy in VM for Kerberos

Today I was trying to setup a kerberos server as a virtual machine.  To my surprise, the kdb5_util was hanging/freezing when running, "kdb5_util create -s" at the step 'Loading random data'.

I did some research and apparently this has to do with the operating system not having enough 'entropy' to create 'quality randomness' for cryptography.   Here is a definition I found helpful:

In computing, entropy is the randomness collected by an operating system or application for use in cryptography or other uses that require random data. This randomness is often collected from hardware sources, either pre-existing ones such as mouse movements or specially provided randomness generators.

Apparently you can see how much 'entropy' your operating system has available by looking at the number contained in the file "/proc/sys/kernel/random/entropy_avail"

When the creation of the kerberos db was hanging, that file was reporting an entropy value of around 75.

So, I found this handy utility called 'haveged' (HArdware Volatile Entropy Gathering and Expansion), available here: http://www.issihosts.com/haveged/downloads.html

More info: http://www.irisa.fr/caps/projects/hipsor/

Of course I had to install a C++ compiler: "yum install -y gcc-c++.x86_64"
Then, $ tar -xvf haveged-x.x.tar.gz
Change into the haveged directory,
$ ./configure
$ make
$ make install
$ haveged -w 1024  #(this starts the daemon)
$ echo "/usr/local/sbin/haveged -w 1024" >> /etc/rc.local  #(this starts the daemon on boot)

Once this was done, my entropy available was showing values between 2000 and 3000.  When I then attempted to create the kerberos DB, it finished within a second! No problems!


...

[Edit]
Looks like I also needed to clear out the mess of 'principal' files in /var/kerberos/krb5kdc/ before re running the create.

"Cloud Manager" for Netbeans

Working on a side project to help automate server maintenance tasks for various open source distributed services.

Zookeeper, Storm, Accumulo, Hadoop, CentOS are the current software packages I want to manage with this tool.  The reason for providing it as a netbeans platform application is for a few reasons:

• Java can run on any platform.
• You don't need to know java to run a netbeans platform application.
• If you already know java, you can contribute to this application through netbeans.

If you want to contribute code or ideas for the project, you can do so through github.

https://github.com/charlescva/cloud-manager

Currently the tool allows you to add some server nodes, create actions for those nodes, and even assign a UI to the action for easier use.  JAXB is used for marshalling xml.  XSDs were generated against the XML on the Accumulo monitor.

SSH code is integrated.  One can easily deploy Storm topologies with the nimbus node action.

Export OVF from VMware Player

Get OVF Tool: https://my.vmware.com/web/vmware/details?downloadGroup=OVFTOOL350&productId=352

From CLI (windows for me, but should be same on other platforms):

C:\Program Files\VMware\VMware OVF Tool\> ovftool "C:\Users\Sam\Documents\Virtual Machines\VmDemo\VmDemo.vmx" C:\Users\Sam\Desktop\VmDemo.ovf

Opening VMX source: C:\Users\Sam\Documents\Virtual Machines\VmDemo\VmDemo.vmx
Opening OVF target: C:\Users\Sam\Desktop\VmDemo.ovf
Writing OVF package: C:\Users\Sam\Desktop\VmDemo.ovf
Transfer Completed
Completed successfully

C:\Program Files\VMware\VMware OVF Tool> _

Java SSH Library

http://www.ganymed.ethz.ch/ssh2/ is the perfect library for making common SSH/SCP calls.  It even support SFTP.  I am using it in my management platform application.  I included the source package with mine, as well as the software license.  I hope that is adequate for anyone seeking to ensure I am not violating any laws in regard to sharing source.  I have also made a good effort to publicize my use of this code, as to not insinuate that it is in any way mine.

Elaborate Linux Script for determining IP

Sometimes the ordering of your Ethernet adapters are not the same on a specific machine vs the others in your cluster.  So simply grepping the IP based on adapter name can be a little tricky.  So I just made this script to do it for you.

#! /usr/bin/env bash
for network in $(cat /proc/net/dev | grep ':' | cut -d: -f1 | awk '{ print $1 }'); do
if [ "$network" != "lo" ]
then
echo $(/sbin/ifconfig $network | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}')
fi
done

Clearly you could modify the if condition to fit your needs.

Change Linux system clock safely with bash script

I learned the hard way a month or two back that you cannot just change the server time when running zookeeper with distributed cloud services on a single server.  Often changing the system clock to a time which is greater than the zookeeper session time (2x the tick time, ~ 4000 ms by default) you can easily break zookeeper session synchronization.

The fix is to change the time little by little until it is where you want it.  Here is an example in which I speed up the clock by 1 second every 10 seconds.  This has been tested and worked while cloud services are running without breaking anything.

#! /usr/bin/env bash

#Increase server time by 960 seconds(16mins) over a 9600 second(2hrs40mins) period.

for i in {1..960}

do

later=$(date --date="+1 second")

echo $later

date -s "$later"

sleep 10

done