Search This Blog

Friday, November 1, 2013

Export OVF from VMware Player

Get OVF Tool: https://my.vmware.com/web/vmware/details?downloadGroup=OVFTOOL350&productId=352

From CLI (windows for me, but should be same on other platforms):

C:\Program Files\VMware\VMware OVF Tool\> ovftool "C:\Users\Sam\Documents\Virtual Machines\VmDemo\VmDemo.vmx" C:\Users\Sam\Desktop\VmDemo.ovf

Opening VMX source: C:\Users\Sam\Documents\Virtual Machines\VmDemo\VmDemo.vmx
Opening OVF target: C:\Users\Sam\Desktop\VmDemo.ovf
Writing OVF package: C:\Users\Sam\Desktop\VmDemo.ovf
Transfer Completed
Completed successfully

C:\Program Files\VMware\VMware OVF Tool> _

Thursday, August 29, 2013

Java SSH Library

http://www.ganymed.ethz.ch/ssh2/ is the perfect library for making common SSH/SCP calls.  It even support SFTP.  I am using it in my management platform application.  I included the source package with mine, as well as the software license.  I hope that is adequate for anyone seeking to ensure I am not violating any laws in regard to sharing source.  I have also made a good effort to publicize my use of this code, as to not insinuate that it is in any way mine.

Wednesday, August 28, 2013

Elaborate Linux Script for determining IP

Sometimes the ordering of your Ethernet adapters are not the same on a specific machine vs the others in your cluster.  So simply grepping the IP based on adapter name can be a little tricky.  So I just made this script to do it for you.

#! /usr/bin/env bash
for network in $(cat /proc/net/dev | grep ':' | cut -d: -f1 | awk '{ print $1 }'); do
if [ "$network" != "lo" ]
then
echo $(/sbin/ifconfig $network | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}')
fi
done


Clearly you could modify the if condition to fit your needs.

Monday, July 29, 2013

Change Linux system clock safely with bash script

I learned the hard way a month or two back that you cannot just change the server time when running zookeeper with distributed cloud services on a single server.  Often changing the system clock to a time which is greater than the zookeeper session time (2x the tick time, ~ 4000 ms by default) you can easily break zookeeper session synchronization.

The fix is to change the time little by little until it is where you want it.  Here is an example in which I speed up the clock by 1 second every 10 seconds.  This has been tested and worked while cloud services are running without breaking anything.

#! /usr/bin/env bash

#Increase server time by 960 seconds(16mins) over a 9600 second(2hrs40mins) period.

for i in {1..960}
do
later=$(date --date="+1 second")
echo $later
date -s "$later"
sleep 10
done

Monday, June 10, 2013

MongoDB Unclean Shutdown

My laptop battery ran out of juice and dumped all standby data from RAM.  When I booted the system back up, and tried to start my mongodb instance.  I received the following error:

Unclean shutdown detected.
Please visit http://dochub.mongodb.org/core/repair for recovery instructions.

If you don't want to read through the documentation to fix the error above re-run mongod with the following usage:

mongod(.exe) --repair --repairpath <(c:)/path/to/data> --dbpath <(c:)/path/to/data>


Once it is complete, start mongo db as usual.


Successful Windows Console Output:
C:\dev\mongodb-win32-i386-2.4.4\bin>mongod --repair --repairpath c:/dev/data --d
bpath c:/dev/data
Mon Jun 10 19:22:16.312
Mon Jun 10 19:22:16.312 warning: 32-bit servers don't have journaling enabled by
 default. Please use --journal if you want durability.
Mon Jun 10 19:22:16.328
Mon Jun 10 19:22:16.359 [initandlisten] MongoDB starting : pid=3936 port=27017 d
bpath=c:/dev/data 32-bit host=samson-pc
Mon Jun 10 19:22:16.359 [initandlisten]
Mon Jun 10 19:22:16.359 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary.
Mon Jun 10 19:22:16.359 [initandlisten] **       32 bit builds are limited to less than 2GB of data (or less with --journal).
Mon Jun 10 19:22:16.359 [initandlisten] **       Note that journaling defaults to off for 32 bit and is currently off.
Mon Jun 10 19:22:16.359 [initandlisten] **       See http://dochub.mongodb.org/core/32bit
Mon Jun 10 19:22:16.359 [initandlisten]
Mon Jun 10 19:22:16.359 [initandlisten] ** NOTE: your operating system version does not support the method that MongoDB
Mon Jun 10 19:22:16.359 [initandlisten] **       uses to detect impending page faults.
Mon Jun 10 19:22:16.359 [initandlisten] **       This may result in slower performance for certain use cases
Mon Jun 10 19:22:16.359 [initandlisten]
Mon Jun 10 19:22:16.359 [initandlisten] db version v2.4.4
Mon Jun 10 19:22:16.375 [initandlisten] git version: 4ec1fb96702c9d4c57b1e06dd34eb73a16e407d2
Mon Jun 10 19:22:16.375 [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=0, build=6002, platform=2, service_pack='Service Pack 2') BOOST_LIB_VERSION=1_49
Mon Jun 10 19:22:16.375 [initandlisten] allocator: system
Mon Jun 10 19:22:16.375 [initandlisten] options: { dbpath: "c:/dev/data", repair: true, repairpath: "c:/dev/data" }
Mon Jun 10 19:22:16.578 [initandlisten] finished checking dbs
Mon Jun 10 19:22:16.578 dbexit:
Mon Jun 10 19:22:16.578 [initandlisten] shutdown: going to close listening sockets...
Mon Jun 10 19:22:16.578 [initandlisten] shutdown: going to flush diaglog...
Mon Jun 10 19:22:16.578 [initandlisten] shutdown: going to close sockets...
Mon Jun 10 19:22:16.578 [initandlisten] shutdown: waiting for fs preallocator...
Mon Jun 10 19:22:16.578 [initandlisten] shutdown: closing all files...
Mon Jun 10 19:22:16.578 [initandlisten] closeAllFiles() finished
Mon Jun 10 19:22:16.578 [initandlisten] shutdown: removing fs lock...
Mon Jun 10 19:22:16.578 dbexit: really exiting now

Wednesday, June 5, 2013

Java and Google Finance

Here are a couple code snippets using JDK6 to fetch data from the Google Finance website.  
Here are some default variables.  The base URL is the google site that will return a json string with requested symbols.  The symbols file is sitting on my Desktop and is just a CSV with 1 column.  Each record is a stock symbol. e.g. JNJ
The Boolean friendlyPrint variable determines the output format.
 private static String defaultBaseUrl = "http://www.google.com/finance/info?infotype=infoquoteall&q=";  
  private static String defaultSymbolsFile = System.getProperty("user.home")  
   + "/Desktop/NasdaqSymbols.csv";  
  private static boolean friendlyPrint = true;  
This main method first creates an ArrayList of the first 100 stock symbols from the CSV, and then appends them with a comma to the base url.  The URL is submitted and the response is parsed into a Simple-JSON Array. Last, the data is displayed on the console.
  public static void main(String[] args) throws Exception {  
  ArrayList symbols = FileUtils.getTextListValues(defaultSymbolsFile);  
  StringBuilder syms = new StringBuilder();   
  String sb = Util.fetchURL(defaultBaseUrl + syms);  
  JSONArray json = (JSONArray) new JSONParser().parse(sb.substring(3));  
  console.utils.GoogleFinancePrinter.printJsonArray(json, friendlyPrint);  
  }  
This is the method called above to fetch the data using the URLConnection object. It reads each line of the response and passes it into a stringbuilder. Then returns the final string.
 static String fetchURL(String url) throws Exception {   
  URL gf = new URL(url);   
  URLConnection yc = gf.openConnection();   
  BufferedReader in = new BufferedReader(new InputStreamReader(   
   yc.getInputStream()));   
  StringBuilder sb = new StringBuilder();   
  String inputLine;   
  while ((inputLine = in.readLine()) != null)   
   sb.append(inputLine);   
  in.close();   
  return sb.toString();   
  }   
Here is the method that prints the data on the console. There is more data in the json object than what is displayed in the friendly condition. These are just a few.
  public static void printJsonArray(JSONArray json, boolean friendly) {  
  if (friendly) {  
   for (int i = 0; i < json.size(); i++) {  
   System.out.println("ID: "  
    + ((JSONObject) json.get(i)).get("id"));  
   System.out.println("Symbol: "  
    + ((JSONObject) json.get(i)).get("t"));  
   System.out.println("Name: "  
    + ((JSONObject) json.get(i)).get("name"));  
   System.out.println("Type: "  
    + ((JSONObject) json.get(i)).get("type"));  
   System.out.println("Exchange: "  
    + ((JSONObject) json.get(i)).get("e"));  
   System.out.println("LastTrade: $"  
    + ((JSONObject) json.get(i)).get("l"));  
   System.out.println("Last Trade: "  
    + ((JSONObject) json.get(i)).get("lt"));  
   System.out.println("Change: "  
    + ((JSONObject) json.get(i)).get("c") + "("  
    + ((JSONObject) json.get(i)).get("cp") + "%)");  
   System.out.println("------------------------------");  
   }  
  } else {  
   System.out.print(json.toJSONString());  
  }  
  }  

Thursday, May 30, 2013

Accumulo 1.4.3 tablet servers failing to connect with master

--- UPDATE
Apparently a race condition exists in 1.4.3 which can sometimes prevent a tablet server from starting.  This bug has been fixed in version 1.4.4.  Upgrading to 1.4.4 should prevent this race condition from occuring and your tablet servers should start without issue.


I setup Accumulo 1.4.3 with a single hdfs data node and tablet
server.  Added a bit of data to it and once my additional hardware
resources were free up I am now trying to add 3 additional tablet
servers.  I already setup 3 hdfs datanodes, so I wanted to just run the
tserver processes on the same 3 servers:

Node1, Node2, Node3


I keep seeing this error with one or two nodes:

Uncaught exception in TabletServer.main, exiting
        java.lang.RuntimeException: java.lang.RuntimeException: Too many
retries, exiting.
                at
org.apache.accumulo.server.tabletserver.TabletServer.announceExistence(T
abletServer.java:2684)
                at
org.apache.accumulo.server.tabletserver.TabletServer.run(TabletServer.ja
va:2703)
                at
org.apache.accumulo.server.tabletserver.TabletServer.main(TabletServer.j
ava:3168)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
                at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
                at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.accumulo.start.Main$1.run(Main.java:89)
                at java.lang.Thread.run(Thread.java:662)
        Caused by: java.lang.RuntimeException: Too many retries,
exiting.
                at
org.apache.accumulo.server.tabletserver.TabletServer.announceExistence(T
abletServer.java:2681)
                ... 8 more


After verifying there were no issues with my host names, slaves, ips, dns, etc.  I shot an email out to the accumulo dev list.  Someone suggested that I disable IPv6 on the server.  Apparently Accumulo does not perform well with IPv6 enabled.

See the post just below this one for details on how to properly disable IPv6 in Centos 6.

Disable IPv6 in CentOS 6

#for persistence over reboots:
add the following to /etc/sysctl.conf:
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

#during runtime:
sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1

I did a # service network restart, just for kicks.

Wednesday, May 29, 2013

Export virtual machine from ESX server to OVF file

Previously I have posted an article on how to upload an OVA file to an VMware ESX server.  I recently had a task doing the opposite.  Here is an example using ovftool to download a vm from ESX server and creating portable OVF to deploy on another VMware server.

C:\Program Files\VMware\VMware OVF Tool>ovftool --noImageFiles "vi://root:password@esxHostname/ha-datacenter?ds=[datastore1] JBOSS 5.0.1.GA/JBOSS 5.0.1.GA.vmx" C:\Users\Charles\Desktop\l1-server.ovf


I have underlined above what you will need to change for your environment.  You can determine where your .vmx file is by looking at the disk settings for your VM in vSphere:




Saturday, April 6, 2013

Cloudera CDH3 HDFS Fuse on CentOS 6.2

Goto this site (or the RPM Repo for your version of cloudera) and download 'lib-hdfs' and 'fuse' RPMs.

http://archive.cloudera.com/redhat/6/x86_64/cdh/3u2/RPMS/x86_64/

On your server that you want the mount point, install 'fuse' and 'fuse-libs' from the BaseRepo.

# yum install -y fuse
# yum install -y fuse-libs

Then Install the RPMs you downloaded (in this order)

# rpm -Uvh hadoop-0.20-libhdfs-0.20.2+923.142-1.x86_64.rpm
# rpm -Uvh hadoop-0.20-fuse-0.20.2+923.142-1.x86_64.rpm

# mkdir /mnt/fuse
# hadoop-fuse-dfs dfs://(namenode-hostname):(port) /mnt/fuse

Done!

The default port for a namenode is 8020


Tuesday, January 29, 2013

Deploy VMware OVA to Remote ESX

Often my customer site provides us with VMs that contain the framework which replicates their environment.  The big issue we have is that with only 8GB of RAM on the older corporate issued development laptops floating around, a VM that needs 6GB to run doesnt leave much space for jboss, eclipse and gwt developer. (And the usual Pandora, Outlook and Instant message client).

A work around I discovered is to deploy these virtual machines to a dedicated server running VMware ESX.

Traditionally I would upload an ISO to the datastore, create a VM configuration.  Mount the ISO, install the OS and configure the network settings manually.

If you already have a VM configured to use DHCP, then its a breeze with the ovftool offered by VMware.

The command is as follows:

ovftool -ds=datastore1 -nw="VM Network" -st=OVA C:\Users\Charles\Downloads\GM_Dev_Local-OWF6.ova vi://root:[password]@[host]



After 100%




That's it!

JBoss (Too Many Files Open)

I was redeploying JBoss on to a new server today.  It was already configured so i didn't think i would have any issues when starting.  However i quickly saw many java.io.Exceptions in the server log.

After a quick bit of research I discovered that the user I created to run JBoss, (webadmin) needed an increase in the maximum number of files that it could have open in the system at one time.

If using Cent OS 6.2  do so by editing the limits.conf file in the /etc/security folder. ( you will need to be root ).

Below is a screenshot of where i increased the limit to 200000 for webadmin.


After logging out and logging back in as webadmin.  the command # ulimit -Hn reported 200000 as the new limit on file descriptors.

After this i started JBoss and it worked like a charm.