Search This Blog

Monday, January 19, 2015

Docker Python API

Here is an example Python script using the docker-py API.  In this example, I start 3 containers.  One with Accumulo, one with Apache YARN, and one with Geoserver.  I am also linking the containers so that they have hosts file entries to support the hostname lookups.

additionally i have declared some volumes that i bind to the host's home folder under /geomesa-docker-volumes/*

If you get version mismatch errors, just modify the version in the get_client_unsecure function.

#!/usr/bin/env python
# The unsecure client requires that your Docker daemon is listening on port 5555 in addition to the default unix socket.
# DOCKER_OPTS="-H unix:///var/run/docker.sock -H tcp://127.0.0.1:5555"
# $ sudo service docker(.io) restart

__author__ = 'championofcyrodiil'

import docker
import getpass
from subprocess import call

geoserver_image = "user:geoserver"
accumulo_image = "user:accumulo"
yarn_image = "user:yarn"
remote_docker_daemon_host = "127.0.0.1"
unsecure_docker_port = 5555

def get_client_unsecure(host, port):
    client = docker.Client(base_url="http://%s:%s" % (host, port), version="1.10")
    return client


if __name__ == '__main__':
    # unsecured connection on localhost (127.0.0.1)
    dc = get_client_unsecure(remote_docker_daemon_host, unsecure_docker_port)

def start_geomesa(user):
    #accumulo container
    accumulo_volumes = ['/opt/accumulo/accumulo-1.5.2/lib/ext/', '/data-dir/', '/data']
    accumulo_container = \
        dc.create_container(image=accumulo_image,
                            name=(user + 's-accumulo'),
                            tty=True,
                            stdin_open=True,
                            hostname='accumulo',
                            ports=[2181, 22, 50070, 50095, 50075, 9000, 9898, 3614],
                            volumes=accumulo_volumes,
                            mem_limit="4g")

    accumulo_binds = {
        '/home/' + getpass.getuser() + '/geomesa-docker-volumes/accumulo-libs':
        {
            'bind': '/opt/accumulo/accumulo-1.5.2/lib/ext/',
            'ro': False
        },
        '/home/' + getpass.getuser() + '/geomesa-docker-volumes/accumulo-data':
        {
            'bind': '/data-dir/',
            'ro': False
        },
        '/home/' + getpass.getuser() + '/geomesa-docker-volumes/hdfs-data':
        {
            'bind': '/data/',
            'ro': False
        }
    }
    dc.start(accumulo_container, publish_all_ports=True, binds=accumulo_binds)

    #YARN CONTAINER
    yarn_container = dc.create_container(image=yarn_image,
                                         name=(user + 's-yarn'),
                                         stdin_open=True,
                                         tty=True,
                                         hostname='yarn',
                                         ports=[8088, 8042, 22], mem_limit="2g")

    link = {(user + 's-accumulo'): 'accumulo'}
    dc.start(yarn_container,
             publish_all_ports=True,
             links=link)

    #geoserver container
    geoserver_container = dc.create_container(image=geoserver_image,
                                              name=(user + 's-geoserver'),
                                              stdin_open=True,
                                              tty=True,
                                              hostname='geoserver',
                                              ports=[8080, 22, 7979], mem_limit="2g")
    link = {(user + 's-accumulo'): 'accumulo', (user + 's-yarn'): 'yarn'}
    dc.start(geoserver_container,
             publish_all_ports=True,
             links=link)

start_geomesa('test')
call("./geomesa_info.py")

RabbitMQ handshake_timeout

Currently I am maintaining an Openstack cluster deployed via Mirantis Fuel 5.1 (Icehouse). Things were going well for a while, but at some point there were a lot of delays in requests to the APIs to perform various tasks such as creating an instance, volume, mounting, etc. This would cause failures and would regularly leave openstack objects in an inconsistent state. This is very frustrating and difficult to diagnose because you will see errors all over the place.

The issue for us was the system swappiness default setting of 60 with Centos 6. This caused a lot of messages to take longer than the rabbitmq default of 3 seconds, resulting in a timeout and failed request.

As root on all openstack controllers:
# sysctl vm.swappiness=10
# swapoff /dev/mapper/os-swap

Additionally it looks like mirantis fuel used LVM. This is likely a slower file system than ext4 native on non lvm partitioned disks.

 Also make sure you have enough RAM to disable swap. More importantly, make sure you have enough RAM for your openstack controller.

see: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/s-memory-tunables.html

Update: This has been added to launchpad as a bug in 5.1, 6.0 and 6.1: https://bugs.launchpad.net/fuel/+bug/1413702

Saturday, November 1, 2014

CentOS 6.5, Custom Linux Kernel 2.6.36.4 with UnionFS

CentOS 6.5 is currently using Linux Kernel 2.6.32.   There are many blogs, posts and snippets around that will lecture you about the 'standards' of Enterprise Linux and force the idea down your throat that if you change the CentOS kernel, you are wrong and your operating system will be flawed and unsupported, etc. etc...

In reality, you may have an environment where you can only install and use CentOS 6.5 instances.  However, you may be in a position in which you can integrate custom code.  In my situation, I am looking to upgrade the kernel so I can use unionfs.  UnionFS is used in LTSP to create a virtual filesystem in which tmpfs is the writeable layer for the read-only root filesystem which provides the OS via NFS/nbd.

This script should work with CentOS 6.5 from a 'Minimal' ISO install.  It should be run as root, and should be the first thing you do with a fresh install.  Once you have verified the kernel is working, you can then install updates and go from there.

One thing to note, this is a CUSTOM solution for a specific situation.  This script will add an exclusion to the YUM configuration so that kernel updates are no longer downloaded.  Use this script at your own risk and be sure to test thoroughly in a VM before trying to run this on any systems used by others.

Friday, September 19, 2014

Just Install Docker on Ubuntu 14.04 64-bit...

It has been a while since I have posted, so I'm making this one short.  Ever want a clean and concise script written in Bash that will install the latest version of Docker (And stay up to date)?  Well, here. Note that Docker wants you to pick the DNS.  So I've used 8.8.8.8 (Google-DNS) for this example.

Note: This will also install the kernal extras to enable AUFS support.
#!/bin/bash
#Update our local package index
sudo apt-get update
#Make sure Apt supports HTTPS
[ -e /usr/lib/apt/methods/https ] || {
  apt-get install -y apt-transport-https
}
# Get server key for repo
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
# Update Repo Sources...
[ -e /etc/apt/sources.list.d/docker.list ] || {
  rm /etc/apt/sources.list.d/docker.list
}
sudo echo "deb https://get.docker.io/ubuntu docker main" > /etc/apt/sources.list.d/docker.list
sudo apt-get update
# ensure  Linux kernel extra modules are installed which support cpu cgroups, etc...
sudo apt-get install -y linux-image-extra-$(uname -r)
# install docker
sudo apt-get install -y lxc-docker
# update docker DNS and listen on localhost tcp
sudo echo 'DOCKER_OPTS="-dns 8.8.8.8 -H unix:///var/run/docker.sock -H tcp://127.0.0.1:5555"' > /etc/default/docker

Tuesday, September 2, 2014

Netflix Finally works on Linux

After years of waiting.  Netflix finally works with the baseline install (and apt-get dist-upgrade) of Ubuntu 14.04 LTS.  Additionally, the performance seems to be much better than Silverlight plugin on Windows 7.

I also installed a few packages from the blog article posted here: http://www.omgubuntu.co.uk/2014/08/netflix-linux-html5-support-plugins

The post at the link above is by far the most thorough and well documented guide to configuring Netflix to work with Ubuntu.  I did have to reboot my system after performing the steps, otherwise Netflix throws an error which I now cannot replicate to provide further information.

Thanks to the perseverance of the Ubuntu community, Netflix is finally supported and the Ubuntu distribution is going to get a lot more footing in the war of operating systems for desktop PCs.  Since it's still free, I'll be sticking with Ubuntu before spending any money on a Windows operating system.

Wednesday, August 20, 2014

Using USB Devices on Solaris (ZFS)

UPDATE:
Solaris 11 does not support the ZFS filesystem provided by 'native' ZFS for linux, and vice-versa.  Thus, you should export your data as NTFS so that everyone is happy, as NTFS is the most supported filesystem for 'read-only'.  Otherwise it's Solaris->Solaris only.  I have not tested native ZFS with import/export, but i suspect the version of zpool (5000 or something like that) will work fine between fedora type distros and ubuntu.


Recently I needed to export a large amount of data on our Solaris NFS Server.  However, getting this information straight off this robust server is not as intuitive or straight forward as you might first think.  Many filesystems are not natively supported by solaris, thus can cause a lot of headache trying to figure out how to use fdisk and format.  Additionally, with block sizes of 4096 (like on a Seagate 3TB hard drive) it may not even be compatible with UFS.

Before you begin, take a look at this chart on wikipedia which breaks down the various versions of zpool and zfs.  This will only work if you are using the proper version of zpool/zfs between hosts.  Currently the Native Linux ZFS project uses a zpool version not support by Solaris 11 and vice versa.  So ZFS cannot be imported between hosts: http://en.wikipedia.org/wiki/ZFS#List_of_operating_systems_supporting_ZFS

ZFS is a good solution and can be imported on other linux distributions such as CentOS or Ubuntu.  Here is a synopsis in which I exported a large amount of data with the label "TwitterFeeds"

List available drives
$ format -e
...    other disks likely shown here...
54. c9t0d0 <Seagate-Expansion Desk-0604 cyl 45597 alt 2 hd 255 sec 63> /pci@0,0/pci108e,cb84@2,1/hub@6/storage@2/disk@0,0
          /pci@0,0/pci108e,cb84@2,1/hub@6/storage@2/disk@0,0
Created a ZFS pool on the USB drive, added a ZFS file system, chowned it with my default user and started rsync with nohup to log (rsync.out)
$ sudo zpool create TwitterFeeds c9t0d0
$ sudo zfs create TwitterFeeds/export 
$ sudo chown -R user:group /TwitterFeeds/export
$ nohup rsync -r --progress /sasdata/TwitterFeeds /TwitterFeeds/export > rsync.out 2>&1&
Example entry of output in rsync log:
bytesize Percent% xferRate Time(file#, to-check=filesremain/estimatedtotal)Absolute/File/Path/Filename.ext
805313211 100%   29.23MB/s    0:00:26 (xfer#364, to-check=1008/1417)TwitterFeeds/08/19/2014/05/17/22/10FEB25021318-S3DM_R5C4-053771096010_01_P001.DAT
You can use tail -f on the rsync log to periodically view progress.
$ tail -f /TwitterFeeds/export/rsync.out
Once rsync has completed, the ZFS File System and Zpool are unmounted and removed from your available zpools.
$ sudo zpool export TwitterFeeds
The USB Drive can now be physically removed and plugged into another computer.  Plug USB Drive in to target system.  ZFS detects any moved or renamed devices, and adjusts the configuration appropriately. To discover available pools, run the zpool import command with no options. To import a pool, specify the name as an argument to the import command (TwitterFeeds).  By default, the zpool import command only searches devices within the /dev/dsk directory. If devices exist in another directory, or you are using pools backed by files, you must use the -d option to search alternate directories.  This may be required when using CentOS/Ubuntu with ZFS.
$ zpool import TwitterFeeds
$ zpool import -d /dev/rdsk/c9t0d0 TwitterFeeds #specifying disk device path manually
Once Imported, "$ sudo zfs get all" should show a PROPERTY mount point for the zpool to access the data. From this point you should be able to use native OS filesystem utils like cp, rm, chmod, and others.

Reference Links:
Managing ZFS Storage Pools
Managing ZFS File Systems (Not the Physical Devices!)
Using ZFS on Linux

Friday, August 8, 2014

Fetching Artifactory Maven Dependencies via Python

This clever python script will fetch your dependencies identified in a YAML file.  Don't forget that python uses white space for scope.  So make sure if you're going to copy and paste, to get it right.  The script uses artifactory's GAVC API.  This is the same API used by maven plugin for artifactory.  A nice feature is that you can run it multiple times, and by comparing md5 hashes, it will only download JAR files that have changed.  Also make note that javadoc, sources and POMs are omitted in the condition on line 67.

Because Python has a great API for Docker as well.  I will be using this code to implement something like fig to automatically deploy containers in my environment which can pull latest dependencies from Artifactory for installation.

First, an example of the YAML:

artifacts:
- artifact:
     artifactid:     accumulo-core
     groupid:     org.apache.accumulo
     version:     1.5.1

- artifact:
     artifactid:     accumulo-fate
     groupid:     org.apache.accumulo
     version:     1.5.1

- artifact:
     artifactid:     accumulo-trace
     groupid:     org.apache.accumulo
     version:     1.5.1



The Script:

#!/usr/bin/env python
import yaml
import hashlib
import os
import sys
import httplib
import json
import urllib2
from urlparse import urlparse

__author__ = 'champion'
artifactory_url = "art.mydomain.com:8081"
#local download folder
local_folder = "./deps"
conn = httplib.HTTPConnection(artifactory_url)


def download(filename, remote):
    print "\nDownloading: " + remote
    req = urllib2.urlopen(remote)
    blocksize = 16 * 1024
    with open(local_folder + "/" + filename, 'wb') as fp:
        while True:
            chunk = req.read(blocksize)
            if not chunk:
                break
            fp.write(chunk)
        fp.close()


def main():
    if not os.path.exists(local_folder):
        os.mkdir(local_folder)

    # Take last arg as filename
    filename = sys.argv[-1]

    if os.path.isfile(filename):
        stream = open(filename, 'r')
        yaml_instance = yaml.safe_load(stream)
        stream.close()

        artifacts = yaml_instance["artifacts"]

        print "\nFetching Artifacts in '" + filename + "' from Artifactory... "

        #for each element in YAML...
        for artifact in artifacts:
            entry = artifact["artifact"]
            artifact_version = str(entry["version"])
            artifact_groupid = str(entry["groupid"])
            artifact_artifactid = str(entry["artifactid"])

            # Create API call
            api_call = "/artifactory/api/search/gavc?g=" + artifact_groupid + "&a=" + artifact_artifactid + "&v=" + artifact_version

            # GET the results
            conn.request("GET", api_call)
            r1 = conn.getresponse()

            # If GET was Successful
            if r1.status == 200 and r1.reason == "OK":
                uris = json.loads(r1.read())["results"]
                # Omit Javadoc, Sources, POMs...
                for uri in uris:
                    link = uri["uri"]
                    if not link.endswith("pom") and not link.endswith("sources.jar") and not link.endswith("javadoc.jar"):
                        #Request the Artifact information
                        conn.request("GET", link)
                        artifact_json = conn.getresponse().read()
                        artifact_props = json.loads(artifact_json)

                        downloaduri = artifact_props["downloadUri"]
                        md5 = artifact_props["checksums"]["md5"]
                        fname = urlparse(downloaduri).path.split('/')[-1]

                        #Always Download Dep, unless conditions change.
                        omit_dl = False
                        if os.path.exists(local_folder + "/" + fname):
                            print "\nLocal Copy of '" + fname + "' Exists, checking md5..."
                            print "Remote MD5: " + md5
                            curr_md5 = hashlib.md5(open(local_folder + "/" + fname).read()).hexdigest()
                            print " Local MD5: " + curr_md5
                            if curr_md5 == md5:
                                omit_dl = True  # conditions changed

                        if not omit_dl:
                            download(fname, downloaduri)
                        else:
                            print "Hashes match, omitting download..."
                    else:
                        #artifact is not the binary jar
                        continue

            else:
                print "Artifact was not found in Artifactory."

        conn.close()
        print "Done."
    else:
        print "YAML file: '" + sys.argv[-1] + "' not found."



main()