Collecting Apache Storm time series data with Graphite

Naturally, Apache Storm processes data as tuples over time.  This is an ideal framework to utilize for streaming data through a 'pipe line'.  However, maintaining a record of this information is not necessarily an inherent feature of storm.  For metrics collection and analysis, I used Graphite.  Graphite is available in the Fedora EPEL and a great option for quickly collecting some metrics and generating a graph w/ overlaying analytical functions.

The easiest way to collect information is by going straight to the source.  The Nimbus server.  Using the NimbusClient class, available from the package backtype.storm.generated, you can easily review the ClusterSummary and iterate through each TopologySummary's ID to retrieve TopologyInfo and each topology's ExecutorSummary, to determine emitted and transferred tuples. Additionally, you can get information about errors, threads, and likely other details related to Apache Storm topologies I have not collected in my example code.

First, Lets look at how I will push data in to the carbon-cache daemon.
https://github.com/charlescva/graphite-common/blob/master/src/main/java/zkCliTest.java#L44

   // TCP Stream to carbon-cache.
    private void graphite(Map metrics) {
        // Current Time-Stamp for test
    long epoch = System.currentTimeMillis()/1000;
   
        try { // output  stream to the host on default port.
        Socket conn          = new Socket("cabon-cache.novalocal", 2003);
        DataOutputStream dos = new DataOutputStream(conn.getOutputStream());
        // graphite syntax map the #ngsm to output stream.
            for  (String metric : metrics.keySet() ){
                dos.writeBytes(metric +
            " " + metrics.get(metric) +
            " " + epoch + "\n");
            }
        //CLOSED CONNECTION.
            conn.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

The above code simply opens a TCP socket to the carbon-cache, writes a Map of metrics with the current timestamp, and closes the connection.  Easy.  The Take-away here, is the syntax. '[metric_path] [value] [time]' as String.

It would be redundant for me to display all of the Java code here: https://github.com/charlescva/graphite-common/blob/master/src/main/java/zkCliTest.java#L125 which essentially takes a Nimbus client connect, and builds a model of the nimbus state with your topoloy as each child node of the root context.

So, rather than bore you.  Here is a screenshot!

As you can see, the "My Topology" is a test, and clearly a static source that is linear.  But all and all, you can quickly get some good information.  Feel free to comment, as I find this article particularly interesting.

Apt-cacher with Docker

I have been working with docker from www.docker.io for couple of weeks and I love it.  I have made containers for Postgres w/ postgis extensions, Accumulo/HDFS/Zookeeper running pseudo-distributed, geoserver and as of yesterday, an apt-cacher for our Ubuntu 14.04 workstations at the office.  Here is all you need!

You can get the /etc/apt-cacher/apt-cacher.conf from an ubuntu system after installing the apt-cacher package.

#
# example ubuntu 14.04 apt-cacher
#

FROM ubuntu:14.04
MAINTAINER championofcyrodiil.blogspot.com

USER root
RUN apt-get update
RUN apt-get -y -q install apt-cacher
ADD apt-cacher.conf /etc/apt-cacher/apt-cacher.conf
ADD start-cacher.sh /root/start-cacher.sh
RUN chmod +x /root/start-cacher.sh
EXPOSE 3142


#Default Docker Run Comamnd(s)
CMD ["/root/start-cacher.sh"]


And of course, the 'start-cacher.sh' bash script.  Make sure it has been chmod +x so it's executable.

#!/bin/bash
/usr/sbin/apt-cacher -R 3 -d -p /var/run/apt-cacher.pid
tail -f /var/log/apt-cacher/access.log
wait


Once that is done, you will want to place all three files in to a single folder, cd to that folder and build the docker image, here is my example:


 $ sudo docker build -t local:apt-cacher .

Note the period at the end to specify the context of the current directory with the scripts.  Next it is time to run the container, make sure the host (-h) matches the daemon listening host specified in /etc/apt-cacher/apt-cacher.conf.

$ sudo docker run -d -p 3142:3142 -h apt-cacher local:apt-cacher

$ sudo docker ps -sl

CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS                    NAMES               SIZE

7b6a7ff25691        local:apt-cacher    /root/start-cacher.s   18 hours ago        Up 18 hours         0.0.0.0:3142->3142/tcp   clever_hoover       361.4 MB

For client systems to use the proxy, add the file /etc/apt/apt.conf.d/01proxy with the contents:

Acquire::http::Proxy "http://$DOCKERHOST:3142";

Like so,

$ sudo echo 'Acquire::http::Proxy "http://:3142";' >> /etc/apt/apt.conf.d/01proxy