Parsing tcpdump/pcap files in bash, without Wireshark.

Sometimes I work in an environment in which software access is restricted.  Things like Wireshark are DEFINITELY not allowed.

However, I'm still expected to troubleshoot network connectivity issues.  Luckily, tcpdump exists in the EPEL Repo, but leaves much to be desired when trying to read the PCAP files.  I decided to read up on the PCAP spec, and create my own parser in bash.  Crazy? Maybe.  But, it does work.

Another nice capability here is that using something like vim, you can easily tweak this script to dump application data for further debugging.  For example, switch your HTTPd to run in Non-SSL, run a capture, and dump the application data.  Of course, you may have to work out the correct byte offset for your packets. As this is a bare minimal implementation of a pcap parser and does not nearly support the full scope of capability provided by libpcap.

The real issue here is that I have not taken the time to parse some flags which consist of single bits.  There is also much work to be done in regard to writing the conditionals for various Ethernet, IP Routing and TCP frames.  But this is a good baseline to start.

To use this script just run 'tcpdump -w file.pcap -X' to run a capture.  Once done, pass the file as an argument to the pcap-analyzer bash script, and you will be able to read your frames on the CLI!  This should work out of the box with Centos/RHEL 6 Minimal, with the addition of the tcpdump RPM.

Here is a screenshot.

As you can see frame 2 starts to break because of the parsing.  Some flags actually mean, the header is going to be 28 bytes instead of 26.  As a result of the byte variations, the byte iterator ends up on some obscure part of the stream I don't handle, and things get start to break.  Git forks are welcome if you like this little utility and are interested in making it more robust.

https://github.com/charlescva/common/blob/master/pcap-analyzer.sh

One final important characteristic about Ethernet wire data is that it is ALWAYS Big Endian.  This is confusing because most CPUs are little endian.  To work around this issue, I use both 'hexdump' and 'od' because one enforces big endian while the other is CPU dependent.

Other resources Used:
http://www.tcpdump.org/manpages/pcap-savefile.5.html
http://www.tcpdump.org/linktypes.html
https://en.wikipedia.org/wiki/Ethernet_frame#Ethernet_II
https://en.wikipedia.org/wiki/IPv4#Packet_structure
https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_segment_structure

Apache Nifi Behind Apache (httpd) (SSL to Non-SSL)

ProxyPreserveHost On


    ProxyPass "http://localhost:8080/nifi" max=20 ttl=120 retry=300
    ProxyPassReverse  "http://localhost:8080/nifi"
    RequestHeader add X-ProxyScheme "https"
    RequestHeader add X-ProxyHost "proxyserver.acme.com"
    RequestHeader add X-ProxyPort "443"



    ProxyPass "http://localhost:8080/nifi-api" max=20 ttl=120 retry=300
    ProxyPassReverse "http://localhost:8080/nifi-api"
    RequestHeader add X-ProxyScheme "https"
    RequestHeader add X-ProxyHost "proxyserver.acme.com"
    RequestHeader add X-ProxyPort "443"



    ProxyPass "http://localhost:8080/nifi-docs" max=20 ttl=120 retry=300
    ProxyPassReverse "http://localhost:8080/nifi-docs"
    RequestHeader add X-ProxyScheme "https"
    RequestHeader add X-ProxyHost "proxyserver.acme.com"
    RequestHeader add X-ProxyPort "443"

NAB Show 2016

NABShow 2016 was a great experience.  I have never taken interest in the production of movies, radio, or any form of media in my life, but this event has changed my perspective dramatically.  This will be my first in a series of posts, in which I will share photos and information I learned throughout this event.

A recent gig has required me to take an interest in video production.  It is not always common to have an employer pay for the attendance of a conference, but I think it is important.  If you take away anything from this post, it should be that self-enrichment is required to succeed in any field.  And you should push your employer to give you an opportunity to pursue it.

The show was in Las Vegas, Nevada.  Being from Virginia, it was a long flight.  However, after meeting so many people from Europe and China, I cannot complain.  I stayed at a cheaper Hotel/Casino called "Circus Circus".  

The "Hotel and Casino" was dated and odd with the whole circus theme, but the room was clean and the last night I was there, I had fun losing a few dollars at the slot machines and roulette.  They do however have a good Italian place with the best meatball sub, and a great steakhouse that has class and has been around for years. ProTip: The steakhouse is reservation only, however if you are alone, you can eat at the classy bar. Also, I can say that I appreciate not seeing one single clown the whole time I was there.  I think acrobats and dancers have mostly replaced them.

The event is Monday through Thursday every year.  I attended Tues & Wed, and had enough for my first visit.  However, if I get the chance to go again, I'll plan it better.  The hotel is only a 15 minute walk from the convention center.  The Las Vegas Convention Center is HUGE.  To give you an idea of what I mean, the "Westgate" is 200,000 sq ft (19,000 m2) and has it's own station on the Vegas monorail.

As you can see the convention center is much larger than the Westgate, located to the left of the blue "North Hall" in the picture above.  So be prepared to walk several MILES at this show.

There were so many exhibitors I could not count them all.  But I learned a lot from some of the free sessions that were included with the Exhibits only pass.  Make sure you go to those while you are there. They are usually only 45 minutes, but worth the time.

I realized I was at a whole new type of tech event when the first thing I came across was being called "IP Hybrid Routing".  My first thought was, "What to they mean by 'hybrid'?  Is there some other kind of digital routing that is more efficient than TCP/IP that I don't know about?"  

Turns out there is.  Video and audio signals have been around a lot longer than the internet.  These original analog signals gave birth to the broadcasting industry.  As a result, the industry vendors behind analog equipment went straight to using hardware components.  Custom hardware for digital signal processing was faster than computer processors (or GPUs), and could perform the digital operations as quickly as they were needed to encode and decode the signals being recorded live.

As a result, in 2016 there are thousands of vendors of this equipment worldwide.  And they are all chomping at the bit to crack in to the cloud.

I attended two sessions by Josh Kolden regarding his C4 ID system.  His system essentially uses natural language semantics and byte hashes to create a metadata tagging system for all of the artifacts used in video production.  There is a whitepaper that goes in to more detail available at his site, http://cccc.io. He also stresses the importance of using a RESTful API with JSON as the glue between your software components.  He named several but Avid and Nuke are two that come to mind.  If you are in to software engineering, it shouldn't take much to wrap your mind around how quickly the cloud is going to (or already is) provide a major role in the future of media production.  The C4 ID system was used to produce "The Suitcase" a movie currently in post-production and will be released later this year. It was very interesting to hear about the challenges getting this metadata collection process integrated with the movie production staff.

I spent the rest of my first day looking at Drones, LED displays and other gadgets.  There was a TON of them.

Dual-Prop Airplane Drone

Stabilized Sensor Suite

Octo-copter with gimbals and professional video camera

With adjustable carbon-fiber chasis

These enormous LED displays are high resolution at a distance, and can be scaled to any size

The LED screens are an array of squares with an input, output and LCD display showing about 14 volts and 37 degrees Celsius.

You can even make cubes with live video for your nightclub.

In my next post, I will go over more equipment and talk a bit more about the current VR industry, the future of VR, and post more photos of the event.

http://lmgtfy.com/?q=NABShow

QuickTip: Tree Alias

Sometimes directory structures are a bit of a nest.  You often want to see the full structure, but may not have access to packages like 'tree'.  I found a great expression and decided to wrap it into an aliased command in ~/.bash_profile

alias tree="ls -R | grep \":$\" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/   /' -e 's/-/|/'"

Now, when you want to see a complex directory structure in the shell, just change to the working directory, and type `tree`

Custom `tree` command

Dynamic Adaptive Streaming over HTTP (DASH) on WildFly

I've been away for a while.  October 10th, 2015 I started a new project at work, and that has kept me busy.

As demand has increased on internet media, the core web infrastructure has evolved to support new use cases for some pretty old standards.

HTTP has been used since the inception of the web around 1989.  The core concept being, a client's request receives a peer's response.  Originally a content length header element was used to box the request and response body in to some static size.  This was used for a variety of reasons, and was also manipulated to perform denial-of-service (DoS) attacks.  This is now used to stream partial offsets of a video file, allowing the player to start fetching segments with a lower bit-rate.  Hence, dynamic adaptive streaming over http.

You can check out the wildfly and castlab's code at my github public repository: https://github.com/charlescva/mobile-dashjs

Notice the Content-Length is determined by the offset as provided by the MPD and Initial MP4 containing the Metadata about each stream.

You can review the source, but the steps are as follows:

1. Obtain a standard MP4 example video.
2. Configure Apache to host the files in the directory dashencrypt is using. This is currently hard-coded in the VideoRegistration.java.
3. Add Video using the Add a Video tab.  The JAX-RS enabled VideoService.java will handle the request, and dash the file for you.
4. Upon success, you will see the entry for the video appear on the "Video Player" tab.

Observing the Console.  You can see the logger is outputting the steps as it processes the request.

I am still getting my feet wet as well, and came across a great article on the following website, https://arashafiei.wordpress.com/2012/11/13/quick-dash/.  

I'll be working on integrating a "live" stream in which a imaging device like /dev/video0 (webcam) will be used to generate the video segment data, while the MPD (Manifest) and initial MP4 file containing the Movie Box (moov) and/or Fragment Box (moof) are updated on the fly.  Essentially, the goal is to enable "DASHing" of a live video feed.

There was a problem playing this protected content. (Error Code: 3336)

Once again, the powers at be, controlling DRM software integration at Google and Hulu have decided to disregard the users of 'free' software and instead prioritize users of Windows, Apple and Google Chrome/Android devices.  QoS for the mainstream is understandable.  But there is a difference between low quality code that is not efficient or ideal, and code that simply doesn't work.

What is more annoying, is that these legal issues are masquerading as 'error codes'.  It can be determined I'm using linux via browser request headers, and IF the OS is linux, and the error is DRM related, just be honest and tell me that it's not supported on this platform, because of a recent update.

I'm sure they have been forced to enabled DRM, and waited until the last minute to do so.  And didn't have time to rework the baseline framework to support HTML5 for linux users.  I'm sure they'll do it eventually, but for now are covering their lack of planning with an error code.

This is not the first time this has happened.  We all remember the days of Linux tweaks getting Netflix to work using VMs, Wine, and god knows what else.  Finally, after years of complaining, Netflix allowed HTML5, and all is well... for now.

It seems that a new update from Hulu and Google does not permit Google Chrome running on Linux variants to support DRM protected content. This is apparently due to licensing issues, copyrights, and cost.  It would seem that Hulu blindly added the DRM requirements, without considering the repercussions (Or they just don't care).

Not surprising, small businesses with a great idea like Hulu once was, has become a draconian enterprise in which legal pressure and money woes force a quality product to become sub-par and only available to paid platforms.

After working a decade in software engineering, I have realized that what is important to those of us creating the software for the rest of society, is not relevant to the mainstream.  The same users who would tell you Linux is 'shit' because you can't be productive, are the ones who will ultimately whine and complain like a child whenever their Windows boot partition shits the bed.

As expected, the Linux base is now responding with the usual rhetoric about leaving Hulu and not giving them any more money.  There are some workarounds being expressed.  But likely I'll switch to either my Windows box, or I'll end up going w/ Firefox + HAL.

 For now, here are the most informative forums with users doing exactly what I described above.

• https://productforums.google.com/forum/#!topic/chrome/H_WqDsZFS9w
• http://ubuntuforums.org/showthread.php?t=2290743 (Marked "Solved", even though it isnt)
• http://www.linuxquestions.org/questions/linux-software-2/note-hulu-playback-in-chrome-no-longer-working-4175550849/

Looking forward to the comments from those of you who 'know' whats really going on.

Collecting Apache Storm time series data with Graphite

Naturally, Apache Storm processes data as tuples over time.  This is an ideal framework to utilize for streaming data through a 'pipe line'.  However, maintaining a record of this information is not necessarily an inherent feature of storm.  For metrics collection and analysis, I used Graphite.  Graphite is available in the Fedora EPEL and a great option for quickly collecting some metrics and generating a graph w/ overlaying analytical functions.

The easiest way to collect information is by going straight to the source.  The Nimbus server.  Using the NimbusClient class, available from the package backtype.storm.generated, you can easily review the ClusterSummary and iterate through each TopologySummary's ID to retrieve TopologyInfo and each topology's ExecutorSummary, to determine emitted and transferred tuples. Additionally, you can get information about errors, threads, and likely other details related to Apache Storm topologies I have not collected in my example code.

First, Lets look at how I will push data in to the carbon-cache daemon.
https://github.com/charlescva/graphite-common/blob/master/src/main/java/zkCliTest.java#L44

   // TCP Stream to carbon-cache.
    private void graphite(Map metrics) {
        // Current Time-Stamp for test
    long epoch = System.currentTimeMillis()/1000;
   
        try { // output  stream to the host on default port.
        Socket conn          = new Socket("cabon-cache.novalocal", 2003);
        DataOutputStream dos = new DataOutputStream(conn.getOutputStream());
        // graphite syntax map the #ngsm to output stream.
            for  (String metric : metrics.keySet() ){
                dos.writeBytes(metric +
            " " + metrics.get(metric) +
            " " + epoch + "\n");
            }
        //CLOSED CONNECTION.
            conn.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

The above code simply opens a TCP socket to the carbon-cache, writes a Map of metrics with the current timestamp, and closes the connection.  Easy.  The Take-away here, is the syntax. '[metric_path] [value] [time]' as String.

It would be redundant for me to display all of the Java code here: https://github.com/charlescva/graphite-common/blob/master/src/main/java/zkCliTest.java#L125 which essentially takes a Nimbus client connect, and builds a model of the nimbus state with your topoloy as each child node of the root context.

So, rather than bore you.  Here is a screenshot!

As you can see, the "My Topology" is a test, and clearly a static source that is linear.  But all and all, you can quickly get some good information.  Feel free to comment, as I find this article particularly interesting.