Search This Blog

Saturday, June 11, 2016

Apache Nifi Behind Apache (httpd) (SSL to Non-SSL)

ProxyPreserveHost On


    ProxyPass "http://localhost:8080/nifi" max=20 ttl=120 retry=300
    ProxyPassReverse  "http://localhost:8080/nifi"
    RequestHeader add X-ProxyScheme "https"
    RequestHeader add X-ProxyHost "proxyserver.acme.com"
    RequestHeader add X-ProxyPort "443"



    ProxyPass "http://localhost:8080/nifi-api" max=20 ttl=120 retry=300
    ProxyPassReverse "http://localhost:8080/nifi-api"
    RequestHeader add X-ProxyScheme "https"
    RequestHeader add X-ProxyHost "proxyserver.acme.com"
    RequestHeader add X-ProxyPort "443"



    ProxyPass "http://localhost:8080/nifi-docs" max=20 ttl=120 retry=300
    ProxyPassReverse "http://localhost:8080/nifi-docs"
    RequestHeader add X-ProxyScheme "https"
    RequestHeader add X-ProxyHost "proxyserver.acme.com"
    RequestHeader add X-ProxyPort "443"


Thursday, April 21, 2016

NAB Show 2016

NABShow 2016 was a great experience.  I have never taken interest in the production of movies, radio, or any form of media in my life, but this event has changed my perspective dramatically.  This will be my first in a series of posts, in which I will share photos and information I learned throughout this event.

A recent gig has required me to take an interest in video production.  It is not always common to have an employer pay for the attendance of a conference, but I think it is important.  If you take away anything from this post, it should be that self-enrichment is required to succeed in any field.  And you should push your employer to give you an opportunity to pursue it.


The show was in Las Vegas, Nevada.  Being from Virginia, it was a long flight.  However, after meeting so many people from Europe and China, I cannot complain.  I stayed at a cheaper Hotel/Casino called "Circus Circus".  

The "Hotel and Casino" was dated and odd with the whole circus theme, but the room was clean and the last night I was there, I had fun losing a few dollars at the slot machines and roulette.  They do however have a good Italian place with the best meatball sub, and a great steakhouse that has class and has been around for years. ProTip: The steakhouse is reservation only, however if you are alone, you can eat at the classy bar. Also, I can say that I appreciate not seeing one single clown the whole time I was there.  I think acrobats and dancers have mostly replaced them.

The event is Monday through Thursday every year.  I attended Tues & Wed, and had enough for my first visit.  However, if I get the chance to go again, I'll plan it better.  The hotel is only a 15 minute walk from the convention center.  The Las Vegas Convention Center is HUGE.  To give you an idea of what I mean, the "Westgate" is 200,000 sq ft (19,000 m2) and has it's own station on the Vegas monorail.


As you can see the convention center is much larger than the Westgate, located to the left of the blue "North Hall" in the picture above.  So be prepared to walk several MILES at this show.

There were so many exhibitors I could not count them all.  But I learned a lot from some of the free sessions that were included with the Exhibits only pass.  Make sure you go to those while you are there. They are usually only 45 minutes, but worth the time.


I realized I was at a whole new type of tech event when the first thing I came across was being called "IP Hybrid Routing".  My first thought was, "What to they mean by 'hybrid'?  Is there some other kind of digital routing that is more efficient than TCP/IP that I don't know about?"  

Turns out there is.  Video and audio signals have been around a lot longer than the internet.  These original analog signals gave birth to the broadcasting industry.  As a result, the industry vendors behind analog equipment went straight to using hardware components.  Custom hardware for digital signal processing was faster than computer processors (or GPUs), and could perform the digital operations as quickly as they were needed to encode and decode the signals being recorded live.

As a result, in 2016 there are thousands of vendors of this equipment worldwide.  And they are all chomping at the bit to crack in to the cloud.



I attended two sessions by Josh Kolden regarding his C4 ID system.  His system essentially uses natural language semantics and byte hashes to create a metadata tagging system for all of the artifacts used in video production.  There is a whitepaper that goes in to more detail available at his site, http://cccc.io. He also stresses the importance of using a RESTful API with JSON as the glue between your software components.  He named several but Avid and Nuke are two that come to mind.  If you are in to software engineering, it shouldn't take much to wrap your mind around how quickly the cloud is going to (or already is) provide a major role in the future of media production.  The C4 ID system was used to produce "The Suitcase" a movie currently in post-production and will be released later this year. It was very interesting to hear about the challenges getting this metadata collection process integrated with the movie production staff.

I spent the rest of my first day looking at Drones, LED displays and other gadgets.  There was a TON of them.

Dual-Prop Airplane Drone

Stabilized Sensor Suite

Octo-copter with gimbals and professional video camera

With adjustable carbon-fiber chasis

These enormous LED displays are high resolution at a distance, and can be scaled to any size

The LED screens are an array of squares with an input, output and LCD display showing about 14 volts and 37 degrees Celsius.

You can even make cubes with live video for your nightclub.

In my next post, I will go over more equipment and talk a bit more about the current VR industry, the future of VR, and post more photos of the event.

Wednesday, March 23, 2016

QuickTip: Tree Alias

Sometimes directory structures are a bit of a nest.  You often want to see the full structure, but may not have access to packages like 'tree'.  I found a great expression and decided to wrap it into an aliased command in ~/.bash_profile

alias tree="ls -R | grep \":$\" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/   /' -e 's/-/|/'"

Now, when you want to see a complex directory structure in the shell, just change to the working directory, and type `tree`

Custom `tree` command

Sunday, November 15, 2015

Dynamic Adaptive Streaming over HTTP (DASH) on WildFly

I've been away for a while.  October 10th, 2015 I started a new project at work, and that has kept me busy.

As demand has increased on internet media, the core web infrastructure has evolved to support new use cases for some pretty old standards.

HTTP has been used since the inception of the web around 1989.  The core concept being, a client's request receives a peer's response.  Originally a content length header element was used to box the request and response body in to some static size.  This was used for a variety of reasons, and was also manipulated to perform denial-of-service (DoS) attacks.  This is now used to stream partial offsets of a video file, allowing the player to start fetching segments with a lower bit-rate.  Hence, dynamic adaptive streaming over http.

You can check out the wildfly and castlab's code at my github public repository: https://github.com/charlescva/mobile-dashjs

Notice the Content-Length is determined by the offset as provided by the MPD and Initial MP4 containing the Metadata about each stream.


You can review the source, but the steps are as follows:

  1. Obtain a standard MP4 example video.
  2. Configure Apache to host the files in the directory dashencrypt is using. This is currently hard-coded in the VideoRegistration.java.
  3. Add Video using the Add a Video tab.  The JAX-RS enabled VideoService.java will handle the request, and dash the file for you.
  4. Upon success, you will see the entry for the video appear on the "Video Player" tab.
Observing the Console.  You can see the logger is outputting the steps as it processes the request.
I am still getting my feet wet as well, and came across a great article on the following website, https://arashafiei.wordpress.com/2012/11/13/quick-dash/.  

I'll be working on integrating a "live" stream in which a imaging device like /dev/video0 (webcam) will be used to generate the video segment data, while the MPD (Manifest) and initial MP4 file containing the Movie Box (moov) and/or Fragment Box (moof) are updated on the fly.  Essentially, the goal is to enable "DASHing" of a live video feed.



Friday, August 21, 2015

There was a problem playing this protected content. (Error Code: 3336)



Once again, the powers at be, controlling DRM software integration at Google and Hulu have decided to disregard the users of 'free' software and instead prioritize users of Windows, Apple and Google Chrome/Android devices.  QoS for the mainstream is understandable.  But there is a difference between low quality code that is not efficient or ideal, and code that simply doesn't work.

What is more annoying, is that these legal issues are masquerading as 'error codes'.  It can be determined I'm using linux via browser request headers, and IF the OS is linux, and the error is DRM related, just be honest and tell me that it's not supported on this platform, because of a recent update.

I'm sure they have been forced to enabled DRM, and waited until the last minute to do so.  And didn't have time to rework the baseline framework to support HTML5 for linux users.  I'm sure they'll do it eventually, but for now are covering their lack of planning with an error code.

This is not the first time this has happened.  We all remember the days of Linux tweaks getting Netflix to work using VMs, Wine, and god knows what else.  Finally, after years of complaining, Netflix allowed HTML5, and all is well... for now.

It seems that a new update from Hulu and Google does not permit Google Chrome running on Linux variants to support DRM protected content. This is apparently due to licensing issues, copyrights, and cost.  It would seem that Hulu blindly added the DRM requirements, without considering the repercussions (Or they just don't care).

Not surprising, small businesses with a great idea like Hulu once was, has become a draconian enterprise in which legal pressure and money woes force a quality product to become sub-par and only available to paid platforms.

After working a decade in software engineering, I have realized that what is important to those of us creating the software for the rest of society, is not relevant to the mainstream.  The same users who would tell you Linux is 'shit' because you can't be productive, are the ones who will ultimately whine and complain like a child whenever their Windows boot partition shits the bed.

As expected, the Linux base is now responding with the usual rhetoric about leaving Hulu and not giving them any more money.  There are some workarounds being expressed.  But likely I'll switch to either my Windows box, or I'll end up going w/ Firefox + HAL.

 For now, here are the most informative forums with users doing exactly what I described above.

Looking forward to the comments from those of you who 'know' whats really going on.

Thursday, August 20, 2015

Collecting Apache Storm time series data with Graphite

Naturally, Apache Storm processes data as tuples over time.  This is an ideal framework to utilize for streaming data through a 'pipe line'.  However, maintaining a record of this information is not necessarily an inherent feature of storm.  For metrics collection and analysis, I used Graphite.  Graphite is available in the Fedora EPEL and a great option for quickly collecting some metrics and generating a graph w/ overlaying analytical functions.

The easiest way to collect information is by going straight to the source.  The Nimbus server.  Using the NimbusClient class, available from the package backtype.storm.generated, you can easily review the ClusterSummary and iterate through each TopologySummary's ID to retrieve TopologyInfo and each topology's ExecutorSummary, to determine emitted and transferred tuples. Additionally, you can get information about errors, threads, and likely other details related to Apache Storm topologies I have not collected in my example code.

First, Lets look at how I will push data in to the carbon-cache daemon.
https://github.com/charlescva/graphite-common/blob/master/src/main/java/zkCliTest.java#L44

   // TCP Stream to carbon-cache.
    private void graphite(Map metrics) {
        // Current Time-Stamp for test
    long epoch = System.currentTimeMillis()/1000;
   
        try { // output  stream to the host on default port.
        Socket conn          = new Socket("cabon-cache.novalocal", 2003);
        DataOutputStream dos = new DataOutputStream(conn.getOutputStream());
        // graphite syntax map the #ngsm to output stream.
            for  (String metric : metrics.keySet() ){
                dos.writeBytes(metric +
            " " + metrics.get(metric) +
            " " + epoch + "\n");
            }
        //CLOSED CONNECTION.
            conn.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
The above code simply opens a TCP socket to the carbon-cache, writes a Map of metrics with the current timestamp, and closes the connection.  Easy.  The Take-away here, is the syntax. '[metric_path] [value] [time]' as String.

It would be redundant for me to display all of the Java code here: https://github.com/charlescva/graphite-common/blob/master/src/main/java/zkCliTest.java#L125 which essentially takes a Nimbus client connect, and builds a model of the nimbus state with your topoloy as each child node of the root context.

So, rather than bore you.  Here is a screenshot!


As you can see, the "My Topology" is a test, and clearly a static source that is linear.  But all and all, you can quickly get some good information.  Feel free to comment, as I find this article particularly interesting.

Thursday, August 6, 2015

Intel NUC won't turn on (after Ubuntu suspend)

UPDATE: THE SOLUTION TO THIS PROBLEM IS IN THE COMMENTS!  READ ON IF YOU WANT TO MAKE SURE YOUR PROBLEM REQUIRES THE SOLUTION!

Just got my NUC back up and running before posting this blog.  It doesn't surprise me much, as we have had a lot of issues with the Core i3 model at work.  I will say though, after ordering around 30 of these things, we have not had a single one brick or be dead on arrival.  Anyway, back to this issue.

I have been using a Microsoft Remote to power my NUC on and off via the Infrared sensor on the front.  I had to install a custom kernel module to get the operating system to understand the signal correctly.  As a result, I noticed if I use the remote to suspend the unit, I have to use the remote to turn it back on.  Pressing the soft switch on the unit itself does nothing.  I'm not sure if this issue is w/ the linux community drivers, intel, or both, but it is likely to be related.

The issue seems to be with the way the NUC hibernates after a software power down signal.  The BIOS/CMOS (whatever it's called these days), seems to store a bit somewhere in memory along with the system clock information related to the power state.  So the only way to get my NUC to power back on was to remove the battery from the motherboard, wait about five minutes for good measure, replace it, put the NUC back together, and boot on up.

To get to the bottom of the motherboard, you should first remove RAM and SSD (unless you really don't care about them.)  I was able to keep wifi connectors attached, but be gentle not to yank them off.  There are two black phillips head (+) screws located on the sides.  Remove these in front of a cat so he/she will look directly at where they land when you drop them on the floor.

Tricky Part: Turn the NUC upside down w/ your hand on the bottom to prevent the board from falling out and gently pry the back out w/ your finger.  Once the back is lifted up, you can shift/slide the board 1/16 of an inch (or about 1.5 cm) towards the rear to get the headphone jack and IR sensor clear of the holes in the chassis.

Battery is located to the left and removed via 2-pin connector

At this point the motherboard should be out and you can flip it around and get to the battery.

Also interesting is that on first boot, my mSATA SSD could not be detected.  Of course this initially had me worried, but after a second reboot, the OS discovered the disk just fine.

Not sure if a firmware update would have fixed this, but I've been running this NUC for more than 6 months without issue and am not going to bother w/ a firmware update at this time.

Likely related is the kernel modules used to power on/off the NUC via remote.  Here are the ones I have loaded:

charlie@NUC:~$ sudo lsmod | grep -i IRir_sony_decoder        12713  0 ir_lirc_codec          13021  0 lirc_dev               19980  1 ir_lirc_codecir_mce_kbd_decoder     13214  0 ir_sanyo_decoder       12839  0 ir_jvc_decoder         12751  0 ir_rc6_decoder         12874  0 ir_rc5_decoder         12710  0 ir_nec_decoder         12915  0 nuvoton_cir            17778  0 rc_core                28124  12 lirc_dev,nuvoton_cir,ir_lirc_codec,ir_rc5_decoder,ir_nec_decoder,ir_sony_decoder,ir_mce_kbd_decoder,ir_jvc_decoder,ir_rc6_decoder,ir_sanyo_decoder,rc_rc6_mce

And some additional information about my model/firmware.

BIOS Information
Vendor: Intel Corp.
Version: WYLPT10H.86A.0038.2015.0410.1634
Release Date: 04/10/2015
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 6656 kB
Characteristics:
PCI is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
BIOS ROM is socketed
EDD is supported
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 kB floppy services are supported (int 13h)
3.5"/2.88 MB floppy services are supported (int 13h)
Print screen service is supported (int 5h)
Serial services are supported (int 14h)
Printer services are supported (int 17h)
ACPI is supported
USB legacy is supported
BIOS boot specification is supported
Targeted content distribution is supported
UEFI is supported
BIOS Revision: 4.6
Base Board Information
Manufacturer: Intel Corporation
Product Name: D34010WYK
Version: H14771-303
Serial Number: XXXXXXXXXXXX (redacted)
Asset Tag:
Features:
Board is a hosting board
Board is replaceable
Location In Chassis: To be filled by O.E.M.
Chassis Handle: 0x0003
Type: Motherboard

Ubuntu 14.04.2 (amd64)
Linux Kernel 3.13.0-61