Search This Blog

Sunday, November 15, 2015

Dynamic Adaptive Streaming over HTTP (DASH) on WildFly

I've been away for a while.  October 10th, 2015 I started a new project at work, and that has kept me busy.

As demand has increased on internet media, the core web infrastructure has evolved to support new use cases for some pretty old standards.

HTTP has been used since the inception of the web around 1989.  The core concept being, a client's request receives a peer's response.  Originally a content length header element was used to box the request and response body in to some static size.  This was used for a variety of reasons, and was also manipulated to perform denial-of-service (DoS) attacks.  This is now used to stream partial offsets of a video file, allowing the player to start fetching segments with a lower bit-rate.  Hence, dynamic adaptive streaming over http.

You can check out the wildfly and castlab's code at my github public repository:

Notice the Content-Length is determined by the offset as provided by the MPD and Initial MP4 containing the Metadata about each stream.

You can review the source, but the steps are as follows:

  1. Obtain a standard MP4 example video.
  2. Configure Apache to host the files in the directory dashencrypt is using. This is currently hard-coded in the
  3. Add Video using the Add a Video tab.  The JAX-RS enabled will handle the request, and dash the file for you.
  4. Upon success, you will see the entry for the video appear on the "Video Player" tab.
Observing the Console.  You can see the logger is outputting the steps as it processes the request.
I am still getting my feet wet as well, and came across a great article on the following website,  

I'll be working on integrating a "live" stream in which a imaging device like /dev/video0 (webcam) will be used to generate the video segment data, while the MPD (Manifest) and initial MP4 file containing the Movie Box (moov) and/or Fragment Box (moof) are updated on the fly.  Essentially, the goal is to enable "DASHing" of a live video feed.

Friday, August 21, 2015

There was a problem playing this protected content. (Error Code: 3336)

Once again, the powers at be, controlling DRM software integration at Google and Hulu have decided to disregard the users of 'free' software and instead prioritize users of Windows, Apple and Google Chrome/Android devices.  QoS for the mainstream is understandable.  But there is a difference between low quality code that is not efficient or ideal, and code that simply doesn't work.

What is more annoying, is that these legal issues are masquerading as 'error codes'.  It can be determined I'm using linux via browser request headers, and IF the OS is linux, and the error is DRM related, just be honest and tell me that it's not supported on this platform, because of a recent update.

I'm sure they have been forced to enabled DRM, and waited until the last minute to do so.  And didn't have time to rework the baseline framework to support HTML5 for linux users.  I'm sure they'll do it eventually, but for now are covering their lack of planning with an error code.

This is not the first time this has happened.  We all remember the days of Linux tweaks getting Netflix to work using VMs, Wine, and god knows what else.  Finally, after years of complaining, Netflix allowed HTML5, and all is well... for now.

It seems that a new update from Hulu and Google does not permit Google Chrome running on Linux variants to support DRM protected content. This is apparently due to licensing issues, copyrights, and cost.  It would seem that Hulu blindly added the DRM requirements, without considering the repercussions (Or they just don't care).

Not surprising, small businesses with a great idea like Hulu once was, has become a draconian enterprise in which legal pressure and money woes force a quality product to become sub-par and only available to paid platforms.

After working a decade in software engineering, I have realized that what is important to those of us creating the software for the rest of society, is not relevant to the mainstream.  The same users who would tell you Linux is 'shit' because you can't be productive, are the ones who will ultimately whine and complain like a child whenever their Windows boot partition shits the bed.

As expected, the Linux base is now responding with the usual rhetoric about leaving Hulu and not giving them any more money.  There are some workarounds being expressed.  But likely I'll switch to either my Windows box, or I'll end up going w/ Firefox + HAL.

 For now, here are the most informative forums with users doing exactly what I described above.

Looking forward to the comments from those of you who 'know' whats really going on.

Thursday, August 20, 2015

Collecting Apache Storm time series data with Graphite

Naturally, Apache Storm processes data as tuples over time.  This is an ideal framework to utilize for streaming data through a 'pipe line'.  However, maintaining a record of this information is not necessarily an inherent feature of storm.  For metrics collection and analysis, I used Graphite.  Graphite is available in the Fedora EPEL and a great option for quickly collecting some metrics and generating a graph w/ overlaying analytical functions.

The easiest way to collect information is by going straight to the source.  The Nimbus server.  Using the NimbusClient class, available from the package backtype.storm.generated, you can easily review the ClusterSummary and iterate through each TopologySummary's ID to retrieve TopologyInfo and each topology's ExecutorSummary, to determine emitted and transferred tuples. Additionally, you can get information about errors, threads, and likely other details related to Apache Storm topologies I have not collected in my example code.

First, Lets look at how I will push data in to the carbon-cache daemon.

   // TCP Stream to carbon-cache.
    private void graphite(Map metrics) {
        // Current Time-Stamp for test
    long epoch = System.currentTimeMillis()/1000;
        try { // output  stream to the host on default port.
        Socket conn          = new Socket("cabon-cache.novalocal", 2003);
        DataOutputStream dos = new DataOutputStream(conn.getOutputStream());
        // graphite syntax map the #ngsm to output stream.
            for  (String metric : metrics.keySet() ){
                dos.writeBytes(metric +
            " " + metrics.get(metric) +
            " " + epoch + "\n");
        } catch (IOException e) {
The above code simply opens a TCP socket to the carbon-cache, writes a Map of metrics with the current timestamp, and closes the connection.  Easy.  The Take-away here, is the syntax. '[metric_path] [value] [time]' as String.

It would be redundant for me to display all of the Java code here: which essentially takes a Nimbus client connect, and builds a model of the nimbus state with your topoloy as each child node of the root context.

So, rather than bore you.  Here is a screenshot!

As you can see, the "My Topology" is a test, and clearly a static source that is linear.  But all and all, you can quickly get some good information.  Feel free to comment, as I find this article particularly interesting.

Thursday, August 6, 2015

Intel NUC won't turn on (after Ubuntu suspend)


Just got my NUC back up and running before posting this blog.  It doesn't surprise me much, as we have had a lot of issues with the Core i3 model at work.  I will say though, after ordering around 30 of these things, we have not had a single one brick or be dead on arrival.  Anyway, back to this issue.

I have been using a Microsoft Remote to power my NUC on and off via the Infrared sensor on the front.  I had to install a custom kernel module to get the operating system to understand the signal correctly.  As a result, I noticed if I use the remote to suspend the unit, I have to use the remote to turn it back on.  Pressing the soft switch on the unit itself does nothing.  I'm not sure if this issue is w/ the linux community drivers, intel, or both, but it is likely to be related.

The issue seems to be with the way the NUC hibernates after a software power down signal.  The BIOS/CMOS (whatever it's called these days), seems to store a bit somewhere in memory along with the system clock information related to the power state.  So the only way to get my NUC to power back on was to remove the battery from the motherboard, wait about five minutes for good measure, replace it, put the NUC back together, and boot on up.

To get to the bottom of the motherboard, you should first remove RAM and SSD (unless you really don't care about them.)  I was able to keep wifi connectors attached, but be gentle not to yank them off.  There are two black phillips head (+) screws located on the sides.  Remove these in front of a cat so he/she will look directly at where they land when you drop them on the floor.

Tricky Part: Turn the NUC upside down w/ your hand on the bottom to prevent the board from falling out and gently pry the back out w/ your finger.  Once the back is lifted up, you can shift/slide the board 1/16 of an inch (or about 1.5 cm) towards the rear to get the headphone jack and IR sensor clear of the holes in the chassis.

Battery is located to the left and removed via 2-pin connector

At this point the motherboard should be out and you can flip it around and get to the battery.

Also interesting is that on first boot, my mSATA SSD could not be detected.  Of course this initially had me worried, but after a second reboot, the OS discovered the disk just fine.

Not sure if a firmware update would have fixed this, but I've been running this NUC for more than 6 months without issue and am not going to bother w/ a firmware update at this time.

Likely related is the kernel modules used to power on/off the NUC via remote.  Here are the ones I have loaded:

charlie@NUC:~$ sudo lsmod | grep -i IRir_sony_decoder        12713  0 ir_lirc_codec          13021  0 lirc_dev               19980  1 ir_lirc_codecir_mce_kbd_decoder     13214  0 ir_sanyo_decoder       12839  0 ir_jvc_decoder         12751  0 ir_rc6_decoder         12874  0 ir_rc5_decoder         12710  0 ir_nec_decoder         12915  0 nuvoton_cir            17778  0 rc_core                28124  12 lirc_dev,nuvoton_cir,ir_lirc_codec,ir_rc5_decoder,ir_nec_decoder,ir_sony_decoder,ir_mce_kbd_decoder,ir_jvc_decoder,ir_rc6_decoder,ir_sanyo_decoder,rc_rc6_mce

And some additional information about my model/firmware.

BIOS Information
Vendor: Intel Corp.
Version: WYLPT10H.86A.0038.2015.0410.1634
Release Date: 04/10/2015
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 6656 kB
PCI is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
BIOS ROM is socketed
EDD is supported
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 kB floppy services are supported (int 13h)
3.5"/2.88 MB floppy services are supported (int 13h)
Print screen service is supported (int 5h)
Serial services are supported (int 14h)
Printer services are supported (int 17h)
ACPI is supported
USB legacy is supported
BIOS boot specification is supported
Targeted content distribution is supported
UEFI is supported
BIOS Revision: 4.6
Base Board Information
Manufacturer: Intel Corporation
Product Name: D34010WYK
Version: H14771-303
Serial Number: XXXXXXXXXXXX (redacted)
Asset Tag:
Board is a hosting board
Board is replaceable
Location In Chassis: To be filled by O.E.M.
Chassis Handle: 0x0003
Type: Motherboard

Ubuntu 14.04.2 (amd64)
Linux Kernel 3.13.0-61

Thursday, June 4, 2015

Openstack DHCP Lease renewing every minute on all VMs?!

Been having performance issues w/ the Openstack cluster. User's have mentioned that occasionally VMs are unresponsive.

I came across this article describing some of the common issues w/ DHCP, Improving DHCP Performance in OpenStack.

The article explains that the default lease duration is 120 seconds. In order to ensure dhcp clients (VMs) don't lose their IP, the client daemon attempts a renewal halfway through the lease duration. Which is approximately +/-60 seconds.

 In my case, I have about 20 virtual machines hitting a single DHCP agent every 60 seconds.

VMs on an Openstack network (private) will not be accessible (from public) during this time, causing periods of 'lock up' for users.

The fix is to simply increase DHCP lease duration to something a bit more sane for your platforms.
In my case, I chose 600 seconds, thus givng a nice 5 minute window between lease renewals.

However, if running Juno with HA, pacemaker will maintain a running dhcp agent on any host in your cluster.  This can be confusing since you will see the agent running on a node, even though the service is 'stopped'.

The fix is just to update the config on your nodes, and then disable/enable the resource via pacemaker.

# For each node in your cluster, from a controller as root run:

[root@node-54 ~]# ssh node-46 -C "echo dhcp_lease_duration = 600 >> /etc/neutron/dhcp_agent.ini"
[root@node-54 ~]# ssh node-47 -C "echo dhcp_lease_duration = 600 >> /etc/neutron/dhcp_agent.ini"
[root@node-54 ~]# ssh node-50 -C "echo dhcp_lease_duration = 600 >> /etc/neutron/dhcp_agent.ini"
[root@node-54 ~]# ssh node-51 -C "echo dhcp_lease_duration = 600 >> /etc/neutron/dhcp_agent.ini"
[root@node-54 ~]# ssh node-52 -C "echo dhcp_lease_duration = 600 >> /etc/neutron/dhcp_agent.ini"
[root@node-54 ~]# ssh node-53 -C "echo dhcp_lease_duration = 600 >> /etc/neutron/dhcp_agent.ini"
[root@node-54 ~]# ssh node-54 -C "echo dhcp_lease_duration = 600 >> /etc/neutron/dhcp_agent.ini"

Next, disable and then enable the service:

[root@node-54 ~]# pcs resource disable p_neutron-dhcp-agent
[root@node-54 ~]# pcs resource enable p_neutron-dhcp-agent

[root@node-54 ~]# neutron agent-list
| id                                   | agent_type         | host             | alive | admin_state_up | binary                    |
| 11a5d775-7f10-4856-b905-a6176e9cb689 | Open vSwitch agent | | :-)   | True           | neutron-openvswitch-agent |
| 14b47084-d1f5-428c-8fed-746a5d5e743f | Metadata agent     | | :-)   | True           | neutron-metadata-agent    |
| 311fe911-503d-498f-a027-72414c6c8664 | DHCP agent         | | :-)   | True           | neutron-dhcp-agent        |
| 326c7dcc-03b0-4830-8a4e-6b58f3af2445 | Open vSwitch agent | | :-)   | True           | neutron-openvswitch-agent |
| 3867b579-1bc2-4aee-b9b6-5bc21f61f514 | Open vSwitch agent | | :-)   | True           | neutron-openvswitch-agent |
| 3de48cde-8dba-4042-9a04-baa9f3d411f9 | Open vSwitch agent | | :-)   | True           | neutron-openvswitch-agent |
| 5b85f41e-2671-4bf6-8dc5-243a0ecb55b3 | L3 agent           | | :-)   | True           | neutron-l3-agent          |
| 61b3c026-99e0-4c50-a440-339b7085d428 | L3 agent           | | :-)   | True           | neutron-l3-agent          |
| 6e9b071d-5417-45a2-abf6-72b2691fd464 | Open vSwitch agent | | :-)   | True           | neutron-openvswitch-agent |
| 91e124a6-6594-4fb4-b48c-06b697cbf437 | L3 agent           | | :-)   | True           | neutron-l3-agent          |
| 9881ddce-f177-4120-9019-1fc26eee19ca | Open vSwitch agent | | :-)   | True           | neutron-openvswitch-agent |
| 9e259924-80ce-44c8-9704-8cb8ed35d751 | Metadata agent     | | :-)   | True           | neutron-metadata-agent    |
| c5b75fd5-f83b-4afb-a4ff-271c92d61695 | Open vSwitch agent | | :-)   | True           | neutron-openvswitch-agent |
| cd99597e-ead6-4eb3-a729-4bc609955ee6 | Metadata agent     | | :-)   | True           | neutron-metadata-agent    |
[root@node-54 ~]# neutron agent-show  311fe911-503d-498f-a027-72414c6c8664
| Field               | Value                                                   |
| admin_state_up      | True                                                    |
| agent_type          | DHCP agent                                              |
| alive               | True                                                    |
| binary              | neutron-dhcp-agent                                      |
| configurations      | {                                                       |
|                     |      "subnets": 5,                                      |
|                     |      "use_namespaces": true,                            |
|                     |      "dhcp_lease_duration": 600,                        |
|                     |      "dhcp_driver": "neutron.agent.linux.dhcp.Dnsmasq", |
|                     |      "networks": 5,                                     |
|                     |      "ports": 39                                        |
|                     | }                                                       |
| created_at          | 2015-06-04 17:35:57                                     |
| description         |                                                         |
| heartbeat_timestamp | 2015-06-04 17:36:46                                     |
| host                |                                        |
| id                  | 311fe911-503d-498f-a027-72414c6c8664                    |
| started_at          | 2015-06-04 17:35:57                                     |
| topic               | dhcp_agent                                              |

The result is that the client will repeat DHCPREQUEST until the service is back up, and start using the new lease time,

root@hannibal:~# tail -f /var/log/syslog
Jun  4 17:35:02 hannibal dhclient: DHCPACK of from
Jun  4 17:35:02 hannibal dhclient: bound to -- renewal in 51 seconds.
Jun  4 17:35:53 hannibal dhclient: DHCPREQUEST of on eth0 to port 67 (xid=0x50f4018e)
Jun  4 17:36:35 hannibal dhclient: message repeated 6 times: [ DHCPREQUEST of on eth0 to port 67 (xid=0x50f4018e)]
Jun  4 17:36:37 hannibal dhclient: DHCPNAK from (xid=0x50f4018e)
Jun  4 17:36:37 hannibal dhclient: DHCPDISCOVER on eth0 to port 67 interval 3 (xid=0x75f1e5db)
Jun  4 17:36:37 hannibal dhclient: DHCPREQUEST of on eth0 to port 67 (xid=0x75f1e5db)
Jun  4 17:36:37 hannibal dhclient: DHCPOFFER of from
Jun  4 17:36:37 hannibal dhclient: DHCPACK of from
Jun  4 17:36:37 hannibal dhclient: bound to -- renewal in 255 seconds.
Jun  4 17:40:52 hannibal dhclient: DHCPREQUEST of on eth0 to port 67 (xid=0x75f1e5db)
Jun  4 17:40:52 hannibal dhclient: DHCPACK of from
Jun  4 17:40:52 hannibal dhclient: bound to -- renewal in 211 seconds.
Jun  4 17:44:23 hannibal dhclient: DHCPREQUEST of on eth0 to port 67 (xid=0x75f1e5db)
Jun  4 17:44:23 hannibal dhclient: DHCPACK of from
Jun  4 17:44:23 hannibal dhclient: bound to -- renewal in 262 seconds.
Jun  4 17:45:01 hannibal CRON[4249]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jun  4 17:48:45 hannibal dhclient: DHCPREQUEST of on eth0 to port 67 (xid=0x75f1e5db)
Jun  4 17:48:45 hannibal dhclient: DHCPACK of from
Jun  4 17:48:45 hannibal dhclient: bound to -- renewal in 249 seconds.

"Attempting to work around these performance problems by significantly increasing IP lease time will cause a huge problem with respect to the release of IP addresses by neutron if your cloud loads dynamically change. By default, neutron will allocate an IP address to a VM for 24 hours, independent of the actual lease time. Also, by default, neutron will not release an IP address until 24 hours after an instance has been terminated."


Thursday, February 26, 2015

Recover Openstack Ceph data with missing/no monitor(s)

Recently ceph monitors got blown away.  Along with this was all of the metadata associated with the monitors.  Using this technique I was able to recover some of my data, but it was a lot of sleuthing

In the top left corner are the script running in a loop over all of the unique 'header' files from the various osds.

The main script is in the top right corner.  Essentially we traverse the servers (nodes) and ceph osd instances throughout the cluster, collecting files (with find) that match the wildcard and are bigger than a byte.

The "wildcard" is the key, "13f2a30976b17" which is defined as replicated header file names for each rbd image on your ceph cluster.  If you had 10 images, with 3 replicas, you would find 30 header files in your cluster, with identical names for the replicas.  This would be okay, even if they are on the same server; because they are in separate osd data folders.

Using SSH we fetch a list of all the files on an osd instance and dump to a temp file.  We do a cut on the slash(/) folder separator and dump a list of the files in a new file and remove the temp.

We then dump all the files into a csv, with the osd node location in column 1 and the file name in column 2.  the -u switch only snags unique instances, so replicas are dropped.

We then execute a little script called scp obs.  the tricky part here is the backslash in the ceph file names. use double quotes in the scp command and escape the \ with \\.   So that's 3 slashes surrounded in double quotes w/ the scp command.

finally once we have all the object files.  we 'dd' them together ans the final output.

Two quick notes,

in my cut command i use column #8 and #9.  Thinking about it, this could give you a different result depending on where your osd data folder is.  Mine is the default path, /var/lib/ceph/osd/ceph-0/current/4.4f/

For my convenience, at the end I mv the "raw" file to qcow2, since I know that is what these images are.  This is based on the output of hexdump -C -n 4 -s 0 $first-block, where the first-block is the object with 16 zeroes. (the first block in the object group).  It basically tells me the header of the first block which is 'QFI' for qcow2.

I even converted one of the qcow2 files to a VDI and booted in successfully in virtualbox.

The bash scripts can be found here:

It is the next morning, and I let my script run overnight. Check it out. :)

Wednesday, February 18, 2015

Administering Fuel with Openstack Juno Services

I have recently started using Openstack in an environment with 'production' value.  By this I mean that our Openstack instance is becoming a critical component of our business infrastructure; and at this point several development support services are tenants within it.

Openstack is not an easy solution.  Almost every core service is distributed, decentralized, and utilizes the full scope of their dependencies. This results in good news, and bad news.  The good news is that your infrastructure is so loosely coupled, that failures will USUALLY be localized to a specific process or configuration setting.  The bad news is, until you learn the terminology and components, you'll be running around like a mad man trying to find the various configs and error logs.


First you will need to ensure your file system is stable.  Ceph has been with Openstack since for a long time.  Yes it is different than any other file system you're likely used to.  This means you'll have to learn something new.  One of the biggest issues with migration and spawning VMs can stem from failures to read/write RAW data to the distributed file system.

The best thing to do first, is read over this paper on RUSH, or replication under scalable hashing:

The gist of this paper should help you to understand that Ceph clients in Openstack use the jenkins hash ( with a tree of weighted buckets (CRUSH Map, and a map defaulting of 256 placement groups ( to figure out where objects are stored.  Also that Ceph is not a file system, per say, but an "object store".  This means there is no central server the clients must negotiate with to read and write object data.  The Ceph documentation is phenomenal, and you should familiarize yourself with it is much as you can.  Most of your questions are answered in the documentation, you'll just need to be patient, read it all at a decent pace, and let the information resonate with your mind for a night before digging in to it again.  After a couple of days it will start to make more sense.  Here are some common commands to take a peak at:

  • ceph osd tree
  • ceph -w
  • ceph osd reweight (don't just run this randomly, understand what it does first)
Also keep in mind there have been bug reports regarding applying a new Crush map to a running cluster.  So spend a lot of time looking at a sample crush map in a test cluster before applying a new one.  It is likely that you can resolve a lot of your issues by using reweight and or modifying the number of replicas in largely used storage pools. like your Openstack volumes, images and compute pool for ephemeral storage

RBD (Rados Block Device)

RBD is used on top of the Ceph object store.  This provides the API Openstack uses to connect your volumes and images to the hypervisor you're using (Hopefully QEMU, because I like it and want it supported).  Here are some helpful commands:
  • rados df
  • rbd import
  • rbd export
  • rbd ls|rm|mv
  • qemu-img convert (although not rbd specific, relvent when dealing with RAW rbd images and compressing them to qcow2 for moving across the network)
In an earlier post on this blog, you will see my experience upgrading openstack.  In there you will see where I manually migrated each of my VMs from an Icehouse cluster to Juno.  I had some hardware constraints and it was tough, but in the end it worked very well. 

nova,cinder,glance CLI

You won't get by on the UI alone.  The bash command line for an openstack controller is your best tool.  Don't be afraid to poke around the databases on mysql for cinder, glance and nova.  Use the nova, glance and cinder tools with the 'help' argument and read the usage.  These tools are required to communicate with the API in a standardized way that is supported by the developers of Openstack.   If you're using 3rd party providers like Mirantis Fuel for Openstack, then you will need to use their documentation for maintaining Openstack environments.  Be advised, some of these 3rd party tools are lacking support and capability to perform some of the tasks you will need to know to properly maintain the environment.

Here are the ones to know:
  • nova boot
    • --availability-zone
    • --nic id
    • --flavor
    • flags for Volume or Image backed.
  • nova services-list
  • nova service-delete (Gets mention for not in Havana, but is in Juno!)
Seriously though, use mysql and don't be affraid to adjust the instances metadata.  Sometimes a VM is actually OFF, but the Horizon UI will show it as 'Shutting Down...' or 'Running'.  You can verify the status of your VM by SSHing into the compute node hosting the instance, and as root running:

# ps -ef | grep kvm

You'll see the instance id in the run command, as well as a bunch of other args.  Be advised, the domain.xml virsh uses is generated in code by python and uses the information in mysql to do so.  So modifying things like the video driver or video ram, require changes to the flavor and image metadata. I recently saw in Juno an option to nova boot with args passing metadata key values to set in the virsh domain, although I have not tried it yet.  I believe it is here:, and the boot option appears to be --image-width .


Neutron is a bit overwhelming.  Just know that the Open vSwitch service on your compute nodes handle the networking for the VMs running there.  Just because your L3 Agent(s) are down and you cannot get to the VM using it's public IP, does not mean that the VM is Off, it just means that the external connection isnt being routed.  Ensure all of these services are running and configured correctly.  This section is intentionally short because of the vast configuration options with neutron.
  • neutron list-agents
Last I need to thank the developers at Mirantis Fuel and others hanging out on the freenode IRC channel #fuel.  I could not have learned as much as I know at this point, without the help of a few users in there.  Thank you guys for your gracious support throughout my adoption of Openstack.

Monday, February 2, 2015

maven: Target server failed to respond

Was getting an exception with the maven (3.2.2) wagon plugin version 2.6.

Looking deeper, this plugin uses wagon-provider 2.6 which depends on httpclient 4.3.  This can be seen here:
and looks like this:

 </dependency> ...  

The exception seen when running a maven build looks similar to this:

Caused by: org.apache.maven.wagon.providers.http.httpclient.NoHttpResponseException: The target server failed to respond

This bug:

indicates that there is a bug with httpclient in the versions 4.3 >= 4.3.4 & 4.4 Alpha1 when using a proxy (like apache httpd) between the client and server, without client authentication enabled.  The bug causes the no response exception later in the wagon plugin called via the WagonRepositoryConnector class.

Specifically, the bug occurs when authentication is disabled because as of 4.3.x the has a function to create tunnel to target, (createTunnelToTarget) that when authentication is disabled, the for loop does not exit properly and the request is never completed.

In our case, just upgrading to 3.2.5 worked.   Another option is to include the lightweight http provider with wagon configuration.  this uses the Java HTTP libraries instead of apache's implementation.

Tuesday, January 27, 2015

How to configure Dell BMC

When configuring a Dell PowerEdge C6220 (Or Similar) to be remotely administrated with BMC you will need to modify the BIOS configuration.  In this example, a dedicated NIC is used.
  1. Boot the Server and Press F2 to enter the BIOS.
  2. From the BIOS, use the right arrow key to navigate to the "Server" menu, then move down to the BMC Administration.
  3. Configure the IP Address for the BMC and set the Inteface from "Shared-NIC" to "Dedicated NIC".
  4. Press Esc to get back to the main menu, then use the right arrow key and navigate all the way to last menu and Save the configuration. (Do not exit the BIOS at this time.)
  5. Connect an Ethernet cable to the Dedicated BMC port (Identified with a open ended wrench icon) and plug the other end into your network LAN switch.
  6. From another PC, use a web browser to connect to http:// and use the username root/root to log in.

Once you are able to log in to the console, you will likely want to configure the remote KVM.  This is slightly more complex:
  1. Navigate to the vKVM settings and click the Launch link, then click "Launch Java KVM Client", this should launch a JNLP file with javaws. However, since Java 1.7.0_51, self-signed code can not be executed.  The work around create a file: ~/.java/deployment/security/exception.sites and add the following lines:
    1. http://
    2. https://
  2. Now, when you run the JNLP KVM client, you will be allowed to authorize the execution of self signed code. 

If you have configured everything correctly you should be able to see the BIOS where we left off in Step 4 above:


vMedia is used to map the server's CD ROM drive to your PC's CDROM, or a disk image (*.iso,*.dmg).  It is executed from the web console as well.

Additional Information:

Use the PowerEdge BMC getting started guide: poweredge-c6105_User's Guide_en-us.pdf
If you get an error running the KVM 'login denied/not authroized', edit the username/password inside the JNLP file. Change it to 'root/root' or whatever the security credentials are configured as. Then relaunch Java KVM ($ javaws ~/Downloads/viewer.jnlp) - Info about self signed code execution in jre 1.7.0 update 51
 The Advocent KVM and vMedia require port 2068 to be accessible from the client to the server. (This means that if there is a firewall between the client and the server the client intends to connect to, an exception must be made from the client to the destination port 2068 on the server.

Monday, January 26, 2015

Upgrading Openstack with Fuel

DISCLAIMER:  This is a development environment as well as a work in progress.  Do not attempt this on a 'production' system without going through the process first on a non-production system and learning how to trouble shoot the system at various points.

Also, this is an 'in place' upgrade.  Which means you need to have approximately less than 49% of your hardware resources used, so you can create a new environment and migrate everything.  Unless you only plan to migrate a few things.  However, this is an entirely different subject.

I have tried to break this process down in to the following steps:

  1. Plan Upgrade
  2. Migrate Instances(VMs) to free up additional nodes.
  3. Remove free Node's Ceph OSD from Ceph Cluster
  4. Clean up Neutron agents.
  5. Disable Nova Services
  6. Upgrade your Fuel Server
  7. Deploy new Environment (Juno here)
  8. Export Volumes from Old (Icehouse) Environment
  9. Import Volumes as Images and boot new VMs w/ new Volume
  10. Repeat Step 9 for all instances to keep, and Delete the old Environment

We have been running a "steady" instance of Fuel 5.1 with 3 controllers configured with "HA" (High-Availability) and 4 Nodes operating as both CEPH and Compute nodes.

We started finding small timeout issues and bugs w/ 5.1, so decided it was time to upgrade.  6.0 is out, so we had to download the 5.1.1 update, and the 6.0 update.  And run them both in order.  Here are the links to the software, and installation guide(s).  Please be advised, this information is current as of January 26, 2015. - Latest Download.  (Fuel 6.0 for me today.)

Step 1.) Plan

Our plan is to decommission a single Compute, Storage - Ceph OSD node, and a single Controller.  Fuel will not like running HA with 2 controllers.  But it should be okay to deal with it while we migrate our environment.

Step 2.) Migrate instances and free up a node.

Live Migration Guide - Using this guide, I was able to determine the instances I had running on Node18, and migrate them to a node that had less.

[root@node-10 ~]# nova host-describe
| HOST             | PROJECT                          | cpu | memory_mb | disk_gb |
| | (total)                          | 24  | 72492     | 8731    |
| | (used_now)                       | 13  | 27136     | 260     |
| | (used_max)                       | 10  | 20480     | 200     |
| | dc25784fc9d94e58b3887045756cf9e8 | 8   | 16384     | 160     |
| | 0dc5c66d16b04d48b07c868cc195f46a | 2   | 4096      | 40      |
[root@node-10 ~]# nova list --host --all-tenants
| ID                                   | Name     | Status | Task State | Power State | Networks                            |
| b58af781-bb57-4c35-bbe5-4153e2d4bb6e | alfresco | ACTIVE | -          | Running     | net04=, |
| 2006d7db-d18e-4390-ae1c-40dd77644853 | hannibal | ACTIVE | -          | Running     | ONR=,      |

In my case, node-15 had the least number of instance, so I decided to migrate the two instance on 18 over to 15; starting with 'alfresco'.

[root@node-10 ~]# nova live-migration b58af781-bb57-4c35-bbe5-4153e2d4bb6e
ERROR: HTTPConnectionPool(host='', port=8774): Max retries exceeded with url: /v2/0dc5c66d16b04d48b07c868cc195f46a/servers/b58af781-bb57-4c35-bbe5-4153e2d4bb6e/action (Caused by : )

Notice the ERROR message.  I believe this is because the migration took longer than expected.  However, I verified through the nova list command, as well as the Horizon UI that the server was still migrating hosts, so i waited.  It finished within 5 minutes, so i then verified:

[root@node-10 ~]# nova list --host --all-tenants
| ID                                   | Name     | Status | Task State | Power State | Networks                       |
| 2006d7db-d18e-4390-ae1c-40dd77644853 | hannibal | ACTIVE | -          | Running     | ONR=, |
[root@node-10 ~]# nova list --host --all-tenants
| ID                                   | Name            | Status | Task State | Power State | Networks                             |
| 2e1057ee-48d5-4b7f-aa9e-14b0103535ec | Mantis          | ACTIVE | -          | Running     | ONR=,       |
| b58af781-bb57-4c35-bbe5-4153e2d4bb6e | alfresco        | ACTIVE | -          | Running     | net04=,  |
| 655fba2a-9867-4305-935c-e6b3c3a84368 | docker-registry | ACTIVE | -          | Running     | net04=,   |
| f368bab9-e054-4bda-84ee-e5633e6381cb | docker01        | ACTIVE | -          | Running     | DS Network=, |
| b336499d-7314-464f-9f98-ee1ed0ddd787 | inventory       | ACTIVE | -          | Running     | net04=,   |
| 1b57d04c-29c7-4a1b-8cac-114f491ec5d3 | onr-node-4      | ACTIVE | -          | Running     | ONR=,       |

Also notice, the public IP did not change.  This is good. :)  Repeat this process to free up a second node to support ceph install on juno, or find a new server in your budget to use.

Step 3.) Remove the Ceph OSD from Ceph cluster

Since I am removing Node-18, I will remove the ceph instance from OSD via Node-18!

List some pools for sanity.

 [root@node-18 ~]# ceph osd lspools
0 data,1 metadata,2 rbd,3 images,4 volumes,5 .rgw.root,6 compute,7 .rgw.control,8 .rgw,9 .rgw.gc,10 .users.uid,11 .rgw.buckets.index,12 .rgw.buckets,

Determine which OSD ID this node has.  #6 in this case

[root@node-18 ~]# ps -ef | grep ceph
root      3258     1  2  2014 ?        1-01:46:17 /usr/bin/ceph-osd -i 6 --pid-file /var/run/ceph/ -c /etc/ceph/ceph.conf --cluster ceph
root     11490 10726  0 17:25 pts/0    00:00:00 grep ceph

Mark it for removal

[root@node-18 ~]# ceph osd out 6
marked out osd.6. 

Watch the rebalance happen

[root@node-18 ~]# ceph -w
    cluster 994f6ed1-69c0-4e8b-8c76-fc1186c7eda5
     health HEALTH_WARN mon.node-10 low disk space; mon.node-12 low disk space; mon.node-13 low disk space
     monmap e3: 3 mons at {node-10=,node-12=,node-13=}, election epoch 676, quorum 0,1,2 node-10,node-12,node-13
     osdmap e217: 7 osds: 7 up, 6 in
      pgmap v6442467: 5312 pgs, 13 pools, 1301 GB data, 301 kobjects
            2304 GB used, 5895 GB / 8199 GB avail
                   1 active+clean+scrubbing+deep
                5311 active+clean

You will see lot of entries that describe what is happening.  Most importantly something like this:

9854/630322 objects degraded (1.563%)

This means that ceph's RBD objects (which keep your openstack data), do not have enough replicas for 9854 objects.  It will copy the replica to another host, so you will have to wait until all objects that were on your OSD node are rebalanced.  This will utilize a lot of network I/O, and your active VMs will suffer.  So warn your users before doing this.

Again, ensure this is done on any additonal nodes you want to delete.

Step 4.) Cleanup Neutron

You may notice that once your node is gone, there are some stale neutron agents marked dead.

The following command will list all dead (xxx) agents and show details.  Change 'agent-show' to 'agent-delete' to remove them permenantly:

for i in $(neutron agent-list | grep "xxx" | awk '{print $2}'); do neutron agent-show $i; done;

Step 5.) Cleanup Nova Services

Just like the neutron services, the nova services on your old node may still show up in horizon as 'down'.  You can use something like the command below to disable them.

for i in $(nova service-list | grep node-13 | awk '{print $2}'); do nova service-disable $i; done;

I have did not figure out how to delete the services.  But it doesn't really matter, because I will be deleting the entire environment once the upgrade and migrations are complete.

Step 6.) Upgrade to Fuel 6.0 if you didn't already.

Once your instances have been all migrated.  You should be able to use the Fuel UI and decomission the node and one controller.  Run the either before or after.  I did it before migrating instances.

Step 7.) Create the new Environment with Fuel UI.

You should now have a free controller and nodes to build a Juno Openstack environment to start migrating your instances from the old Icehouse Openstack environment.  Hopefully with virtually 0 downtime.

Ceph installation requires at least 2 nodes.

Step 8.) Export Volumes from Old Environment

There are likely a variety of ways to import/export volumes in openstack.  I have found the following method works well.

First, find a place on your old controller w/ extra disk.  Generally /var/lib/mongo has a lot of space w/ default partitioning.  Locate the UUID for a volume using nova or cinder list.  instance IDs are used for the ephemeral disks, volume IDs are used for 'volume' disks. Make sure the Instance using the disk is shut off.

Export it with rbd, then compress it to qcow2 so you can pass it over the Fuel network to your other environment's controller.  In this example, I am exporting a 'Volume' as 'raw' disk, then converting it.

[root@node-10 mongo]# rbd export --pool=volumes volume-0f2a87ec-74c5-4356-a4e7-12fffd6fe5ea docker-registry.raw                                                                                                                               
Exporting image: 100% complete...done.
[root@node-10 mongo]# qemu-img convert -f raw -O qcow2 ./docker-registry.raw docker-registry.qcow2

To support SCP, on the old environment controller modify /etc/ssh/sshd_config and set PasswordAuthentication to 'yes' which is at the bottom of the file. (Also int he middle of the file, but commented out).  Then # useradd temp and set the password, # passwd temp
#service sshd restart, and you should now be able to scp the data from the new controller.

Step 9.) Import Volumes as Images and launch

Here, my new Juno environment's controller is 'node-35'

[root@node-35 ~]# scp temp@node-10:/var/lib/mongo/docker-registry.qcow2 .                                                                   
Warning: Permanently added 'node-10,' (RSA) to the list of known hosts.
temp@node-10's password:
docker-registry.qcow2                                                  65% 7489MB 90.3MB/s    00:21 ETA

Get the byte size first:
[root@node-35 ~]# ls -al docker-registry.qcow2 

-rw-r--r-- 1 root root 11334057984 Feb 11 16:34 docker-registry.qcow2

And now, import it as an image with glance:

[root@node-35 ~]# glance image-create --size 11334057984 --name docker-registry --store rbd --disk-format qcow2 --container-format bare --file ./docker-registry.qcow2

Importing into Glance while watching Ceph/rbd

| Property         | Value                                |
| checksum         | a209fafa8ae5369e0a93b30e41c4e27c     |
| container_format | bare                                 |
| created_at       | 2015-02-11T16:40:09                  |
| deleted          | False                                |
| deleted_at       | None                                 |
| disk_format      | qcow2                                |
| id               | 22209ea8-2287-425b-9e45-c79ec210d380 |
| is_public        | False                                |
| min_disk         | 0                                    |
| min_ram          | 0                                    |
| name             | docker-registry                      |
| owner            | aeea9a5fd7284450a3468915980a8c45     |
| protected        | False                                |
| size             | 11334057984                          |
| status           | active                               |
| updated_at       | 2015-02-11T16:47:53                  |
| virtual_size     | None                                 |

At this point, you should be able to launch a new instance from a converted Image->Volume, and specify this glance image as the source, and create a new Volume specifying the size of the original volume.  In my case with this docker registry, it was 100GB, even though qcow compressed it down to 11GB.
Create Volume from Image

Create Volume from Image with Specified Size of Original Volume (Not qcow size!)
Booting instance from new Volume

If at any point the UI has an error.  Just watch the osd pool stats # watch ceph osd pool stats volumes
You should see client io that is pretty heavy.  When it is done, you can refresh your volumes on horizon UI and it should be there as 'Available'.

Step 10.) "Rinse and Repeat"

Go ahead and repeat step 8/9 for all the instances you want migrated.  Set up your public IPs, update your external DNS entries, etc. And wait a day to make sure things are stable.  Afterwards go ahead and delete the old environment, add the free nodes to your new environment, and migrate some instances to lighten the load on your first couple of nodes, and you should be good to go!