Search This Blog

Wednesday, February 18, 2015

Administering Fuel with Openstack Juno Services

I have recently started using Openstack in an environment with 'production' value.  By this I mean that our Openstack instance is becoming a critical component of our business infrastructure; and at this point several development support services are tenants within it.

Openstack is not an easy solution.  Almost every core service is distributed, decentralized, and utilizes the full scope of their dependencies. This results in good news, and bad news.  The good news is that your infrastructure is so loosely coupled, that failures will USUALLY be localized to a specific process or configuration setting.  The bad news is, until you learn the terminology and components, you'll be running around like a mad man trying to find the various configs and error logs.


First you will need to ensure your file system is stable.  Ceph has been with Openstack since for a long time.  Yes it is different than any other file system you're likely used to.  This means you'll have to learn something new.  One of the biggest issues with migration and spawning VMs can stem from failures to read/write RAW data to the distributed file system.

The best thing to do first, is read over this paper on RUSH, or replication under scalable hashing:

The gist of this paper should help you to understand that Ceph clients in Openstack use the jenkins hash ( with a tree of weighted buckets (CRUSH Map, and a map defaulting of 256 placement groups ( to figure out where objects are stored.  Also that Ceph is not a file system, per say, but an "object store".  This means there is no central server the clients must negotiate with to read and write object data.  The Ceph documentation is phenomenal, and you should familiarize yourself with it is much as you can.  Most of your questions are answered in the documentation, you'll just need to be patient, read it all at a decent pace, and let the information resonate with your mind for a night before digging in to it again.  After a couple of days it will start to make more sense.  Here are some common commands to take a peak at:

  • ceph osd tree
  • ceph -w
  • ceph osd reweight (don't just run this randomly, understand what it does first)
Also keep in mind there have been bug reports regarding applying a new Crush map to a running cluster.  So spend a lot of time looking at a sample crush map in a test cluster before applying a new one.  It is likely that you can resolve a lot of your issues by using reweight and or modifying the number of replicas in largely used storage pools. like your Openstack volumes, images and compute pool for ephemeral storage

RBD (Rados Block Device)

RBD is used on top of the Ceph object store.  This provides the API Openstack uses to connect your volumes and images to the hypervisor you're using (Hopefully QEMU, because I like it and want it supported).  Here are some helpful commands:
  • rados df
  • rbd import
  • rbd export
  • rbd ls|rm|mv
  • qemu-img convert (although not rbd specific, relvent when dealing with RAW rbd images and compressing them to qcow2 for moving across the network)
In an earlier post on this blog, you will see my experience upgrading openstack.  In there you will see where I manually migrated each of my VMs from an Icehouse cluster to Juno.  I had some hardware constraints and it was tough, but in the end it worked very well. 

nova,cinder,glance CLI

You won't get by on the UI alone.  The bash command line for an openstack controller is your best tool.  Don't be afraid to poke around the databases on mysql for cinder, glance and nova.  Use the nova, glance and cinder tools with the 'help' argument and read the usage.  These tools are required to communicate with the API in a standardized way that is supported by the developers of Openstack.   If you're using 3rd party providers like Mirantis Fuel for Openstack, then you will need to use their documentation for maintaining Openstack environments.  Be advised, some of these 3rd party tools are lacking support and capability to perform some of the tasks you will need to know to properly maintain the environment.

Here are the ones to know:
  • nova boot
    • --availability-zone
    • --nic id
    • --flavor
    • flags for Volume or Image backed.
  • nova services-list
  • nova service-delete (Gets mention for not in Havana, but is in Juno!)
Seriously though, use mysql and don't be affraid to adjust the instances metadata.  Sometimes a VM is actually OFF, but the Horizon UI will show it as 'Shutting Down...' or 'Running'.  You can verify the status of your VM by SSHing into the compute node hosting the instance, and as root running:

# ps -ef | grep kvm

You'll see the instance id in the run command, as well as a bunch of other args.  Be advised, the domain.xml virsh uses is generated in code by python and uses the information in mysql to do so.  So modifying things like the video driver or video ram, require changes to the flavor and image metadata. I recently saw in Juno an option to nova boot with args passing metadata key values to set in the virsh domain, although I have not tried it yet.  I believe it is here:, and the boot option appears to be --image-width .


Neutron is a bit overwhelming.  Just know that the Open vSwitch service on your compute nodes handle the networking for the VMs running there.  Just because your L3 Agent(s) are down and you cannot get to the VM using it's public IP, does not mean that the VM is Off, it just means that the external connection isnt being routed.  Ensure all of these services are running and configured correctly.  This section is intentionally short because of the vast configuration options with neutron.
  • neutron list-agents
Last I need to thank the developers at Mirantis Fuel and others hanging out on the freenode IRC channel #fuel.  I could not have learned as much as I know at this point, without the help of a few users in there.  Thank you guys for your gracious support throughout my adoption of Openstack.

No comments:

Post a Comment