Apt-cacher with Docker

I have been working with docker from www.docker.io for couple of weeks and I love it.  I have made containers for Postgres w/ postgis extensions, Accumulo/HDFS/Zookeeper running pseudo-distributed, geoserver and as of yesterday, an apt-cacher for our Ubuntu 14.04 workstations at the office.  Here is all you need!

You can get the /etc/apt-cacher/apt-cacher.conf from an ubuntu system after installing the apt-cacher package.

#
# example ubuntu 14.04 apt-cacher
#

FROM ubuntu:14.04
MAINTAINER championofcyrodiil.blogspot.com

USER root
RUN apt-get update
RUN apt-get -y -q install apt-cacher
ADD apt-cacher.conf /etc/apt-cacher/apt-cacher.conf
ADD start-cacher.sh /root/start-cacher.sh
RUN chmod +x /root/start-cacher.sh
EXPOSE 3142


#Default Docker Run Comamnd(s)
CMD ["/root/start-cacher.sh"]


And of course, the 'start-cacher.sh' bash script.  Make sure it has been chmod +x so it's executable.

#!/bin/bash
/usr/sbin/apt-cacher -R 3 -d -p /var/run/apt-cacher.pid
tail -f /var/log/apt-cacher/access.log
wait


Once that is done, you will want to place all three files in to a single folder, cd to that folder and build the docker image, here is my example:


 $ sudo docker build -t local:apt-cacher .

Note the period at the end to specify the context of the current directory with the scripts.  Next it is time to run the container, make sure the host (-h) matches the daemon listening host specified in /etc/apt-cacher/apt-cacher.conf.

$ sudo docker run -d -p 3142:3142 -h apt-cacher local:apt-cacher

$ sudo docker ps -sl

CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS                    NAMES               SIZE

7b6a7ff25691        local:apt-cacher    /root/start-cacher.s   18 hours ago        Up 18 hours         0.0.0.0:3142->3142/tcp   clever_hoover       361.4 MB

For client systems to use the proxy, add the file /etc/apt/apt.conf.d/01proxy with the contents:

Acquire::http::Proxy "http://$DOCKERHOST:3142";

Like so,

$ sudo echo 'Acquire::http::Proxy "http://:3142";' >> /etc/apt/apt.conf.d/01proxy

Screen Locking with X and C

I had an issue recently with screen locking not working properly in LTSP.  The shadow file is not available on the file system of the thick client, so when the screen lock is run as a local program, the user can not unlock to get back to the desktop.

The work around was to install xscreensaver, which authenticates using a PAM module when the screen is locked.  However, the 'hook' which tells the system to 'lock' after a determined idle time is disabled to prevent gnome from locking out the user.

So, I created my own C application which runs in the background and monitors for user mouse movement or keystrokes.  There are 4 threads:

• Timer Thread (Track user idle time, locks screen if threshold is reached.)
• Screen Watch Thread (Start timer if screen is ever Unlocked)
• Mouse Thread (Restart timer if mouse is moved)
• Keyboard Thread (Restart timer if keystroke is made)

I used several sources and although this is not the most elegant solution it does work.  I would appreciate any feed back, as this is the first C application I have ever written, except for "Hello world."




#include
#include
#include
#include
#include
#include

typedef struct Timer
{
double elapsed;
struct timespec start,finish;
}Timer;


//Predefine global vars.
double timeLimit = 300.0; //seconds
int verboseClock = 0; //1-True,0-False
int verboseMouse = 0; //1-True,0-False
int verboseKeyboard = 0; //1-True,0-False

//do not modify these
volatile int movement = 0;
volatile int screenlocked = 0;
pthread_t threads[4];
int rc_one;
int rc_two;
int rc_three;
int rc_four;


void *MouseWatch(void *threadid) {
//This threa will watch the mouse and ensure the timer is reset if event fired.
FILE *fp;
char buffer[3];
/* Open the command for reading from xinput. */
fp = popen("xinput --test 9", "r");
if (fp == NULL) {
printf("Failed to watch mouse\n");
exit;
}
printf("Watching Mouse...\n");
/* Read the output a line at a time - output it. */
  while (fgets(buffer, sizeof(buffer)-1, fp) != NULL) {
if(verboseMouse) {  
printf("%c", buffer[0]);
}
    if(buffer[0]=='m') { 
//motion detected.
movement = 1;

}
  }
pclose(fp);
}



void *LockTimer(void *threadid) {
//This thread acts as a timer until Locked, provided mousemove==0 the whole time.
Timer stopwatch;
//start timer.
clock_gettime(CLOCK_MONOTONIC, &stopwatch.start);
//while time < timeLimit seconds.
while(stopwatch.elapsed < timeLimit ) {
if(movement == 1) {
movement = 0;
//mouse moved, restart the clock
clock_gettime(CLOCK_MONOTONIC, &stopwatch.start);
}
//get elapsed time.  if clock has exceeded threshold, lock screen.
       sleep(1);
clock_gettime(CLOCK_MONOTONIC, &stopwatch.finish);
       stopwatch.elapsed = (stopwatch.finish.tv_sec - stopwatch.start.tv_sec);
       stopwatch.elapsed += (stopwatch.finish.tv_nsec - stopwatch.start.tv_nsec) / 1000000000.0;
if(verboseClock) {
printf("elapsed time: %f\n",stopwatch.elapsed);
}
if(screenlocked) { break; }
}
system("xscreensaver-command -lock");
screenlocked = 1;

while(screenlocked) {
sleep(2);
}

printf("restarting the timer\n");
rc_two = pthread_create(&threads[2], NULL, LockTimer, (void *)2);
if(rc_two)  {
printf("ERROR; return code from pthread_create() is %d\n", rc_two);
                exit(-1);
}

pthread_exit(NULL);
}

void *KeyboardWatch(void *threadid) {
//This threa will watch the keyboard and ensure the timer is reset if key is pressed.
FILE *fp;
char buffer[3];
/* Open the command for reading from xinput. */
fp = popen("xinput --test 10", "r");
if (fp == NULL) {
printf("Failed to watch keyboard\n");
exit;
}
printf("Watching Keyboard...\n");
/* Read the output a line at a time - output it. */
  while (fgets(buffer, sizeof(buffer)-1, fp) != NULL) {
    if(verboseKeyboard) { printf("%s", buffer); }
    if(buffer[0]=='k') { 
//key press detected.
movement = 1;
}
  }
pclose(fp);
}

void *ScreenWatch(void *threadid) {
//This thread will watch the screen and ensure timer is started when screen is 'U'nlocked.
FILE *fp;
char buffer[50];
/* Open the command for reading. */
  fp = popen("xscreensaver-command -watch", "r");
  if (fp == NULL) {
    printf("Failed to run command\n" );
    exit;
  }
printf("Watching Screen...\n");
  /* Read the output a line at a time - output it. */
  while (fgets(buffer, sizeof(buffer)-1, fp) != NULL) {
   
printf("%s", buffer);

    if(buffer[0]=='U') { 
screenlocked=0;
} 
if(buffer[0]=='L') {
screenlocked=1;
}
  }
}



int main()
{
        printf("In main: creating threads. \n");
//start mouse watch thread
rc_one = pthread_create(&threads[1], NULL, MouseWatch, (void *)1);
        if(rc_one)  {
                        printf("ERROR; return code from mouse thread is %d\n", rc_one);
                        exit(-1);
                }
//start timer thread
        rc_two = pthread_create(&threads[2], NULL, LockTimer, (void *)2);
if(rc_two)  {
                        printf("ERROR; return code from timer thread is %d\n", rc_two);
                        exit(-1);
                }
//start Screen watch thread
rc_three = pthread_create(&threads[3], NULL, ScreenWatch, (void *)3);
        if(rc_three)  {
                        printf("ERROR; return code from screen watch thread is %d\n", rc_one);
                        exit(-1);
                }
//start keyboard watch thread
rc_four = pthread_create(&threads[4], NULL, KeyboardWatch, (void *)4);
        if(rc_four)  {
                        printf("ERROR; return code from keyboard thread is %d\n", rc_four);
                        exit(-1);
                }
//LOOP Forever until user Kills main thread. (Ctrl+C)
while(1) {
sleep(10);
}

}

nomachine (!M) NX client on ubuntu LTSP thick/fat clients

The NX client/server software released from the company NoMachine, is fantastic.  I did discover that their Linux x86 installer does not properly handle missing dependencies on a CentOS 6.5 Minimal installation.  But it was my fault for mistakenly choosing the wrong binary from their download site.  Nevertheless, A few yum installs later, and even their 32 bit client is fantastic! Of course, I'm running their x64 client in our development environment now.

There is a lot to take in when you first run the client tool.  Its classy simple icon, installer, and website pitch, leads you to believe this magic tool will just work without a lot of features.  And for me, it did.  Every time. On different platforms.  The streaming of multimedia through the nx client over seperate UDP port is genius. And there are tons of small features that make this worth the effort.

There isn't a custom view you can't get with their client.  All of your devices are integrated as well.  I haven't even started to use collaboration tools.  But they are there if you need them, and even include recording video sessions.

I am running their software on the newest Ubuntu 14.04 LTSP "fat" client desktop.  I initially ran into an issue with the thick client LDM sessions not properly locking the screen.  Filed a bug report with launchpad here: https://bugs.launchpad.net/ubuntu/+source/unity/+bug/1316320.  I was able to work around by installing the classic xscreensaver package with the gl extras.  Not only is this batch of screen savers really cool, but it enables you to create a desktop and unity bar shortcut that will activate the xscreensaver-command lock.  Just make sure you enable the authentication dependencies on on the client image as well.  Otherwise your screen cannot be unlocked if your client session is logged in with a terminal server account that isn't also on the image.

The hardware I chose was the Intel NUC.  There are several variations to this model, but the model #D34010WYKA worked great for me.  With a modern monitor and using the display port as the primary output.  Running this PC without hard disk or wireless networking is very snappy.  With the NoMachine client on top of that, I can stream Youtube video with audio, flawlessly from a headless KVM running CentOS 6 and a "nohup" GDM session.  It was quite impressive.

My next investigation will be deeper into virus scanning solutions for the LTSP environment.  Although the client image is read-only.  A user can still execute downloaded code from their home folder, or temporary write space on the RAMfs (/tmp).  They could exploit vulnerabilities on your network systems and create back door entry points, causing information leakage, and more.  Often users who aren't intentionally malicious will pick up these Trojans and viruses from various websites.  Having a modern virus scanning engine will stop a lot of this junk.  It may not stop someone creating custom code and targeting your network specifically.  But it will help ensure avoidable accidents don't happen.

ClamAV is looking like a good bet.  It is an open source (GPL) antivirus engine.  But McAfee has a lot of years under their belt and is already on the approved list for a lot of organizations.  Corporate solutions tend to have a bit smaller footprint than the Best Buy 1st year free edition you often get with buying a PC from a partnered vendor.  And paid products usually include personal support.

Thin Client Computing with Ubuntu Linux

I used Ubuntu 12.04 ALTERNATIVE ISO to perform the F4 Mode "LTSP Installation" during ISO boot.

LTSP is a Thin Client solution for Linux operating systems.  This was chosen because of the preferred use of Ubuntu Linux for development.  Benefits of LTSP are as follows:

·      Reduced Costs – Thin Clients require fewer resources than traditional Thick clients and therefore have a lower procurement cost.

·      No Licensing Fees – LTSP is open source software released under GPLv2 License.

·      Less Maintenance – Single point of control is the operating system image on the thin clients.

·      Security – LTSP clients are secured via SSH and are restricted to their own LAN.

·      LTSP Display Manager (LDM) – Python application for remote desktop SSH sessions.  KDM/GDM do not support remote SSH sessions.

Typical LTSP Layout

LTSP is typically run from a single server with two network cards that piggyback the LTSP isolated LAN and the larger network.  The LTSP Server uses NAT to provide connections between thin clients and the rest of the resources on the larger network.  This allows more control over the connections between developers and network systems and services.  Developers still have access to web services and network bound APIs they need, without necessarily having access to sensitive management protocols and systems.

LTSP supports a concept called ‘screen scripts’.  Multiple screen scripts can be run at the same time on different virtual consoles. (Ctrl + Alt + F[1-9])  User’s can toggle between screens while Screen six (or seven?) is reserved for the LDM.   Screen scripts can also be used to enable rdesktop for connecting to a Windows Server. 

LTS Configuration allows many custom configurations to be applied per client machine.  Here are some examples:

[AA:BB:CC:DD:EE:FF]

# Use nvidia driver for this thin client, overriding auto-detected driver

XSERVER = nvidia

[FF:EE:DD:CC:BB:AA]

# Set Screen 7 of this client to an RDP session rather than LDM

SCREEN_07 = “rdesktop 192.168.0.253”

A new feature of LTSPv5 is the ability to run linux applications installed on the chroot (“change root”, the image used by the thin clients) environment from within the LDM session.  This means reduced server load, enables use of graphics intense multimedia applications, and enables use of applications that require direct hardware access.  Drawbacks include increased chroot maintenance and increased hardware requirements on thin clients.

Local devices can also be supported with thin clients; so removable media such as CDROM and USB Flash drives can still be used on the thin clients.

Printers are supported and spooling is done on the server. No client-side print driver management required.

Sound is redirected from the server to the client using PulseAudio.  This network-aware client-server sound system can easily go through NAT firewalls.

Although not yet tested, LTSP also supports use of “Thick” clients.  Also known as Fat Clients.  These client machines would have a larger network block device root file system containing a complete OS with all desired additional programs (i.e. Chrome). Since processes are running on the client rather than the server, an admin cannot kill them from a central location.  Internet connectivity is provided directly to the client, so the client needs to be directed to an Internet gateway.

https://help.ubuntu.com/community/UbuntuLTSP/

ORACLE initialization or shutdown in progress

Our database guy was trying to perform some operations on the oracle 11g database today and got the following error:

ORA-011033: ORACLE initialization or shutdown in progress

This could have a lot of underlying errors, but in our situation this occurs when the power is unexpectedly turned off to the server, causing the database transaction logs to not be closed properly.

Provided that you configured RMAN backups when you installed the instance,  (Which should have been rather apparent during the installation of the software and the creation of the database) recovery from this situation should be quite smooth.

I will highlight the commands manually entered.  Anything else is a response from command(s).

First, we log into the server using the oracle account you created before installing the database software.  Once you are logged into the operating system, on the console you will want to run sqlplus as the SYSDBA account, and properly shutdown the database. (It's probably stuck trying to initialize but cannot, since it will attempt to start on boot when the power came back on)  Then you will want to start the instance, manually mount the control files, but NOT open the database yet, just quit.

ORACLEBIR:/export/home/oracle$ sqlplus '/as sysdba'

SQL> shutdown abort

ORACLE Instance shut down.

SQL> startup nomount

ORACLE Instance started

SQL> alter database mount;

SQL> quit

At this point, your instance has been started and the database files have been mounted.  Now we run RMAN or the Recovery Manager.  Here we will request that RMAN performs a recovery.  If you get more errors or this does not work for you, it would seem your backup and recovery settings are not configured properly, and you are really in trouble.  Otherwise, it should look something like this:

ORACLEBOX:/export/home/oracle$ rman

RMAN> connect target

connected to target database: DBV2 (DBID=2494479496, not open)

using target database control file instead of recovery catalog

RMAN> recover database;

Starting recover at 09-APR-14

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=161 device type=DISK

starting media recovery

archived log for thread 1 with sequence 871 is already on disk as file /u1/app/oracle/oradata/dbv2/redo01.log

archived log file name=/u1/app/oracle/oradata/dbv2/redo01.log thread=1 sequence=871

media recovery complete, elapsed time: 00:00:07

Finished recover at 09-APR-14

RMAN> quit

If you see something like above, congratulations, your database has recovered.  Now, let's go back and actually open the database and reset the logs.  Also, make sure to start enterprise manager, and that your backup admin knows how to connect and use this tool to some degree.  Although some do not like the Enterprise Manager, it is essential to automate tasks for anyone who is not familiar with some of the basic operations.  And you might get hit by a bus, leaving all the work in your colleague's lap.

ORACLEBOX:/export/home/oracle$ sqlplus '/as sysdba'

SQL> alter database open resetlogs;

SQL> quit

ORACLEBOX:/export/home/oracle$ emctl start dbconsole

Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.0

Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.

https://ORACLEBOX:1158/em/console/aboutApplication

Starting Oracle Enterprise Manager 11g Database Control .................... started.

------------------------------------------------------------------

Logs are generated in directory /u1/app/oracle/product/11.2.0.3/dbhome_1/ORACLEBOX_dbv2/sysman/log

Your EM Login:

https://ORACLEBOX:1158/em/console/logon/logon

Backup and Recovery Guide for Oracle RDBMS

Administration Guide for Oracle 11g

Cloudera SCM Agent Error

"This host had been out of contact with Cloudera Manager for too long. The host's Cloudera Manager agent's software version could not be determined."

Today I saw this error pop up on the CM4 hosts monitor.  Running /etc/init.d/cloudera-scm-agent status only confirmed that the agent was running.  However I needed to review the logs to find the error.

The log for the agent is located at /var/log/cloudera-scm-agent/cloudera-scm-agent.log

The error reported looked like this:

[08/Apr/2014 15:58:09 +0000] 1228 MainThread agent        ERROR    Heartbeating to prodsrv01vmid.saic.com:7182 failed.
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 741, in send_heartbeat
    self.master_port)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 471, in __init__
    self.conn.connect()
  File "/usr/lib64/python2.6/httplib.py", line 720, in connect
    self.timeout)
  File "/usr/lib64/python2.6/socket.py", line 553, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
gaierror: [Errno -5] No address associated with hostname

The problem was, that when the system rebooted, the file /etc/cloudera-scm-agent/config.ini was modified:

[General]
# Hostname of Cloudera SCM Server
server_host=prodsrv01vmid.saic.com

The DNS server had an old host name entry for the IP address my Cloudera SCM Server was now using.  When the system restarted the agent, I believe a DNS lookup was performed using the IP and resolved the old host name.  My cluster uses /etc/hosts files to maintain name resolution, so I'm not 100% sure yet why this happened, but I speculate it is a result of the socket library in python, used by the cloudera SCM agent.

Resolved by changing the server_host value back to the host with the SCM server running on it.  Then restarted the cloudera-scm-agent service.

Parrot AR Drone 2.0

So I am starting a new hobby.  The Parrot AR Drone 2.0 is an awesome piece of equipment.

http://ardrone2.parrot.com/

Here are some interesting points.

• 720p Camera mounted to the front
• GPS Supported
• Linux 2.6
• MAVlink compatable for use with Qgroundcontrol
• Uses Wifi 2.4 GHz 802.11b/g/n standards for control and configuration
• And more...

I have only embarked on a single outdoor test flight.  And it was very successful.  Since then I have ordered the GPS module and the 2000mah batteries with balance changer for quick charges.  Once everything comes, I should be able to support continuous flight.

There are also a lot of mods that can be performed.  Camera mods, hull mods, transceiver mods, and more.

I have also started looking into an opensource solution on github called ardrone_automony. Linked here: https://github.com/AutonomyLab/ardrone_autonomy So far I have been successful at connecting to the drone and looking at settings, however I have not yet been able to issue commands.  Ultimately I would like to be able to map my joystick input.

Do you work with these types of devices? let me know your thoughts.

UPDATE!!!
I was able to configure both a Joystick and a Gamepad to control the AR Drone using the SDK from Parrot.   Once you get the code to compile and run in QT, it is quite easy to use with a little testing.

Also got the GPS working with qgroundcontrol.  this was really a cool experience and I have had many flights via GPS.  WARNING, pay attention to altitude settings and ensure the MAX altitude is configured on the drone before using the MAVlink GPS.  If your drone's max altitude is set to 3 M, then it doesnt matter that you may have 10 M in the GPS flight.