Search This Blog

Monday, May 21, 2018

Repairing Corrupted LUKS Encrypted Filesystem

This morning I noticed that my USB backup drive would not mount properly.  The first thing I did was check the dmesg log from Debian/Linux...

[1962217.824829] usb 1-5.1: New USB device found, idVendor=0bc2, idProduct=3300
[1962217.824832] usb 1-5.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[1962217.824834] usb 1-5.1: Product: Desktop         
[1962217.824835] usb 1-5.1: Manufacturer: Seagate 
[1962217.824837] usb 1-5.1: SerialNumber: 2GHNRSVE    
[1962217.825589] usb-storage 1-5.1:1.0: USB Mass Storage device detected
[1962217.825927] scsi host4: usb-storage 1-5.1:1.0
[1962218.832372] scsi 4:0:0:0: Direct-Access     Seagate  Desktop          0130 PQ: 0 ANSI: 4
[1962218.833253] sd 4:0:0:0: Attached scsi generic sg1 type 0
[1962218.833370] sd 4:0:0:0: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[1962218.833691] sd 4:0:0:0: [sdb] Write Protect is off
[1962218.833694] sd 4:0:0:0: [sdb] Mode Sense: 2f 08 00 00
[1962218.833991] sd 4:0:0:0: [sdb] No Caching mode page found
[1962218.833996] sd 4:0:0:0: [sdb] Assuming drive cache: write through
[1962218.842262]  sdb: sdb1
[1962218.843668] sd 4:0:0:0: [sdb] Attached SCSI disk
[1962219.029068] sd 4:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE
[1962219.029072] sd 4:0:0:0: [sdb] tag#0 Sense Key : Hardware Error [current] [descriptor] 
[1962219.029074] sd 4:0:0:0: [sdb] tag#0 Add. Sense: No additional sense information
[1962219.029077] sd 4:0:0:0: [sdb] tag#0 CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
[1962219.111836] sd 4:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE
[1962219.111839] sd 4:0:0:0: [sdb] tag#0 Sense Key : Hardware Error [current] [descriptor] 
[1962219.111841] sd 4:0:0:0: [sdb] tag#0 Add. Sense: No additional sense information
[1962219.111844] sd 4:0:0:0: [sdb] tag#0 CDB: ATA command pass through(12)/Blank a1 06 20 da 00 00 4f c2 00 b0 00 00
[1962251.479639] JBD2: Invalid checksum recovering block 144 in log
[1962251.574260] JBD2: recovery failed
[1962251.574265] EXT4-fs (dm-4): error loading journal
[1962258.650930] JBD2: Invalid checksum recovering block 144 in log
[1962258.749283] JBD2: recovery failed
[1962258.749288] EXT4-fs (dm-4): error loading journal

This generally means that there were issues with the unmount and the "checksum"  for block 144 in the journal does not match the actual checksum from disk.  This means there is an error reading the journal, and to prevent further corruption EXT4-fs will not mount via device mapper.

So how do we fix this?  If this was not encrypted you could google this error and find out to run FSCK and hopefully be done with it.

Lets start with a quick reminder of the utility `lsblk`.

root@CLCFQ92:~# lsblk --fs /dev/sdb
NAME                                          FSTYPE      LABEL  UUID   
└─sdb1                                        crypto_LUKS        ce6cebbc-5026-4f47-9a22-da4aecfd26ad 
  └─luks-ce6cebbc-5026-4f47-9a22-da4aecfd26ad ext4        Backup e7bab3dc-87ce-4a7d-b758-34e2839b51f0 

 This 'console' output does not render well above using the blog defaults, so I'll have to modify it for clarity, but the thing to notice is that the File System type for sdb1 is NOT ext4, it is crypto_LUKS.  There is then an extended partition that is 'ext4' called luks-.  This luks partition is the one you want to run fsck on.  NOT /dev/sdb1 (or /dev/sdb).

So lets first open the encrypted fs...

root@CLCFQ92:~# cryptsetup luksOpen /dev/sdb1 corrupted
Enter passphrase for /dev/sdb1: 
Now, lets recover the device via the name ("corrupted") we mapped to the luks partition... 
root@CLCFQ92:~# fsck /dev/mapper/corrupted 
fsck from util-linux 2.29.2
e2fsck 1.43.4 (31-Jan-2017)
Backup: recovering journal
JBD2: Invalid checksum recovering block 144 in log
Journal checksum error found in Backup
Backup was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (436669919, counted=433657941).
Fix? yes
Free inodes count wrong (121861163, counted=121856397).
Fix? yes

Backup: ***** FILE SYSTEM WAS MODIFIED *****
Backup: 245363/122101760 files (0.4% non-contiguous), 54719555/488377496 blocks
Next, disable/close the mapped name to the luks partition:
root@CLCFQ92:~# cryptsetup luksClose /dev/mapper/corrupted
Then mount/open the filesystem the way you normally would using gnome or whatever.  Check dmesh logs for confirmation!

At this point it may be good to look at using the `smartctl` utility, as your disk may be old/dying...

root@CLCFQ92:~# smartctl -a /dev/sdb
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
  1 Raw_Read_Error_Rate     0x000f   118   100   006    Pre-fail  Always       -       200743680
  3 Spin_Up_Time            0x0003   094   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       72
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   073   063   030    Pre-fail  Always       -       24115411
  9 Power_On_Hours          0x0032   077   077   000    Old_age   Always       -       20273
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       39
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   053   043   045    Old_age   Always   In_the_past 47 (Min/Max 42/47 #179)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       19
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       767
194 Temperature_Celsius     0x0022   047   057   000    Old_age   Always       -       47 (0 19 0 0 0)
195 Hardware_ECC_Recovered  0x001a   022   021   000    Old_age   Always       -       200743680
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       12697 (213 75 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       3453343066
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       2807657092

Based on the output from the SMART controller, this disk is having a lot of errors and should be backed up and disposed of.

Friday, March 2, 2018

Easy install of NVIDIA SDK on Linux

Spent some time looking around for the best solution to this problem.  I think that it is best to download the 1.8GB installer from nvidia.  Then execute something like the following:

./ --no-drm --no-opengl-libs --extract=/home/ottch/Development/nvidia_cuda_9_1/ --toolkit --toolkitpath=/home/ottch/Development/nvidia_cuda_9_1/ --silent

Next you'll need to execute the cuda-linux .bin file that is generated in the toolkitpath by the last command.  Ensure to set the prefix to something other than /usr/local/cuda-9.1 if you don't have sudo.

Tuesday, September 26, 2017

Export oVirt Image/Template for VMware

oVirt supports exporting your VM or template to multiple formats.  You can find a decent overview of the utility here, [[]]

 QCOW2 (KVM, Xen) - *.qcow2 QED (KVM) - *.qed raw - *.raw (Sometimes no file extension provided) VDI (VirtualBox) - *.vdi VHD (Hyper-V) - *.vpc VMDK (VMware) - *.vmdk

The utility to perform Exports is 'qemu-img' with the sub-command 'convert'

For example, I currently have a VM Template Image in my GlusterFS storage domain.  This storage domain is mounted through the hypervisor "rosie-carreiro", so I first SSH in to the hypervisor, and 'cd' to the path where the RAW template resides.

Using the oVirt admin console, I am able to identify that the template/image has a UUID of, bbb58b82-7252-41e2-9b28-72cbf74d98e4.
There are other CLI tools that allow you to query storage domains and resolve UUIDs to names, however the UI is pretty intuitive for new users.

When you run 'df -h' on an oVirt hypervisor, you can see the mount points used:  399G   25G  374G   7% /rhev/data-center/mnt/       847G  136G  711G  17% /rhev/data-center/mnt/     8.2T  761G  7.4T  10% /rhev/data-center/mnt/                   1.9T   82G  1.8T   5% /rhev/data-center/mnt/glusterSD/

In this case the Template is in the GlusterFS "gv0" volume, so I first 'cd' to /rhev/data-center/mnt/rosie-carreiro... and find the directory with the UUID, bbb58b82-7252-41e2-9b28-72cbf74d98e4.

  [root@devops bbb58b82-7252-41e2-9b28-72cbf74d98e4]# ls -ahlt  total 33G  drwxr-xr-x. 9 36 36 4.0K Aug 25 19:08 ..  drwxr-xr-x. 2 36 36 4.0K Aug 25 13:16 .  -rw-r--r--. 1 36 36  318 Aug 25 13:16 4b66fd92-99a6-4651-8288-24f50fd3596a.meta  -rw-rw----. 1 36 36  40G Aug 25 13:16 4b66fd92-99a6-4651-8288-24f50fd3596a  -rw-rw----. 1 36 36 1.0M Aug 25 12:58

You can also view the metadata which specifies the format and description.

  [root@devops bbb58b82-7252-41e2-9b28-72cbf74d98e4]# cat 4b66fd92-99a6-4651-8288-24f50fd3596a.meta  DOMAIN=8fe3f7f3-15c9-4376-b20e-5e237b31b185  CTIME=1503680317  FORMAT=RAW  DISKTYPE=2  LEGALITY=LEGAL  SIZE=83886080  VOLTYPE=SHARED  DESCRIPTION={"DiskAlias":"Windows7_Disk1","DiskDescription":""}  IMAGE=bbb58b82-7252-41e2-9b28-72cbf74d98e4  PUUID=00000000-0000-0000-0000-000000000000  MTIME=0  POOL_UUID=  TYPE=SPARSE  GEN=0  EOF

You can now export the template/image to your desired format:

  [root@rosie-carreiro bbb58b82-7252-41e2-9b28-72cbf74d98e4]# qemu-img convert -f raw -O vmdk -o adapter_type=lsilogic 4b66fd92-99a6-4651-8288-24f50fd3596a /root/DevImage-1.1.vmdk
Once completed, verify:  [root@rosie-carreiro bbb58b82-7252-41e2-9b28-72cbf74d98e4]# ls -halt /root/DevImage-1.1.vmdk  -rw-r--r--. 1 root root 33G Aug 27 16:58 /root/DevImage-1.1.vmdk

NOTE: adapter_type=lsilogic is important because qemu-img VMDK output defaults to using IDE.  This means that you cannot later extend the size of the disk in VMware, as the option will actually be grayed out/disabled in vSphere.  Use the lsilogic scsi controller format to support extending your disks.

Thursday, August 3, 2017

Setting VirtualBox SMBIOS Settings to match your PC

In some cases you may want your VM to emulate your physical machine.  For example, if you want to set the Serial Number/Service Tag to match your laptop, you can do this with the following steps:

  1. sudo dmidecode -t system
  2. VBoxManage setextradata "VM name" "VBoxInternal/Devices/pcbios/0/Config/DmiSystemSerial" "#######"
In Step two above, you would not use the #######, but instead use the Serial Number displayed when running dmidecode.

Essentially what we are doing is configuring the SMBIOS (System Management BIOS) on the VirtualMachine to have the same strings/settings as the actual PC.  This can be useful in a scenario in which you are running some software within a virtual machine that expects the hardware to be of a specific vendor or serial number.

More information about the SMBIOS can be found here:

More information about setting VirtualBox VM SMBIOS data can be found here:

And last, more information about how to use dmidecode on your Linux PC can be found here:

This technique can be used to "spoof" the VM's Product ID, Serial Number, BIOS Version, Manufacturer and really any other SMBIOS value.


Saturday, August 6, 2016

Linux, Newest Kernel, Latest Hardware, Windows 10

About eight years ago I installed Ubuntu Intrepid Ibex, which I believe was Ubuntu 8.10.  I had a Dell desktop at work and they just rolled out, "Trusted Desktop".  The original intention of trusted desktop is simply to ensure an end-user's workstation is safe, free of viruses, and has the most recent security updates.  This helps to keep malware and attackers out of the corporate network when users plug in to it daily.

The problem for me is that because the corporation was so large, they did not ask or care about the developer's needs.  The first version of trusted desktop they rolled over top of Windows XP completely removed Administration capability, as well as installed several incompatible drivers and encryption software combinations.  Eventually these systems began showing the "Blue screen of death", and crawling to a halt under the poorly designed implementation of trusted platform computing.

At this point I saw where Windows was going with my company and discovered they would "allow" employees to use Linux if they signed a waiver and ensured they implemented a compliant disk encryption policy.  So I was off, installing Ubuntu, Virtualbox and Windows XP VM to support Windows specific work software I needed to use.  In this case it was mostly Microsoft Office and the ActiveID client drivers for my smartcard to work with the VPN software.

I didn't need wireless because this was a desktop plugged in at work.  However I did have two outputs on my video card and hoped I would be able to run a similar dual head/monitor display, the way I had been doing on XP for a couple of years.  I quickly had flashbacks of trying to get wireless drivers working on Mandrake Linux, circa 2004.  It was only a bit painful getting Ibex to use my Radeon correctly, but after all of this I was content for about a year.

Unfortunately, a year later support for 8.10 died out, and it was time to upgrade to 9.04, Jaunty. In that release, support for my old ATI Radeon GPU was dropped in favor of the newest version of X that was shipping w/ Jaunty.  I felt betrayed by the Linux community for leaving my crappy old GPU behind.  I had no idea how to write my own driver then, had my own work to do, and so I gave up and went back to Windows XP and then 7 for a long time.  Of course, later I would provide systems support for about fifty developers running Ubuntu 14.04 LTS, but this topic would cause us to digress as I usually do.

Well, this past week I have gone back to Linux, this time choosing Debian Jessie 8.  I chose Debian because I am feeling unsure about where Canonical will be taking Ubuntu in the future, and I wanted something I was already familiar with in regard to package management.  For example, I have recently used a Slackware 10 distribution on an old laptop I have, but it leaves much to be desired in regard to community support for the latest hardware.

After choosing Debian for my OS, I got a new Dell Precision 7710 laptop.  This thing really packs a punch with:
  • Intel® Core™ i7-6820HQ CPU @ 2.70GHz
  • Intel Skylake GPU w/ CPU
  • Nvidia® Quadro® M3000M w/4GB GDDR5
  • 32GB (4x8GB) 2133MHz DDR4 SDRAM, Non-ECC
  • Hynix 512GB M.2 PCIe NVMe Class 40 Solid State Drive

You can imagine my disappointment when after 8 years passing since I ran Linux on Dell for my personal system, my desktop booted with a notification that my Cinnamon desktop was being rendered in software mode.  Furthermore, my wireless interface was nowhere to be found.

The first issue was with a feature in the BIOS being enabled called Optimus.  I don't want to make this entire post about Optimus, so I'll use a reference from Wikipedia. "Nvidia Optimus is a computer GPU switching technology created by Nvidia which, depending on the resource load generated by client software applications, will seamlessly switch between two graphics adapters within a computer system in order to provide either maximum performance or minimum power draw from the system's graphics rendering hardware."

Apparently the Dell supported version of Ubuntu knows how to handle configuration this correctly.  However, Debian Jessie out of the box could not.  I chose not to attempt to enable Optimus (Optimux would have been a better name.) support on Debian, but found some interesting work here, that may assist if you choose to do so.  For me, I have simply disabled the setting in the BIOS, and have opted for the NVidia GPU to be my primary display adapter.  Perhaps later I will investigate Optimus further.

However this led me to my next issue, in that my GPU is not yet supported by Nouveau.  The Quadro M3000M is part of the Maxwell-2 series.

NVIDIA Quadro Mobile Specification Comparison (High-End)
Quadro M5000MQuadro M4000MQuadro M3000M
CUDA Cores153612801024
Memory Clock5GHz GDDR55GHz GDDR55GHz GDDR5
Memory Bus Width256-bit256-bit256-bit
ArchitectureMaxwell 2Maxwell 2Maxwell 2

A quick review of the list at will show you that this GPU is not supported.  So, I had to install the bundle available from Nvidia.  The important thing to remember when bundling the manufacture's driver is to ensure DKMS (Dynamic Kernel Module Support) is installed and that Nvidia uses DKMS to manage the kernel module between kernel updates.

So with my GPU now working, it was time to figure out the issue with my wireless.  A quick `lspci | grep Wireless` resolved,

02:00.0 Network controller: Intel Corporation Wireless 8260 (rev 3a)

The firmware for Intel wireless adapters generally ends up in a package called, 'iwlwifi' and once installed on most systems, will match the vendor id on your hardware during modprobe, and load the correct module.  As luck would have it, this card's module was not added/patched until November 2015.

This means my driver doesn't really show up until the November 2015 timeframe, which is kernel 4.3.  Turns out Debian has backports up to Kernel 4.6! So, upgrading with the backports should include the firmware for my wifi card.  Backports warns you to only update a package if you NEED it, and encourages you not to upgrade everything.  However, so much depends on the Linux kernel, that upgrading that specific package creates a dependency for many other backport requied packages.  As a result, I ended up using backports for pretty much all of my software.  Which is Virtualbox, X, and Gnome3.

To perform this upgrade, first gloss over,  Then, after you have added the backports repo, run the following,

$ sudo apt-get -t jessie-backports install linux-image-amd64

Once this is complete, rebooting your PC should prompt you for the latest kernel at the Debian GRUB screen, in my case this is 4.6+74~bpo8+1.  If I understand this naming convention correctly, it means
"Linux Kernel 4.6, Build number 74, Backport for 8.0 Build 1".  Which is quite the mouthful.

As a bonus, the above kernel will also include support for the latest bluetooth adapter!  The end result of all this work was a bleeding edge opensource operating system using Virtualbox 5.0 (from backports) which has USB 3.0 support and 3D acceleration!

Thursday, July 28, 2016

Parsing tcpdump/pcap files in bash, without Wireshark.

Sometimes I work in an environment in which software access is restricted.  Things like Wireshark are DEFINITELY not allowed.

However, I'm still expected to troubleshoot network connectivity issues.  Luckily, tcpdump exists in the EPEL Repo, but leaves much to be desired when trying to read the PCAP files.  I decided to read up on the PCAP spec, and create my own parser in bash.  Crazy? Maybe.  But, it does work.

Another nice capability here is that using something like vim, you can easily tweak this script to dump application data for further debugging.  For example, switch your HTTPd to run in Non-SSL, run a capture, and dump the application data.  Of course, you may have to work out the correct byte offset for your packets. As this is a bare minimal implementation of a pcap parser and does not nearly support the full scope of capability provided by libpcap.

The real issue here is that I have not taken the time to parse some flags which consist of single bits.  There is also much work to be done in regard to writing the conditionals for various Ethernet, IP Routing and TCP frames.  But this is a good baseline to start.

To use this script just run 'tcpdump -w file.pcap -X' to run a capture.  Once done, pass the file as an argument to the pcap-analyzer bash script, and you will be able to read your frames on the CLI!  This should work out of the box with Centos/RHEL 6 Minimal, with the addition of the tcpdump RPM.

Here is a screenshot.

As you can see frame 2 starts to break because of the parsing.  Some flags actually mean, the header is going to be 28 bytes instead of 26.  As a result of the byte variations, the byte iterator ends up on some obscure part of the stream I don't handle, and things get start to break.  Git forks are welcome if you like this little utility and are interested in making it more robust.

One final important characteristic about Ethernet wire data is that it is ALWAYS Big Endian.  This is confusing because most CPUs are little endian.  To work around this issue, I use both 'hexdump' and 'od' because one enforces big endian while the other is CPU dependent.

Other resources Used:

Saturday, June 11, 2016

Apache Nifi Behind Apache (httpd) (SSL to Non-SSL)

ProxyPreserveHost On

    ProxyPass "http://localhost:8080/nifi" max=20 ttl=120 retry=300
    ProxyPassReverse  "http://localhost:8080/nifi"
    RequestHeader add X-ProxyScheme "https"
    RequestHeader add X-ProxyHost ""
    RequestHeader add X-ProxyPort "443"

    ProxyPass "http://localhost:8080/nifi-api" max=20 ttl=120 retry=300
    ProxyPassReverse "http://localhost:8080/nifi-api"
    RequestHeader add X-ProxyScheme "https"
    RequestHeader add X-ProxyHost ""
    RequestHeader add X-ProxyPort "443"

    ProxyPass "http://localhost:8080/nifi-docs" max=20 ttl=120 retry=300
    ProxyPassReverse "http://localhost:8080/nifi-docs"
    RequestHeader add X-ProxyScheme "https"
    RequestHeader add X-ProxyHost ""
    RequestHeader add X-ProxyPort "443"