Tuesday, September 20, 2016

Part 2 - Mounting and imaging Logical Volume Manager (LVM2)

The continuation of Part 1
Quickly summarizing and picking up where things previously left off (from this link for Part 1), the two 100GB Logical Volume Manager (LVM2) physical file segments have been mounted as a single 215GB logical volume and can now be accessed as a live file system. 

Summary and Review
To quickly review, at this point two LVM2 files called LinuxSvr-CLONE_1-flat.001 and LinuxSvr-CLONE_2-flat.001 have been successfully mapped to the path /dev/mapper/VolGroup[redacted] and then mounted at /media/sansforensics/. Viewing the console output confirms that the two 100GB files have been successfully "stitched together" as a single 215GB volume and mounted at:

Imaging the Mounted LVM2 Volume
With that confirmation in place, the forensic imaging process can now begin. Using the mapped file above as input along with dcfldd (just because I like the feedback/status as compared to standard 'dd'), the output will be written to a generically named LVM2_200GB.001 file using /dev/mapper/VolGroup[redacted]as the input file. We'll go ahead and fast forward to the end of the imaging process since it takes several hours to complete, as shown below:
As displayed above, the conclusion of dcfldd doesn't appear to show any errors, so it's time to cd into the directory and take a quick look at the results. At a high level view, the (almost) 215GB image file size looks good...

So let's mount it and see if it's valid image file. For this example, the pre-existing folder /mnt/aff/ on the SIFT workstation is randomly selected as the mount point.
The mount command appears to be successful so we'll go ahead and cd into it and list the directory contents to see if there's a valid file system to navigate:
Success! We now have a dd image file of the entire logical volume composed of the original split physical VMDK files. 

Unmounting and Detaching the Volumes
And so at this point the obvious question is how to get all of this stuff undone as well as cleanly unmounting/detaching:
(1) the 215GB dd output file that was just created and (2) both source 100GB dd files used to create it.

A simple sudo umount /mnt/aff command is the first option (although I forgot to 'sudo' during the first attempt in the example, hence the initial error message). So let's unmount it, then cd back into the directory and use ls to see if there's still something there:
OK, it's no longer mounted. So item-1 is now resolved (unmounting the newly created 215GB volume / image file). So it's time to move along since a quick glance at the GUI file manager shows the original source volume (created from the two merged physical VMDK files) that was used to generate the 215GB dd image file, is still mounted:
Maybe trying to unmount it by referencing the full media path (/home/sansforensics/[redacted long hex name]) will eject the volume?
No feedback after issuing the command but the GUI view hasn't changed - the 215GB volume is still there as shown previously in Figure-7. But since this is the world of LVM, delving into those tools seems to be the best bet. A quick scan with the LVM sudo pvs command (previously described in Part 1 at this link) shows that the two LVM2 physical VMDK files are still attached: 

They need to be detached by using the LVM command set. The previously used losetup command can be used with the -d option (detach) to accomplish that. We'll run sudo losetup -d on both loop devices to detach them cleanly:
OK then. That should have worked -- at least 'in theory'. But this is the cold cruel real world of technology, where 'the practice' often works differently than 'the theory'. Time to perform another sudo pvs command to scan for physical volumes again:
Gack. Apparently they aren't going to be detached without a fight. They're still mapped on loop0 and loop1. And that's key. The mappings need to be detached -- forcibly. From my own non-scientific observations, there seems to be a 50-50-ish chance of success getting the logical volume LVM2 files detached with the losetup -d command.

So the next step is to query the device mapper for some information about the mappings to see if that might be the likely cause of the problem. Using the command sudo dmsetup info for a quick peek into the state of things, the following information is returned:
OK, the logical volume group is still there, explicitly identified by its Logical Volume Manager UUID (the long redacted hex string) at the bottom of Figure-12 above. So it's time for some tough love, which means forcibly detaching the mapped device with the follow up command of sudo dmsetup remove VolGroup[redacted]

Note that the targeted device to detach, VolGroup[redacted], was pinpointed by using the item listed on the Name: row (resulting from the previous dmsetup info command above).
Did it FINALLY work? Yet again, a quick scan with the sudo pvs command can report what the real status is (all three of the previous commands are shown consecutively here):
Victory at last. Nothing reported this time by pvs. And as another sanity check, the file manager GUI is ALSO showing that the volume has finally been successfully ejected:
The end has arrived. The individual (smaller) physical disk images files have been individually imaged and then successfully combined into the larger LVM2 logical volume spanning across the physical files. Along with much troubleshooting along the way.

The LVM2 volume was successfully mounted, imaged as one large LVM2 into dd format, then verified to be a functional image file, along with the larger/combined dd file being unmounted without issues.

And finally, the source image files (used to generate the combined volume that was ultimately imaged) were detached as an active device while using a few troubleshooting techniques to find the correct way to detach them.

In conclusion I'll admit this seems to be a fairly niche DFIR topic for the moment but I suspect it's going to grow to be a fairly common issue. Hopefully this posting will help others figure things without having to reinvent the forensic wheel. 

I'd also like to thank those who responded to me about other ways to approach this. I'll be experimenting with them to see if there's another way / better way to perform this activity or provide some alternatives.

Monday, September 5, 2016

Mounting and imaging Logical Volume Manager (LVM2)

Welcome to my first attempt at a blog. Hopefully this will help someone somewhere out there someday in the DFIR community. Since this turned out to be a longer topic than I originally thought, or I'm more long-winded than I originally thought, I'll go ahead and say up front that this is being divided into Part 1 (this entry) and a follow-up Part 2 available at this link.

Defining the Problem

So, to get right to it, here's the challenge that gave rise to this blog entry. My day job at a Fortune 500 corporation is primarily focused on digital forensics, usually of the "deep dive" variety. In the course of investigations that involve disk-based analysis of a given server, dealing with physical hard drives/RAIDS/etc tends to be a very rare occurrence. 

The data almost always is in the form of a bunch of files from a virtualized environment (VMware for the content below). More specifically, those files are server clones -- including "pooled storage volumes" (for lack of a better description), meaning a single virtual storage volume written across two or more physical VMDK files. 

As a "bonus", most forensic tools (Encase, FTK, TSK, etc) might be able to correctly identify the file type of those VMDK files as LVM (technically, "LVM2" or "type 8e") but they can't stitch the multiple files back together for you into a single volume for purposes such as mounting an intact file system and/or creating a forensic image of the entire thing.

After working on several of these cases and poking around online for answers (while not finding much of anything about LVM from the forensic angle), hopefully the info here can shave off a few hours from the research/try/fail cycle for someone else out there in DFIR-land.

CAVEAT-1: This is still a work in progress, with experimentation and everything else underway while trying to figure out some of the LVM quirks. So if there's a better way to do it than I'm laying out here, other ideas are definitely welcome. 

CAVEAT-2: Most of the investigations I work are intrusions, malware, APT related, etc. So the examples here are not shown with the "typical" read only (mount -o ro) options, that would be more of a concern for cases involving human resources, law enforcement, etc. Aside from protecting the original evidence / source VM files (which are on a physical write blocker while imaging them) the main concerns here are to get the files assembled, mounted, imaged, and start the analysis asap.

What is LVM / LVM2?

In a nutshell, LVM is a software layer on top of (or "around"?) the virtual hard disks and partitions, which creates an abstraction for administrative usage to manage hard drive replacement, re-partitioning, backups, and re-sizing disk volumes on the fly. Very handy to address on-demand storage requirements as they change.

The main concern here, for DFIR purposes, is how to get the thing mounted and forensically imaged. But anyone wanting more in-depth information about LVM technology can get get a crash course while reviewing these links (at the very least, I recommend checking out the graphic at  https://en.wikipedia.org/wiki/File:Lvm.svg ):

So Let's Get To It
When it comes to doing the analysis and experimentation here, the only things we'll be using are: 
  1. A Windows host system (Server 2012R2 in this case but anything Win7 and later should work just as well), 
  2. A Windows-based forensic tool to examine image files (FTK Imager in this case), 
  3. Virtual machine guest software, i.e., VMware Workstation, to run 
  4. The SANS SIFT forensic platform, available at this link.

And so without further adieu, let's get started. As mentioned above, it's far more common for a pile of cloned VM files to arrive for analysis than for actual server physical hard drives, disk arrays, other RAID, etc, showing up. 

In this case, we'll be dealing with Linux Logical Volume Manager server files (with the ext4 file system buried somewhere in there), consisting of 2 storage volumes. One volume is completely self-contained within a single physical VMDK file (numbered 1 below) while the other larger volume is split across 2 physical VMDK files (numbered 2 and 3 below) -- which will cause mounting and imaging problems. The larger/split volume is the star of this show.

The goal is to merge the physical files from numbers 2 and 3 into one logical volume in order to build out the full file and folder structure to be forensically imaged for later analysis.
In my very-unscientific testing (scrambling around looking for answers while semi-stuck), it seems like recognition of LVM2 files is a hit-or-miss proposition for standard forensic imaging tools. I'm not sure why. But in this instance FTK Imager detected LVM2, so it will be used to take a quick look at the cloned VM files. 

After loading the CLONE_1-flat and CLONE_2-flat VMDK files (File >> Add Evidence Item...) with the "Image Files" option under the "Select Source" dialog and then expanding the drop-downs under the "Evidence Tree", it shows the following:
We have confirmation that these are Logical Volume Manager files with the [LVM2] label next to the red arrows, which I believe (but still need to research to be certain) is derived from the metadata in the first 1MB of each file. 

We also see the administratively assigned Volume Group names (redacted, obviously). Additional key data is found to the right, under the "File List" column, identifying which segment (physical file number) it is along with the total number of expected files in the entire logical group. 

So we'll go ahead and image the two VMDK clones as "Raw DD format" (unsplit/unsegmented) files under two separate jobs on FTK Imager. Flash forward and upon completion we see the results -- two individual raw DD image files. Both image files are named after their source VMDK counterparts:
Note that the output image files were written to the Linux-individual-vols directory (unnecessarily redacted by me but... oh well), apart from the original Source_VM input file directory. 

Now it's time to jump over to VMware, fire up the SIFT Workstation, open a shell and see what we have. Logging in as the default user (sansforensics) we'll use ls -l to see the directory contents and make sure the Windows directory containing the CLONE image files is shared with the guest VM SIFT workstation:
So we have access and the two physical LVM2 disk files appear to be imaged into dd format... but are they valid? How do they get mounted and then "merged" or "stitched together" or whatever you want to call it? Let's see if a simple attempt to mount them at /media/sansforensics will work:

OK, so that didn't work. How about using the mmls command from The Sleuth Kit (TSK) to see if we can get some information about the partitions and file systems?

That didn't work either. Maybe running sudo fdisk -l will find something:
Alright. Now we're getting somewhere... just not a valid partition table, though. Both of the CLONE files produce the same error message, BTW. But the information about the disk appears to be valid. Or at least "realistic".

At this point it became time to do some research and here's where we'll fast forward ahead. After reading various online forums written by frustrated-Linux-admin-after-frustrated-Linux-admin, it turns out there's a missing Linux package that needs to be installed... lvm2. And we can confirm if it needs to be installed by issuing the  lvm command:


 As instructed above, we'll download and install it with sudo apt-get install lvm2...

Enter "Y", hit enter, and wait a few minutes. Voila - the package was downloaded and installed successfully. Now what? As it turns out, the world of Logical Volume Management has a laundry list of commands to learn... luckily we only need to know a few of them. 

A quick crash course is available at: 

Or a more in-depth review of the commands at:

For basic forensic mounting and imaging purposes, the primary commands we're concerned with are:

  • pvs - reports information about LVM physical volumes
  • pvscan - scans all supported LVM block devices
  • lvdisplay - logical volume in-depth information
They'll be used to obtain data needed to feed two other standard commands:
  • losetup - mount/unmount the files as loop devices 
  • dmsetup - manages the device-mapper driver and can force unmounting if losetup doesn't successfully do it (which is often the case)

Returning to the example, first we'll make sure we're in the same directory with the CLONE dd image files (at /mnt/hgfs/Examples/VM_dd_Files/Linux-individual-vols) and then mount the files as loop devices with the command losetup. Note that everything from here onward will require sudo / superuser privileges, so you may want to just sudo -i before doing any of this.

To make this work, it was necessary to put each individual CLONE dd image file on successive loop devices at /dev/loop0 and /dev/loop1, as shown below. If you want it read only, use losetup -r as the command. Just beware that using the -r option sometimes makes LVM uncooperative (not sure why, but "it just happens").

There's no feedback as to whether the operation succeeded or failed, so maybe no news is good news. Now it's time to break out the LVM commands and find out where things stand. Using the sudo pvs command, we can see some information about the imaged VMware clones, including the volume group (redacted) that they belong to:
That looks "promising" but it's not enough information to figure out (1) were these physical LVM files successfully combined into one virtual volume? (2) If so, where is it? And (3) can we navigate through it like a "live" mounted file system? It's time to get more in-depth information by using the sudo lvdisplay command:
Now we have some useful information from the logical volume manager, including LV Size being reported at 200 GB, which would appear to confirm that the two physical dd image files are now properly stitched together and mounted as a single virtual volume. 

But the LV Path is also included (showing the device path), which will be useful to know later, in case the device wasn't automatically mounted during the losetup activities. Other potentially useful data includes the VG Name created by the admin and the UUID (a large string of random hex, similar to ef674641-3ab9-8e5c-1dbd-eac810715327a). Let's try sudo fdisk -l again to see if anything has changed:
Ah. Now there's some very specific information about the 200GB logical volume mapped at /dev/mapper/VolGroup[redacted]. So let's change directories to /dev/mapper and confirm it's there and then try to mount it at /media/sansforensics/ (the local user). Note one more "weird" LVM thing: Sometimes the previous losetup commands [Figure-9 above] seem to automatically mount the device in the local user /media directory. Other times it has to be done manually (I haven't found rhyme-or-reason for it). In short, the mount step here may be unnecessary. But we'll run the commands (since that's how it behaved while making the screen shots):
No error messages so it looks like the fully assembled logical volume (ext4 file system from a Linux server) should now be properly mounted. But the proof is in the file system navigation, so let's cd into /media/sansforensics and ls -hl the contents to see what happens...
BuenĂ­simo! Some actual file and folder structure (most names redacted) from a Linux host. How about a cross-check with the Nautilus file manager? Looks good:
As shown above under the Devices heading, the 200GB (technically, 215GB) volume icon appears. If the losetup command initially maps it AND mounts it automatically, the volume "should" show up here... in which case we could have just right-clicked to get the path of the mount point (instead of going through all of the activity above from Figure-9 to Figure-13).

So it looks like we have a properly mounted volume, the Logical Volume Manager software "barrier" is gone, and valid file/folder structure now exists for navigation. If there's a targeted examination to perform, this might be as far as we need to go since the LVM2 volume is fully accessible.

Otherwise if a full forensic image is required, the entirely assembled logical volume can now be imaged. After that, we'll verify that it's a valid image file and then undo all of the "LVM stuff" up to this point, like detaching the loop devices. But since this is a good stopping point, reviewing the imaging and unmounting procedures for the LVM volume will be continued separately in Part 2 at this link.