Part 2 - Mounting and imaging Logical Volume Manager (LVM2)

The continuation of Part 1
Quickly summarizing and picking up where things previously left off (from this link for Part 1), the two 100GB Logical Volume Manager (LVM2) physical file segments have been mounted as a single 215GB logical volume and can now be accessed as a live file system. 

Summary and Review
To quickly review, at this point two LVM2 files called LinuxSvr-CLONE_1-flat.001 and LinuxSvr-CLONE_2-flat.001 have been successfully mapped to the path /dev/mapper/VolGroup[redacted] and then mounted at /media/sansforensics/. Viewing the console output confirms that the two 100GB files have been successfully "stitched together" as a single 215GB volume and mounted at:

Figure-1

Imaging the Mounted LVM2 Volume
With that confirmation in place, the forensic imaging process can now begin. Using the mapped file above as input along with dcfldd (just because I like the feedback/status as compared to standard 'dd'), the output will be written to a generically named LVM2_200GB.001 file using /dev/mapper/VolGroup[redacted]as the input file. We'll go ahead and fast forward to the end of the imaging process since it takes several hours to complete, as shown below:

Figure-2

As displayed above, the conclusion of dcfldd doesn't appear to show any errors, so it's time to cd into the directory and take a quick look at the results. At a high level view, the (almost) 215GB image file size looks good...

Figure-3

So let's mount it and see if it's valid image file. For this example, the pre-existing folder /mnt/aff/ on the SIFT workstation is randomly selected as the mount point.

Figure-4

The mount command appears to be successful so we'll go ahead and cd into it and list the directory contents to see if there's a valid file system to navigate:

Figure-5

Success! We now have a dd image file of the entire logical volume composed of the original split physical VMDK files. 

Unmounting and Detaching the Volumes
And so at this point the obvious question is how to get all of this stuff undone as well as cleanly unmounting/detaching:
(1) the 215GB dd output file that was just created and (2) both source 100GB dd files used to create it.

A simple sudo umount /mnt/aff command is the first option (although I forgot to 'sudo' during the first attempt in the example, hence the initial error message). So let's unmount it, then cd back into the directory and use ls to see if there's still something there:

Figure-6

OK, it's no longer mounted. So item-1 is now resolved (unmounting the newly created 215GB volume / image file). So it's time to move along since a quick glance at the GUI file manager shows the original source volume (created from the two merged physical VMDK files) that was used to generate the 215GB dd image file, is still mounted:

Figure-7

Maybe trying to unmount it by referencing the full media path (/home/sansforensics/[redacted long hex name]) will eject the volume?

Figure-8

No feedback after issuing the command but the GUI view hasn't changed - the 215GB volume is still there as shown previously in Figure-7. But since this is the world of LVM, delving into those tools seems to be the best bet. A quick scan with the LVM sudo pvs command (previously described in Part 1 at this link) shows that the two LVM2 physical VMDK files are still attached: 

Figure-9

They need to be detached by using the LVM command set. The previously used losetup command can be used with the -d option (detach) to accomplish that. We'll run sudo losetup -d on both loop devices to detach them cleanly:

Figure-10

OK then. That should have worked -- at least 'in theory'. But this is the cold cruel real world of technology, where 'the practice' often works differently than 'the theory'. Time to perform another sudo pvs command to scan for physical volumes again:

Figure-11

Gack. Apparently they aren't going to be detached without a fight. They're still mapped on loop0 and loop1. And that's key. The mappings need to be detached -- forcibly. From my own non-scientific observations, there seems to be a 50-50-ish chance of success getting the logical volume LVM2 files detached with the losetup -d command.

So the next step is to query the device mapper for some information about the mappings to see if that might be the likely cause of the problem. Using the command sudo dmsetup info for a quick peek into the state of things, the following information is returned:

Figure-12

OK, the logical volume group is still there, explicitly identified by its Logical Volume Manager UUID (the long redacted hex string) at the bottom of Figure-12 above. So it's time for some tough love, which means forcibly detaching the mapped device with the follow up command of sudo dmsetup remove VolGroup[redacted]. 

Note that the targeted device to detach, VolGroup[redacted], was pinpointed by using the item listed on the Name: row (resulting from the previous dmsetup info command above).

Figure-13

Did it FINALLY work? Yet again, a quick scan with the sudo pvs command can report what the real status is (all three of the previous commands are shown consecutively here):

Figure-14

Victory at last. Nothing reported this time by pvs. And as another sanity check, the file manager GUI is ALSO showing that the volume has finally been successfully ejected:

Figure-15

Conclusion
The end has arrived. The individual (smaller) physical disk images files have been individually imaged and then successfully combined into the larger LVM2 logical volume spanning across the physical files. Along with much troubleshooting along the way.

The LVM2 volume was successfully mounted, imaged as one large LVM2 into dd format, then verified to be a functional image file, along with the larger/combined dd file being unmounted without issues.

And finally, the source image files (used to generate the combined volume that was ultimately imaged) were detached as an active device while using a few troubleshooting techniques to find the correct way to detach them.

In conclusion I'll admit this seems to be a fairly niche DFIR topic for the moment but I suspect it's going to grow to be a fairly common issue. Hopefully this posting will help others figure things without having to reinvent the forensic wheel. 

I'd also like to thank those who responded to me about other ways to approach this. I'll be experimenting with them to see if there's another way / better way to perform this activity or provide some alternatives.