ZFS forensics scrollback script

From Siwiki

Jump to: navigation, search

Contents

[edit] What is this?

This is a python script that enables to get back already deleted files or partitions. This is highly experimental, but I managed to get back a moths work when all the partitions were deleted and created new ones on top by accident. This will only work if the pool is not very full and there was not very much activity after the unwanted action.

[edit] How it works

  • The script finds all uberblocks, reads their metadata and orders them by time
  • Enables you to destroy all the uberblocks that were created after the event that you want to scroll back to.

(Uberblocks are created every time data gets edited in ZFS. Every uberblock creates a new "path" to the data, and the new path includes data changes, by deleting the last uberblocks, ZFS will use the last "path" witch leads to the old data.) This might not be 100% true, but this is how i understand it. Shortly ZFS amnesia.

[edit] Thoughts

  • It might be that reverting to a TXG where the pool was in an exported state, then there is a better chance of importing it after.
  • Don't know how to do this with multiple disc arrays. But i think you just have to delete the uberblocks from all the discs. Could someone test this please?
  • A zpool recovery feature that rolls back several transactions integrated into SXCE release, build 128. You can read about this feature here:

[zpool recovery] and [ZFS Storage Pool Recovery]. This can be used to make ZFS recover even if you damage the uberblocks to a bad "state".

[edit] Example walkthrough 

#lets create a 500mb file.
mkfile 500m test_filesystem 
#mount it as a lofi device that will be called /dev/lofi/1
lofiadm -a test_filesystem 
#now we make a pool.
zpool create testpool /dev/lofi/1
#lets just verify all that.
zpool list
NAME       SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
...
testpool   492M    73K   492M     0%  ONLINE  -
...
#i will make 2 different filesystems
#later i will delete some files and write new and delete the other completely.
zfs create testpool/a
zfs create testpool/b
zfs set mountpoint=/export/home/mardicas/TEST/a testpool/a
zfs set mountpoint=/export/home/mardicas/TEST/b testpool/b

#i will use some homevideos for testing
du -sh sourcefiles/*
 140M	sourcefiles/2009.05.07
 232M	sourcefiles/2009.05.08
#lets copy them.
cp sourcefiles/2009.05.07/* a/
cp sourcefiles/2009.05.08/* b/
#make sure they are written.
sync
#i will export and import the pools to make it easier to find the right place later.
zpool export testpool
zpool import -d /dev/lofi/ testpool
#now lets do some damage!(you can even copy new files on top)
rm -rf a/*
zfs destroy testpool/b
#well lets copy something else there now...
cp /etc/* a/
sync
#OOPS i should not have deleted those videos, not to mention the entire testpool/b.
#(now i follow the procedure described in the How-to)
zpool status testpool
zpool history -il testpool
dd if=test_filesystem of=test_filesystem.bu
ls -l test_filesystem
-rw------- 1 mardicas staff 524288000 2010-01-10 20:40 test_filesystem
#524288000/512=1024000
#make zfs deal gracefully with errors! 
#This is probably ignored later because we delete the uberblocks and ZFS will get an Amnesia.
zpool set failmode=continue testpool
#run the script
./zfs_revert-0.1.py -bs=512 -tb=1024000 test_filesystem
#when we printed out the history earlier there is a line:
#2010-01-10.20:31:31 zpool export testpool [user root on opensolaris:global]
#this is where we would like to be!
#the program returned the uberblocks ranging from 4-146, and we will delete all of them after 
#TXID	DATE			Timestamp	Uberblock locations(4 of them)
#103	10 Jan 2010 20:31:31	1263148291	[462, 974, 1023438, 1023950]
rm -rf /etc/zfs/zpool.cache 
#Now i reseted the machine! (brutally) Reboot might not work here...
lofiadm -a test_filesystem 
#lets import the filesystem and tell it to handle errors gracefully(again)
zpool import -d /dev/lofi/ testpool && zpool set failmode=continue testpool
zpool list
NAME       SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
...
testpool   492M   373M   119M    75%  ONLINE  -
...
zfs list
NAME                         USED  AVAIL  REFER  MOUNTPOINT
testpool/a                   140M  87.5M   140M  /export/home/mardicas/TEST/a
testpool/b                   232M  87.5M   232M  /export/home/mardicas/TEST/b
mkdir restored
mkdir restored/a
mkdir restored/b
cp /export/home/mardicas/TEST/a/* restored/a/
cp /export/home/mardicas/TEST/b/* restored/b/
/export/home/mardicas/TEST/b/m2u00794.mpg: I/O error
#seems like we did lose one file.
#Probably because we did copy file to A and they were written somewhere where the m2u00794.mpg file was before
#lets make sure everything is like it was before!
diff -r restored/a/ sourcefiles/2009.05.07/
#looks good :-)
diff -r restored/b/ sourcefiles/2009.05.08/
Only in sourcefiles/2009.05.08/: m2u00794.mpg
#well i was asking for it :P

[edit] How-To

First check the pool status: 

zpool status zones

Get the disc name e.g:c2t60060E800457AB00000057AB00000146d0

Now we look up the history of the pool so we can find the timeline and some uberblocks(their TXG-s) where to scroll back: 

zpool history -il zones

Save this output for later use.

You will defently want to backup the disk before you continue from this point: 

ssh root at host "dd if=/dev/dsk/c..." | dd of=Desktop/zones.dd


The script accepts 2 arguments: 

-bs  is block size, by default 512 (never tested)
-tb is number of blocks:[this is mandatory, maybe someone could automate this]
  • To find the block size in solaris you can use 
prtvtoc /dev/dsk/c2t60060E800457AB00000057AB00000146d0 | grep sectors

From there look at the "sectors" row.

  • If you have a file/loop device just sizeinbytes/blocksize=total blocks

Now run the script for example: 

./zfs_revert.py -bs=512 -tb=41944319 /dev/dsk/c2t60060E800457AB00000057AB00000146d0

This will use dd, od and grep(GNU) to find the required information. This script should work on linux and solaris.


Something like this should appear, if not then something is wrong: 

TXG, time-stamp, unixtime, addresses(there are 4 copy's of uberblocks)
411579  05 Oct 2009 14:39:51    1254742791      [630, 1142, 41926774, 41927286]
411580  05 Oct 2009 14:40:21    1254742821      [632, 1144, 41926776, 41927288]
411586  05 Oct 2009 14:43:21    1254743001      [644, 1156, 41926788, 41927300]
411590  05 Oct 2009 14:45:21    1254743121      [652, 1164, 41926796, 41927308]


  • Take a wild guess witch block might be the one, it took me about 10 tries to get it right, i don't know what are the "good" blocks. The history we printed out earlyer might help!
  • Enter the last TXG you want to KEEP.
  • The script writes zeroes to all of the uberblocs after the TXG inserted.
  • Now clear the ZFS cache and reboot, cross fingers. 
rm -rf /etc/zfs/zpool.cache && reboot
  • First try to import the pool if it is not imported yet.
  • Make sure to set the failmode right after!!! It might freze otherwize.(no use issueing it before, because ZFS has amnesia!) 
zpool import -f zones && zpool set failmode=continue zones

Now see if it can import it or fail miserably. There is a good chance that you will hit Corrupt data and unable to import, but as i said earlier it took me about 10 tries to get it right. I did not have to restore the whole disk every time, i just took baby steps and every time deleted some more blocks until found something stable (it will still crash after few minutes, but this is enough time to get back conf files or some code)

[edit] The script

Can be downloaded from http://www.loovsys.eu/redmine/projects/zfsrevert Please send any patched or improoved versions to mardicas@gmail.com and i will upload them!

Authors: Martin Vool - mardicas@gmail.com

[edit] References

I got so far by the help of these materials!

Retrieved from "http://www.solarisinternals.com/wiki/index.php/ZFS_forensics_scrollback_script"
Solaris Internals
Personal tools
The Books
The Ads