Clearing Cache & Fixing VMs – VMWare Workstation 8.0.2

Hello!  I have not posted since the summer and will try to update more often.  I try to only put tips and tricks on the Internet when I cannot find an article or blog already posted on the topic.

Anyone who knows me knows I run many operating systems and programs on a Virtual Machine (VM) for testing.  I have two main bash scripts I run in Linux on a regular basis that I find useful every day.  To understand these bash scripts, please note I keep my virtual machines in the directory /home/VMs.  The first script clears out the cache kept by a running VM while the second script scans for and repairs problems on virtual disks.

#!/bin/bash

pwd=$(pwd)
cd /home/VMs
chmod -R 755 *

for i in `ls -alh /home/vms | grep ^d | awk '{print $8}' | grep -v '\.'`
    do rm -rfv /home/vms/$i/caches
       rm -rfv /home/vms/$i/*.lck
       rm -rfv /home/vms/$i/*.log
    done

cd $pwd

This script goes to that directory (/home/VMs) and lists any directories in that directory looking for any not containing a period/dot in their name.  The script recursively deletes any cache directories, lock (.lck) files, and any log files (.log).  I do not run this script if I am troubleshooting a Virtual Machine, but do run this script when I wish to tidy up the files – especially before backing them up.

#!/bin/bash

echo   ## "sudo echo" if you want to gain sudo privileges for commands below
cd /home/VMs

for DIR in `ls`
    do
    cd /home/VMs/$DIR
    CHARS=$(echo $DIR | wc -m)
    let SPACES=$((35-$CHARS))
    echo -en "$DIR"
    for (( i=1; i<=$SPACES; i++))
        do
            echo -en " "
        done
    TEST=$(find ./ -name "*.vmdk" | wc -l)
    if [[ "$TEST" != "0" ]]
        then
            for FILES in `ls`
                do
                vmware-vdiskmanager -R $FILES
            done
        else
        echo -en "There are no VMDK files in this directory.\n"    
    fi
done
echo -e "\n\n\n"

If you wanted to substitute a disc/disk wiping utility there, you could.  I chose normal removal because I do not need a secure wipe for VMWare’s temporary files.

This last script, also written in bash, will recurse through the same directories in /home/VMs and attempt to repair the virtual disks.  Whether you create your virtual disks via a single file or with multiple, this script scans for the VMDKs until it finds one it can scan and/or repair.

I prefer to collect the sudo permission immediately instead of pausing while /home/VMS is scanned because asking for a password immediately after invocation associates the permissions with the program invoked.  In order words, it does not leave someone to wonder why they are being asked for credentials seconds or minutes into the program.  However, please note sudo access is not necessary to run VMWare’s vdiskmanager.  I leave it for sudo to do because some of my VMs are not writable by the username I use on the given machine that hosts the VM.

Automating tshark on Debian-based systems

On some servers, I need to be able to have tcpdump or wireshark running.  Since most servers do not run X-windows, I prefer to reach them and perform all work via the command-line.  Every five minutes, on one such server, I have a cron job running that first checks all running processes, then starts tshark if it is not already running.

On Debian-based systems, perform the following if you do not already have a copy of tshark running.

$ sudo apt-get update
$ sudo apt-get install tshark

 .

Here is the BASH script I have named “TS_start” and is placed in a scripts folder “/root/scripts” which only root can see or change [hopefully, haha].

!/bin/bash
OriginalDirectory=$(pwd)
WorkingDirectory='/home/user/captures'
DateTimeEpoch=$(date +%s)
DateTime=$(date +%Y%m%d\_%H%M)
Hour=$(date +%H)
YesterdayEpoch=$(echo "$DateTimeEpoch - 86400" | bc)
YesterdaysDate=$(date --date "Jan 1, 1970 00:00:00 +0000 + $YesterdayEpoch seconds" \+\%Y\%m\%d)
TSRunning=$(ps -ef | grep -vi grep | grep -i tshark | wc -l)

if [ "$TSRunning" -ge "1" ]
   then
   echo &> /dev/null
   else
   /usr/bin/tshark -a files:100 -b filesize:30000 -w "$WorkingDirectory" -f "$(cat /root/scripts/TS_filter)" -t ad -i eth0 &> /dev/null &
  fi

##
## Change All Permissions after 0100 local time
 if [ "$Hour" -ge "1" ]
    then
      chmod 777 $WorkingDirectory/*
      chown user:user $WorkingDirectory/*
    else
    cd $pwd
 fi

 .

First, I declare all variables.  Some of these variables are not currently being used, but are there in case I want to zip or delete yesterday’s capture files.  I left them in there for your viewing, but one can decide to archive, delete, or otherwise process capture files.  I sometimes rsync them to a central location for analysis if they’re a day old.

Please note the “TSRunning” variable.  I list all processes running, pipe them into grep where I grep for everything except grep (ignoring case with the switch -i ).  I pipe the output from those commands into a non-case-sensitive grep for tshark, and pipe that output into word count minus L, which gives me the total count of lines containing that string.  This variable decides whether or not tshark will be called during the next section of conditional programming.

The if-then-else conditional construct says if the string “TSRunning” is greater than or equal to the string “1″, then basically do nothing.  I put an echo into “/dev/null” because sometimes I have an inbox full of empty mail messages from cron when mail is running on the same system.  The “&>” means standard output and error output gets redirected.

On that same if-than-else conditional check, if “TSRunning” is equal to zero or less, the else section will be executed.  Here, tshark is called with many options:

-a            condition which will cause tshark to stop
-b            ring the directory (capture buffer) with this many files
-w            where to write the files
-f            what filters (if any) to use
-t            timestamp format (ad is absolute with date)
-i            interface to capture on

 .

“tshark -h” will give one a list of options available.  I would like to focus on the filter for just one moment.  In the same directory as this BASH script, I keep a file called “TS_filter” which has a one-liner explaining to tshark what I do or do not want to capture.  Doing so saves space on disk as well as time when sorting or filtering during analysis.

$ cat /root/scripts/TS_filter
(not host 1.2.3.4 && not port 22) && (not host 5.6.7.8 && not port 80)

 .

Explaining all options with wireshark and tshark filters would take many blog pages.  Here is a good start on how to write simple filters.  I usually prevent the bulk of management traffic (such as my getting into the capture server or moving files to/from the same server) from making it into the files I must later sift through.

Finally, I redirect all output (standard output and standard error) to “/dev/null” and run the whole process in the background with the final ampersand (&).

 .

The very last if-then-else set of conditions changes the permissions and owner of the files in the capture file directory at 01:00 local time.  This was in there along with the other variables I have in there relating to yesterday’s date.  The reason is, if I have to capture in a data center for a few days, I will want to archive or rsync older files.

I use tshark daily, and find it to be a wonderful networking and security tool.

 .

 

Loop device limit in Ubuntu 10.10 affects Truecrypt

If you are a fan of Truecrypt like many computer users, you have no doubt wondered if there are any limitations. In my five years using it, I had not reached any real limits other than limits of FAT, which do not permit file sizes above 4 GB. However, recently, I was mounting a whole bunch of truecrypt volumes at once, and ran into a problem. Here is a copy of the BASH script I was using:

#!/bin/bash

## run this as root or be prepared to enter the admin (sudo)  password
##
PASS=$1

if [ "$PASS" == "$NULL" ]
  then
    echo -e "\n\n\nYou must supply the password in ‘single quotes’ \n\n\n"
  else

truecrypt -t -k="" -protect-hidden=no -password=$PASS /home/truecrypt/vol1 /home/user/tc1 -fs-options="umask=000,uid=1000,gid=1001"
truecrypt -t -k="" -protect-hidden=no -password=$PASS /home/truecrypt/vol2 /home/user/tc2 -fs-options="umask=000,uid=1000,gid=1001"
truecrypt -t -k="" -protect-hidden=no -password=$PASS /home/truecrypt/vol3 /home/user/tc3 -fs-options="umask=000,uid=1000,gid=1001"
truecrypt -t -k="" -protect-hidden=no -password=$PASS /home/truecrypt/vol4 /home/user/tc4 -fs-options="umask=000,uid=1000,gid=1001"
truecrypt -t -k="" -protect-hidden=no -password=$PASS /home/truecrypt/vol5 /home/user/tc5 -fs-options="umask=000,uid=1000,gid=1001"
truecrypt -t -k="" -protect-hidden=no -password=$PASS /home/truecrypt/vol6 /home/user/tc6 -fs-options="umask=000,uid=1000,gid=1001"
truecrypt -t -k="" -protect-hidden=no -password=$PASS /home/truecrypt/vol7 /home/user/tc7 -fs-options="umask=000,uid=1000,gid=1001"
truecrypt -t -k="" -protect-hidden=no -password=$PASS /home/truecrypt/vol8 /home/user/tc8 -fs-options="umask=000,uid=1000,gid=1001"
truecrypt -t -k="" -protect-hidden=no -password=$PASS /home/truecrypt/vol9 /home/user/tc9 -fs-options="umask=000,uid=1000,gid=1001"

  fi

When I invoked this, either directly or via a cron job, I kept getting an error message stating truecrypt was unable to create the loop device. Unfortunately, that last volume I was attempting to mount had been just created, so I spent time and effort creating new volumes and playing with the truecrypt settings.

After searching and finding some answers, I found out Ubuntu 10.10 comes with loop devices numbered zero through seven. I then found snippets from various sites explaining portions of what has to be done.

 .

Here is what worked for me…. First, edit or create the file called “/etc/modprobe.conf” and place the following line into it:

####
options loop max_loop=64

 .

Next, create new loop devices eight through some number up to 255. I have selected 127.  You do not have to start on 8 like I did; you can create whatever ranges (0 – 255) you want.

for i in `seq 8 127`; do
mknod -m0660 /dev/loop$i b 7 $i
chown root .disk /dev/loop$i
done

 .

Since I was affecting modprobe, which has to do with kernel modules, I rebooted.

This happy ending inspired me to pass it on.  Here is a list of sites I read which helped me to come up with this solution:

 .

http://www.brandonhutchinson.com/Creating_additional_loop_devices.html
http://homepage.smc.edu/morgan_david/linwin/encryptedfilesystem.htm
http://planet.admon.org/create-additional-loop-devices-in-linux

 .