Repairing quotas after you delete them all

As I mentioned in my earlier post, I managed to delete the quotas on one of my Isilon clusters by accident. Still haven’t figured out exactly how it happened, but it happened.

By a happy coincidence, we do dumps of our quota lists on a daily basis (I recommend you do too). The command you could use for this is:

isilon-1# isi quota quotas list --format csv

From there, I cut it down to an output that looks like:

type,path,hard-threshold

and tossed that into a file called quotadata.txt.

Then I used this script:

#!/bin/bash
OIFS=$IFS
IFS=','
INPUT=./quotadata.txt
[ ! -f $INPUT ] &while read TYPE QPATH SIZE
do
type=$TYPE
qpath=$QPATH
size=$SIZE
if [ -z $size ]; then
isi quota quotas create $qpath $type
else
isi quota quotas create $qpath $type --hard-threshold $size --container=yes
fi
done <$INPUT
IFS=$OIFS
isi quota quotas list

It threw a bunch of errors about stdin not being a tty, but those are safely ignored (and you could probably fix them through some kind of flag on the isi quota quotas commands.

In any case, that put all my quotas back. At that point, it was a (relatively) simple matter of running the quotascan job. 

As usual with my Isilon posts, this is in regards to OneFS 7.1.0. 

Moving data between quota protected directories on Isilon

Updated version of the script here: https://unscrupulousmodifier.wordpress.com/2015/10/08/moving-data-between-quota-protected-directories-on-isilon-take-ii/

In the current versions of Isilon OneFS, it is impossible to move files and directories between two directories with quotas on them (regardless of the directory quota type; even if it’s advisory, it won’t allow it). This is really annoying, and although I’ve put in a feature request for it, who knows if it will ever be fixed. So I wrote this script that will make a note of the quota location and threshold (if it’s a hard threshold), remove the quota, move the items, and reapply the quotas.

#!/bin/bash

#Tests whether there is a valid path
testexist () {
        if [ ! -r $1 ]; then
                echo "$1 is an invalid path. Please try again."
                exit
        fi
}

#Iterates through path backwards to find most closely related quota
findquota () {
        RIGHTPATH=0
        i=`echo $1 | awk -F'/' '{print NF}'` #define quantity of fields
        while [ $RIGHTPATH -eq 0 ]; do
                QUOTA=`echo $1 | cut -d"/" -f "1-$i"`
                if [ -n "`isi quota list | grep $QUOTA`" ]; then
                        RIGHTPATH=1
                fi
                i=$(($i-1))
        done
        echo $QUOTA
}

testquota () {
        if [ "$1" = "-" ]; then
                echo "No hard directory quota on this directory."
                exit
        fi
}

if [[ $# -ne 2 ]]; then
        #Gets paths from user
        echo "Enter source:"
        read SOURCE
        echo "Enter target:"
        read TARGET
else
        SOURCE=$1
        TARGET=$2
fi

testexist $SOURCE
testexist $TARGET

#Verifies paths with user
echo "Moving $SOURCE to $TARGET. Is this correct? (y/n)"
read ANSWER
if [ $ANSWER != 'y' ] ; then
        exit
fi

#Defines quotas
SOURCEQUOTA=$(findquota $SOURCE)
TARGETQUOTA=$(findquota $TARGET)

#Gets size of hard threshold from quota
SOURCETHRESH=$(isi quota view $SOURCEQUOTA directory | awk -F" : " '$1~/Hard Threshold/ {print $2}')
TARGETTHRESH=$(isi quota view $TARGETQUOTA directory | awk -F" : " '$1~/Hard Threshold/ {print $2}')
testquota $SOURCETHRESH
testquota $TARGETTHRESH

echo $SOURCEQUOTA $SOURCETHRESH
echo $TARGETQUOTA $TARGETTHRESH

isi quota quotas delete --type=directory --path=$SOURCEQUOTA -f
isi quota quotas delete --type=directory --path=$TARGETQUOTA -f

isi quota quotas view $SOURCEQUOTA directory
isi quota quotas view $TARGETQUOTA directory

mv $SOURCE $TARGET

isi quota quotas create $SOURCEQUOTA directory --hard-threshold=$SOURCETHRESH --container=yes
isi quota quotas create $TARGETQUOTA directory --hard-threshold=$TARGETTHRESH --container=yes

isi quota quotas view $SOURCEQUOTA directory
isi quota quotas view $TARGETQUOTA directory

Here’s how I use it:

bash /ifs/data/scripts/qmv /ifs/source/path /ifs/target/path

First I’ve got some functions in there:
testexist (): test if it’s a sane path
findquota (): find the quota info for the given path
testquota (): check if it’s a hard quota. If it’s not, the script fails, because that’s all we use around here. Feel free to fix it up and post something better in the comments.

Then we get to the bit where if it’s not given two arguments for source and target, it asks for them. It then tests if the source and target both exist. Please note that this script expects a fully qualified path including the bit you want to move for the source, and the place you want to move it for the target (ie, not source=/ifs/data/somethingdir/something target=/ifs/data/otherdir/something).

Of course, there’s a bit of error checking you’ll pretty much start ignoring and answering y to all the time…

Then we find the quotas for the directories. What the findquota () function does is it iterates back through the path until it finds an actual quota on it. I think this will break if you have nested quotas, but again, feel free to fix it up and let me know. It’ll then throw out which quota applies. Once it’s found both the quota paths, it saves the hard threshold in a variable. Now we’ve got variables for the source quota directory, the target quota directory, and both of their hard thresholds.

From there, it’s an easy move to delete the quotas, move the actual data, and then put the quotas back.

Don’t forget to use the –container=yes flag on those isi quota quotas create commands if you don’t want to show your end users the entire size of the filesystem.

**** Please note, and I found this after I made this post… if you comment out the echo $QUOTA line in the findquota () function, it kinda breaks the whole script. And then deletes all of your quotas without asking you. So, uh, don’t comment that out. That echo is what populates the $SOURCEQUOTA and $TARGETQUOTA variables. ****

This script works as of OneFS 7.1. I make no guarantees they won’t switch around the isi commands again in their quest to make commands as long and convoluted as possible.

Say, what’s this blog thing here?

Oh yeah, I have a blog. Well, we’ll see what happens, but I need a place to stash a few things that keep getting lost in my bash histories across various servers.

For starters, here’s quick find line that will get everything in a directory that’s

a) older than a month and
b) not from the 1st of the month

find /directory/subdirectory/* -maxdepth 1 -mtime +30 -ls | awk '$9 != /1/ { print $11 }' | parallel rm -rf {}

Now, why would you want to do this? I did because one of my dbas had been doing full backups nightly to our primary storage for years without trimming his backups. So we had terabytes of old (largely unimportant) stuff on relatively expensive and scarce storage. I talked to him, and he asked me to keep the data per the parameters above. Our company retention policy for backups is 30 days, hence the very short window.

So let’s break this down. The find command is a little tricky in and of itself. First you want to do a find across multiple directories, but not go too deep. That’s the /directory/subdirectory/* bit, with a maxdepth of 1, with a modification time of more than 30 days. Then list that. That looks like this:

[root@server ~]# find /mnt/mysql/* -maxdepth 1 -mtime +30 -ls
....
234199799 35514 -rw-r--r-- 1 root root 30984020 May 27 2013 /mnt/mysql/mysql2/clustrix-job_manager-201305270010.tar.gz
234181155 2690 -rw-r--r-- 1 root root 2187282 May 25 2013 /mnt/mysql/mysql2/wip-201305250100.tar.gz
354760343 98 -rw-r--r-- 1 root root 27568 Jun 15 2013 /mnt/mysql/mysql2/clustrix-probes-201306150011.tar.gz
257679475 26 -rw-r--r-- 1 root root 1374 Jun 2 2013 /mnt/mysql/mysql2/clustrix-gbrowse_login-201306020001.tar.gz
395629190 45898 -rw-r--r-- 1 root root 31066606 Jun 22 2013 /mnt/mysql/mysql2/clustrix-job_manager-201306220010.tar.gz
357046308 8874 -rw-r--r-- 1 root root 5904716 Jun 12 2013 /mnt/mysql/mysql2/clustrix-ror-201306120009.tar.gz
259828901 26 -rw-r--r-- 1 root root 6290 Jun 2 2013 /mnt/mysql/mysql2/clustrix-smith_lemur-201306020011.tar.gz
354511053 242 -rw-r--r-- 1 root root 75127 Jun 16 2013 /mnt/mysql/mysql2/clustrix-parts-201306160004.tar.gz
356484133 730 -rw-r--r-- 1 root root 449046 Jun 18 2013 /mnt/mysql/mysql2/clustrix-mad-201306180013.tar.gz
....
498553872 134570 -rw-r--r-- 1 root root 117480405 Mar 30 2012 /mnt/mysql/wiki-db/wiki-201203300855.tar.gz
3097297952 198146 -rw-r--r-- 1 root root 173105695 Nov 2 2013 /mnt/mysql/wiki-db/wiki-201311020855.tar.gz
244561447 167874 -rw-r--r-- 1 root root 114123533 Feb 25 2012 /mnt/mysql/wiki-db/wiki-201202250855.tar.gz
356159441 190722 -rw-r--r-- 1 root root 166592128 Jul 15 2013 /mnt/mysql/wiki-db/wiki-201307150855.tar.gz
4132816213 188730 -rw-r--r-- 1 root root 128309505 Oct 3 2012 /mnt/mysql/wiki-db/wiki-201210030855.tar.gz
2347163665 253386 -rw-r--r-- 1 root root 172236350 Oct 18 2013 /mnt/mysql/wiki-db/wiki-201310180855.tar.gz
636357288 204698 -rw-r--r-- 1 root root 166829424 Jul 22 2013 /mnt/mysql/wiki-db/wiki-201307220855.tar.gz
3504126126 155002 -rw-r--r-- 1 root root 135339049 Apr 4 2013 /mnt/mysql/wiki-db/wiki-201304042055.tar.gz

At that point, we pipe it into the awk statement, which does a search against the 9th field and makes sure it’s not 1 (and only 1, ie not 17 or 21), and then prints the 11th field. So you get something like this:

[root@server ~]# find /mnt/mysql/* -maxdepth 1 -mtime +30 -ls | awk '$9 != /1/ { print $11 }'
....
/mnt/mysql/mysql2/clustrix-genie-201306140014.tar.gz
/mnt/mysql/mysql2/clustrix-job_manager-201306150010.tar.gz
/mnt/mysql/mysql2/clustrix-genie-201306220014.tar.gz
/mnt/mysql/mysql2/clustrix-scheduleit-201305241800.tar.gz
/mnt/mysql/mysql2/clustrix-looger_lemur-201306180002.tar.gz
/mnt/mysql/mysql2/clustrix-scheduleit-201306021201.tar.gz
/mnt/mysql/mysql2/clustrix-geci_lemur-201306180002.tar.gz
/mnt/mysql/mysql2/clustrix-ahrens_lemur-201306100015.tar.gz
/mnt/mysql/mysql2/clustrix-campus_security-201306040600.tar.gz
/mnt/mysql/mysql2/clustrix-galaxy-201306200013.tar.gz
/mnt/mysql/mysql2/clustrix-qstatworld-201306070015.tar.gz
/mnt/mysql/mysql2/clustrix-zhang_lemur-201305270012.tar.gz
/mnt/mysql/mysql2/clustrix-gbrowse_login-201305310001.tar.gz
....
/mnt/mysql/wiki-db/wiki-201211162055.tar.gz
/mnt/mysql/wiki-db/wiki-201203300855.tar.gz
/mnt/mysql/wiki-db/wiki-201311020855.tar.gz
/mnt/mysql/wiki-db/wiki-201202250855.tar.gz
/mnt/mysql/wiki-db/wiki-201307150855.tar.gz
/mnt/mysql/wiki-db/wiki-201210030855.tar.gz
/mnt/mysql/wiki-db/wiki-201310180855.tar.gz
/mnt/mysql/wiki-db/wiki-201307220855.tar.gz
/mnt/mysql/wiki-db/wiki-201304042055.tar.gz

Now, why do that whole /directory/subdirectory/* thing? It’s for parallel. If you’re not familiar with parallel, go ahead and install it and read the man page. Yeah, I’m that kind of unix admin. Well, sometimes. Basically, it takes an input and multithreads it. It’s quite useful (if dangerous) with rsync, rm, and other commands of that ilk.

So what this one does is it parallelizes over 500 threads (the -P 500) the command (rm -rf) against the input from the pipeline ({}). I tend to use a very high threadcount, because what I often run into is that it’ll torch the easy stuff quickly and then chug along on the 2-10 highly nested directories. That way, I don’t have to wait for all the easy (ie, not very nested, relatively large file) stuff to go through before it starts working on the highly nested tiny file stuff. I’m usually running this on a 16 core machine, for reference. YMMV.

Please note that I do NOT attach that rm -rf {} bit before I’ve tested the first two parts of the command!

nowarrantyorguaranteeisexpressedorimpliedalldatalossresultingfromyouruseofthesecommandsispurelyyourresponsibilityauthorwillnotcomeandrecoveryourdataifyoublowitupbyaccident