Breaking down the monster, part the second

Let’s do this thing.

This little snippet tells the bash builtin time to output in realtime seconds to one decimal place (like the comment says). Very handy modifier, and of course there are others.

#defines output of time in realtime seconds to one decimal place

Here's a function to create directories. Seems kind of silly, but if you don't check if the directory is created first, mkdir throws an error, and who wants that?
#creates directory to write to
createdir () {
	if [ ! -d $1 ]; then
		mkdir -p $1

Now we get to the meat of the thing. There’s 5 functions to do the writes, reads, and shuffled reads. But wait, that’s only 3 operations, you say! Well, object is different. So there are two separate functions for writing and reading from the object store. Obviously these would need to be heavily modified for any object based storage that’s not Scality, since we’re using their sproxyd method at this time. I haven’t looked into what Swift or S3 looks like, but it’s probably fairly similar, syntax wise.

So, writes:

#write test
writefiles () {
	#echo WRITE
	for i in `seq 1 $filecount`; do 
		#echo -n .
		dd if=/dev/zero of=$path/$i-$suffix bs=$blocksize count=$blockcount 2> /dev/null

Pretty straightforward, here. All the variables, aside from i, are globally defined outside the function. The commented out echo -n . just creates some pretty …….. stuff, if you’re into that sort of thing.

Same with reads:

#read test
readfiles () {
	#echo READ
	for i in `seq 1 $filecount`; do 
		#echo -n .
		dd if=$path/$i-$suffix of=/dev/null bs=$blocksize 2> /dev/null
		#dd if=$path/$i-$suffix of=/dev/null bs=$blocksize

Now the shuffled writes. I played with a few ways of doing this (like writing out the list to a file, then reading that file), but writing to a file and reading from it is expensive, which gives bum results on the timing. So, an array it is, created by shuffling (shuf -i 1-$filecount) a list of numbers. There’s still some debugging code commented out in here.

#shuffled read test
shufreadfiles () {
	filearray=(`shuf -i 1-$filecount`)
	for i in ${filearray[*]}; do 
		#echo -n .
		#echo $path/$i-$suffix
		dd if=$path/$i-$suffix of=/dev/null bs=$blocksize 2> /dev/null
		#dd if=$path/$i-$suffix of=/dev/null bs=$blocksize

Now we get to the object stuff. I wanted to eliminate the need for reading from a file to do the writes, so I jiggered up curl to take stdin from dd. This is done with the -T- flag. It doesn’t work in some circumstances (which I will detail in a later post when I talk about Isilon RAN object access), but it does here with plain ‘ol unencrypted http calls.

scalitywrite () {
    for i in `seq 1 $filecount`; do
        dd if=/dev/zero bs=$blocksize count=$blockcount 2> /dev/null | curl -s -X PUT ;
         http://localhost:81/proxy/bparc$fspath/$i-$suffix -T- > /dev/null

So there’s a lot of mess in here where I was trying to get rid of output from curl and dd. This is fairly difficult. dd outputs to stderr as a normal program does, but curl need to be silenced (-s) as well as sending its stdout to /dev/null. The other note is that Scality sproxyd can be written to in two ways; with a hash that contains some metadata about how to protect the files and where to put them, or as a path. The path is hashed by the system, and the object is written that way. Note that you CAN’T do a file list, and it’s not stored in the system by the path. The full path can be retrieved, but not searched for.

The read is much simpler

scalityread () {
    for i in `seq 1 $filecount`; do
        curl -s -X GET http://localhost:81/proxy/bparc/$fspath/$i-$suffix > /dev/null

OK, in the next post, I’ll get to the heart of the script, where it calls all of these here functions.


About kcarlile
Twitter: @overclockdlemon

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: