Addendum to mongodb on rocksdb-Haswell

So I finally got around to doing the install on the physical box today, and on the initial make static_lib, I got this:

>/opt/src/rocksdb>make static_lib
 GEN util/
 CC db/c.o
 /tmp/ccActpEz.s: Assembler messages:
 /tmp/ccActpEz.s:14025: Error: no such instruction: `shlx %rsi,%r9,%rax'
 /tmp/ccActpEz.s:40750: Error: no such instruction: `shlx %rcx,%r11,%rsi'
 /tmp/ccActpEz.s:40764: Error: no such instruction: `shlx %rax,%r11,%rdx'
 /tmp/ccActpEz.s:41219: Error: no such instruction: `shlx %r8,%rsi,%r9'
 /tmp/ccActpEz.s:41231: Error: no such instruction: `shlx %rdx,%rsi,%rax'
 /tmp/ccActpEz.s:41497: Error: no such instruction: `shlx %rbx,%rsi,%rdi'
 /tmp/ccActpEz.s:41511: Error: no such instruction: `shlx %r14,%rsi,%rdx'

The machine I was installing on has Haswell CPUs, which was what was causing this error. So we had to download/compile/install binutils-2.2.5, and then add that to the path. The link is here.


Installing MongoDB with RocksDB as a storage engine

I honestly don’t really understand most of the words I just wrote in that headline. Nevertheless, when our dba or our software team asks us to install something, install it we do.

This one was fairly tricky, and I’m hoping this post will help some people.

First, some words about our environment. We run Scientific Linux 6.x in our datacenter primarily, and we maintain a shared /usr/local directory (mounted via nfs) across most of the servers. This makes installing software somewhat tricky in two areas: Most new/Big Data/open source/whatever software these days is expecting the latest and greatest code, which tends to show up in Ubuntu. Unfortunately, Ubuntu is not all that well suited to some other stuff we do, and tends to not be as easy to manage in the datacenter. The other trickiness is that we want to avoid the default install location of /usr/local for a lot of software, unless it’s being installed for our HPC cluster.

Aaaaanyway. So that basically meant I had to install a number of packages for this to work, as well as compiling with a non-standard location for gcc, g++, etc. And I also had to install the software itself in a non-standard location. I also had to call a newer version of Python than is standard on SL (and you DON’T want to install over the standard version!)

I started out from this blog post. I’m sure those instructions work for a standard install on Ubuntu 14.xx or later, but they need a little expanding for RHEL/CentOS/Scientific Linux.

Set up the path and install some prereqs:
export PATH=/usr/local/gcc/bin:/usr/local/python-2.7.8/bin:$PATH
yum install libmpc --enablerepo=epel
yum install snappy-devel zlib-devel bzip2-devel

Install the rocksdb libraries:

git clone
cd rocksdb
git checkout mongorocks

Change the INSTALL_PATH line in Makefile change to preferred install location (/opt/rocksdb):

INSTALL_PATH ?= /opt/rocksdb

And make/make install

make static_lib
sudo make install

Download scons, which you will need to build mongo. Note that the version of scons available in the standard repos is not new enough, so you get to compile this from source, too:

curl -O

Install and add it to your path:

python install --prefix=/opt/scons
export PATH=/opt/scons/bin:$PATH

Download the latest version of Mongo that works with rocksdb:

git clone
cd mongo
git checkout v3.0-mongorocks

Here’s the tricky bit. Scons seems to be rather poorly documented, or at least it’s so flexible that documenting it isn’t helpful. These are the commands I figured out, bearing in mind that our version of GCC (4.9.1) is at /usr/local/gcc and I installed the rocks libraries at /opt/rocksdb:

scons all --cxx=/usr/local/gcc/bin/g++ --cc=/usr/local/gcc/bin/gcc --cpppath=/opt/rocksdb/include --libpath=/opt/rocksdb/lib --rocksdb=1 --prefix=/opt/mongo

scons --cxx=/usr/local/gcc/bin/g++ --cc=/usr/local/gcc/bin/gcc --cpppath=/opt/rocksdb/include --libpath=/opt/rocksdb/lib --rocksdb=1 --prefix=/opt/mongo install

So I’m not certain I need all that jazz for the install line, but it seemed to work, so better safe than sorry. This took about 4 hours to compile in the little VM I was running it in, so I didn’t feel like rebuilding over and over again once I had it actually functional.

Now, in my case, I found that because I couldn’t figure out how to make scons use static libraries and actually work, my dba will need to use this command before running mongo:

export LD_LIBRARY_PATH=/usr/local/gcc/lib64

If anyone can figure out how to make the static libraries with a non-standard location, please include it in the comments. I tried –static, but that caused the compile to throw errors about not being able to find lpthread (??), which it finds just fine without the static libraries.