If you application uses valloc(), be careful to use with libmalloc.so on Solaris 10.
Solaris 10's libmalloc (at least for several versions I tested) does
not have implementation of valloc(), then valloc() is called from libc
but free() is called from libmalloc, this will cause core dump.
You can use DTrace to check it. For example:
~/tmp$ dtrace -qn 'pid$target::valloc:entry {ustack();}' -c ./a.out
To check if you libmalloc has valloc:
$ elfdump /usr/lib/libmalloc.so|grep valloc
Saturday, April 20, 2013
Four Years
Four years ago today, my colleague called me while I was washing dishes after dinner. He said: "our company is sold! check your email now..."
Yes, I cannot forget it for ever. The bitterness is still deeply in my heart.
Yes, I cannot forget it for ever. The bitterness is still deeply in my heart.
Wednesday, January 30, 2013
Build Mesos on Solaris
Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. The purpose to build it is that I want to have experiments on Spark , a popular framework for cluster computing (e.g. big data analysis).
Mesos integrated a lots of third party software, the build process on Solaris is not very smooth. At first, I tried Solaris Studio (SunCC), but had some troubles. To save time, I decided to use gcc.
The build environment is: Solaris 11 11/11 sparc; gcc is 4.5.2 from Solaris 11 IPS repository; mesos 0.9.0.
wget http://ftp.gnu.org/gnu/automake/automake-1.11.6.tar.gz
CC=cc CXX=CC ./configure --program-suffix=-1.11
gmake;gmake install (in /usr/local/bin)
- modify configure.ac
in "solaris" section:
CC=cc
CXX=CC
CFLAGS="$CFLAGS"
CXXFLAGS="$CXXFLAGS"
LIBS="$LIBS -lsocket -lnsl -lproject -lproc -lresolv -lsendfile -lxnet"
then in "JAVA_LDFLAGS" section, add something:
...
elif test "$OS_NAME" = "solaris"; then
JAVA_LDFLAGS=""
for arch in sparc; do
dir="$JAVA_HOME/jre/lib/$arch/server"
if test -e "$dir"; then
# Note that these are libtool specific flags.
JAVA_LDFLAGS="-L$dir -R$dir -ljvm"
break;
fi
done
fi
...
- execute autoconf.
- run configure:
./configure CC=gcc CXX=g++ CFLAGS="-m32 -pthreads" CXXFLAGS="-m32 -pthreads" JAVA_HOME=/usr/java --prefix=/opt/mesos
compile process.cpp and future.hpp failed because syntax error in assembly.
using CC -S, then cc -c process.s, we found it use "pause" instruction.
Solution: use smt_pause()
2) process.cpp in libprocess
ssize_t length = sendfile(s, fd, offset, size);
=>
ssize_t length = sendfile(s, fd, &offset, size);
3)pid.cpp in libprocess
for gethostbyname2_r, it is not availabe on Solaris, has to modify the codes to use gethostname_r
4)port_posix.h, atomic_pointer.h
for macros and memory barriers.
5)getpwuid_r in zookeeper.c
getpwuid_r(uid, &pw, buf, sizeof(buf), &pwp))
smiliar as gethostbyname2
6)recordio.c in zookeeper
redefined htonll:
recordio.h:int64_t htonll(int64_t v);
On solaris, the prototype is: uint64_t htonll(uint64_t hostlonglong);
linux doesn't have htonll
Solution: comment out htonll
7)mt_adapter.c in zookeeper
atomic ops: fetch_and_add
8)cli.c in zookeeper
ctime_r(&tctime, tctimes)
Solaris requires: char *ctime_r(const time_t *clock, char *buf, int buflen);
9)mesos
./common/utils.hpp:359:17: error: ‘NAME_MAX’ was not declared
Solution: define it #define NAME_MAX 255
10)slave/solaris_project_isolation_module.hpp
Solaris project implementation is not completed, so got compiling error.
Workaround: skip it by modifying macro definition
11) protobuf is compiled in 64-bit by default.
Solution: reconfigure in thirdparty/protobuf according to config.log and add -m64 in CFLAGS and CXXFLAGS: (it's better to add -m32 in top level of mesos configure).
12) /usr/lib/python2.6/pycc complained:
cc: No valid input files specified, no output generated
because the src file is c++: native/proxy_executor.cpp
Workaround (by looking at /usr/lib/python2.6/pycc, pycc and pyCC is same):
in ~/Downloads/mesos-0.9.0/src/python
$ PYCC_CC=g++ PYCC_CXX=g++ LDFLAGS="-lnsl -lresolv -lsendfile -lsocket" python setup.py build
(It seems on Solaris 11.1 there's no such issue).
13) gmake test
need -lxnet
14) got runtime error:
if (errno != EINPROGRESS) {...
in fact, errno is 0 here.
Reason: not with "-pthreads" in build libprocess.
Solution: also add "-pthreads" in configure.ac so that it take effects for each thread having a private copy of errno.
- I also built mesos 0.9.0 on Solaris 11.1 x64, the code modifications are less because the platform is x64.
- The modified files are now put into github.
Mesos integrated a lots of third party software, the build process on Solaris is not very smooth. At first, I tried Solaris Studio (SunCC), but had some troubles. To save time, I decided to use gcc.
The build environment is: Solaris 11 11/11 sparc; gcc is 4.5.2 from Solaris 11 IPS repository; mesos 0.9.0.
Preparation Steps:
- need to build automake-1.11.6 (automake-1.11.2 from Solaris 11 is not enough for mesos 0.9.0)wget http://ftp.gnu.org/gnu/automake/automake-1.11.6.tar.gz
CC=cc CXX=CC ./configure --program-suffix=-1.11
gmake;gmake install (in /usr/local/bin)
- modify configure.ac
in "solaris" section:
CC=cc
CXX=CC
CFLAGS="$CFLAGS"
CXXFLAGS="$CXXFLAGS"
LIBS="$LIBS -lsocket -lnsl -lproject -lproc -lresolv -lsendfile -lxnet"
then in "JAVA_LDFLAGS" section, add something:
...
elif test "$OS_NAME" = "solaris"; then
JAVA_LDFLAGS=""
for arch in sparc; do
dir="$JAVA_HOME/jre/lib/$arch/server"
if test -e "$dir"; then
# Note that these are libtool specific flags.
JAVA_LDFLAGS="-L$dir -R$dir -ljvm"
break;
fi
done
fi
...
- execute autoconf.
- run configure:
./configure CC=gcc CXX=g++ CFLAGS="-m32 -pthreads" CXXFLAGS="-m32 -pthreads" JAVA_HOME=/usr/java --prefix=/opt/mesos
Porting issues and solutions:
1) process.cpp in libprocesscompile process.cpp and future.hpp failed because syntax error in assembly.
using CC -S, then cc -c process.s, we found it use "pause" instruction.
Solution: use smt_pause()
2) process.cpp in libprocess
ssize_t length = sendfile(s, fd, offset, size);
=>
ssize_t length = sendfile(s, fd, &offset, size);
3)pid.cpp in libprocess
for gethostbyname2_r, it is not availabe on Solaris, has to modify the codes to use gethostname_r
4)port_posix.h, atomic_pointer.h
for macros and memory barriers.
5)getpwuid_r in zookeeper.c
getpwuid_r(uid, &pw, buf, sizeof(buf), &pwp))
smiliar as gethostbyname2
6)recordio.c in zookeeper
redefined htonll:
recordio.h:int64_t htonll(int64_t v);
On solaris, the prototype is: uint64_t htonll(uint64_t hostlonglong);
linux doesn't have htonll
Solution: comment out htonll
7)mt_adapter.c in zookeeper
atomic ops: fetch_and_add
8)cli.c in zookeeper
ctime_r(&tctime, tctimes)
Solaris requires: char *ctime_r(const time_t *clock, char *buf, int buflen);
9)mesos
./common/utils.hpp:359:17: error: ‘NAME_MAX’ was not declared
Solution: define it #define NAME_MAX 255
10)slave/solaris_project_isolation_module.hpp
Solaris project implementation is not completed, so got compiling error.
Workaround: skip it by modifying macro definition
11) protobuf is compiled in 64-bit by default.
Solution: reconfigure in thirdparty/protobuf according to config.log and add -m64 in CFLAGS and CXXFLAGS: (it's better to add -m32 in top level of mesos configure).
12) /usr/lib/python2.6/pycc complained:
cc: No valid input files specified, no output generated
because the src file is c++: native/proxy_executor.cpp
Workaround (by looking at /usr/lib/python2.6/pycc, pycc and pyCC is same):
in ~/Downloads/mesos-0.9.0/src/python
$ PYCC_CC=g++ PYCC_CXX=g++ LDFLAGS="-lnsl -lresolv -lsendfile -lsocket" python setup.py build
(It seems on Solaris 11.1 there's no such issue).
13) gmake test
need -lxnet
14) got runtime error:
if (errno != EINPROGRESS) {...
in fact, errno is 0 here.
Reason: not with "-pthreads" in build libprocess.
Solution: also add "-pthreads" in configure.ac so that it take effects for each thread having a private copy of errno.
Other notes:
- Building mesos 0.9.0 fails with gcc 4.7 on Linux. Most current linux versions come with this gcc version.- I also built mesos 0.9.0 on Solaris 11.1 x64, the code modifications are less because the platform is x64.
- The modified files are now put into github.
Monday, October 29, 2012
Micro Benchmark MongoDB 2.2 performance on Solaris
I 100% agree on the statement by
MongoDB:
“MongoDB does not publish any official benchmarks. We recommend running application performance tests on your application's work-load to find bottleneck and for performance tuning.”
However I don't have real world workload, so I just tried some micro benchmarks to observe the behaviors of MongoDB and OS. Although the result numbers mean nothing, but I would share some findings here.
1) Using JS Benchmark Harness
MongoDB provides JS
Benchmarking Harness as a QA baseline perf measurement tool, not
designed to be a "benchmark". This is a good start for
having a first look at MongoDB performance. The harness is very easy
to setup. However, there are a few things to be considerded.
The sample code on that web page is
really really a micro benchmark. I tested it against MongoDB 2.2 for
Solaris x64 and got suboptimal result comparing against Linux
version. After analyzing the workload characteristics, it is more
like a multi-threaded malloc and small TCP/IP packet ping-pong
testing.
By passing LD_PRELOAD_64=libmtmalloc.so
to starting mongod, I got the performance on Solaris parallel to
Linux. If the test client and sever are on separate systems, I may
also need disable nagle algorithm: $ sudo ndd -set /dev/tcp
tcp_naglim_def 1
The harness also has an interesting
feature: RAND_INT [ min , max , multiplier ], it looks like
we are able to only touch a fix fraction of data during the testing.
Two things need be considered here:
- I looked at the current harness implementation, RAND_INT is translated to rand(), this is not really random for big (millions of records) data sets. The fix is using lrand48() instead.
- MongoDB uses mmap to cache data, like many other databases, it is still a page-level cache rather than row-level cache. So if your record size is small, RAND_INT [ 1, 10000000, 10 ] doesn't make you only touch 1/10 data, rather it makes you touch all the data.
2) Using YCSB.
YCSB
is an extensive load testing tool. But its tests codes for mongodb is
a little outdated. I need modify a little bit to add more
writeConcern type.
YCSB's testing driver has some
limitations:
- You can set read/write proportion, but they are in same thread context, which means writes can block reads. So I prefer to put them in separate simultaneous jobs in testing.
The “recordcount” also implicitly set the max Id of data to be tested. when testing mongodb, the small number means only a few data files are mapped into the memory during transaction phase. So setting “recordcount”in transaction phase is not the right way to test against only small portion of the data.
3) Solaris related stuff.
The Solaris version of mongodb 2.2 has a large binary size compared to Linux, although it nearly does not affect the performance, but I don't like it. A quick check on its build info got “GCC 4.4 on snv_89 January 2008”, too old. This should be fixed by adding GCC option "-fno-function-sections" and "-fno-data-sections".
When starting mongod for Solaris, a warning message shows: “your operating system version does not support the method that MongoDB uses to detect impending page faults. This may result in slower performance for certain use cases”. After browsing the source codes, I found processinfo support is not there. So I added the Solaris support, currently the functions that count are ProcessInfo::blockInMemory() and ProcessInfo::blockCheckSupported().
The mongodb source code says “madvise not supported on solaris yet”, this is funny. Solaris certainly supports madvise. But madvise() is only useful when you understand your workloads. So I don't think this piece of code of calling madvise() is important.
ZFS and UFS.
==========
Since mongodb uses mmap(), it leaves a lot of things to the OS file system. UFS is a traditional file system, it uses traditional page cache (cachelist ) for caching file data. ZFS has quite a lot features beyond a file system, ZFS has its own ARC cache. The physical memory usage can be inspected using mdb ::memstat command:
Since mongodb uses mmap(), it leaves a lot of things to the OS file system. UFS is a traditional file system, it uses traditional page cache (cachelist ) for caching file data. ZFS has quite a lot features beyond a file system, ZFS has its own ARC cache. The physical memory usage can be inspected using mdb ::memstat command:
# echo "::memstat"|mdb -k Page Summary Pages MB %Tot ------------ ---------------- ---------------- ---- Kernel 293720 1147 7% ZFS File Data 85347 333 2% Anon 138902 542 3% Exec and libs 1638 6 0% Page cache 27118 105 1% Free (cachelist) 3036514 11861 73% Free (freelist) 576129 2250 14% Total 4159368 16247 Physical 4159367 16247
In my test, ZFS is very good
performance in data loading. However, because ZFS has its own cache,
if the data is not mmaped, it will be searched firstly from cachelist
then ZFS cache, if it's not there, then data is read from the disk into ARC cache, then data is mapped into mongod process address space as page cache. Using ZFS need more memory and when all data cannot fit in
physical memory, there would be fights for memory between cachelist
and ZFS cache. Tweaking ZFS parameters (manually set ARC cache size, adjust "primarycache" property, etc) did not help in my tests. For read intensive workload, using SSD as 2nd-level ARC cache will help. In addition, depending the workload and data characteristics, adjusting ZFS recordsize or disabling ZFS prefetching may worth a try.
An interesting madvise option for UFS
is MADV_WILLNEED, when this option is set, the system will try to
pull all data into the memory (quick warm) while during this period the mongod could not response to clients. So if your whole dataset
can fit into the physical memory and you can stand the short period of unresponsive to outside during startup, you can consider using it because it
warms fast and get peak performance quickly.
Wednesday, August 22, 2012
Build CouchDB 1.2 on Solaris 11 (SPARC)
I want to play a little bit with Document-oriented DB. The first try was MongoDB, however, building MongoDB on SPARC is a horrible experience: MongoDB is written for x86/x64; in addition, it uses scons as its build tool and scons doesn't work with IPS-based Solaris Studio 12.3 (I need hack scons to make it work), in fact in MongoDB's source code, it never consider other compilers but VC and GCC. Therefore, I decided to try CouchDB. Allthough CouchDB has a lot of dependencies, but they are friendly for porting.
1) ICU (International Components for Unicode)
This is easy on Solaris 11:
Note: the ICU library 4.6 is built with Sun CC, so the C++ code that will link to it also should be built with Sun CC.
2) Build Erlang
Solaris 11 11/11 IPS repository provides Erlang 5.6.5, this is too old. CouchDB 1.1.x or later requires Erlang >= 5.7.3. Although Erlang website says they do daily build for Solaris Sparc (may be because Ericsson was Sun's shop), but why don't they publish the binary package for Solaris? Fortunately, the build procedure is not that difficult:
- Grab the source code from Erlang website. I chose 14B04 which version is 5.8.5.
-
Ignore configure warnings unless configure fails.
- fix erts/emulator/drivers/common/inet_drv.c:
3) Build SpiderMonkey 1.8.5
- Download source code from here.
- It's interesting that Mozilla's project need autoconf-2.13, so I had to build this old version.
Get the source from GNU webiste, then:
- gmake; sudo gmake install (in /usr/local)
4) Build CouchDB 1.2
Before proceeding, I need to fix something:
- apache-couchdb-1.2.0/src/couchdb/priv/Makefile.in:
replace "-Wall -Werror" with "-v" because I use Sun CC.
-
(because configure.ac always choose libmozjs185-1.0 instead of libmozjs185 due to existence of libmozjs185-1.0.a)
- apache-couchdb-1.2.0/src/snappy/google-snappy/snappy-stubs-internal.h:
solaris does not have byteswap.h, I replaced it with byteorder.h and defined a few macros.
- Run test:
$ export PATH=/usr/perl5/5.12/bin:$PATH
$ gmake check
Wait for a while, all tests should be successful.
- sudo gmake install
- create a script couchdb.sh in /opt/couchdb1.2/bin (since I don't want the couchdb output to be at arbitrary place).
#!/usr/bin/bash
BIN_DIR=$(dirname $0)
cd $BIN_DIR
./couchdb -o ../var/log/couchdb.stdout -e ../var/log/couchdb.stderr ${1+"$@"}
- modify /opt/couchdb1.2/etc/couchdb/default.ini, change bind_address to let couchdb accessible from anywhere
bind_address = 0.0.0.0
- start couchdb: /opt/couchdb1.2/bin/couchdb.sh -b
Now, enjoy CouchDB!
UPDATE (Sept 2012):
- I also tested Erlang 15B01 and latest 15B02, erts/emulator/drivers/common/inet_drv.c has already been fixed. But for building couchdb, we need to add something to /opt/local/lib/erlang/usr/include/erl_driver.h:
#if defined(__sun)
#include <unistd.h> /* for ssize_t */
#endif
1) ICU (International Components for Unicode)
This is easy on Solaris 11:
# pkg install developer/icu (this will also cause library/icu to be installed)Note: the ICU library 4.6 is built with Sun CC, so the C++ code that will link to it also should be built with Sun CC.
2) Build Erlang
Solaris 11 11/11 IPS repository provides Erlang 5.6.5, this is too old. CouchDB 1.1.x or later requires Erlang >= 5.7.3. Although Erlang website says they do daily build for Solaris Sparc (may be because Ericsson was Sun's shop), but why don't they publish the binary package for Solaris? Fortunately, the build procedure is not that difficult:
- Grab the source code from Erlang website. I chose 14B04 which version is 5.8.5.
-
./configure --prefix=/opt/local (I use default gcc-45 from Solaris 11 IPS repository, since it is said erlang uses some GCC features (label vars?).Ignore configure warnings unless configure fails.
- fix erts/emulator/drivers/common/inet_drv.c:
ifreq.ifr_hwaddr.sa_data would cause compiling error because ifreq does not have that member. Since Solaris 11 has defined both SIOCGIFHWADDR and SIOCGENADDR, we only need SIOCGENADDR.
//#ifdef SIOCGIFHWADDR
// if (ioctl(desc->s, SIOCGIFHWADDR, (char *)&ifreq) < 0)
// break;
// buf_check(sptr, s_end, 1+2+IFHWADDRLEN);
// *sptr++ = INET_IFOPT_HWADDR;
// put_int16(IFHWADDRLEN, sptr); sptr += 2;
// /* raw memcpy (fix include autoconf later) */
// sys_memcpy(sptr, (char*)(&ifreq.ifr_hwaddr.sa_data),
IFHWADDRLEN);
// sptr += IFHWADDRLEN;
//#elif defined(SIOCGENADDR)
#ifdef SIOCGENADDR
3) Build SpiderMonkey 1.8.5
- Download source code from here.
- It's interesting that Mozilla's project need autoconf-2.13, so I had to build this old version.
Get the source from GNU webiste, then:
./configure --prefix=/usr/local --program-suffix=-2.13
cd js-1.8.5/js/src
/usr/local/bin/autoconf-2.13
CC=cc CXX=CC ./configure- gmake; sudo gmake install (in /usr/local)
4) Build CouchDB 1.2
Before proceeding, I need to fix something:
- apache-couchdb-1.2.0/src/couchdb/priv/Makefile.in:
replace "-Wall -Werror" with "-v" because I use Sun CC.
-
cd /usr/local/lib; ln -s libmozjs185.so.1.0 libmozjs185-1.0.so(because configure.ac always choose libmozjs185-1.0 instead of libmozjs185 due to existence of libmozjs185-1.0.a)
- apache-couchdb-1.2.0/src/snappy/google-snappy/snappy-stubs-internal.h:
solaris does not have byteswap.h, I replaced it with byteorder.h and defined a few macros.
...
#else
//#include
#include
#define bswap_16(x) BSWAP_16(x)
#define bswap_32(x) BSWAP_32(x)
#define bswap_64(x) BSWAP_64(x)
....
- configure
./configure CC=cc CXX=CC LDFLAGS="-R /usr/local/lib" --with-erlang=/opt/local/lib/erlang/usr/include --with-js-lib=/usr/local/lib/ --with-js-include=/usr/local/include/js/ --prefix=/opt/couchdb1.2
- Run test:
$ export PATH=/usr/perl5/5.12/bin:$PATH
$ gmake check
Wait for a while, all tests should be successful.
- sudo gmake install
- create a script couchdb.sh in /opt/couchdb1.2/bin (since I don't want the couchdb output to be at arbitrary place).
#!/usr/bin/bash
BIN_DIR=$(dirname $0)
cd $BIN_DIR
./couchdb -o ../var/log/couchdb.stdout -e ../var/log/couchdb.stderr ${1+"$@"}
- modify /opt/couchdb1.2/etc/couchdb/default.ini, change bind_address to let couchdb accessible from anywhere
bind_address = 0.0.0.0
- start couchdb: /opt/couchdb1.2/bin/couchdb.sh -b
Now, enjoy CouchDB!
UPDATE (Sept 2012):
- I also tested Erlang 15B01 and latest 15B02, erts/emulator/drivers/common/inet_drv.c has already been fixed. But for building couchdb, we need to add something to /opt/local/lib/erlang/usr/include/erl_driver.h:
#if defined(__sun)
#include <unistd.h> /* for ssize_t */
#endif
- Running couchdb testsuite in browser (Erlang 14B04 and 15B01) caused erlang process crash. I posted the core dump analysis and temporary workaround in erlang mailing list. Using 15B02 doesn't have this problem, but still should be careful when the Erlang application on Solaris reloads crypto.so, see discussions here. One solution might be adding "-z nodelete" LDFLAGS in lib/crypto/c_src/sparc-sun-solaris2.11/Makefile.
Subscribe to:
Posts (Atom)