I have released Criterium, a new project for benchmarking code in Clojure. I found Brent Broyer's article on Java benchmarking which explains many of the pitfalls of benchmarking on the JVM, and Criterion, a benchmarking library in Haskell.
The main issues with benchmarking on the JVM are associated with garbage collection, and with JIT compilation. It seems from Broyer's articles that we can mitigate the effects but not completely eliminate them, and Criterium follows his advice. Both of the above libraries use the bootstrap technique to estimate mean execution time and provide a confidence interval, and Criterium does likewise. At the moment the confidence intervals are biased and I still need to implement BCa or ABC to improve these.
One of the functions that I wanted to benchmark originally involved reading a file. Criterium does not yet address clearing I/O buffer cache, and I am not sure on the best way forward on this. On Mac OS X, the purge command can be used to clear the caches, and on Linux this can be achieved by writing to /proc/sys/vm/drop_caches. On the Mac at least, this causes everything to grind to halt for about five seconds, and there are then some file reads as whatever processes are running read things in again. This sort of behaviour doesn't lend itself to inclusion in a timing loop... Any suggestions?
Discuss this post here.
Published: 2010-02-19
