criterium.core documentation
Criterium measures the computation time of an expression. It is designed to address some of the pitfalls of benchmarking, and benchmarking on the JVM in particular.
This includes: - statistical processing of multiple evaluations - inclusion of a warm-up period, designed to allow the JIT compiler to optimise its code - purging of gc before testing, to isolate timings from GC state prior to testing - a final forced GC after testing to estimate impact of cleanup on the timing results
Usage: (use 'criterium.core) (bench (Thread/sleep 1000) :verbose) (with-progress-reporting (bench (Thread/sleep 1000) :verbose)) (report-result (benchmark (Thread/sleep 1000)) :verbose) (report-result (quick-bench (Thread/sleep 1000)))
References: See http://www.ellipticgroup.com/html/benchmarkingArticle.html for a Java benchmarking library. The accompanying article describes many of the JVM benchmarking pitfalls.
See http://hackage.haskell.org/package/criterion for a Haskell benchmarking library that applies many of the same statistical techniques.
*final-gc-problem-threshold*
Fraction of excution time allowed for final cleanup before a warning is issued.
*max-gc-attempts*
Maximum number of attempts to run finalisers and gc.
*sample-count*
Number of executions required
*target-execution-time*
Target elapsed time for execution for a single measurement.
*warmup-jit-period*
Time period used to let the code run so that jit compiler can do its work.
StateChanged
Interrogation of differences in a state.
bench
Convenience macro for benchmarking an expression, expr. Results are reported to out in human readable format. Options for report format are: :os, :runtime, and :verbose.
benchmark
Benchmark an expression. This tries its best to eliminate sources of error. This also means that it runs for a while. It will typically take 70s for a fast test expression (less than 1s run time) or 10s plus 60 run times for longer running expressions.
benchmark*
Benchmark a function. This tries its best to eliminate sources of error. This also means that it runs for a while. It will typically take 70s for a fast test expression (less than 1s run time) or 10s plus 60 run times for longer running expressions.
bootstrap
Bootstrap a statistic. Statistic can produce multiple statistics as a vector so you can use juxt to pass multiple statistics. http://en.wikipedia.org/wiki/Bootstrapping_(statistics)
bootstrap-bca
Bootstrap a statistic. Statistic can produce multiple statistics as a vector so you can use juxt to pass multiple statistics. http://en.wikipedia.org/wiki/Bootstrapping_(statistics)
default-reducer
A function that can be used to reduce any function output.
estimate-execution-count
Estimate the number of executions required in order to have at least the specified execution period, check for the jvm to have constant class loader and compilation state.
execute-expr
A function to execute a function the given number of times, timing the complete execution.
execute-expr-core
See execute-expr-core-timed-part. Here we create the Java Object array in which execute-expr-core-timed-part will store return values from calling (f). Even though storing the return values in that array is probably enough not to allow the JVM to optimize away calls to (f), go ahead and use reduce-with to combine up to +max-obj-array-size+ return values together and return that.
execute-expr-core-timed-part
Performs the part of execute-expr where we actually measure the elapsed run time. Evaluates (f) n times, each time saving the return value as an Object in the Java Object array ret-vals-arr, i.e. ret-vals-arr is mutated.
The idea is that except for the call to (f), the only things done during each iteration are a few arithmetic operations and comparisons to 0 on primitive longs, and the aset to store the return value.
The JVM is not free to optimize away the calls to f because the return values are saved in ret-vals-arr.
This array is at most +max-obj-array-size+ elements long, to save memory. An artificially intelligent JVM might be able to determine that if n is larger than +max-obj-array-size+, some of the return values are overwritten and thus those calls need not be made. I doubt we will see that kind of optimization any time soon, and perhaps some JVM rules even prohibit doing so since the writes to ret-vals-arr could potentially be read by a different thread.
extract-report-options
Extract reporting options from the given options vector. Returns a two element vector containing the reporting options followed by the non-reporting options
final-gc
Time a final clean up of JVM memory. If this time is significant compared to the runtime, then the runtime should maybe include this time.
force-gc
Force garbage collection and finalisers so that execution time associated with this is not incurred later. Up to max-attempts are made.
heap-used
Report a (inconsistent) snapshot of the heap memory used.
jvm-compilation-state
Returns the total compilation time for the JVM instance.
jvm-jit-name
Returns the name of the JIT compiler.
memory
Report a (inconsistent) snapshot of the memory situation.
os-details
Return the operating system details as a hash.
outlier-effect
Return a keyword describing the effect of outliers on the estimate of mean runtime.
outlier-significance
Find the significance of outliers given boostrapped mean and variance estimates. See http://www.ellipticgroup.com/misc/article_supplement.pdf, p17.
outliers
Find the outliers in the data using a boxplot technique.
quick-bench
Convenience macro for benchmarking an expression, expr. Results are reported to out in human readable format. Options for report format are: :os, :runtime, and :verbose.
quick-benchmark
Benchmark an expression. Less rigorous benchmark (higher uncertainty).
quick-benchmark*
Benchmark an expression. Less rigorous benchmark (higher uncertainty).
report
Print format output
run-benchmark
Benchmark an expression. This tries its best to eliminate sources of error. This also means that it runs for a while. It will typically take 70s for a quick test expression (less than 1s run time) or 10s plus 60 run times for longer running expressions.
run-benchmarks-round-robin
Benchmark multiple expressions in a 'round robin' fashion. Very similar to run-benchmark, except it takes multiple expressions in a sequence instead of only one (each element of the sequence should be a map with keys :f and :expr-string). It runs the following steps in sequence:
Execute each expr once
Run expression 1 for at least warmup-jit-period nanoseconds so the JIT has an opportunity to optimize it. Then do the same for each of the other expressions.
Run expression 1 many times to estimate how many times it must be executed to take a total of target-execution-time nanoseconds. The result is a number of iterations n-exec1 for expression 1. Do the same for each of the other expressions, each with the same target-execution-time, each resulting in its own independent number of executions.
Run expression 1 n-exec1 times, measuring the total elapsed time. Do the same for the rest of the expressions.
Repeat step 4 a total of sample-count times.
runtime-details
Return the runtime details as a hash.
scale-time
Determine a scale factor and unit for displaying a time.
system-properties
Return the operating system details.
time-body
Returns a vector containing execution time and result of specified function.
time-body-with-jvm-state
Returns a vector containing execution time, change in loaded and unloaded class counts, change in compilation time and result of specified function.
timestamp
Obtain a timestamp
timestamp-2
Obtain a timestamp, possibly using MXBean.
warmup-for-jit
Run expression for the given amount of time to enable JIT compilation.
with-progress-reporting
Macro to enable progress reporting during the benchmark.
state-changed?
Check to see if a state delta represents no change
state-delta
Return a state object for the difference between two states