Benchmarking Javatm Execution Environments on Linux

The following set of benchmarks were run on an IBM ThinkPad A31p Laptop, running Red Hat Linux 9. This machine sports a P4 Mobile 1.7 GHz processor, and 512MB RAM.

Some effort was made to ensure that the machine was quiet during the performance tests. The numbers below represent the best of three runs.

If you have any questions, comments or suggestions, send them to green@redhat.com.

ApacheBench + Jetty

The Apache Software Foundation provides a simple tool called ApacheBench for measuring the performance of http servers. In this case, we are measuring the popular Jetty web server delivering static content. The following graph shows the number of requests handled per second (y-axis) vs the number of concurrent requests (x-axis).

The big drop-offs appear to have something to do with unusual kernel interactions, as most of the CPU time is being spent in the kernel.

Note that IKVM and Kaffe are unable to run this benchmark.

  1. gcj 3.3 compiled with -O2
  2. IBM JDK 1.4.1
  3. Sun JDK 1.4.2-beta

SciMark 2.0a

SciMark 2.0 is a number crunching bechmark. It reports a composite score, which is an approximate measure of Mflops (millions of floating point operations per second).

I ran the benchmark with two data sets (provided with SciMark). The small data set is used to isolate memory-hierarchy effects, to focus on the efficiency of the Java execution environment. The larger data set represents out-of-cache problem sizes.

In the following chart, bigger is better:

  1. gcj 3.2.2 compiled with -O3 -fomit-frame-pointer -march=pentium4 -finline-functions -finline-limit=800 -ffast-math -mfpmath=sse
  2. IKVM.NET + mono 0.24
  3. Sun JDK 1.4.2-beta
  4. IBM JDK 1.4.1
  5. gcj 3.2.2 compiled with -O3 -fno-bounds-check -fomit-frame-pointer -march=pentium4 -finline-functions -finline-limit=800 -ffast-math -mfpmath=sse
  6. Kaffe 1.1.0
  7. gcj HEAD (2003-06-30) compiled with -O3 -fno-bounds-check -fomit-frame-pointer -march=pentium4 -finline-functions -finline-limit=800 -ffast-math -mfpmath=sse
  8. gcj HEAD (2003-06-30) compiled with -O3 -fomit-frame-pointer -march=pentium4 -finline-functions -finline-limit=800 -ffast-math -mfpmath=sse

ShellSort

The ShellSort benchmark was written by Norman Hendrich. Bryce McKinley added some timers to isolate the fast start-up times for gcj-built executables.

In the following chart, smaller is better:

  1. gcj 3.2.2 compiled with -O3 -fno-store-check -fomit-frame-pointer -march=pentium4 -finline-functions -finline-limit=800
  2. IBM JDK 1.4.1
  3. Sun JDK 1.4.2-beta
  4. IKVM.NET + mono 0.24
  5. Kaffe 1.1.0
  6. gcj 3.2.2 compiled with -O3 -fomit-frame-pointer -march=pentium4 -finline-functions -finline-limit=800

Javascript Interpreter

This benchmark is simply the amount of real time (as reported by time) required to run a "Hello World" javascript from the command line using the Mozilla rhino javascript interpreter. This benchmark demonstrates the fast startup time for gcj-built programs -- making javascript a practical shell scripting language.

Note that IKVM was unable to run this benchmark.

In the following chart, smaller is better:

  1. gcj 3.2.2 compiled rhino with -O3 -fomit-frame-pointer -march=pentium4 -finline-functions -finline-limit=800
  2. Kaffe 1.1.0
  3. Sun JDK 1.4.2-beta
  4. IBM JDK 1.4.1
Java and all Java-based marks are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. All other trademarks and tradenames belong to their respective owners.