On the representativeness of embedded Java benchmarks

Java has become one of the predominant languages for embedded and mobile platforms due to its architecturally neutral design, portability, and security. But Java execution in the embedded world encompasses Java virtual machines (JVMs) specially tuned for the embedded world, with stripped-down capabilities, and configurations for memory-limited environments. While there have been some studies on desktop and server Java, there have been very few studies on embedded Java. The non proliferation of embedded Java benchmarks and the lack of widespread profiling tools and simulators have only exacerbated the problem. While the industry uses some benchmarks such as MorphMark, MIDPMark, and EEMBC Java Grinder Bench, their representativeness in comparison to actual embedded Java applications has not been studied. In order to conduct such a study, we gathered an actual mobile phone application suite and characterized it in detail. We measure several properties of the various applications and benchmarks, perform similarity/dissimilarity analysis and shed light on the representativeness of current industry standard embedded benchmarks against actual mobile Java applications. It was observed that for many characteristics, the applications had a broader range, indicating that the benchmarks were under representing the range of characteristics in the real world. Furthermore, we find that the applications exhibit less code reuse/hotness compared to the benchmarks. We also draw comparisons of the embedded benchmarks against popular desktop/client Java benchmarks, such as the SPECjvm98 and DaCapo. Interestingly, embedded applications spend a significant amount of time in standard library code, on average 65%, suggesting to the usefulness of software and hardware techniques to facilitate pre-compilation with out the real time resource overhead of JIT.