Designing an Efficient and Extensible Robustness Benchmark of a Real-Time Operating System

An important step in the development of a Real-Time Operating System (RTOS) is the validation of its tolerance properties. An abnormal input (fault injection) has become one of the most efficient ways to test the software robustness. In this paper we have designed a comprehensive robustness benchmark to evaluate the current popular RTOSs by faults injection. Firstly, we provide a set of uniform application program interfaces to ensure that the benchmark can be easily ported to a new RTOS. Then a package testing method has been used to improve the testing efficiency. Finally, a comprehensive robustness evaluation model is provided for the quantitative evaluation of RTOS robustness. Three popular RTOSs (Ucos2.62, Vxworks5.4 and Rtems4.10) have been evaluated with the help of our benchmark and we have found that Rtems performs best in robust evaluation, while Vxworks performs worst.