Thelonious

If you ever play a false note, play it again, and play it loud.

— Thelonious Monk

Introduction

Thelonious is a tool for automatic fault localization. It links SUnit test results to methods under test (MUT), and calculates the suspiciousness of being faulty for each MUT.

A blueprint visualization of the results helps the programmer to decide where to start looking for the cause of a test failure.

Thelonious was inspired by the Java tool EzUnit. It uses an adapted version of the Spy Framework.

Screenshot

Here we injected a single fault in the Graph-ET package and ran Thelonious:

Usage

Select packages under test and corresponding test packages in Nautilus and start Thelonious via the context menu. Choose different fault locators and visualizations in the Pharo System Settings Browser.

Blueprint legend

Fault locators

At this moment Thelonious includes the following fault locators:

• Failure Ratio, relates the number of failed tests for a MUT to the total number of tests covering this MUT.
• Failure Coverage, relates the number of failed tests for a MUT to the total number of failed tests in the System under test (SUT).
• Failure Accountability, combines Failure Ratio and Failure Coverage. At this moment, Failure Accountability is the most promising fault locator.
• Singe Method Under Test, marks a MUT with the highest suspiciousness, if the MUT is the only method called by a failing test. This fault locator is only useful when combined with other fault locators.
• Ochiai, is a very popular fault locator. It also happens to be the square root of Failure Accountability.

You can add your own fault locators by subclassing FaultLocator and implementing possibilityFor: aTheloniousMethod. You can choose the active fault locator at the Thelonious section of the Pharo System Settings Browser.

Limitations

Due to it's implementation, using method wrappers, there are some limitations to the nature of systems under test (SUT), that can be analyzed with Thelonious at this moment:

• If the SUT accesses the context of a method (via #thisContext) test results may be incorrect. The reason is that the context of an instrumented method (more precisely it's sender) is different from the context of a non-instrumented method.
• Pragma calls are not taken into account at this moment.

Accuracy of results

Thelonious results with the Failure Accountability (or Ochiai) fault locator are pretty accurate, if the reason for a test failure is a single fault. As soon as several faults are the reason for a test failure, accuracy of Thelonious results declines.

Installation

Thelonious currently works on any Pharo 2.0 image. Install it by executing:

Gofer it   
    smalltalkhubUser: 'MartinWalk' project: 'Thelonious';  
    configuration;  
    loadStable

Contact

If you have any questions, please contact me at wm@fastmail.fm