HDB-a high level debugging

This paper presents a new high level debugging tool, HDB, for debugging large scientific programs running on a moderate number of processors. The unique feature of HDB is that checksums are used to compress arrays and groups of variables without losing meaningful information for debugging. Using checksums makes it possible to use invariance assertions to detect misbehavior of a program at a place near the source of the error. Tracing the checksums allows the tracing of a large amount of data with a small amount of output. Comparing the traced checksums of a program and the traced checksums of its reference copy can rapidly reduce the potential error sources to a small number of subroutines. These subroutines can then be directly probed for further investigation. If desired, a debugger providing break points and single stepping source code can be used in conjunction with HDB. Examples show that using the HDB method can rapidly uncover bugs hidden in both parallel and sequential programs.