Cognitive Evaluation of a Physician Data Query Tool for a National ICU Registry: Comparing Two Think Aloud Variants and Their Application in Redesign

Applying usability methods in formative evaluations of interactive healthcare information systems design is recognized as of extreme importance to the final success of these systems. However, it seems that the merits of specific methodological approaches for conducting these studies have received little attention. This study reports on a cognitive evaluation of a Physician Data Query Tool, which offers physicians the opportunity to query quality of care data collected by the Dutch National Intensive Care Evaluation (NICE) foundation. A comparison in terms of usefulness and utility of two variants of the Think Aloud method is addressed, the Concurrent and Retrospective Think Aloud. These methods are well known in the field of Human Computer Interaction in the context of usability evaluation. The results of this research indicate that though both methods have their disadvantages and benefits, in redesigning the Physician Data Query tool the Retrospective Think Aloud provided more useful input to the Tool's redesign. However, in deciding which method to apply in a formative evaluation study, end users' cognitive workload of performing the system's tasks and the system characteristics need to be considered as well