A systematic assessment of rigor in information systems ranking-type Delphi studies

The ranking-type Delphi method is well suited as a means for consensus-building by using a series of questionnaires to collect data from a panel of geographically dispersed participants. This method allows a group of experts to systematically approach a particular task or problem. While information systems researchers have been using this method for almost three decades, no research to date has attempted to assess the extent to which Delphi studies have been rigorously conducted. Using the guidelines that have been prescribed by the leading Delphi methodologists, our descriptive review reveals many positive signs of rigor such as ensuring the anonymity of experts and providing clear and precise instructions to participants. Nevertheless, there are still several areas for improvement, such as reporting response and retention rates, instrument pretesting, and explicitly justifying modifications to the ranking-type Delphi method.