Citizen Engineering: Methods for "Crowdsourcing" Highly Trustworthy Results

Citizen Engineering seeks to leverage a large number of ordinary citizens to solve real-world problems. Emerging information technologies provide us with opportunities to answer a long-standing challenge in citizen engineering -- can we effectively extract reliable results from a myriad of crowd inputs of varying quality? To investigate efficient approaches to achieving this "wisdom of crowds", we established a prototype site, where 242 students, acting as surrogate citizen engineers, signed up, logged in, and performed engineering tasks -- tagging photographs of earth-quake damage. Based on the analysis of user online behaviors, we developed an operable data mining algorithm to retrieve highly trustworthy results from thousands of limited size submissions collected from a cohort of contributors. By converging weight assignments and crowd consensus step- by-step, this extraction algorithm improves the quality of the results over time.