中央研究院  |   資訊科學研究所  |   多媒體網路與系統實驗室 
On formal models for social verification
(NOTE: Sheng-Wei Chen is also known as Kuan-Ta Chen.)

Abstract
The introduction of the ESP Game and other Games With A Purpose (GWAP) has demonstrated the potential of human computation in solving AI-hard problems. In such systems, users are normally required to input answers for questions proposed by the system, e.g., descriptions about a picture or a song. Since users may bring up irrelevant inputs intentionally or carelessly, and often the system does not have "correct" answers, we have to rely on the users to verify answers from others. We call this kind of mutual verification of users' answers "social verification."

In this paper, we propose formal models for two fundamental social verification mechanisms, simultaneous verification and sequential verification, in human computation systems. By adopting a game-theoretic approach, we perform an equilibrium analysis which explains the effect of each verification mechanism on a system's outcome. Our analysis results show that sequential verification leads to a more diverse and descriptive set of outcomes than simultaneous verification, though the latter is stronger in ensuring the correctness of verified answers. Our experiments on Amazon Mechanical Turk, which asked users to input textual terms related to a word, confirmed our analysis results. We believe that our formal models for social verification mechanisms will provide a basis for the design of future human computation systems.


Citation
Ho Chien-Ju and Chen Kuan-Ta, "On formal models for social verification," In Proceedings of Human Computation Workshop 2009, pages 62-69, 2009.

BibTex
@INPROCEEDINGS{ho09:model,
  AUTHOR     = {Ho Chien-Ju and Chen Kuan-Ta},
  TITLE      = {On formal models for social verification},
  BOOKTITLE  = {Proceedings of Human Computation Workshop 2009},
  PAGES      = {62--69},
  YEAR       = {2009},
  LOCATION   = {Paris France},
  DOI        = {http://doi.acm.org/10.1145/1600150.1600172}
}
Downloads