From BELIV 2010

Presentation

Notes

- small jobs @ amazon

- automatic recruitment and payment, prevent people going the tasks multiple times

- pretty good gender balance, broader age pool

- faster at tasks, but some unexplained differences

- personality may differ (unusual population)

- design, may have cheaters (accuracy/quality) problems, may be solved by bonus system, build checks into data collection to filter

- good for: simple interaction, short performance time (less drop out), responses that are hard to fake

- demographics: cannot trust

q and a:

- self report, people are more willing to self report when f2f, may not a population difference

- cheaper: 6-10X cost saving, but time saving is more

- human study report? yes, pretty fast, publish IRB protocol?

- faster, accuracy? no difference

- more intensive tasks with more money? long tasks, high drop out, paid $2, need to figure out how much to pay for the long task

- vis research, monitor, graphic cards etc critical and not controlled? doesn't go away with any online studies, make sure that's something not critical, most people have similar pixel counts and default colors

- self report? no, but should do

- connectivity speed

- can collect a lot of client info with javascript

- mostly US, for our case, mostly looking for perceptual / cognitive effects so pretty culturally neutral, but a consideration. benefits is could get people from different countries and compare, but not sure if ip logging is reliable.

- how do you account for questions that you don't know the answers to, no ground truth? add some ground truth, and intersubject agreement, task dependent.