Publications: 51 | Followers: 1

Crowdsourcing & Crowdfunding platforms

Publish on Category: Birds 0

The future of crowdworkAniketKitturet al., (2013).The future of crowd work,CSCW’13
MejdlSafranFall 2013SIUC
More complex, creative and highly valued work
“aplatformis needed for managing pools of tasks and workers. Complex tasks must bedecomposedinto smaller subtasks, eachdesignedwith particular needs and characteristics which must beassignedto appropriate groups of workers who themselves must be properlymotivated, selected (e.g., throughreputation), and organized (e.g., throughhierarchy). Tasks may be structured through multi-stageworkflowsin which workers maycollaborateeither synchronously or asynchronously. As part of this,AImay guide (and be guided by) crow workers. Finally,quality assuranceis needed to ensure each worker’s output is of high quality and fits together.”
Research challenges in 12 major areas
The future ofcrowd work processesThe future ofcrowd computationThe future ofcrowd workers
Crowd Work: Workflow
Motivation/GoalsComplex crowd work cannot accomplished by simple parallel approaches (e.g., aggregating multiple independent judgments)Workflows are needed that facilitatedecomposingtasks into subtasks,managingthe dependencies between subtasks, andassemblingthe results.
CrowdWork: Workflow
Research ProposalImproving existing workflows toward moregeneral tasksandcomplex problemsthat have no clearly defined solution (creativity, civic planning, etc.)Comparing to simple workflow, crowd workflow could involve a muchlarger scale of operation, an a much moreheterogeneousset of actorsHow to validate?We must experiment on a large space of parameters, incentives and decompositions.Costs of doing so can be reduced throughmodelsof worker behavior, or by reusing provendesign patterns.
Crowd Work: Task Assignment
Motivation/GoalsSharing limited resources(workers) requirescoordination.While workers arecontinuouslyemployed with tasks that match their interests, requesters will see their tasks completed quickly.ESP game and Galaxy Zoo use FCFS modeloDeskand AMT use market model
Crowd Work: Task Assignment
Research ProposalData partitioning, OS scheduling and failover!Or new theoretical model.“Several workers in our survey complained that they spent too much energy finding appropriate tasks” (Task Recommendation).Research question: should tasks bepulledby workers orpushedby platforms?
Crowd Work:Quality Control
Motivation/GoalsWhy? High throughput, low transaction costs ands complex/subjective tasks.Cheating or game the system.Workers with low expertise and requesters with unclear instructions.
Crowd Work: Quality Control
Research ProposalUp-front task design: peer-review, agreement filters (writing an essay!!), optimize instructions, etc.Post-hocresultanalysis: gold standard data (prevent biases from dominating the results) and workers’ votes (costs more).Better approach: use ML to predict the quality of a worker’s output from their behavior.Open-ended work and highly skilled tasks??Is it possible to robustly measure worker’s skills at tasks such as audio engineering and poetry? Should we rely on peer evaluation?
Crowd Work:Real-time Crowd Work
Motivation/GoalsWe need to createflash crowds: groups of individuals who arrive moments after a request and can work synchronously.All on-demand CS are limited by the crowd latency.
Crowd Work: Real-time Crowd Work
Research ProposalFast recruitmentis a major research area in real-time CS.Time-limitedtasks (e.g., search for a missing person and timed competitions)E-maila set of workers the night before the study and announced a time for the experiment.Payingworkers a small wage to stay on call.They need to be modeled usingqueuing theory.Scaling upto increased demand for real-time workersMaking workersefficient enoughto collectively generate results ahead of time deadlines
Crowd Computation: Crowds Guiding AIs
Motivation/GoalsIn human computation, people act as computational components and perform the work that AI systems lack the skills to complete.By tapping into crowd intelligence, computational systems can support a much broader set of tasksCrowd workers may end up training machines to replace them!!Gathered data is used to train algorithms (e.g., Google image search engine, birds count project)
Crowd Computation: Crowds Guiding AIs
Research ProposalIntegrating crowds more deeply into algorithms.Design ML algorithms that more deeply understand the human nature of these labels.
Crowd Computation: AIs GuidingCrowds
Motivation/GoalsHumans can introduce errors, and errors are amplified as they propagate through the crowd.Integrate crowds inside of software and use the software to help guide crowd work.E.g., ML model can determine which work products may still be improved and then assign workers most likely to make such improvements.
Crowd Computation: AIs Guiding Crowds
Research ProposalAIs can point out what others have done in similar contexts.We need to move from a setting where simple AIs completely determine a workflow to a richer, mixed setting where crowds and AIs jointly teach each other, and jointly control the work process.
Crowd Computation: Crowdsourcing Platforms
Motivation/GoalsLimiting ourselves to exiting platforms greatly restricts the scope and nature of change we can enact.
Crowd Computation: Crowdsourcing Platforms
Research ProposalDifferent platforms for different aims.MobileWorks(mClerk): through SMS, more populations.Vending machine with purpose (exam grading) targeting experts.Labor exchanged policy issuesHow can platforms disclose enough info to be trusted as a source of worker quality while also maintaining privacy?Security concerns both attacks on platforms or use of platforms for launching attacks
Crowd Workers: Reputation
Motivation/Goals“Graduated from UC Berkeley and working Google”Traditional employers judge a prospective employee's education and work history through interviews, transcripts, and references.“I have an accepting rate of 4 out of 5 on AMT”
Crowd Workers: Reputation
Research ProposalWe wantbetter reputation rankings for workers to be established with theplatformsReputation systems must be robust to cheating and gaming while supporting low-transaction cost hiring.One way:a web of trust! (workers and requesters validate each other.What ifmaliciousbuild their ownweb of trust!!! (gaming the system)Technical tools for sharing information about workers requester abuses!! (privacy)
Crowd Workers: Motivation & Rewards
Motivation/GoalsCrowd workers are a diverse and multifaceted population with a range of motives and experiencesWorker said “Mturkshould encourage and reward requesters that provide clear instructions, quick payment, and higher pay”
Crowd Workers: Motivation & Rewards
Research ProposalPast research:Requesters should clearly understand and communicate desired behaviorsUnderstand and worker motivations and incentives with these desired behaviorsDesign the requests and incentive structures in order to achieve both effective task completion and worker satisfactionBeyond financial incentivesBuild systems that support workers’ diverse motivations.





Make amazing presentation for free
Crowdsourcing & Crowdfunding platforms