Paper Reading #21: Towards maximizing the accuracy of human-labeled sensor data

Title:
Towards maximizing the accuracy of human-labeled sensor data

Comments:
Cindy Skach
Luke Roberts


Reference:
Rosenthal, Stephanie, and Dey, Anind.  Towards maximizing the accuracy of human-labeled sensor data.  IUI '10.  ACM, New York.  http://delivery.acm.org/10.1145/1730000/1720006/p259-rosenthal.pdf?key1=1720006&key2=7883922031&coll=DL&dl=ACM&ip=165.91.4.135&CFID=17152168&CFTOKEN=47763320

Summary:
This study was focused on researching seeing the factors that influence the way people label things.  More specifically the study was related to the labeling of online content.  Some websites attempt to label their content through computer generation while others rely on user to label the data for them.  When the latter is done, there are usually many users who do this in order to increase the accuracy of the label.  The investigation focuses on uncertainty, amount and level of context, prediction of an answer, and request for user feedback concerning labeling.

The study was done to see the relation between people labeling their own data and labeling strangers data.  The result of this experiment found there was little difference in the way these labelings occurred.  The results found that there there were combinations of the focus of the investigation (uncertainty, amount and level of context, prediction of an answer, and request for user feedback) that would aid the most in maximizing labeling accuracy among users.  It was found that all these were important and helpful to generate more accurate labels on data.  The limitation was that providing additional information did not have an effect on their accuracy.  This was seen as a result of the user going in with notions based upon the raw data and didn't need to formalize any other thoughts in his head about the situation.

Discussion:
The paper began in utter confusion for me as I had very little idea what the paper was actually talking about until I was almost done with the introduction.  The title and abstract did not give me near the amount of information needed in order to understand in any sort of what what the study was trying to accomplish.  


Aside from that, the idea of labeling and the accuracy to which people do so reminded me of a few things I have done in school.  In my ecology class we all had to grade each others assignments online.  In order to calibrate our grading, we were all given three sample papers to correct and grade and off of that a program would analyze our conclusions in order to grade we should get on our papers.

I don't think that this study was important at all.  I didn't understand why they were doing it as they did not have a good explanation in any part of the paper.  I couldn't follow it completely.  I would say that for future work, aside from writing a decent report, would be to try and make an algorithm to better label instead of just having people label.