Monday, May 14, 2012

In which I battle MTurk External Hits and (eventually) WIN

Here are some lessons I learned recently about Mechanical Turk and launching external hits. Painful, painful lessons. Makes me not want to play around with MTurk too much in the future.
  1. The only way to officially make an "external hit" is through the command line tools or maybe one of the fancy tools built on top of the command line tools. You can't do it through the mturk requester website unless by "external" you mean a link to another website and maybe asking your workers to copy and paste some field back into the mturk website.
  2. The command line tools are scarily old. Even though this link was last updated OVER FOUR YEARS AGO, it seems to be the way to go. Here are brief instructions about getting set up: 
    1. If you're not on windows, don't click the "DOWNLOAD" button because that will lead you astray and fetch you a .exe! Look on down the page for Unix Downloads / Command Line Tools without JRE.
    2. No need to compile or really install anything because it's all shell scripts and java code. That's nice, at least.
    3. Go into bin/ and put in your access code and key. ALSO CHANGE http to https, because Amazon is fancy and secure these days but doesn't update their mturk developer code!
    4. Do this: export JAVA_HOME=/usr
    5. Okay, great! Now you can run things like ''... 
    6. Also note you can play around in the sandbox by commenting out lines in The sandbox balance should be like, $10,000... Ooooh Ahh!
  3. The examples are okay, but not informative enough to like, have an external hit with more than one variable.
  4. Don't be a fool, like me, and keep running ./ to add more hits and then wonder why the things you see on the webpage appear to have no bearing on reality/the things you just changed. Adding new hits via literally adds new hits and doesn't replace old ones, even in magical sandbox land. So when you refresh to look at a particular hit, you'll get a random one from all the ones you've created.
    1. Reset account is your friend! cd ../bin/ && ./ -force
  5. FU#$KING ampersands!!!! url=${helper.urlencode($urls)} means nothing to me when it says 'url' in three different places AND refreshing the page gives me random unintelligible crap that may or may not also still say 'url' somewhere. 
    1. When it says... [Fatal Error] :6:117: The reference to entity "centerLong" must end with the ';' delimiter.
      [ERROR] Error creating HIT 1 (44.5628547668457): [6,117] The reference to entity "centerLong" must end with the ';' delimiter. 
    2. You say... &amp; like so: <ExternalURL>$centerLats&amp;centerLong=$centerLongs&amp;bundleId=$bundleIds</ExternalURL>
  7. I thought the hit batch input file needs tabs, and commas don't seem to be okay. That is pretty annoying because I made my vim not use tabs AGES AGO.
The end. The rest, like the code for pulling out your variables from the URL and then submitting your external hit form data seems pretty self explanatory.

Tuesday, May 8, 2012

CrowdCamp 2012 reflections

I went to this amazing workshop this weekend at CHI. It was a 2-day work shop where we came up with lots of ideas and pitches, broke into small groups, and MADE THINGS. We also played random physical games including an intense match of dodgeball that came down to Steve Drucker vs. a group of five. I'm hoping someone else summarizes all the amazing projects. I'm just going to write down some reflections based on my group's project and dinner discussions.

Context: My group's goal was to create an interesting creative artifact, like a book or collection of stories or photos. We eventually settled on a theme of trying to capture a snapshot of Mechanical Turkers lives and the interesting stories they each want to tell. We made a few separate mturk hits asking people to submit a photo (either through email or through a webcam) and write a story, based on one of a number of different prompts. Throughout the weekend, we experimented with many different story prompts, yielding a variety of responses, some more interesting or more authentic (and not ripped off wikipedia) than others. There will be an official blog post soon, but here's a link (that won't be around forever) to some of the pictures and stories:

chi and crowdcamp

The three things I want to discuss, then, are 1) feedback cycles 2) spontaneity and creativity and 3) the end artifact versus the process. Also, for fun, jumpstarting memes at the end of the post!

Feedback Cycles

An initial idea that kind of got buried under other ideas was to have creative cycles or iterations in which the people making (pieces of) the final creative artifact built off of or saw other people's work. If I could go back in time, I would push that idea more, but I have my own examples of it working awesomely.

There's a fun little crowdsourced book of cat stories ( put together by Bjoern Hartmann, one of the other guys at the workshop. I adore the idea behind it and that the book actually exists, and I own a copy. But frankly, I'm disappointed in the quality of the cat stories. I'm sure these cats, my cats, other cats have done far more interesting things. Perhaps the silliest thing my cats have done is "drive" their cardboard firetruck into the bedroom in the middle of the night. PICTURE! I had to go look at photos to come up with that story there. I am fairly sure the cats in the book have done more exciting things than what ended up in print.

The photos and stories that came back from workers this weekend (excluding the fake ones) were kind of bland. My hypothesis is that if you ask someone on the internet (a mechanical turk worker specifically) to write/draw/do something interesting completely out of the blue, they probably won't come up with something that interesting. First, it's kind of awkward to be paid to be creative (but some of our turkers were pretty creative, so maybe that's not a huge problem). Second and more importantly, they're not primed to say anything interesting right then and there.

Spontaneous Creativity

The corollary to the above hypothesis is that you could very easily prime these people to be creative by showing them other people's stories. Maybe they'll try to write something less boring than someone else. Maybe another person's story will trigger a memory. Maybe knowing someone else (especially another turker) might see it will raise the bar.

It's also possible that the prompts just weren't the right ones to spark creativity. Consider: "Write a story about your cat" vs. "What's the most surprising thing your cat has ever done?" We looked at some of the original photos people took in response to prompts like "your most prized possession, your hands, your shoes" and came up with new themes that were a little more flippant, but elicited some enjoyable photos. These new prompts included "make a hat" and "make a scene out of 2-5 objects around you". Personally, I enjoyed the photos and stories from these prompts much more.

Artifacts vs Process

There was a comment at dinner about a "saturation" of stories in the world. I'm not sure I agree, but I definitely think there's more being produced than there is time to consume it. Consider a recent film called "Life in a Day" that was compiled out of thousands of youtube videos all filmed on the same day. It took a fancy director to turn 4,500 hours of video into a manageable 1.5 hours. Now it's all about the artifact, but the experience of submitting a video has value, too, even if the final video was never made, I would argue.

I am most interested in designing opportunities to _participate_ in creative, collaborative experiences. You could take up drawing, or you could take up drawing within a social and constrained framework, like drawing a picture of an animal every day for a month and sharing those drawings with others. Like this!

Jumpstarting Memes

The idea of jumpstarting memes using mechanical turk came up during a coffee break. Our photos responses to "make a hat out of something" are pretty good. Kurtis Heimerl (had to stalk list of attendees and google images to remember name) put some mturk jobs out there asking people to take 2 photos, one where their clothes were on properly, and one where their clothes were on backwards but they were facing backwards, too. By the second day, he had a handful of pretty hilarious photos. If you ever see such a meme on the internet, know it started at CrowdCamp 2012!