In which I battle MTurk External Hits and (eventually) WIN

Here are some lessons I learned recently about Mechanical Turk and launching external hits. Painful, painful lessons. Makes me not want to play around with MTurk too much in the future.

1. The only way to officially make an "external hit" is through the command line tools or maybe one of the fancy tools built on top of the command line tools. You can't do it through the mturk requester website unless by "external" you mean a link to another website and maybe asking your workers to copy and paste some field back into the mturk website.
2. The command line tools are scarily old. Even though this link was last updated OVER FOUR YEARS AGO, it seems to be the way to go. Here are brief instructions about getting set up: 
1. If you're not on windows, don't click the "DOWNLOAD" button because that will lead you astray and fetch you a .exe! Look on down the page for Unix Downloads / Command Line Tools without JRE.
2. No need to compile or really install anything because it's all shell scripts and java code. That's nice, at least.
3. Go into bin/mturk.properties and put in your access code and key. ALSO CHANGE http to https, because Amazon is fancy and secure these days but doesn't update their mturk developer code!
4. Do this: export JAVA_HOME=/usr
5. Okay, great! Now you can run things like 'getBalance.sh'... 
6. Also note you can play around in the sandbox by commenting out lines in mturk.properties. The sandbox balance should be like, $10,000... Ooooh Ahh!
3. The examples are okay, but not informative enough to like, have an external hit with more than one variable.
4. Don't be a fool, like me, and keep running ./run.sh to add more hits and then wonder why the things you see on the webpage appear to have no bearing on reality/the things you just changed. Adding new hits via run.sh literally adds new hits and doesn't replace old ones, even in magical sandbox land. So when you refresh to look at a particular hit, you'll get a random one from all the ones you've created.
1. Reset account is your friend! cd ../bin/ && ./resetAccount.sh -force
5. FU#$KING ampersands!!!! url=${helper.urlencode($urls)} means nothing to me when it says 'url' in three different places AND refreshing the page gives me random unintelligible crap that may or may not also still say 'url' somewhere. 
6. THIS IS THE MAGIC KEY/POTION/FORMULA/INCANTATION:
1. When it says... [Fatal Error] :6:117: The reference to entity "centerLong" must end with the ';' delimiter.
   [ERROR] Error creating HIT 1 (44.5628547668457): [6,117] The reference to entity "centerLong" must end with the ';' delimiter. 
2. You say... &amp; like so: <ExternalURL>http://mypage.com/external_thing.html?centerLat=$centerLats&amp;centerLong=$centerLongs&amp;bundleId=$bundleIds</ExternalURL>
7. I thought the hit batch input file needs tabs, and commas don't seem to be okay. That is pretty annoying because I made my vim not use tabs AGES AGO.

The end. The rest, like the code for pulling out your variables from the URL and then submitting your external hit form data seems pretty self explanatory.

CrowdCamp 2012 reflections

I went to this amazing workshop this weekend at CHI. It was a 2-day work shop where we came up with lots of ideas and pitches, broke into small groups, and MADE THINGS. We also played random physical games including an intense match of dodgeball that came down to Steve Drucker vs. a group of five. I'm hoping someone else summarizes all the amazing projects. I'm just going to write down some reflections based on my group's project and dinner discussions.

Context: My group's goal was to create an interesting creative artifact, like a book or collection of stories or photos. We eventually settled on a theme of trying to capture a snapshot of Mechanical Turkers lives and the interesting stories they each want to tell. We made a few separate mturk hits asking people to submit a photo (either through email or through a webcam) and write a story, based on one of a number of different prompts. Throughout the weekend, we experimented with many different story prompts, yielding a variety of responses, some more interesting or more authentic (and not ripped off wikipedia) than others. There will be an official blog post soon, but here's a link (that won't be around forever) to some of the pictures and stories: http://23.21.100.126/view.html



The three things I want to discuss, then, are 1) feedback cycles 2) spontaneity and creativity and 3) the end artifact versus the process. Also, for fun, jumpstarting memes at the end of the post!

Feedback Cycles

An initial idea that kind of got buried under other ideas was to have creative cycles or iterations in which the people making (pieces of) the final creative artifact built off of or saw other people's work. If I could go back in time, I would push that idea more, but I have my own examples of it working awesomely.

There's a fun little crowdsourced book of cat stories (http://bjoern.org/projects/catbook/) put together by Bjoern Hartmann, one of the other guys at the workshop. I adore the idea behind it and that the book actually exists, and I own a copy. But frankly, I'm disappointed in the quality of the cat stories. I'm sure these cats, my cats, other cats have done far more interesting things. Perhaps the silliest thing my cats have done is "drive" their cardboard firetruck into the bedroom in the middle of the night. PICTURE! http://www.flickr.com/photos/kaflurbaleen/4968850498/ I had to go look at photos to come up with that story there. I am fairly sure the cats in the book have done more exciting things than what ended up in print.

The photos and stories that came back from workers this weekend (excluding the fake ones) were kind of bland. My hypothesis is that if you ask someone on the internet (a mechanical turk worker specifically) to write/draw/do something interesting completely out of the blue, they probably won't come up with something that interesting. First, it's kind of awkward to be paid to be creative (but some of our turkers were pretty creative, so maybe that's not a huge problem). Second and more importantly, they're not primed to say anything interesting right then and there.

Spontaneous Creativity

The corollary to the above hypothesis is that you could very easily prime these people to be creative by showing them other people's stories. Maybe they'll try to write something less boring than someone else. Maybe another person's story will trigger a memory. Maybe knowing someone else (especially another turker) might see it will raise the bar.

It's also possible that the prompts just weren't the right ones to spark creativity. Consider: "Write a story about your cat" vs. "What's the most surprising thing your cat has ever done?" We looked at some of the original photos people took in response to prompts like "your most prized possession, your hands, your shoes" and came up with new themes that were a little more flippant, but elicited some enjoyable photos. These new prompts included "make a hat" and "make a scene out of 2-5 objects around you". Personally, I enjoyed the photos and stories from these prompts much more.

Artifacts vs Process

There was a comment at dinner about a "saturation" of stories in the world. I'm not sure I agree, but I definitely think there's more being produced than there is time to consume it. Consider a recent film called "Life in a Day" that was compiled out of thousands of youtube videos all filmed on the same day. It took a fancy director to turn 4,500 hours of video into a manageable 1.5 hours. Now it's all about the artifact, but the experience of submitting a video has value, too, even if the final video was never made, I would argue.

I am most interested in designing opportunities to _participate_ in creative, collaborative experiences. You could take up drawing, or you could take up drawing within a social and constrained framework, like drawing a picture of an animal every day for a month and sharing those drawings with others. Like this! http://www.flickr.com/photos/kaflurbaleen/sets/72157626192792880/

Jumpstarting Memes

The idea of jumpstarting memes using mechanical turk came up during a coffee break. Our photos responses to "make a hat out of something" are pretty good. Kurtis Heimerl (had to stalk list of attendees and google images to remember name) put some mturk jobs out there asking people to take 2 photos, one where their clothes were on properly, and one where their clothes were on backwards but they were facing backwards, too. By the second day, he had a handful of pretty hilarious photos. If you ever see such a meme on the internet, know it started at CrowdCamp 2012!

The tines of my heart: three or four flavors of head-crabs

What kind of ridiculous title is that?! I am referring, of course, to Derek Yu’s indie game rant about head crabs, and also Adam calling my trio of proposed research interests “prongs”, which has then morphed into “tines” because forks have tines. And then there’s an extra one that doesn’t fit into a research-I-do-at-school category.

Prong #1: The complete pipeline

I’ve worked on two research projects now that actually fit together quite nicely. PointCraft feeds directly off the output from PhotoCity and could even benefit from PhotoCity’s revival, while the artifacts generated in PointCraft could in fact feed back into PhotoCity. Take pictures of something you care about that get used to generate a point cloud, trace the point cloud in 3D to get a cleaned up solid mesh, and if necessary, go back out and take more photos to capture the details that the are missing from the mesh. The whole process is collaborative so that a single person need not follow the entire pipeline unless she wants to, but any person can provide useful input at any point along the way: by taking photos, but pruning out erroneous photo matches in the point cloud, by modeling a mesh, and by marking the areas of the model that need more data.

Prong #2: Repurposing “gamification” for good

Currently, when crowdsourcing researchers hear the term “gamification”, they think of how to incentivize people to do work without having to pay them. I advocate borrowing rule systems, mechanics and interfaces from games to design better (richer?) crowdsourcing systems. Not for “motivation” or to candy coat an otherwise dull task. Crowdsourcing via actual money is useful, effective, and extremely valuable to study the economics of, and the absolute wrong thing to do is to inappropriately replace money with “fun” via what I shall refer to as manipulative gamification. This type of gamification serves to provide extrinsic motivation*, essentially changing the user himself to make him “want” or “need” to perform a task. Human beings seek purpose and want to contribute to goals larger than themselves, and there are certain kinds of tasks (e.g. creative tasks) that people are already driven to do, that need not be financially compensated and in fact would be difficult to assign monetary value.

There is a guy, Daniel Pink (who possibly ripped off someone else’s ideas and rewrote them in a more digestible fashion) who states that the three components of intrinsic motivation are: autonomy, challenge/mastery, and purpose. Games totally nail the first two on the head, and to combine them with a purpose gives you the whole freakin’ package.

I can think of a few groups of related crowdsourcing systems.

1.     Shallow games with a purpose: I don’t actually mean shallow in a bad way (though I do mean manipulative gamification in a bad way). I mean that the action the player takes isn’t that complicated and thus there’s not much of an opportunity to be challenged and develop mastery. Games like this include many of Luis von Ahn’s games, like the ESP Game, Peekaboom, etc. The verification mechanisms in these games are really clever, but I personally don’t get anything out of playing except knowing I’m contributing to a good cause.

2.     Inappropriate use of game mechanics: I’m not going to name names right now (because I don’t feel like looking it up) but I saw a talk recently that suggested using the slingshot mechanism from angry birds to place the appropriate geometric concept on the correct part of a diagram. If you’re just supposed to put the piece in the right place, why use a damn slingshot? It’s just going to make things imprecise and terrible for the kids who know where exactly to put the piece. I’ve also seen a game about typing a word so a beaver can walk over a log without falling into a gap? And matching words while simultaneously playing some sort of space shooting game? It is extremely important to have the mechanics actually fit with the goal, otherwise you’re probably doing whatever Jesse Schell was complaining about (in his rant at the latest GDC (2012) when he lit a $50 bill on fire) and wasting people’s money and time by having no idea what you’re doing.

3.     Complex crowdsourcing tasks that aren’t games: Wikipedia and Open Street Map and other things that people contribute to out of the goodness of their hearts hit those same motivation bullet points we mentioned earlier, but they’re not called games. That’s fine! That’s good! But they could be games. (Possibly, or it might make them seem too cheesy and drive people away. But it could drive other people in!) (Actually I think there’s something particular about providing information/knowledge that means you should take things seriously and not turn it into a game) Think about it like this: take a system like one of the above, see if any of the actions that people do map onto actions that players commonly perform in games, and then reframe the task as a game. Thinking about something as a game could give people a familiar handle to latch on to so they can understand it better. It could also attract people who are okay with the role of “gamer” but don’t want to take on the new, more specific, official-sounding role of “contributor to Wikipedia” or “mechanical turk worker”. Maybe that’s too much responsibility, whereas a game seems more accessible, easier to try out.

4.     Complex crowdsourcing tasks that are games: This category is TOO SMALL and that is why I care about this at all. I can think of multiple scientific discovery games (just a few years ago, these games weren’t numerous enough to have their own category but I think they are now!) including Foldit, EteRNA, Phylo, and the unreleased DNA Game that my lab is working on. There is also my previous game, PhotoCity, which is about crowdsourcing the photo-collection process by having players go out into the world and use real cameras to take photos. Luis von Ahn is also developing a language learning game called DuoLingo that will magically translate text AND teach its participants a new language. Of all these games, PhotoCity might be the simplest in terms of having parallels to traditional games – you try and capture territory on a map and turn the flags into your team’s color and maybe spawn more flags – because the scientific discovery games especially have had to develop entirely new interfaces and come up with really useful feedback mechanisms.

Popping off the stack, or getting out of that crazy list of crowdsourcing things (which doesn’t even include the systems with monetary incentives)…

My point is that there is a lot to borrow from the world of game design that can help structure a crowdsourcing task. Games are all about mastery, developing skills and ramping up the challenges according to your skills, and if we want to crowdsource more complicate things besides labeling images and transcribing text (or breaking big tasks into chunks a la map reduce) we need environments that let them develop their skills over the longer term. ...To be expanded upon!

*There’s something complicated about extrinsic motivation and the badges/leaderboards/points/etc. that get shimmied into other scenarios. It’s not all bad, but I think the good that can arise has to do with these things providing appropriate feedback or some other positive feeling, like discovering the secret recipe in Foursquare to unlock a particular quirky badge. 

Prong #3: World-wide photo graph

This idea stems from the fact that PhotoCity is ridiculously hard to play anywhere outside of UW, Cornell, and a few other choice locations that dedicated souls managed to seed successfully. After telling people about the game, they’d ask, “can I play?” to which I’d say NO, and then feel really bad inside. The servers are down, there’s no disk space, there are no seeds here, the buildings are too shiny, or too featureless, or too tall, blah blah blah. That is a pretty depressing motivation; so let me try to explain what this inspired. It’s the 3D reconstruction technology’s fault that certain environments work so poorly. There is still probably enough information in these pictures to match them to nearby pictures, though, without trying to produce a 3D model (which requires a LOT of matches).  I think there could be a lightweight version of PhotoCity that lets anyone anywhere take a single photo, and if it’s near something else, it matches. If not, it just dumps in in an approximate location on a map, and maybe someone else will come a long and take a photo that connects to it. There would be two outcomes to this: 1) I can tell people about this game and they can immediately go outside and play/contribute. 2) A graph of all these photos and their connections grows over time as people add more photos (and it’d be situated on a map of the world, too) so you’d see these splotches growing and webbing out and merging all over the place. There could be games within this world, like find all the fire hydrants or use a chain of photos to connect the library to the bookstore. Or make the longest path or the shortest path or discover hidden photos by exploring an area until one of your photos matches the hidden one. (Going totally reverse order of how this should be motivated…) The best part of PhotoCity was that you could play it in a way that no other game can be played – by taking pictures with a camera – and it’s a new and crazy and kind of surreal experience. (The closest experience I’ve had is a friend Adrian’s ghost-hunting game where you walk around with a phone that tells you how close and in what direction a “ghost” at a particular GPS location is, and you wind up walking around a familiar environment following a totally abnormal path.) I would like to make this game, because the photography and the game elements and the system architecture would be really, really interesting to me. It’s also highly likely that, if successful, the data collected could feed computer vision researchers a new type of treat they’d never tasted before.

Prong #4: Secret projects

• Sketch-a-bit investigation (using grounded theory)
• A photo version of Sketch-a-bit

The thing that interests me in both cases is observing how other people will interpret and evolve the artwork of others. Sketch-a-bit has some really interesting examples of how artwork evolves or how games are played or how conversations take place (over time and between anonymous users who have no guarantee of actually finding their way back to a past conversation). I would love to have the photo version as a challenge for myself, and to see how others have different answers to the same posed question. I also really enjoy the unguided nature, how once you start the chain reaction, no further intervention is (or should be) needed, and it will likely go in directions never originally predicted. 

San Francisco and GDC

Just got back from GDC 2012, my 3rd (half-) GDC! Here's a general recap of games played, food eaten, experiences had...

Sketch-a-bit Birthday: Two years ago at GDC 2010, Google gave out Android phones, and immediately after the conference, Adam and I went to an Android hackathon at the Hacker Dojo and made Sketch-a-bit. Just under two years later, during this year's GDC, we hit 50,000 sketches! Happy birthday to Sketch-a-bit! For the record, I haven't gotten any free phones from GDC since.

Sketch-a-bit's launch in the Android market! Ah, this photo was taken with my new nexus one showing the screen of Adam's new nexus one.

Hotel: We (a bunch of people from my school) stayed in a suite at the Hotel Des Arts in a room that I shall refer to as "the Boob Room". It was weird art, but otherwise a good setup. Free wifi, muffins and coffee for breakfast, enough space for us all to work and sleep, about a half-mile from the convention center, and a reasonable price.

Hotel des Arts with rooms decorated by local artists.

Irish Pubs: This is not all that remarkable, but over a 4-5 day period, we twice had dinner at Irish pubs around the city. First at the Irish Bank next to the hotel, and then at the Elephant & Castle after seeing Indie Game: The Movie.

Indie Game: The Movie: The movie was fantastic. But I am going to talk more about the process and not the actual movie itself. I had supported it through Kickstarter a long, long time ago (and got a shirt that I then wore to the screening) and it was everything I hoped it would be! Whoever you are, you should definitely go see it. But here are some sort of extraneous points that I also think are interesting: The movie was going to be shown in Santa Cruz last fall (and I even had a ticket) but that screening got pulled because it was so good that it had to go to Sundance. The film crew filmed at a TIGjam in the past, so maybe I (and certainly people I know) are in the background of the special features extra footage stuff. At dinner at one of those Irish restaurants, Jeff told us how he became Executive Producer of the movie by supporting it at one of the super-high tiers on Kickstarter that they hadn't expected anyone to actually go for. Basically, the movie is great and there are more detailed and specific reviews of it out there, but in addition, it's further proof that cool stuff is happening in the world with games and movies and kickstarter and things that people (and entire communities) are crazy-passionate about. Also, making indie games reminds me of grad school.

Indie Game Festival: Speaking of indie games, the summits and tutorials GDC pass got us into the IGF award ceremony where a lot of stuff happened, but most importantly to me, my friends from UC Santa Cruz had their game Prom Week nominated. See how before I said making indie games reminds me of grad school? Turns out making indie games can be what you do in grad school!

Laaasserrrrsssss... at the IGF awards and Game Developers Choice awards

Games Played: (a.k.a. Awesome non-standard games with crazy interfaces or controllers) I played Johann Sebastian Joust and Fingle and Uprok and Mega-GIRP and games on people's phones that they handed me... And there was more epic Team Joust at a party, with 15+ players at a time. Madness!

Johann Sebastian Joust in the expo hall

Uprok start screen - press your colored pedal to join the game!

MEGA-GIRP - like Twister combined with the already horrendously hard GIRP

Conference Talks: There were a few things that were really cool. Like a scripting language for organizing and distributing tasks amongst humans in a crowdsourcing or alternate reality game setting! And a talk about random AI applications that I stumbled into and found 3 nice nuggets of nifty technology, ranging from a computer vision motion-blob-finding technique to neural networks to computing level of detail in a way not simply based on distance. And a beautiful talk from the guy who made Way and now works on Journey about people from anywhere connecting and collaborating and journeying together through games.

A scripting language for people!

Back to food again: I want to share with you my strategy for eating during a convention in SF: Walk  3+ blocks away from the convention center (like towards Market) and find a sandwich. Most people seem to get trapped within the 1-2 block range where everyone else and their coworker is eating. I guess I ate a lot of sandwiches. And a quesadilla in the Mission at 1:30am! And the best chocolate croissant I've ever had in the US at Le Boulange. And swanky mac and cheese! And crepes! And pork buns! I want to go back to SF and eat things.

The end: And that, my friends, is my recap of GDC.

Colored Point Clouds from Kinect

Here's how to use your Kinect as a chintzy laser scanner and capture a point cloud of your own self or your own stuff!

There were a few things on the internet about getting colorless point clouds, and then a preview of a book on Google books that had an image of some code... so retyped that code and hacked some things together to get a pretty simple point cloud capturing program.

Live Processing application with code and kinect display

What you'll need:

1. Processing (http://processing.org/)
2. OpenNI and the Processing wrapper for OpenNI called SimpleOpenNI (http://code.google.com/p/simple-openni/ and installation instructions: http://code.google.com/p/simple-openni/wiki/Installation)
3. This Processing code: http://www.cs.washington.edu/homes/ktuite/colored_point_cloud.pde

After you capture some point clouds (by pressing 'p' and then finding kinect_output.ply) you could go get PointCraft (http://photocitygame.com/pointcraft/) and use that to view and make geometry out of your point cloud.

Point cloud from Kinect loaded into PointCraft!

I feel the slightest bit bad for copying code from a preview of a book that is actually for sale and stuff (this one here: http://books.google.com/books?id=xJOxrWzJ5iUC) but the SimpleOpenNI examples were completely missing and all it really says is "omg you can use this function called alternativeViewPointDepthToImage() to get the depth image and the colored image from the kinect to line up!"

PointCraft Release!

We're officially releasing PointCraft!

Go to the website, watch some videos, sign up, and model things! 

It's a whole new 3D modeling experience! The goal is to make simplified low-polygon-count models that represent the ~million-point pointclouds.

Kathleen's Time Management Strategies

My co-intern, Stacy, from FXPAL a few summers ago invited me to be on a panel on Time Management this year at Grace Hopper! I spoke about "Productive Procrastination" -- making awesome things and sharing them with people, and this blog post talks about that a little bit and some other strategies of mine.

Productive Procrastination
When faced with the choice to get a head start on some not-terribly-crucial work or waste my time on Facebook, I occasionally opt for secret option number three: learn something new and make something for fun. I'll call this "productive procrastination". It's challenging to get started sometimes, so I start small and let myself get sucked into the flow of trying out just one more little experiment before stopping. I also share my discoveries and creations with others, usually by making mini-artifacts and posting them online. The benefits of this are huge: I have cool things to tell people about, I feel motivated and productive, which then often transfers to my "real" work, and I gain skills, ideas, and perspective that help me do my regular work better and faster as well.

A note on Hacking
When I say "productive procrastination", I mean coding/hacking for fun. Maybe even with someone else. And since the GHC panel was targeted mainly at women, I'll say now that I think more women should code for fun because of the extra skills and extra confidence gained. I would tell my younger self to code more outside of class. I wanted and still want to be a better programmer. (At this point, I think I'm quite good and capable of making awesome things, but I still want to improve.) I've gotten loads better by doing fun little projects for the purposes of just trying something new, which don't actually need to accomplish anything, and by learning from coding with other people.

Actual Time Management Strategies
Right, so, here's stuff I didn't talk about in the panel. Somehow, I manage to get things done, and here's how I do that. It's all pretty minimal because I don't want my organizational strategies to stress me out.

Disposable To-Do Lists
In grad school, it's kind of vague what I should be doing and how I should be progressing. I may write out a list of things to do and half-way through, change my mind and realize it's a horrible path and that I should scrap it and try something else. It's much less traumatizing to delete a temporary "todo.txt" or toss a piece of scratch paper than to abandon a nicely written list in a fancy planner or something. Thus, I mostly write things down on scratch paper, whiteboards, post-its and text files.

to-do!

OMG Post-its
Sometimes I write things on post-its. Small, actionable chunks, only. Then I stick them on my whiteboard above my desk, potentially grouped by priority or something, and when I'm sitting at my desk figuring out to do, I just pick a post-it and do it. Then I pull it off the whiteboard when I'm done! Or I pull all the notes off and make new ones if my overall direction has changed.

post-its! it took me 4 years to use my CSE note cube and then i acquired a second one.

Inbox Zero
You're not allowed to stress me out, email! Everything except things I need to actually deal with but don't want to deal with right away get archived. Gmail will find anything I need, and if there's a chance it won't, I'll forward it to myself with some extra searchable keywords.

the only email here is one from stacy telling me to write this blog post

New Trick: Google Calendar for tracking what I've finished
I've only been doing this for a month, but it's incredible. Kind of like writing post-its for what I need to do, I now write calendar events (on a special research calendar) for what already done. It's like if I save the post-its but they didn't take up physical space.

My research project has been grinding along for about a year now, but it's finally coming together and a big deadline is steadily approaching. I needed badly to ramp up the effort, and now I can look back and see that's exactly what has happened over the last 2 weeks. I also vaguely track what's coming up (like what I plan to do tomorrow, or when I should be collecting data and when I should be writing) but I don't let it bog me down. I also have all of my meetings and whatnot scheduled on another google calendar, because I have no brainspace to remember such things.

Slacking Off
I think not working sometimes is an important part of being productive. I am NOT super-productive for hours on end anyway, so I may as well fill the off time with something good. I'll test this hypothesis over the next month, but I think relaxing, eating, drinking beer, exercising, sleeping in, reading, spending time with friends to talk about non-work or to complain about work are excellent extracurricular activities.