Sunday, December 30, 2012

Creating a font from my handwriting

In 2012, kind of as a "resolution", I decided I would either try to make my handwriting neater and more consistent or create a font of my handwriting. (I think my handwriting is fairly neat, but it swaps which direction it's leaning or how close to cursive it is or what percentage of each word devolves into a scribble.) With one day remaining this year, I did the latter and figured out how to make my own font.

Real handwriting with pen on paper.

Digitized handwriting!


This tutorial from the Wired wiki gave me a clue as to what software to use and has some information about scanning and tracing your letters if you write them with pen on paper. 

I also had some trouble figuring out how to make my lowercase letters be the right size, but Chank's tutorial, which looks to be about 15 years old yet is still highly relevant, gave me some clues.
In which all the letters are unintentionally the same size...


  • Wacom Bamboo tablet (mine looks like this, but I think this is the current model)
  • Adobe Illustrator
  • Fontlab's Fontographer 5

Step 1: Draw some letters in Illustrator

Illustrator and I are pretty good friends. Better than me and Photoshop, anyway. It allows me to make some pretty nice drawings and diagrams. I use the brush tool and set it to "2 pt oval". It feels like a fat felt-tip pen. Unlike the pencil tool, it doesn't try to connect with and smooth out other brush strokes, because it deselects the stroke after you draw it. If you actively select an existing brush stroke (hold command, then click on the stroke) you can re-stroke an existing stroke.

Alphabet in Illustrator

Step 2: Get those letters into Fontographer

It's super easy to select a letter in Illustrator, copy it, and then paste it into the appropriate letter in Fontographer. The problem is that it will scale your letter to fit the whole space it allocated for that letter. (There are mysterious lines and margins, but I didn't figure them out.)

The trick I learned from Chank's tutorial was to use some dummy geometry to keep your letter height consistent. That's what all those pink lines are. The height of the dummy geometry should go from the highest point to the lowest point in your entire alphabet.

Copying the letter "A" and the dummy pink vertical line.
Pasting the "A" with vertical line into the font program.
Selecting and deleting the vertical line!

Step 3: Test out your new font!

To get a live preview of your font, go to Window > Open Metrics Window and type some stuff. 
I don't know why it's just showing the outline. It wasn't doing that earlier.

Step 4: Muck with the kerning and spacing

I don't know too much about typography. I know making it look good has to do with the spacing between letters and sometimes the spacing between particular pairs of letters.

In the Metrics window, you can click on letters and drag around their left/right spacing and the kerning thing.
Fixing the "y" to slide underneath other letters.

Step 5: Admire and install!

Before creating your final font files, go to Element > Font Info... and give your font a good name. I discovered that it doesn't like having numbers in the name.

Even though copying each letter into the font was kind of tedious, now I'm done and can write a blog post about it! You can download my font here if you are interested.

Font in Fontographer
Font preview in Font Book

Installing, at least on a mac, is pretty easy. Just open the font in the Font Book app, and it will let you install it. Then you can use it in other programs... maybe! My new font works in some programs, but not Powerpoint, which maybe requires fancier export settings.

Now that I know how, maybe I'll make more fonts in the future! Ones that capture specific moods or are designed for certain aesthetics.

Monday, December 24, 2012

Glue and Magnet Nail Polish

Rambling introduction!

I was at dinner with a friend who had nail polish on one moment and then didn't the next! She then enlightened me about peel-off nail polish and in particular, peel-off base coat that you can put any other nail polish on top of! I used to have lots and lots of colors of nail polish and paint all kinds of things, not just my nails, with it. These days, I have some purple and some teal polish. I don't like wearing it because I don't like taking it off or wearing it when it's chipping off. So I was somewhat intrigued by the prospect of peel-off base coat opening me up to a whole new/old life of colorful nails.

I learned right away that peel-off nail polish is not available here in Seattle. However, according to my friend at dinner, a grad student lady in Australia had acquired some and figured out what was in it and determined it was basically just kids' craft glue. I emailed another friend who happens to live/be/maybe-she-just-returned from Australia to inquire about the existence of such products over there, and she helped me track down the original post of the PhD student bloggress.  After identifying what was in it, original chemistry student lady made her own, and the message of "paint your nails with glue!" has now gone around the internet and propagated through several layers of grad students/nail-painting nerds all the way to me.

You could go look at her blog post, or you could look at pictures of me trying it out.

Painting my nails with glue!

This was the girliest AND most scientific night I've had in a while. 

Glue, ugly nail polish to poach bottle from, and some nail polish remover.

I got "tacky glue" because people kept saying, "it's basically Elmer's glue" and then they'd qualify by saying "PVA glue" or "craft glue" which made me think it was not Elmer's. And this was the only craft glue Fred Meyer had. Since that night, I've learned that actual Elmer's would also be fine, if not better, than this because the Tacky Glue is really thick PVA glue.

Insert glue into empty nail polish bottle, add a little bit of water, and mix it.
I used the nail polish remover to help get the yellow out of the bottle. Then I rinsed it with warm soap and water. You want a nail polish bottle so you can have the brush and not just be smearing glue on your hands with the bottle of glue itself.

Glue on nails. Waiting for it to dry and become clear.
I put glue on my skin as a kid... in order to let it dry and then peel it off like creepy/awesome fake dead skin. Just like back then, dried glue turns clear and stops being wet/tacky. Except the tacky glue is always ever so slightly tacky.

Ponder.... will this even work? Be careful to not get glue in your imaginary beard.

I got Nicole in on this whole ordeal, so right there we have yet another generation of grad student learning to paint her nails with glue.

Applying regular nail polish on top of the glue base coat! Actually it's not regular nail polish... it's magnetic. But more on that at the end!

Magnetic blue and sparkly festive red!

I had never experienced this peel off nail polish stuff before, so of course I tried taking it off the very same night I put it on. It worked pretty darn well!

There were little bits of nail polish left over, though. The solution: more glue. Make it so there aren't naked nail borders for the nail polish to stick to. Although another blog warned of not getting too much glue on your fingers because it'll make it too easy to peel off.

Painting my nails with magnets!

Another nail polish topic that came up that first night was the existence of magnetic nail polish. A polish that has magnetic stuff in it so when you hold a magnet over a freshly painted nail, a pattern emerges.

The company (Essence) that supposedly sells the peel-off base coat in Australia, but not here, does in fact sell the magnetic stuff here, so we were able to give them some business anyway. I got a dark blue/gray color and a starburst magnet pattern. Even though I didn't get all the starburst patterns looking that great, I am super entranced by the subtle patterns my nails had. (I painted them and peeled them off again...) It's the right level of distraction for me when I'm typing and going about my day: "Ooh, shiny..." vs. "Ahh! Why are there bright colors on my hands?!"


Tuesday, December 18, 2012

Ingress Day 1

Some portals near my house changed color over night and I saw the chats of other players scroll by. There are other humans actively playing this game! That is one thing it has going for it. And even though I complained about XM yesterday, it's alright if I just think of it as "pretty blue sparkles!"

We tromped all around in the morning... trying and failing to find breakfast. Intel gathered: 1) Einstein Bagels lied about its opening date. 2) Saley's Crepes is moving away. 3) B&O has moved away. 4) Thomas St. Bistro has no hours posted and might only be open on weekends.

Sometimes I'd pull out my phone and see if there were portals near by. I had collected all the sparkly nuggets I could until I found a portal to use them on. I finally found some at UW in Red Square, but it felt like I'd already captured Red Square through a different game... (here's a video of Red Square reconstructed from PhotoCity photos)

I finally used up some of my points to hack a few portals, collect some arcane items, and then deploy some shields on a different portal. I tried to link a portal to another portal, but I guess it wasn't close enough.

Then my hands got so cold that they fell off.
Brrrr! Hand stump.

Monday, December 17, 2012

Ingress Day 0

I got an Ingress invite, although I'm not sure I entered the activation code in anywhere. Anyway, I have it on a phone now and I tried it out briefly.

Keep in mind, it's cold and dark all the time in Seattle until Spring comes back in June.

The app looks very pretty. It's also trying to be way more epic than I care for. Just tell me your real purpose, app! Not this dressed up fake sci-fi.

*clickity clackity* THIS CHAT IS NOT SECURE! 

Blah blah blah. I believed the Blight in Vernor Vinge's A Fire Upon the Deep was some wicked intelligent mold. But this "exotic matter" I don't buy.

The tutorial leads me through some squelchy grass to slurp up energy. That's okay, I like it when apps change the way I interact with the real world.

Then I go through several more missions that have me standing in the cold, poking buttons on my phone. Hack the portal, collect the thing, deploy the thing, fire the crap, charge the resonator, poke poke poke. There are no actual portals near by so this is all done with a fake test portal. Blah blah blah. I'm just standing in one place except when I have to walk 5 feet to get into our out of the range of something. I can't internalize these mechanics because I don't know why I'm doing them.

I couldn't see any real portals nearby until I went back inside and poked at the map with warmer fingers. They're out there. I guess I'll go by those on the way home and try "playing" "for real".

Taking photos wasn't part of the game *SAD FACE* but I took one anyway.


Meme Impression Quiz

I've recently been learning about face detection/comparison/classification for grad school purposes.

Last week, I built a fun little quizzing application where three random meme faces are shown to the user, the user picks one and imitates it, and then the computer guesses which face the user is imitating.

It compares detected/aligned/cropped faces and features computed on those faces. Here are some side-by-side cropped impressions.
Detected/aligned/cropped face comparisons
It works! Not all the time, but most of the time I can make a face it'll recognize. One of the many next steps is to provide feedback about exactly HOW and WHY the best match matches. Then a person who is not very good at imitating the face could learn to get better.
Not bad, computer. Not bad.

Monday, December 10, 2012

Filling in the gaps with PhotoCity

My research in grad school has revolved mainly around games as a means of crowdsourcing the collection of diverse and useful data. I'm embarking on a new project that, like my original game PhotoCity, also combines computer vision and games. It will become a big chunk of my thesis. My original plan for this post was to write about that new project, after providing context by explaining PhotoCity. It turned out that I just wrote about PhotoCity.

Sidenote: Some game designer folks might read this and be concerned about how I'm using games for the wrong reasons. Gamification is a sin! Games with a purpose are questionable! I have more to say about how I think I'm using games for the right reasons (feedback! learning! mastery!) but I'm not going to go into it now.

This is the finish line graphic for The Big Race 3D.


Around 2006 (while I was working on The Big Race, actually) a project called Photo Tourism came out of UW and was then licensed by Microsoft as Photosynth. It had 3 parts:
  1. Use large photo collections from Flickr. It turns out people have photographed and shared enough images from enough different vantage points to reconstruct popular locations like Trevi Fountain and Notre Dame in 3D. 
  2. Perform magic (I can tell you more about the specifics of this, too, if you want!) to turn that hodge-podge of photos into a 3D scene. Release code online, thus enabling many hobbyists and many grad students to base their research off of this work.
  3. Make a slick viewer that lets you navigate the photos in 3D, because you've already recovered their 3D poses relative to each other and the scene. This is the part that Photosynth mainly focused on.  
A big idea of this work that carried over from this work was that there is so much information already out there on the internet! Look! Every photo of the Colosseum you'd ever want to take has already been taken! (Rome in a day is the scaled-up, even more impressive evolution of Photo Tourism, which attempted to reconstruct Rome from 100,000 tagged Flickr photos.)

Using Rome in a Day to make my point, the data is not all there already. 100,000 photos went in and something like 10,000 were incorporated into models. Trevi Fountain was one of the largest clusters with 1,900 images. Most "rome" photos on Flickr are of people and food and interiors... not images that you could use to reconstruct a city. Even if you take a different source of data, like aerial images or street view images, it's going to be missing information along SOME dimensions (time, point of view, level of detail). Information that a normal person with a camera phone could conceivably provide.

hello, trevi fountain. hello, adam and kathleen and bunny ears.
Adam and Kathleen are not all that impressed.
PhotoCity was this awesome project to crowdsource photos that could be used to reconstruct a location in 3D. Will people be able to rebuild, say, the entire UW campus, knowing that that's their goal and getting feedback along the way? Surprise answer: Yes!! Here's a video of the final UW model.

It was part game, in that your progress was measured by points (real 3D points that your photos had added to the 3D model) and flags (denoting territory on a map that you had contributed to). The game elements weren't as key as the feedback, though. PhotoCity was really a collaborative, interactive version of the original Photo Tourism that showed how every photo was incorporated into a model, what new points it added, where it fit in spatially. And except for the part that already existed to do the 3D reconstruction, I built it all.

Contributing in a way that adds up

The original motivation for PhotoCity was incentivization and accomplishing a task that depended on real people with real cameras walking around in the real world.

The more general motivation that rolls through my head these days is this:
  • Does the data exist for {x}? 
  • If not, can I go out and collect that data myself? Or augment existing data with my own?
  • How do I know I'm collecting useful data? 

For PhotoCity, {x} was photos that could be used to reconstruction locations around the world. Obviously there was a lot of the world not reconstructed within PhotoCity, but it was fun to travel to new places and try to start a new model, or in some rare cases, travel to a new country and add to someone else's existing model.

It wasn't just a game, it was an interactive system built on some really neat technology that people all over could contribute to and help grow.

Human-computer symbiosis 

That's a phrase one of my advisors uses to... sound impressive, I think. It evolved from him talking about Foldit but I don't think it's specific to Foldit anymore.

From my recent blog post on design patterns of crowdsourced art and the last little section, it probably sounds like my main goal is enabling creative collaboration and outlets for people to make useful contributions. I do build things in that direction, but there's more...

Wikipedia is the obvious example of an incredibly useful collaborative ecosystem. You write an article on something you're knowledgeable about, or edit or update other people's articles. It's now an amazing resource (also driving lots and lots of computer science research). In PhotoCity, you'd take a picture of some place you happened to be, or you'd seek out near by locations that needed more data and take pictures there.

The difference between Wikipedia and PhotoCity (besides text info about the world vs. photographic imagery) is that a computer processed your photo to decide where it fit in and how much novel information it provided. It's not just an ecosystem of human writers, editors, and browsers, it's an ecosystem with some structure from motion and a big, fat bundle adjustment at its core. When I put on my grad student hat, I care about humans contributing and collaborating with each other, but also with some underlying technology/system/algorithm/computer process. And sometimes the computer is wrong!

Up next (coming soon in another post)

I had the best intentions of writing this here.. it would help me figure out what I'm doing for the rest of the week. For now, though, you just get a bulleted list:
  • New domain (photos of facial expressions and appearance variations) 
  • Contribute face photos that fill in the facial frontier and make face detection, face recognition, and various facial classifiers better
  • Get feedback on how your face photos are advancing that frontier
  • Give feedback on how the computer should correct or question some of its faulty assumptions
I'd like to have more conversations about this. If you have things you want to say, say them here in the comments or via email or whatever!

Wednesday, November 28, 2012

Forgetting the Code Signing Password to Sketch-a-bit

There are 3 things I remember completely forgetting in my life. In that a) it sort of matters (mattered) that I remember these things and b) even many years later, I have NOT finally remembered. The first was my ATM PIN. It wasn't even a new pin. It just escaped my memory one day and was gone forever. I called Adam and was like, "Hey, did I ever share my pin with you? No? Darn." Later I called the bank and chose a more memorable one.

The second and third things I've forgotten are the code signing passwords to Sketch-a-bit and apparently Impressionist Fingerpaint. In my defense, Adam picked out and typed those passwords, but did show them to me. So they're not in my muscle memory anywhere at all.

Why this sucks

It means we can't update the Sketch-a-bit application at all.

It used to be that you could use a new key and the update would require people to remove the app and reinstall instead of just normal updating. Apparently this happened to Twitter and a few other apps and it was mostly inconvenient and embarrassing.

Now, security in the Google Play store seems tighter. If I try to upgrade my APK file, it yells at me and says, "The apk must be signed with the same certificates as the previous version."

It also means that we can't update Impressionist Fingerpaint, either. I have a feeling that password (since it was invented much more recently) will be easier/possible to recover from someone's brainmeat, though, and it doesn't have very many users.

But I wanted to make Sketch-a-bit suuuuper awesome!!

I still do!! Ahhh, I wish I could just upgrade the main app!

Now that we've seen how people cope with starting from another random user's random image, I kind of want to see what artistry people are capable of if they get to have more choice and agency. To achieve this, I was thinking of having a gallery and letting people select which sketches they want to start from.

I also want to make the system a little less anonymous by having random user ids instead of no identity whatsoever. Right now I can hypothesize about the same artist drawing several images in a row, but there's no way to know for sure.

At a retreat for my lab at UW, I learned that the color red was probably the first color people were aware of after dark and light. Here's a link to a wikipedia page with more info. Adam and I thought it would be totally awesome to introduce RED into Sketch-a-bit, then.

Decisions, decisions

The questions now are whether or not to spend more time trying to remember those passwords. We tried cracking one and learned that it's probably longer than 5 characters.

And then whether or not to release a Sketch-a-bit 2 and how many features to add. Just the bug fixes? Or more of those desired features that I don't really have time to implement? Seems like an entirely new version warrants some significant changes.

Good enough solution: put the APK online

I realized on the bus this morning that this is what I should do to get the fixes that I'd made to the two people who were requesting them. So now it's online. In a slightly secret location. I want to add the user ids and something on the server that tracks users and version numbers, and THEN I'll publicize the location.

Currently there are 3 people out there in the world capable of drawing in red. Here's what one of them made!

What have we learned here?

If you're going to release some code at 2am right before going to sleep, or 7pm right before grabbing a beer, PICK A MEMORABLE PASSWORD for your code signing key. Make a mnemonic for it. Write it down on a post-it. Email the password or the mnemonic or both to yourself. Whatever. In this case, I think losing the password was more trouble than having other people steal it and publish apps under our name.


Life moves on and we're still enabling people to make awesome things!

Tuesday, November 27, 2012

How to train your classifier (on a mac with opencv)

Or rather, how to experiment with using OpenCV to train a classifier and ultimately fail.

The following two links were my guides. They might make better guides than what follows here, but I'm going to write this stuff down for myself.

I wanted to try training my own OpenCV Haar Cascaded Classifier. OpenCV comes with some pre-trained ones for faces and body parts and whatnot. For fun, I tried to train it to recognize my cat Jpeg, which Picasa humorously recognized as a person in a handful of photos last month:

picasa now apparently detects cat faces

Step 1: Download pictures from my own Flickr account

I use this python program to get Flickr image urls:
I added my flickr API key and changed the main of that python file to look through MY pictures for things tagged 'cat'. 

if __name__ == '__main__':
    pix = photos_search('16977922@N00', False, '', '', 'cat')
    for p in pix:
        print p.getLarge()

Then, because bash is one of my best friends these days:

python > cat_pix.txt && for i in `cat cat_pix.txt`; do wget $i; done

Actually the first time I downloaded the pictures, I wanted to resize them with Imagemagick's mogrify (in-place image editing) command so I ran:

for i in *.jpg; do mogrify -resize 1024x1024 $i; done

Step 2: Mark "objects" to make positive examples

One of the links above has an object marking program for windows. This link has one that I was able to compile on my mac. It's kind of amateur code and I have vague plans to rewrite it.

I compiled it with the following command after commenting out two useless includes that were complaining at me. BRIEF INTERJECTION: I just installed OpenCV with Macports. If you do that, your OpenCV stuff will probably be in the same place. Installation of OpenCV didn't work properly until I made sure my macports was up to date.

g++ -I/opt/local/include -L/opt/local/lib -lopencv_core -lopencv_highgui -lopencv_imgproc -lopencv_objdetect objectmarker.cpp -o objectmarker

For context, I'm inside a directory of cat images. Running that objectmarker program is as so:

../objectmarker positive_examples.txt ./

Right, so, run that program from the directory with your images in it because the code doesn't actually build the right path to the image file if you run it from a different directory, even though it looks like it intended to. Then click boxes and press the appropriate SPACE and Capital B keys until all the objects/cats are marked.

Step 3: Get negative examples by making a list of images that don't have the object in them

I wound up using images from flickr tagged "eddie", which is the name of my mom's chubby chihuahua. Pictures of other dogs named Eddie were also included, but that didn't matter. The only thing that mattered was that Jpeg was not in any of the images. Which is true because he's never met any dogs.

The command went something like this:

ls eddie/ > negative_examples.txt

Step 4: Fix the paths

My images were in folders called "jpeg/" and "eddie/" and some of the jpeg images didn't actually have jpeg's face clearly shown. I had to change the image paths in positive_examples.txt and negative_examples.txt from "eddie_4787878333_4dd7bcd3d8_o.jpg" to "eddie/eddie_4787878333_4dd7bcd3d8_o.jpg". I did that with vim and the mass insert thing!

Step 5: Run opencv_createsamples

Finally! Getting to the running of the actual training code. Almost.
So I had these two files:
  • positive_examples.txt
  • negative_examples.txt
Negative examples just had image paths in it. Positive examples had image paths and # of objects in the image and the bounding rectangles for each image. For some reason, you have to convert your handy list to a binary representation to feed to the next step. Actually that reason is that you can use the createsamples program to take one image showing off your object and warp and distort it a whole bunch to create artificial training data. But if you actually have many positive examples of your data, you need to run this to just process it and get it in the right form.

I had 19 positive examples and 17 negative images so I ran this:

/opt/local/bin/opencv_createsamples -vec jpeg_pos.vec -info postive_examples.txt -show -w 32 -h 32

One very important thing I learned is that you should not put a number like 100 into the height and width fields because that will make your computer freak out (have a lot to process) at the next step.

Step 6: Run opencv_traincascade

Finally, for reals. Let's do it. 

/opt/local/bin/opencv_traincascade -data output_jpeg_cascade/ -vec jpeg_pos.vec -bg negative_examples.txt -numPos 19 -numNeg 17 -w 32 -h 32

Your -w and -h numbers have to be the same. When I tried them at 100, the program was trying to use 6GB of ram and making very slow progress. When I knocked it down to 32x32, it finished in under 30 seconds. I hear from those other links on the internet that training can take 5 days with 2000 positive and negative examples. Maybe that'll be me some day, and maybe it wont.

Step 7: Detect cat

I hacked some code online that used the face detector classifier and replaced "haarcascade_frontalface_alt2.xml" or whatever with "output_jpeg_cascade/cascade.xml". I don't remember where the code came from so I don't have a good link that particular resource.

In many cases, it completely failed to detect Jpeg. Even though this image was part of the training:

In another instance, it got different ends of the cat confused: 

Voodoo magic

This was the first time I ever tried anything like this. There's probably a lot I could learn about how to make it work better. At the same time, I wish I could teach the classifier more specific things to look for, or have it tell me exactly why it thinks the cat butt is the face. Feeding lots of data into a black box and crossing my fingers doesn't seem like a fun approach.

Sunday, November 25, 2012

Design Patterns of Crowdsourced Art

Sometimes I try to google "crowdsourced art" in hopes that I'll find a comprehensive list of all such projects. Maybe there's something out there that I'm unaware of... that'll be on that list... that will really make my day or something.

What I'm finding instead is either stuff that I've already heard about or things that fits into the same general pattern, one that I'm frankly getting a little tired of.

So, I'm going to share my own Comprehensive List. And then talk about the most common pattern I've spotted, as well as a few alternative ones.

30 Crowdsourced Art Projects

(Organized by how people contribute)

Drawing w/ spatial context
Webcam photo
Animating/drawing multiple frames
Cats (er, I mean drawing and writing..)
Playing w/ data
Full project remixing w/ assets
Mixed: photos/videos/text/drawings

Design Patterns

Many of the above are like, "Hey everybody, do a thing that's the same thing everyone else is doing, but since you did it, it'll be sorta unique to you! Then we'll put everyone's things together and maybe it'll be kinda neat to see the similarities and differences!" When everyone goes out and does their own thing, and then the results get smooshed back together, I call it the flat contribution model.

Flat Contribution Model (v1)

The Life in a Day project is a great example of how tons of effort and creativity were filtered down into one consumable artifact. An artifact that is meant to accentuate the artists' differences, yet also our shared humanity and whatnot... The idea was that on July 24, 2010, people all over the world would record videos of their daily lives, maybe answering some questions like "What do you love?" and "What do you fear?" and those would be curated and edited into a single movie. In the end, 80,000 submissions comprising 4,500 hours of material were edited down to 92 minutes. Whoa! Since a mere 0.03% of submissions are actually in the movie, it hardly captures the diversity it could have. On the other hand, 92 minutes was good enough for me as a viewer/consumer. There was a TON of material, a lot of which was likely crap, and some amazing editing went into constructing a coherent viewing experience.

The gist of the model is this: There's some prompt, people go out and do it on their own, submit their creations, and then some curator accepts or denies their contribution and integrates it into the One True Artifact.

Flat Contribution Model (v2) 

While I may have been too snarky about Life in a Day, I reaallly like the Johhny Cash Project. Maybe because I'm impatient and the artifact here is only a few minutes long. Maybe because I really like looking at hand-drawn illustrations. Or maybe because I can actually contribute to this project still; the Life in a Day thing is over and I'm not a video person anyway.

The idea with Johnny Cash is that people re-draw frames of a music video in their own artistic styles. Then when you watch the video, different aesthetics flicker in and out, like realistic and impressionistic and sketchy and haunting and fan art and non sequitur... Here's the link again if you don't want to scroll up and find it. Go take a look!

Some differences for version 2 of this flat contribution model are that a) people work from a common input artifact (frames of a video) and b) they're not working totally in isolation. You can see what other people have drawn before you start drawing your own frame. Working from a common artifact is a lot like the map part of map reduce, and there's research on how to actually use the map reduce model with humans.

Hierarchical Contribution Model

I was moving in this direction by mentioning that in the Johnny Cash Project, people can see what other people have done so far, and that may or may not influence how they draw their own frame.

About 2.5 years ago, Adam and I made an Android app called Sketch-a-bit. It's a gray-scale drawing application, which gives it a similar aesthetic to Johhny Cash, but any time you want to open the app and sketch something, your canvas isn't blank, but an image drawn by another user. Every uploaded sketch is added to the global pool of images for another user to randomly download some day. Every sketch also remembers its parent sketch, so the lineage of each drawing can be traced back to the one white canvas that we seeded the system with. Here's a cool skull drawing that you can trace the ancestry of -- look out for a "SHDH" as a children of one of the ancestors. ;)

The gist of this model, and of Sketch-a-bit, is that people build off other people's work, and the "artifact" -- the 80,000 images that people have drawn so far -- is never really done. It's this thing that's alive and continues to grow and evolve over time.

You may have noticed that a large percentage of those projects described above have Aaron Koblin as their instigator/creative director. His work is super cool, but I'm proud of him for finally making something hierarchical -- The Exquisite Forest -- which debuted this summer, about 2 years after Sketch-a-bit.

The projects in the "voting/consensus" category do follow this hierarchical (or evolutionary) model, but instead of contributing by executing some kind of artistic vision, people contribute simply by voting for their preferred automatically generated offspring.

Of the projects listed above, I would say that Sketch-a-bit, Exquisite Forest, Drawception, and especially Monster Mash, are all somewhat related to the old-timey surrealist parlor game Exquisite Corpse. So this hierarchical contribution model isn't really new, I just haven't seen as many instances of it as I would like.

Upside-down Contribution (Distribution?) Model

I totally just invented this one right now based on the two projects in the "Playing w/ data" category. In the House of Cards project, point cloud data from laser scanning Thom Yorke's face is released on google code and anyone can go make stuff with it. In the White Glove Tracking project, Michael Jackson's gloved hand is hand-tracked (by the crowd) through an entire music video of Billie Jean, and then the tracked hand data is released for people to play around with.

Instead of many people contributing to one artifact, many people consume one artifact (the data) and produce many different things from it.

Towards... an Ecosystem of Ecosystems! 

This deserves its own blog post some day. The idea is that I care about how and what people create based on what they've seen other people create, and that the creation process itself is more valuable than the artifact that you get at the end. We've seen a bunch of crowdsourced or collaborative art projects now, some of which are like little ecosystems in which individuals evolve each other's artifacts. Why not evolve the very nature of those creative ecosystems themselves?


Adam and I wrote a paper about our some of our findings with Sketch-a-bit: Emergent Remix Culture in an Anonymous Collaborative Art System. The pictures are from a slide deck I made for the talk I gave at a Human Computation in Games and Entertainment workshop at AAIDE

Monday, May 14, 2012

In which I battle MTurk External Hits and (eventually) WIN

Here are some lessons I learned recently about Mechanical Turk and launching external hits. Painful, painful lessons. Makes me not want to play around with MTurk too much in the future.
  1. The only way to officially make an "external hit" is through the command line tools or maybe one of the fancy tools built on top of the command line tools. You can't do it through the mturk requester website unless by "external" you mean a link to another website and maybe asking your workers to copy and paste some field back into the mturk website.
  2. The command line tools are scarily old. Even though this link was last updated OVER FOUR YEARS AGO, it seems to be the way to go. Here are brief instructions about getting set up: 
    1. If you're not on windows, don't click the "DOWNLOAD" button because that will lead you astray and fetch you a .exe! Look on down the page for Unix Downloads / Command Line Tools without JRE.
    2. No need to compile or really install anything because it's all shell scripts and java code. That's nice, at least.
    3. Go into bin/ and put in your access code and key. ALSO CHANGE http to https, because Amazon is fancy and secure these days but doesn't update their mturk developer code!
    4. Do this: export JAVA_HOME=/usr
    5. Okay, great! Now you can run things like ''... 
    6. Also note you can play around in the sandbox by commenting out lines in The sandbox balance should be like, $10,000... Ooooh Ahh!
  3. The examples are okay, but not informative enough to like, have an external hit with more than one variable.
  4. Don't be a fool, like me, and keep running ./ to add more hits and then wonder why the things you see on the webpage appear to have no bearing on reality/the things you just changed. Adding new hits via literally adds new hits and doesn't replace old ones, even in magical sandbox land. So when you refresh to look at a particular hit, you'll get a random one from all the ones you've created.
    1. Reset account is your friend! cd ../bin/ && ./ -force
  5. FU#$KING ampersands!!!! url=${helper.urlencode($urls)} means nothing to me when it says 'url' in three different places AND refreshing the page gives me random unintelligible crap that may or may not also still say 'url' somewhere. 
    1. When it says... [Fatal Error] :6:117: The reference to entity "centerLong" must end with the ';' delimiter.
      [ERROR] Error creating HIT 1 (44.5628547668457): [6,117] The reference to entity "centerLong" must end with the ';' delimiter. 
    2. You say... &amp; like so: <ExternalURL>$centerLats&amp;centerLong=$centerLongs&amp;bundleId=$bundleIds</ExternalURL>
  7. I thought the hit batch input file needs tabs, and commas don't seem to be okay. That is pretty annoying because I made my vim not use tabs AGES AGO.
The end. The rest, like the code for pulling out your variables from the URL and then submitting your external hit form data seems pretty self explanatory.

Tuesday, May 8, 2012

CrowdCamp 2012 reflections

I went to this amazing workshop this weekend at CHI. It was a 2-day work shop where we came up with lots of ideas and pitches, broke into small groups, and MADE THINGS. We also played random physical games including an intense match of dodgeball that came down to Steve Drucker vs. a group of five. I'm hoping someone else summarizes all the amazing projects. I'm just going to write down some reflections based on my group's project and dinner discussions.

Context: My group's goal was to create an interesting creative artifact, like a book or collection of stories or photos. We eventually settled on a theme of trying to capture a snapshot of Mechanical Turkers lives and the interesting stories they each want to tell. We made a few separate mturk hits asking people to submit a photo (either through email or through a webcam) and write a story, based on one of a number of different prompts. Throughout the weekend, we experimented with many different story prompts, yielding a variety of responses, some more interesting or more authentic (and not ripped off wikipedia) than others. There will be an official blog post soon, but here's a link (that won't be around forever) to some of the pictures and stories:

chi and crowdcamp

The three things I want to discuss, then, are 1) feedback cycles 2) spontaneity and creativity and 3) the end artifact versus the process. Also, for fun, jumpstarting memes at the end of the post!

Feedback Cycles

An initial idea that kind of got buried under other ideas was to have creative cycles or iterations in which the people making (pieces of) the final creative artifact built off of or saw other people's work. If I could go back in time, I would push that idea more, but I have my own examples of it working awesomely.

There's a fun little crowdsourced book of cat stories ( put together by Bjoern Hartmann, one of the other guys at the workshop. I adore the idea behind it and that the book actually exists, and I own a copy. But frankly, I'm disappointed in the quality of the cat stories. I'm sure these cats, my cats, other cats have done far more interesting things. Perhaps the silliest thing my cats have done is "drive" their cardboard firetruck into the bedroom in the middle of the night. PICTURE! I had to go look at photos to come up with that story there. I am fairly sure the cats in the book have done more exciting things than what ended up in print.

The photos and stories that came back from workers this weekend (excluding the fake ones) were kind of bland. My hypothesis is that if you ask someone on the internet (a mechanical turk worker specifically) to write/draw/do something interesting completely out of the blue, they probably won't come up with something that interesting. First, it's kind of awkward to be paid to be creative (but some of our turkers were pretty creative, so maybe that's not a huge problem). Second and more importantly, they're not primed to say anything interesting right then and there.

Spontaneous Creativity

The corollary to the above hypothesis is that you could very easily prime these people to be creative by showing them other people's stories. Maybe they'll try to write something less boring than someone else. Maybe another person's story will trigger a memory. Maybe knowing someone else (especially another turker) might see it will raise the bar.

It's also possible that the prompts just weren't the right ones to spark creativity. Consider: "Write a story about your cat" vs. "What's the most surprising thing your cat has ever done?" We looked at some of the original photos people took in response to prompts like "your most prized possession, your hands, your shoes" and came up with new themes that were a little more flippant, but elicited some enjoyable photos. These new prompts included "make a hat" and "make a scene out of 2-5 objects around you". Personally, I enjoyed the photos and stories from these prompts much more.

Artifacts vs Process

There was a comment at dinner about a "saturation" of stories in the world. I'm not sure I agree, but I definitely think there's more being produced than there is time to consume it. Consider a recent film called "Life in a Day" that was compiled out of thousands of youtube videos all filmed on the same day. It took a fancy director to turn 4,500 hours of video into a manageable 1.5 hours. Now it's all about the artifact, but the experience of submitting a video has value, too, even if the final video was never made, I would argue.

I am most interested in designing opportunities to _participate_ in creative, collaborative experiences. You could take up drawing, or you could take up drawing within a social and constrained framework, like drawing a picture of an animal every day for a month and sharing those drawings with others. Like this!

Jumpstarting Memes

The idea of jumpstarting memes using mechanical turk came up during a coffee break. Our photos responses to "make a hat out of something" are pretty good. Kurtis Heimerl (had to stalk list of attendees and google images to remember name) put some mturk jobs out there asking people to take 2 photos, one where their clothes were on properly, and one where their clothes were on backwards but they were facing backwards, too. By the second day, he had a handful of pretty hilarious photos. If you ever see such a meme on the internet, know it started at CrowdCamp 2012!

Friday, March 23, 2012

The tines of my heart: three or four flavors of head-crabs

What kind of ridiculous title is that?! I am referring, of course, to Derek Yu’s indie game rant about head crabs, and also Adam calling my trio of proposed research interests “prongs”, which has then morphed into “tines” because forks have tines. And then there’s an extra one that doesn’t fit into a research-I-do-at-school category.

Prong #1: The complete pipeline

I’ve worked on two research projects now that actually fit together quite nicely. PointCraft feeds directly off the output from PhotoCity and could even benefit from PhotoCity’s revival, while the artifacts generated in PointCraft could in fact feed back into PhotoCity. Take pictures of something you care about that get used to generate a point cloud, trace the point cloud in 3D to get a cleaned up solid mesh, and if necessary, go back out and take more photos to capture the details that the are missing from the mesh. The whole process is collaborative so that a single person need not follow the entire pipeline unless she wants to, but any person can provide useful input at any point along the way: by taking photos, but pruning out erroneous photo matches in the point cloud, by modeling a mesh, and by marking the areas of the model that need more data.

Prong #2: Repurposing “gamification” for good

Currently, when crowdsourcing researchers hear the term “gamification”, they think of how to incentivize people to do work without having to pay them. I advocate borrowing rule systems, mechanics and interfaces from games to design better (richer?) crowdsourcing systems. Not for “motivation” or to candy coat an otherwise dull task. Crowdsourcing via actual money is useful, effective, and extremely valuable to study the economics of, and the absolute wrong thing to do is to inappropriately replace money with “fun” via what I shall refer to as manipulative gamification. This type of gamification serves to provide extrinsic motivation*, essentially changing the user himself to make him “want” or “need” to perform a task. Human beings seek purpose and want to contribute to goals larger than themselves, and there are certain kinds of tasks (e.g. creative tasks) that people are already driven to do, that need not be financially compensated and in fact would be difficult to assign monetary value.

There is a guy, Daniel Pink (who possibly ripped off someone else’s ideas and rewrote them in a more digestible fashion) who states that the three components of intrinsic motivation are: autonomy, challenge/mastery, and purpose. Games totally nail the first two on the head, and to combine them with a purpose gives you the whole freakin’ package.

I can think of a few groups of related crowdsourcing systems.
1.     Shallow games with a purpose: I don’t actually mean shallow in a bad way (though I do mean manipulative gamification in a bad way). I mean that the action the player takes isn’t that complicated and thus there’s not much of an opportunity to be challenged and develop mastery. Games like this include many of Luis von Ahn’s games, like the ESP Game, Peekaboom, etc. The verification mechanisms in these games are really clever, but I personally don’t get anything out of playing except knowing I’m contributing to a good cause.
2.     Inappropriate use of game mechanics: I’m not going to name names right now (because I don’t feel like looking it up) but I saw a talk recently that suggested using the slingshot mechanism from angry birds to place the appropriate geometric concept on the correct part of a diagram. If you’re just supposed to put the piece in the right place, why use a damn slingshot? It’s just going to make things imprecise and terrible for the kids who know where exactly to put the piece. I’ve also seen a game about typing a word so a beaver can walk over a log without falling into a gap? And matching words while simultaneously playing some sort of space shooting game? It is extremely important to have the mechanics actually fit with the goal, otherwise you’re probably doing whatever Jesse Schell was complaining about (in his rant at the latest GDC (2012) when he lit a $50 bill on fire) and wasting people’s money and time by having no idea what you’re doing.
3.     Complex crowdsourcing tasks that aren’t games: Wikipedia and Open Street Map and other things that people contribute to out of the goodness of their hearts hit those same motivation bullet points we mentioned earlier, but they’re not called games. That’s fine! That’s good! But they could be games. (Possibly, or it might make them seem too cheesy and drive people away. But it could drive other people in!) (Actually I think there’s something particular about providing information/knowledge that means you should take things seriously and not turn it into a game) Think about it like this: take a system like one of the above, see if any of the actions that people do map onto actions that players commonly perform in games, and then reframe the task as a game. Thinking about something as a game could give people a familiar handle to latch on to so they can understand it better. It could also attract people who are okay with the role of “gamer” but don’t want to take on the new, more specific, official-sounding role of “contributor to Wikipedia” or “mechanical turk worker”. Maybe that’s too much responsibility, whereas a game seems more accessible, easier to try out.
4.     Complex crowdsourcing tasks that are games: This category is TOO SMALL and that is why I care about this at all. I can think of multiple scientific discovery games (just a few years ago, these games weren’t numerous enough to have their own category but I think they are now!) including Foldit, EteRNA, Phylo, and the unreleased DNA Game that my lab is working on. There is also my previous game, PhotoCity, which is about crowdsourcing the photo-collection process by having players go out into the world and use real cameras to take photos. Luis von Ahn is also developing a language learning game called DuoLingo that will magically translate text AND teach its participants a new language. Of all these games, PhotoCity might be the simplest in terms of having parallels to traditional games – you try and capture territory on a map and turn the flags into your team’s color and maybe spawn more flags – because the scientific discovery games especially have had to develop entirely new interfaces and come up with really useful feedback mechanisms.

Popping off the stack, or getting out of that crazy list of crowdsourcing things (which doesn’t even include the systems with monetary incentives)…

My point is that there is a lot to borrow from the world of game design that can help structure a crowdsourcing task. Games are all about mastery, developing skills and ramping up the challenges according to your skills, and if we want to crowdsource more complicate things besides labeling images and transcribing text (or breaking big tasks into chunks a la map reduce) we need environments that let them develop their skills over the longer term. ...To be expanded upon!

*There’s something complicated about extrinsic motivation and the badges/leaderboards/points/etc. that get shimmied into other scenarios. It’s not all bad, but I think the good that can arise has to do with these things providing appropriate feedback or some other positive feeling, like discovering the secret recipe in Foursquare to unlock a particular quirky badge. 

Prong #3: World-wide photo graph

This idea stems from the fact that PhotoCity is ridiculously hard to play anywhere outside of UW, Cornell, and a few other choice locations that dedicated souls managed to seed successfully. After telling people about the game, they’d ask, “can I play?” to which I’d say NO, and then feel really bad inside. The servers are down, there’s no disk space, there are no seeds here, the buildings are too shiny, or too featureless, or too tall, blah blah blah. That is a pretty depressing motivation; so let me try to explain what this inspired. It’s the 3D reconstruction technology’s fault that certain environments work so poorly. There is still probably enough information in these pictures to match them to nearby pictures, though, without trying to produce a 3D model (which requires a LOT of matches).  I think there could be a lightweight version of PhotoCity that lets anyone anywhere take a single photo, and if it’s near something else, it matches. If not, it just dumps in in an approximate location on a map, and maybe someone else will come a long and take a photo that connects to it. There would be two outcomes to this: 1) I can tell people about this game and they can immediately go outside and play/contribute. 2) A graph of all these photos and their connections grows over time as people add more photos (and it’d be situated on a map of the world, too) so you’d see these splotches growing and webbing out and merging all over the place. There could be games within this world, like find all the fire hydrants or use a chain of photos to connect the library to the bookstore. Or make the longest path or the shortest path or discover hidden photos by exploring an area until one of your photos matches the hidden one. (Going totally reverse order of how this should be motivated…) The best part of PhotoCity was that you could play it in a way that no other game can be played – by taking pictures with a camera – and it’s a new and crazy and kind of surreal experience. (The closest experience I’ve had is a friend Adrian’s ghost-hunting game where you walk around with a phone that tells you how close and in what direction a “ghost” at a particular GPS location is, and you wind up walking around a familiar environment following a totally abnormal path.) I would like to make this game, because the photography and the game elements and the system architecture would be really, really interesting to me. It’s also highly likely that, if successful, the data collected could feed computer vision researchers a new type of treat they’d never tasted before.

Prong #4: Secret projects

  • Sketch-a-bit investigation (using grounded theory)
  • A photo version of Sketch-a-bit

The thing that interests me in both cases is observing how other people will interpret and evolve the artwork of others. Sketch-a-bit has some really interesting examples of how artwork evolves or how games are played or how conversations take place (over time and between anonymous users who have no guarantee of actually finding their way back to a past conversation). I would love to have the photo version as a challenge for myself, and to see how others have different answers to the same posed question. I also really enjoy the unguided nature, how once you start the chain reaction, no further intervention is (or should be) needed, and it will likely go in directions never originally predicted. 

Sunday, March 11, 2012

San Francisco and GDC

Just got back from GDC 2012, my 3rd (half-) GDC! Here's a general recap of games played, food eaten, experiences had...

Sketch-a-bit Birthday: Two years ago at GDC 2010, Google gave out Android phones, and immediately after the conference, Adam and I went to an Android hackathon at the Hacker Dojo and made Sketch-a-bit. Just under two years later, during this year's GDC, we hit 50,000 sketches! Happy birthday to Sketch-a-bit! For the record, I haven't gotten any free phones from GDC since.

Sketch-a-bit's launch in the Android market! Ah, this photo was taken with my new nexus one showing the screen of Adam's new nexus one.
Hotel: We (a bunch of people from my school) stayed in a suite at the Hotel Des Arts in a room that I shall refer to as "the Boob Room". It was weird art, but otherwise a good setup. Free wifi, muffins and coffee for breakfast, enough space for us all to work and sleep, about a half-mile from the convention center, and a reasonable price.

Hotel des Arts with rooms decorated by local artists.
Irish Pubs: This is not all that remarkable, but over a 4-5 day period, we twice had dinner at Irish pubs around the city. First at the Irish Bank next to the hotel, and then at the Elephant & Castle after seeing Indie Game: The Movie.

Indie Game: The Movie: The movie was fantastic. But I am going to talk more about the process and not the actual movie itself. I had supported it through Kickstarter a long, long time ago (and got a shirt that I then wore to the screening) and it was everything I hoped it would be! Whoever you are, you should definitely go see it. But here are some sort of extraneous points that I also think are interesting: The movie was going to be shown in Santa Cruz last fall (and I even had a ticket) but that screening got pulled because it was so good that it had to go to Sundance. The film crew filmed at a TIGjam in the past, so maybe I (and certainly people I know) are in the background of the special features extra footage stuff. At dinner at one of those Irish restaurants, Jeff told us how he became Executive Producer of the movie by supporting it at one of the super-high tiers on Kickstarter that they hadn't expected anyone to actually go for. Basically, the movie is great and there are more detailed and specific reviews of it out there, but in addition, it's further proof that cool stuff is happening in the world with games and movies and kickstarter and things that people (and entire communities) are crazy-passionate about. Also, making indie games reminds me of grad school.

Indie Game Festival: Speaking of indie games, the summits and tutorials GDC pass got us into the IGF award ceremony where a lot of stuff happened, but most importantly to me, my friends from UC Santa Cruz had their game Prom Week nominated. See how before I said making indie games reminds me of grad school? Turns out making indie games can be what you do in grad school!

Laaasserrrrsssss... at the IGF awards and Game Developers Choice awards

Games Played: (a.k.a. Awesome non-standard games with crazy interfaces or controllers) I played Johann Sebastian Joust and Fingle and Uprok and Mega-GIRP and games on people's phones that they handed me... And there was more epic Team Joust at a party, with 15+ players at a time. Madness!

Johann Sebastian Joust in the expo hall
Uprok start screen - press your colored pedal to join the game!
MEGA-GIRP - like Twister combined with the already horrendously hard GIRP
Conference Talks: There were a few things that were really cool. Like a scripting language for organizing and distributing tasks amongst humans in a crowdsourcing or alternate reality game setting! And a talk about random AI applications that I stumbled into and found 3 nice nuggets of nifty technology, ranging from a computer vision motion-blob-finding technique to neural networks to computing level of detail in a way not simply based on distance. And a beautiful talk from the guy who made Way and now works on Journey about people from anywhere connecting and collaborating and journeying together through games.

A scripting language for people!
Back to food again: I want to share with you my strategy for eating during a convention in SF: Walk  3+ blocks away from the convention center (like towards Market) and find a sandwich. Most people seem to get trapped within the 1-2 block range where everyone else and their coworker is eating. I guess I ate a lot of sandwiches. And a quesadilla in the Mission at 1:30am! And the best chocolate croissant I've ever had in the US at Le Boulange. And swanky mac and cheese! And crepes! And pork buns! I want to go back to SF and eat things.

The end: And that, my friends, is my recap of GDC.