Friday, January 24, 2014

In which I attempt to clone Imgur

I've been working on a crowdsourcing project called the Face Frontier for about a year now. It's primary goal is to amass a new database of facial expressions that spans lots of expressions (beyond the basic six ones of joy, fear, anger, sadness, surprise, and disgust) as well as many different types of faces and many different lighting conditions. We're building the database from scratch (collecting it through the crowd by asking people to take photos of themselves) so that we can have labels for expressions.

The main component of the Face Frontier is the website: (The other main component is the server side computer vision, which trains expression classifiers and handles the face swapping.)

I have changed the website's design A LOT. In the past week, I redesigned it yet again to make it look a lot like imgur.

One of the things Face Frontier does is let you swap your face with a meme.
It'll also let you swap memes with each other...

So if you swap the original (er, more like, version 5.67) Face Frontier...
Previous face frontier
 With imgur...
 You get...
NEW, imgur-inspired face frontier!

The basic components I copied from imgur were:

  • grid of interesting and recent and highly rated content
  • being able to interact with that grid and rate stuff and get involved right away
  • clicking on something on the grid to get more information and rating details about it
  • boxes on the side for 
    • adding your own content directly 
    • getting at content from a different axis
  • prev/next navigation with the keyboard arrow keys when viewing a specific piece of content, so it's possible to just browse content forever... 

Short term results:
  • Lots of positive feedback from the friends who pushed me to do this redesign in the first place
  • A little bit of stickiness (30-40 pictures a day), people actually using it, without it being a direct response to me sharing something online

Obviously I need to get those numbers up so that this dataset can be REALLY BIG and REALLY USEFUL. There's still a lot to improve and a lot of features that would probably make the whole thing even more satisfying... but currently, I'm quite happy with it!

Tuesday, January 7, 2014

What's up with Photosynth 2

I don't work at Microsoft or on Photosynth, but a couple years ago, I had a project called PhotoCity that was sort of the awkward academic sister of Photosynth. Both were descended from the Photo Tourism project and made the tech available to anyone online.

These speculations about Photosynth 2 are reconstructed (hah) from my own experience working on similar things and from a talk Blaise gave at CVPR last summer.

Underlying Geometry 

Take a bunch of photos of the same scene from different angles, and you can reconstruct 3D information about the scene. It's hard to reconstruct a clean, correct, complete 3D model, but you can usually get a sparse idea of what's going on, like a point cloud or depth estimates for each image, as well as how the images relate to one another in 3D space.

Some software and apps that are out there: [bundler + pmvs] [visualsfm] [123d catch]

Dense (PMVS) point cloud reconstruction of Hing Hay Park in Seattle, photographed by Erik Andersen

My notes say that Photosynth reconstructs a point cloud (probably dense, something like PMVS) and rough geometry (probably from the point cloud). It doesn't seem to use the point cloud in the rendering. Instead, each image has its own piecewise planar proxy geometry of the scene. As you move between images, that geometry (or your view of the geometry) is interpolated.

View-dependent textures

I am certain that if you looked at the Photosynth 2 geometry without any texture, it would look like crap. You can sometimes tell how bad the geometry is in the transitions.

Screenshot of mid-transition. There are ghosted objects (that moved as if they were separate geometry) and tears and other issues, but they don't actually detract from the experience. View this ship yourself. 

UPDATE: YES YES LOOK AT THOSE CAMERA FRUSTA AND WACKY GEOMETRY... press 'c' on any synth to get this inside look for yourself!

One piece of magic though, the thing that smooths out all the terrible geometry blemishes, is the fact that you're looking at the beautiful original photos plastered onto the crappy geometry. AND you're looking from an angle that is similar to where the camera was actually taken. In that sense, it doesn't really matter what shape the geometry is because it's just going to look like the original photo.

Sometimes the viewpoint and geometry will be such that you can see past something to a place that's not textured. It looks like they just use some kind of texture hole filling to deal with that.

My friend and colleague Alex Colburn has some really cool work on using view-dependent textures to do cinematic parallax photography and to model and remodel home interiors.

Camera manifolds

Ye olde Photosynth let you take photos all higglety-pigglety and then let you browse them in a similarly haphazard fashion. I think many of the design choices of Photosynth 2 stem directly from trying to address the problems with capture and navigation of the original. (My PhotoCity solution was to show the 3D point cloud instead of the photos or show the point cloud projected on a 2D map.)

Oldschool synth: a sea of photographs. But! Since you're using Photosynth and not PhotoCity, you don't get much feedback about which photos work/fail and you don't get the chance to improve your model except by doing the whole thing again.

The types of manifolds Photosynth 2 supports are (at least):
  1. object: taking photos in a loop looking in at an object
  2. panorama: taking photos in a loop looking out, e.g. around a room or at the vista on top of a mountain
  3. strafe: take photos in a line moving sideways
  4. walk: take photos in a line moving forwards

My notes about types of camera manifolds including: object, panorama, strafe, walk, and concepts like "negative radius panorama" (shooting across a room)

These manifolds give the photographer some constraints about how to shoot, probably makes the computation easier, and makes navigation a hell of a lot easier for viewers (by constraining it).

Navigation UI magic

The camera manifolds make it super easy to navigate. Spin in a circle! Move along a single path! I think the Photosynth 2 people are really proud (and rightfully so) of being able to poke at a synth (via a touch screen) and have it respond in a sensible and pleasing way.

The second piece of navigation magic is how if you stop actively moving around, the synth just drifts to the nearest actual photo. When it's moving, the artifacts kind of blend together and get ignored by your brain. When it stops, you're looking at something free of artifacts, something that looks like a photo because it is a photo. But, unlike a normal photo, you can touch it to get a new view of the scene from a different angle.

A new art medium  

I haven't used Photosynth 2 yet (or even the original because I had my own version) but I can attest to the fact that taking photos for PhotoCity or for any other reconstruction/stitching pipeline deeply changed how I take photos and think about photography. Instead of hiking up a mountain and taking a single photo at the top, I want to take a bunch of photos and reconstruct the whole valley. (But ugh, not enough parallax from one peak!)

I think Photosynth 2 is a little more modest by enabling people to make reaaallly sexy/rich/immersive panorama-like things. Something pretty familiar, but also much enhanced. And on top of that, people will uncover quirks in the medium, like being able to capture dynamic, lively scene action in a stop-motion kind of way. For example, friend/past-colleague/actual Photosynth engineer shot this synth in Hawaii and there are occasional wave splashes!  Like a cinemagraph but in 3D! Compare that to your ghosted/amputated people in normal stitched panoramas.

Sunday, January 5, 2014

Blogpact 2014

I was part of a blogpact last year. A bunch of smart and interesting folks from around the internet (but connected to people I knew) challenged themselves to write a blog post every week. Eventually, it dried up and got down to just 2 or 3 people posting intermittently. That was kind of sad and actually discouraged me from blogging. Thus, I'm starting up a new blogpact with a fresh new batch of people! YOU should totally be part of it!

Yea! Let's blog together!

Steps to be involved: 

  1. Get yourself a blog (I just use blogger) and commit to writing stuff in it on a weekly basis 
  2. Email me (kathleen . tuite AT gmail) the link to your blog's feed (e.g. mine is 
  3. Visit the aggregator website where posts will be slurped up and combined on an hourly basis! 

First week: 

Since I'm just setting this up now, the first week will start today, and the weeks will end on Sundays. That is, your first post should come in by next Sunday, January 12, 2014. If you join later, just start whenever makes sense!


The only penalty for missing a post is that I will eventually add a thing that says how long it's been since each person posted. And, ya know, you'll miss out on the practice of writing something and putting your thoughts out into the world. 


If you need inspiration for what to write about, I have arbitrarily thought of the following topics:
  • Something you learned recently that was cool, maybe why and how you learned it, and if others should learn it, too
  • Something you created recently, whether it be out of foodstuffs, code, clay, robot parts...
  • An introduction to something you consider yourself an expert in
  • An introduction to something you're not an expert in but would like to learn about, thus getting you to learn a bit about it
  • Books you read in 2013 that stuck with you
  • Anything else you did in 2013 that stuck with you
  • What you imagine the world to possibly be like in 5/10/50/100/1000 years or a sci-fi gadget you wish existed now
  • Someone you admire