Sunday, November 1, 2015

JPEG, the nerdiest cat

A little over 5 years ago, Adam and I adopted two cats, who we named JPEG and MPEG (suggested by a good friend with a cat named Pixel). Being young-ish shelter cats, they were the same size when we got them, but they grew into their names with MPEG becoming the larger, more active cat.

MPEG (left) and JPEG (right)

Over the last two days, we were investigating JPEG's recent stomach troubles and why he'd been sporadically vomiting and losing weight, and why diet changes seemed to have no impact. He got an ultrasound on Friday, October 30, which revealed a "poorly encapsulated" mass mostly blocking his GI tract, and nodules and fluid in his abdomen. We were emailed a terrible "J-PEG.pdf" full of passive doctor language and cryptic medical words that basically spelled out BAD NEWS, CAN'T FIX. He had been sedated for the ultrasound, and he was awake when we brought him home, but that evening, he struggled to recover from the anesthesia. We brought him to an emergency vet around 5am on Halloween, two blocks away from where I lived when I first adopted JPEG. He was in poor shape, with a 10% chance of surviving the weekend until we got more biopsy results, so we decided to let him go. We couldn't have known how bad his sickness was until we did this ultrasound, and unfortunately, he was too sick to bounce back from that procedure.

Adam and I sifted through all my online photos for the past 5 years, pulling out 200+ .jpgs of JPEG to post on facebook, but I wanted to specifically highlight JPEG's participation in my image processing and computer vision hacking throughout the years.

JPEG, the computer vision muse:

JPEG, working hard at the computer while sitting on my lap
JPEG reading a research paper
MPEG and JPEG learning about recursion.

That Time Picasa Detected JPEG's Face

Normally, I don't use Picasa. But once day, I let Picasa try to detect faces in my photos... so guess who's kitty face greeted me when I scanned through the results (I still have no idea who those people in the adjacent pictures are):

Picasa face detection - Who's a kitty!

Picasa actually detected several JPEG faces.

Faces of JPEG the cat, as detected by Picasa
I took screenshots of the hilarity, and Picasa recursively went and recognized JPEG's face in those new images, until there were TONS OF JPEG FACES:

JPEG kitty-face explosion!

That Time I Had No Idea How To Do Object Detection And Made a Cat-Butt Detector

Another time, I wanted to understand OpenCV's object detection and train my own detector. I wrote a blog post here. Basically, I figured out OpenCV's pipeline for training a new detector, but I didn't give it good enough data to make it work. Here is JPEG stealing magnets off the fridge, and the detector firing on his butt. 

OpenCV cat-butt-detector

That Time JPEG Helped Me Play With Live Face Detection

One time, I was experimenting with Jason Saragih's realtime face detector (now maintained by Kyle McDonald), available here on GitHub

Sitting on the couch, trying out the face tracker, under JPEG's supervision.

Of course, JPEG and Adam wanted to try it, too.

JPEG's face, loved by computers 
JPEG is less impressed with a face being detected on his arm.

That Time JPEG Was HUGE!

I saw this on the internet once and had to recreate it. Adam's biggest question: "How did you hold the camera??" (One of these hanging on a mirror.)

Look how big JPEG is!! Just kidding.

Those Times JPEG Helped Me Make Panoramas

Of course, JPEG was there when I discovered panorama mode on my camera.

JPEG is chilling on the ledge in the kitchen (left)

JPEG finally being larger than MPEG, thanks to panorama magic.

That Recent Time JPEG Was The Subject of DEEP DREAM

Adam and I wanted to run Deep Dream code ourselves, so we used this ipython notebook and this docker setup that we ran on Digital Ocean. 

Original image: JPEG liked to bring this stuffed penguin around the house.

Eyes and faces everywhere!!

Visualizing one of those lower edgy-layers.

Visualizing a different, low level wiggly layer.

That Time You Read A Blog Post About My Cat

This last picture has nothing to do with computer vision, it's just a time that JPEG got out on the deck of our 3rd floor apartment and then got on the roof to explore and chase pigeons. 

JPEG on the roof

Wednesday, September 16, 2015

Using IPython Notebook with Django on a remote server

IPython Notebook (or Jupyter, whatver) is great. Django is useful and I use it for both work and play. The combination would be greatly useful

My setup is something like this: I have an EC2 machine running Ubuntu, which powers a Django site. I want to be able to launch IPython Notebook on the remote server such that:
  1. It knows about Django and can import my models properly
  2. I can just write the code in a web browser on my local machine
  3. It's secure in that there's not just a stray notebook on the web that anyone can look at

My mom mentioned the Django extension 'shell_plus' the other day, and that turned out to be key to this solution!

Installing stuff:

 (I did this a few days ago and don't remember all the details, I *think* it was this simple.)
  • IPython Notebook
    • pip install "ipython[notebook]"
  • shell_plus via django-extensions
    • pip install django-extensions

Running stuff:

There are a couple steps that I wrote down for myself (and now you) since they are hard to remember until you've done them many times.

First, I connect to the server through an SSH tunnel. On my local machine, I run

ssh -L 9999:localhost:8888

The second port (the 8888) is the remote port. That needs to be 8888 for IPython Notebook, unless you can convince it to run on a different port. The first port (9999) is the local one, and you can change it if you want. 

On the remote server, (in the SSH connection I just made, or a separate one, doesn't matter) I launch IPython Notebook using Django's and shell_plus, with an extra argument to stop it from trying to launch a browser. I want to use a local browser, not a remote one.

./ shell_plus --notebook --no-browser

Back on my local machine, that 9999 I had in the first command comes in to play. I go to Chrome and visit:


And from there, I can do all the normal IPython Notebook and interactive Django shell things! At the same time!

To stop things, I kill the IPython Notebook by killing the ./ process, and then I kill the SSH connection. 

Thursday, June 4, 2015

How Imgur saved Feminist Hacker Barbie behind the scenes

Back in November 2014, the internet erupted in rage over discovering Mattel's/Barbie's "I can be a computer engineer!" book. Spoilers: the book basically tells girls they need boys to code and fix their computer problems for them.

Inspired by the amazing Kate Compton (master of the pink and glittery Crystal Code Palace of hardcore AI tutorials, including the Javascript grammar expansion library Tracery) I whipped up a new meme-generator website called Feminist Hacker Barbie to stoke the flames/give folks on the internet a creative and constructive outlet to rewrite pages of the book themselves.

A new #FeministHackerBarbie page:
Why ELSE would Barbie be hacking on so many computers at once??
Actually there were many great potential explanations for this glorious scene.

This post isn't about #FeministHackerBarbie itself. NPR, Wired, The Verge, and other such coverage exists already.

This post is meant to be a brief glimpse behind the scenes: one particular design choice I made out of love and laziness, and how that totally saved my ass.

Imgur is great

I'm a lurker on Imgur. I don't post, I just observe and absorb the weird community memes. I probably check it more than Facebook because it's more interesting and clever over all. Anyway, Imgur has an API.

Since I was making a website that let people create their own new images, I needed a place to host these images. I have made many a system to accept and process user images (PhotoCity, Sketch-a-bit, Picard) but it's a hassle and there's a risk of running out of disk space or people uploading terrible things. I heard that Imgur had an API, so I used my Barbie hacking as an excuse to learn it and try it out. It was super easy. You post some image data at Imgur, and you get back an image key.

The flow of my website was something like this:

  1. user creates a new image by typing some text that gets rendered into an image using HTML5 canvas stuff
  2. canvas gets converted to binary data with canvas.toDataUrl()
  3. binary encoded data gets sent to imgur, imgur returns a key
  4. imgur key saved by being posted back to feministhackerbarbie site

The TTD metric

What happens when you open up a hole on the internet where users can submit their own content? The internet fills that hole with dicks.

There's a metric called "time to dicks" that is basically how long it takes before users submit penises. They could be photos, drawings, or even penis-shaped SPORE creatures.

For FeministHackerBarbie, there was actually (luckily?) a wave of Richard Stallman pictures before the straight-up porn. Still pretty creepy, though.

When the site got Stallman'd

Wait, I thought the site just allowed you to re-write Barbie pages. What are these other photos doing in there??? It got hacked, kids. The "learning how the internet works" kind of hacking, I think.

Django's CSRF hacked

Let's let barbie explain how the site was hacked:

Conveniently, someone shared the script they were using to hack the site.

  • someone wrote a really simple python script that took an imgur image id as a command line argument
  • the script generated a random number to serve as the csrf (cross site request forgery protection) token
  • then the script posted some data, including the imgur id and the fake csrf token, to a URL on my site called /save_page
  • the result: the imgur image appearing as a recent submission to my website

I was just looking in to WHY this happened. How the hell could a random CSRF token bypass whatever magical security I thought Django was promising me?? It turns out that while Django complains if you DON'T pass CSRF tokens around, it doesn't actually care if the CSRF-checking mechanism is turned on or not. I realize now I should have either:

Back to why I love Imgur

So, hacking was bound to happen. I'm not surprised and I learned a lot from the experience. When it did happen, I didn't actually try to fix it (or enter into an arms race with the hackers), I completely shut down the ability to upload content.

People could still use the site to generate the images and then post them themselves, and it was really cool to see the meme live on on Twitter for a while.

But while the hacking was happening, relying on Imgur to host the images was beneficial in many ways:

  1. No risk of me running out of disk space (the site was hosted on Heroku, which doesn't even have free persistent storage)
  2. I spent all of 0.5 seconds looking at the website when there was porn on it and took refuge in the MySQL command line instead. Since people were posting the same pornographic imgur ids over and over, it was easy to find and delete the duplicate ids. 
  3. A backchannel chatroom claimed they'd successfully posted a variety of terrible, exploitative images, including ones that would be illegal to have on one's computer. But because images went straight to Imgur, none of them ever went through my server. 
  4. Imgur probably has means of dealing with terrible user submitted content since that is their basis of their entire existence. 


I'm trying to come up with some good take-aways here. When you make things on the internet that let people submit their own content, you need to be at least a little bit careful. Here are some ways to be careful:

  1. Imgur is a wonderful home for internet memes including #FeministHackerBarbie, and it might be appropriate for whatever you're trying to do!
  2. Showing unmoderated user submitted content on the front page (like I originally did) is not a great idea. It helped the site take off in the first ~18 hours, but then it was like a siren call to the hacker trolls. "Put porn here to shock and upset the general internet!!"
  3. Check that your CSRF tokens are actually working and protecting your django site. Here's an ever so slightly modified version of the barbie hack code if you wanted to try it on your own django site: 
And that's about it!

Monday, May 18, 2015

Time-lapse photography and The Photo Graph

My labmate and friend at UW has been working on a really cool project that just hit the press today. Here's a video for Ricardo's Time-laps Mining from Internet Photos:

A couple people have sent it my way, with some note like "this seems relevant to your interests!" which it very much is. What may not be apparent is how relevant. How very relevant indeed...

A descendent of Photo Tourism

This project is a descendent of Noah Snavely's Photo Tourism project from almost a decade ago, which was the first to show that computer vision techniques were robust enough to use on community photo collections, images taken by different people with different cameras under different viewing conditions/angles/lighting, to produce results such as the geometric 3D registration of all the photos and some of the geometry in the scene.

Both of these projects are extremely cool. Well, cool is an understatement. Both of these projects use community photos to show us completely new views and perspectives on our world. Look at Photo Tourism and understand the spatial context of the photos, even where people tend to walk and snap shots. Look at the Time-lapses to see the seasons change, buildings grow, glaciers melt...

I traveled to Rome, and I knew from this paper, Finding Paths through the World's Photos, what it would be like to enter the Pantheon. That the crowd would enter on the right and move through in a counterclockwise pattern, with most people taking most of the photos when they entered, and taking fewer and fewer as they reached the exit.

Here's my Pantheon selfie, because honestly, every other picture of this place has already been taken:

Pantheon PhotoCity selfie

Speaking of the Pantheon and other incredible UW projects descended from the Photo Tourism heritage, check out 3D Wikipedia:

From accidental crowd photography to intentional crowd photography 

There's a big problem with all of the projects outlined above: they only work in locations that have been heavily photographed, which means they only really work in popular tourist locations, where people are already taking photographs anyway. To solve this problem, I worked on a crowdsourcing game called PhotoCity. It was a real-world capture the flag game, where players were incentivized to take new pictures and capture new territory, instead of taking the same old iconic shots. In six weeks back in 2011, forty players took over 100,000 photos of University of Washington and Cornell campuses.

There is a need for a source of imagery that's not just "what I saw on my summer vacation," but that stems from people knowing their photos are contributing to time-lapses and 3D reconstructions, and purposefully going out to take pictures. I know a number of old PhotoCity players who are itching for this opportunity to exist again.

Incentivizing strangers on the internet to take photos is scary and daunting. What if they don't want to? How do you get people to do it anyway? These are the questions that tripped me up during my work on PhotoCity.

These days, I can point to projects like Mapillary, which makes streamlined apps and interfaces for collecting crowdsourced streetview images, and say these questions are not that scary. They weren't even that scary in the PhotoCity days. People do want to do want to take photos and to contribute in a way that adds up. They will do a good job, provided you give them good tools and feedback.

A personal note

I was pushed away from my PhotoCity project several years ago. I don't fully remember or understand why. I thought it would be okay, that I would work on something new and it would be fine, but it actually broke me and derailed my grad school career.

Ricardo told me about his time-lapse project back in October at the UW GRAIL retreat. It was one of the incidents that motivated me to quit grad school: This guy is doing awesome things in this domain, and I want to, too. In fact, I want to do the work that would very clearly support this other work and make it more powerful. But for some reason, the lab that specializes in this work was preventing me from doing this work, from continuing my work. Maybe someone from my lab will read this and realize how deeply this is still affecting me.

I quit grad school so I could resume working on crowdsourced photography, but it's taken time to get back on my feet. The situation made me frustrated and mad and depressed. Luckily, since last October, I've stumbled into many great opportunities that are helping me become myself again: creating the Feminist Hacker Barbie meme that swept the internet, programming for fun and profit, publishing papers (even one related to PhotoCity, to appear next month at FDG), teaching at the local university, and working with people who understand my crowdsourced photography ideas. It's a fascinating swirl of realigning priorities -- right now, I really should be grading the midterms for my Interactive Narrative class.

The Photo Graph

I have a plan to revive PhotoCity, or something like it. To build a new crowdsourced photography ecosystem. I am calling it The Photo Graph, because rather than being about capturing flags and building 3D models as with PhotoCity, it will be about building a connected graph of photos. Photos from the International Space Station could connect all the way down to ground-level photos. Photos from the aftermath of the earthquake in Nepal could be registered with the before-photos, to let us see into the past before the natural disaster struck, or to aid in the recovery efforts there.

Progress so far is slow. Unsurprisingly, teaching is all-consuming. So far, I've rebuilt the PhotoCity website. I bought a new domain. I occasionally contribute to Mapillary's reconstruction pipeline OpenSfM, which is like a modern, pythonic version of Noah Snavely's Bundler (the original core of PhotoCity), and dream about how to build a new PhotoCity around it. I've also been working on related ideas, like using crowdsourced photography for event coverage and sharing experiences in real time.

I don't know if or when I'll actually work on the Photo Graph. Maybe soon, or maybe I'll get a real job instead. But this is your warning, Internet, that this idea of The Photo Graph is brewing and lurking and stewing. Maybe it'll come from me, or maybe it'll come from somewhere else. Keep sharing related projects and talking with me about it!

Wednesday, January 7, 2015

How to compile custom Bootstrap themes with Less

I knew how to make custom Bootstrap themes with color variables defined in Less at one point. Maybe I never really knew, but I managed to successfully do it once. I would like to do it again, but I don't remember how! So let's learn together.

The main thing I want to do is define a couple nice main colors, and get a coherent Bootstrap theme. The website Lavish sort of does this, but I think the themes it puts together are kind of unstylish. I used it for Feminist Hacker Barbie, but I want something more refined now and I don't just want a prebuilt Bootswatch theme.

Step 1: Try running 'less' in mac terminal and then remember that that less is a normal little unix command that I totally use now and again.

Step 2: Go ponder this Bootstrap instruction post and realize there are so many other things involved, like Node.js and Grunt. Huh.

Step 3: Download Bootstrap source code from the website, unzip it, enter the directory. Oh hey, there's a grunt directory and this package.json and all the things that the bootstrap guide was mentioning.

Step 4: Read enough to realize I don't want to just run 'grunt' because that'll compile a ton of things and run tests and good software development crap I don't care about right now! Run 'grunt dist' instead. "We regret to inform you that grunt is not installed." That's not what it said but it's how I feel. Hello, you hairy yak, let's give you a haircut.

Step 5: Start shaving the yak. Luckily, the bootstrap page tells me what to do. At this point, I'm basically just live-blogging me reading that page. Sorry. Good thing I had the node package manager already installed. AHH it's not working, why is it not working? The yak, it is hairy:

  1. Got this error: "TypeError: Object Gruntfile.js has no method 'flatten'" so I did this: (which may not be required)
  2. Now this damn computer's all ">> Error: Cannot find module './valid-callable' ... Aborted due to warnings." It even has angry colors.
  3. I don't know or care who's angry right now. Bootstrap suggests deleting the note_modules folder and getting it to regrow, so I'm trying that. 
  4. Omg it made it better! Yay! I know this isn't a yak, but... so fancy! 

Step 6: Try to remember what we were doing. Ah yes. 'grunt dist' seemed to work. It made a 'dist/' folder that looks like what I put in my websites. If I delete it, it grows back when I run the command again. When I do copy bootstrap.min.css to my website, it looks like normal bootstrap. For now...!

Step 7: NOW finally I can go mess with the less variables! They're in 'less/' of course. I'm tweaking variables, then basically running 'grunt dist && cp dist/css/bootstrap.min.css ~/mywebsite/static/css/' repeatedly and refreshing the page. I'm discovering that Lavish seems to change the 'gray-*' less variables to get its effect, which is maybe why it looks cheesy.

Step 8: Fiddle until vaguely satisfied! It probably looks similar. It probably is really similar. I am not really a graphic design person. Did I mention, I totally used Lavish and a photo for inspiration.

Anyway, to summarize, the words you (past me) was looking for was not "make bootstrap with less" but "use grunt to compile bootstrap css from another css language 'less'".

Short summary of steps:

  1. Get bootstrap source code from website
  2. Get npm and grunt-cli as the bootstrap website also says
  3. Go to where you put your bootstrap source code and go into the less/ folder and change things in variables.less! Maybe use Lavish or Bootswatch for inspiration/starter values.
  4. Compile with 'grunt dist'
  5. Copy dist/css/bootstrap.min.css (and/or other relevant files) to where you need it to go
There. So now you (and I) know.