Twisted magic so webpages can wait for longer term process to finish

Computahs!

I occasionally want to show images on my face research website that show off the average of hella (technical term) images blended together. Problem is, sometimes I haven't finished making these images yet, because they can have thousands of images in them and take a couple seconds to produce.

I totally just implemented a thing that is handy. It is a Twisted thing.

A webpage can request something (in my case, an image url), and if the image is ready, it gets it right away.

If the image is not ready, however, the webpage doesn't poll or ask again later, it just sits on the line waiting for the image to be ready. And if it (or the viewer) gets bored, they can refresh/re-ask, and it'll be like, "yo dude, it's still not ready, be patient and you can have it when it's done."

This is one of the things I'm making. It's an average of 600 of my faces!

DIAGRAM

Diagramorama! The jobs are part of the twisted server. And they are actually a dictionary. And all the other boxy things are  browser windows with lil tabs.

The part that makes the work happen

cluster_image_jobs = {}

def makeClusterAverage(cluster_id):

    @defer.inlineCallbacks
    def _makeDeferred():
        # this uses inline callbacks to the thing itself is a deferred
        print "making cluster average", cluster_id
        
        yield askDbForSomeStuff()
        yield doDirtyWorkOfMakingCluster() 
        yield moreStuffThatIsPartOfTheHardWork()
        
        defer.returnValue(cluster_id)

    # CALL the deferred thing to actually start running the job
    # notice how we return it to the outer function 
    # in case you really want to wait for it/add more callbacks 
    
    job = _makeDeferred()

    # the cleanup function
    def _removeJobFromList(cluster_id):
        print "Done with cluster job %d, removing from list" % (cluster_id)
        del cluster_image_jobs[cluster_id]

    # if there's not already a job in the list, add it
    # i only add a job if one's not already there 
    # so an older job can't remove a newer, in progress job from the list
    
    if cluster_id not in cluster_image_jobs:
        job.addCallback(_removeJobFromList)
        cluster_image_jobs[cluster_id] = job

    return job

The part that lets browsers ask for stuff 

@defer.inlineCallbacks
def getClusterImage(cluster_id):

    # if the job is in the list, sit around and wait for it to finish
    if cluster_id in cluster_image_jobs:
        print "Here I am waiting for the job to finish..."
        yield cluster_image_jobs[cluster_id]
        print "Sweet! The job has finished now and I can get on with my life!"
    
    some_id_that_i_needed = yield doSomeDatabaseStuffThatIWantedToDo()
    
    cluster_image = "image%d.jpg" % (some_id_that_i_needed)
    
    print "returning cluster image: ", cluster_image
    defer.returnValue(cluster_image)