Notes on profiling Python applications

I've been watching a looooot of Scott Hanselman lately and in one of his videos, he googles for the sentence "penny pinching in the cloud" and a bunch of his articles are the top hits. He talks about how he blogs about things as a way of maintaining notes for his future self. My articles might not be top hits on google for any sentence but I can still find them in my blogpost history.

Over the weekend, I spent a little bit of time profiling a Python GUI application. It's been a while since I profiled Python code and every single time I forget how to set things up and get it working. There are the usual Python standard library modules cProfile and pstats but there are a lot of powerful visualisation techniques that help you better understand the profiling results.

First of all, when you are profiling a GUI application (or profiling anything for that matter) in Python, the first question to answer is what exactly are you trying to measure? In my case, what I was really trying to measure was the "Time to first interaction" (which is a new word that I learnt recently). Time to first interaction is the time it takes between when a user starts the application (say by double clicking the application icon) and when the user can start interacting with the application.

TFI (Time to First Interaction) is important because we want to provide a good user experience and we don't want them to wait for multiple minutes while the application starts up.

So we now know what I want to measure and why it is important. Let's look at how.

In this specific case, I was debugging an Envisage TasksApplication. This is relevant because the TasksApplication class thankfully emits an application_initialized event when the GUI application has been initialized. So, theoretically, I could try hook up my profiler such that it counts the time between when I start the application and when this event is emitted.

I was lazy. So, instead of figuring out how to get that working, I just decided to close the application as soon as the event is emitted. I can do that by simply saying application.on_trait_change(application.exit, 'application_initialized'). What that does is - when "application_initialized" event is emitted, run "application.exit". And then I just profiled the application lifecycle i.e. from when I start the application to when it stops itself.

I can now write a simply script which starts the application - and then the application immediately stops itself. I can now use cProfile to profile this script using

python -m cProfile -o startstopprofile.prof start_and_stop_application

This should now create a startstopprofile.prof binary file in your current directory. Investigating the contents of the file using the pstats standard library package is a bit of a pain - especially in the case of this GUI application which has multiple dependencies and have a large call stack.

So, we instead turn to GUI tools. I was struggling to remember the visualization tool that an ex-colleague of mine introduced me to (Thanks Pankaj!). And after searching haphazardly for python profile visualization tools, I finally caught a partial word "Snake" which triggered my memory to remember thet tool - snakeviz. It takes a couple of seconds to get a handle on the UI but once you do, it is very useful.

I tried profiling an example script in the Envisage GitHub repository using almost the same process and here is the output of the profiling, visualized using snakeviz.


 I still need to figure out how to visualize the profiling results using flamegraphs but let's leave that for another time.

Popular posts from this blog

Animation using GNUPlot

Image
Animation using GNUPlot I've been trying to create an animation depicting a quasar spectrum moving across the 5 SDSS pass bands with respect to redshift. It is important to visualise what emission lines are moving in and out of bands to be able to understand the color-redshift plots and the changes in it. I've tried doing this using the animate function in matplotlib, python but i wasn't able to make it work - meaning i worked on it for a couple of days and then i gave up, not having found solutions for my problems on the internet. And then i came across this site, where the gunn-peterson trough and the lyman alpha forest have been depicted - in a beautiful manner. And this got me interested in using js and d3 to do the animations and make it dynamic - using sliders etc. In the meanwhile, i thought i'd look up and see if there was a way to create animations in gnuplot and whoopdedoo, what do i find but nirvana! In the image, you see 5 static curv
Read more

Thoughts on the National Deep Tech Startup Policy

The Principal Scientific Advisor to the Government of India published a draft policy to support and nurtue the unique requirements of Deep Tech startups in India. I just finished reading it and put together a few comments in a FOSS United forum thread . I will probably read it again a week or so later and try to update my comments on the thread with any new insights/thoughts I have.
Read more

Arxiv author affiliations using Python

So, I wanted to get author affiliation information from papers on arXiv. arXiv provides with an API to bulk query their database and get information. Following that, I look for the attribute 'arxiv:affiliation' in the html data. Here's the code - import urllib from BeautifulSoup import BeautifulStoneSoup   url = 'http://export.arxiv.org/api/query?search_query=all:astro&start=0&max_results=1000'   data = urllib.urlopen(url).read() soup = BeautifulStoneSoup(data) #print(soup.prettify()) #list = soup.findAll('arxiv:affiliation') #for i in range(len(list)): #        print list[i].contents   test = [tag.string for tag in soup.findAll('arxiv:aiffiliation')] Now, the problem I'm having is that I'm getting affiliation of all authors which I want to split into sets of affiliations of authors of a paper, which I'm stuck on at the moment. Once I get that part, I can move on to the next part of this pet project, displaying these
Read more