Pandas download statistics, PyPI and Google BigQuery - Daily downloads and downloads by latest version

Inspired by this blog post : https://langui.sh/2016/12/09/data-driven-decisions/, I wanted to play around with Google BigQuery myself. And the blog post is pretty awesome because it has sample queries. I mix and matched the examples mentioned on the blog post, intent on answering two questions - 

1. How many people download the Pandas library on a daily basis? Actually, if you think about it, it's more of a question of how many times was the pandas library downloaded in a single day, because the same person could've downloaded multiple times. Or a bot could've.

This was just a fun first query/question.

2. What is the adoption rate of different versions of the Pandas library? You might have come across similar graphs which show the adoption rate of various versions of Windows.

Answering this question is actually important because the developers should have an idea of what the most popular versions are, see whether or not users are adopting new features/changes they provided in a new version, and so on.

https://bigquery.cloud.google.com/dataset/the-psf:pypi is where you can explore the metadata and enter/run the queries you are interested in.

Daily download stats

The query is


SELECT
  STRFTIME_UTC_USEC(timestamp, "%Y-%m-%d") AS yyyymmdd,
  COUNT(*) as total_downloads,
FROM
  TABLE_DATE_RANGE(
    [the-psf:pypi.downloads],
    DATE_ADD(CURRENT_TIMESTAMP(), -61, "day"),
    DATE_ADD(CURRENT_TIMESTAMP(), -1, "day")
  )
WHERE
  file.project = 'pandas'
GROUP BY
  yyyymmdd
ORDER BY
  total_downloads DESC

For those of you not familiar with the SQL-like syntax, the above query creates a new table which contains the dates in one column and the total number of downloads in the other. The download statistics pertain to the Pandas library. And we asked for 60 days worth of data.

After running the query, downloading the data set and plotting, you should see something similar to



If i'm not wrong, the regular dips are on the weekends. There are atleast two weeks which have strong dips mid-week. I haven't explored the dataset further but I wonder why.

Adoption rate of new updates/releases

The query is


SELECT
  STRFTIME_UTC_USEC(timestamp, "%Y-%m") AS yyyymm,
  file.version,
  COUNT(*) as total_downloads,
FROM
  TABLE_DATE_RANGE(
    [the-psf:pypi.downloads],
    DATE_ADD(CURRENT_TIMESTAMP(), -1, "year"),
    CURRENT_TIMESTAMP()
  )
WHERE
  file.project = 'pandas'
GROUP BY
  file.version,
  yyyymm
ORDER BY
  total_downloads DESC
LIMIT 1000

This time, the dataset contains one column pertaining to month, a second pertaining to the version of the Pandas and the third containing total downloads of a particular version of the library on a particular month. All in all, we ask for a full year's worth of download data i.e 12 rows, one for each month's cumulative.

Again, running the query, downloading the data and plotting it should give you something that looks like -



Actually. I lied. Directly plotting the dataset is going to look like a mess because there are a lot more versions than the 7 plotted above. I chose those 7 because they are the most recent versions. And, this too shows some interesting trends. It's interesting to note that Pandas 0.17.x and 0.18.x still have loyal/lazy users.

That's all for now. And I'm also being lazy by not posting the code-snippets I used to generate the plots from the data. But, I thought, this blog post would be a start so, I'll try to write a followup which links to the datasets and the code. Until then ...

Popular posts from this blog

Farewell to Enthought

I started my professional career at Enthought , at the Pune office in India, on Feb 29, 2016. Over the course of eight long years, I spent time at the Austin office in the US, the Cambridge office in the UK, and I worked remotely from India. I was let go on March 27, 2024 . I had an unbelievably great time. I had three amazing mentors - Pankaj Pandey, Senganal Thirunavukkarasu, and Mark Dickinson. Pankaj mentored me in my first year at Enthought. I started paying attention to the craft of software after working with him. Senganal started mentoring me in my second year at Enthought, and continues to mentor me, even after the both of us have moved on from Enthought. He instilled professionalism in me, ensuring that the code I wrote produced precise scientific results, and that I tackled the known unknown aspects of a project first, to ensure timely communication about project progress. I was eager to manage people at the time and Senganal made me realize that I didn't want to manage ...
Read more

Arxiv author affiliations using Python

So, I wanted to get author affiliation information from papers on arXiv. arXiv provides with an API to bulk query their database and get information. Following that, I look for the attribute 'arxiv:affiliation' in the html data. Here's the code - import urllib from BeautifulSoup import BeautifulStoneSoup   url = 'http://export.arxiv.org/api/query?search_query=all:astro&start=0&max_results=1000'   data = urllib.urlopen(url).read() soup = BeautifulStoneSoup(data) #print(soup.prettify()) #list = soup.findAll('arxiv:affiliation') #for i in range(len(list)): #        print list[i].contents   test = [tag.string for tag in soup.findAll('arxiv:aiffiliation')] Now, the problem I'm having is that I'm getting affiliation of all authors which I want to split into sets of affiliations of authors of a paper, which I'm stuck on at the moment. Once I get that part, I can move on to the next part of this pet project, displaying these...
Read more

Elementary (particle physics), my dear Watson

Ground Control to Major Tom:  Shruti Patel checking in. Since my dear friend and fellow-physicist Rahul Poruri is homeward bound, and shall remain there for a week, I'm stepping in to continue his tradition of daily posts to Astronut . Even as I write, I'm not quite sure what this post is going to be about, so please forgive me for occasional rants and digressions.  By the way Rahul, I totally resent you for the name of this blog- I'm never going to start my own blog because there's no way I'll come up with a name as cool as Astronut! For now, I shall try and be happy with being a guest writer! Before I begin, a little bit on who I am and what I do; so that you can trust that my general views on science- particle physics in particular- are not entirely baseless. I graduated with a masters in Physics from IIT Madras in July this year, and I'm currently working towards (an eventual) PhD with the Theory Group at the Deutsches Electronen-Synchrotron (DESY)...
Read more