Skip to main content

Arxiv author affiliations using Python - Part III

After a very long time, I was reminded of this project on Friday. I stopped working on this pet project of mine because I couldn't find a library that let me convert a dataset to a network. I probably (definitely) search hard enough and I would've found a solution had I kept working on the problem for long enough (because look, I have a solution!)!

 The following is the network of affiliations. You can read the last two blogposts I wrote on this topic here and here. You can find the relevant codes in this Github repository. Note that the file in the Github repository will generate this interactive image, containing the network.

Also note that there are a couple of dependencies to be able to run the code. Firstly, the code is written in Python. Secondly, you will need to have the following libraries installed - BeautifulSoup and graph-canvas.

As I've mentioned, albeit briefly, in the previous blog posts, the Python code uses the arXiv API to scrape the most recent 500 papers. It then searches through the file, referred to as soup in the Python code, for the arxiv:affiliation tag, whose string contains the information we want! We finally have a list containing lists of affiliations.

We then convert this list to a dictionary, which is then fed to the the graphcanvas library, which generates this image.

I am going to build on top of this in time so if you're interested, keep checking the blog for future updates.

Popular posts from this blog

Animation using GNUPlot

Animation using GNUPlotI've been trying to create an animation depicting a quasar spectrum moving across the 5 SDSS pass bands with respect to redshift. It is important to visualise what emission lines are moving in and out of bands to be able to understand the color-redshift plots and the changes in it.
I've tried doing this using the animate function in matplotlib, python but i wasn't able to make it work - meaning i worked on it for a couple of days and then i gave up, not having found solutions for my problems on the internet.
And then i came across this site, where the gunn-peterson trough and the lyman alpha forest have been depicted - in a beautiful manner. And this got me interested in using js and d3 to do the animations and make it dynamic - using sliders etc.
In the meanwhile, i thought i'd look up and see if there was a way to create animations in gnuplot and whoopdedoo, what do i find but nirvana!

In the image, you see 5 static curves and one dynam…

on MOOCs.

For those of you who don't know, MOOC stands for Massively Open Online Course.

The internet is an awesome thing. It's making education free for all. Well, mostly free. But it's surprising at the width and depth of courses being offered online. And it looks like they are also having an impact on students, especially those from universities that are not top ranked. Students in all parts of the world can now get a first class education experience, thanks to courses offered by Stanford, MIT, Caltech, etc.

I'm talking about MOOCs because one of my new year resolutions is to take online courses, atleast 2 per semester (6 months). And I've chosen the following two courses on edX - Analyzing Big Data with Microsoft R Server and Data Science Essentials for now. I looked at courses on Coursera but I couldn't find any which was worthy and free. There are a lot more MOOC providers out there but let's start here. And I feel like the two courses are relevant to where I …

Pandas download statistics, PyPI and Google BigQuery - Daily downloads and downloads by latest version

Inspired by this blog post :, I wanted to play around with Google BigQuery myself. And the blog post is pretty awesome because it has sample queries. I mix and matched the examples mentioned on the blog post, intent on answering two questions - 
1. How many people download the Pandas library on a daily basis? Actually, if you think about it, it's more of a question of how many times was the pandas library downloaded in a single day, because the same person could've downloaded multiple times. Or a bot could've.
This was just a fun first query/question.
2. What is the adoption rate of different versions of the Pandas library? You might have come across similar graphs which show the adoption rate of various versions of Windows.
Answering this question is actually important because the developers should have an idea of what the most popular versions are, see whether or not users are adopting new features/changes they provide…