Rahul gives unsolicited advice

If you haven't read my last post , I am taking a look a Stack Overflow's yearly developer survey. After the survey is complete, the release the data collected publicly, which can be found here . At first, I was only looking at numbers from the 2017 survey but there were a few questions I wanted to ask using survey numbers from 2011-2017, such as the ratio of Male/Female developers, among those responded to the developer survey and the change in salary of developers over the last 7 years. Which is when I hit a roadblock - the actual survey data. Real life data is a mess, starting from the weird formats it is saved in to the weird data saved in them. For example, the developer's country/region has been a consistent question over the years but the possible answers have changed e.g. from "United States of America" to "United States". The column name for this data has changed as well, from "country" to "Country" to "What country ...

Prelude If you didn't know, Stack Overflow is a community of 7.3 million programmers, just like you, helping each other. It's a Q&A forum with probably the most comprehensive knowledge of the intricacies of programming languages and associated libraries, knowledge which volunteers contributed. Being part of the programming community, they run a survey every year. The survey is developer-centric and asks questions on what language you use, what language you want to learn/use, how much you get paid, what your title is, where you work from, what your gender is, and so on. It is one of the largest surveys, as far as I know. This blogpost looks at data from the most recent SO survey from 2017. Specifically, it looks at data submitted by Indian developers. Before we take a peek at the data, the data is freely available at this URL . You will also find raw data from surveys through 2011-2017. You might also want to take a look at this blogpost from the SO folks regarding...

On 21 May, I conducted a workshop on Scientific Computing 101 using Python. Over the course of 3 hours, I introduced the Numpy and Pandas packages. The participants practiced using the packages in Jupyter Notebooks. It was a great learning experience for me to make the slides and deliver the material. I did a workshop at IIT Madras earlier on the same topic but this one was better planned and cleaner content. The content was delivered using Jupyter Notebooks, which can be found on my github repository . I have been working on the notebooks to add documentation, references and a few exercises. I will keep adding more exercises in time. Going forward, the participants were interested in a followup workshop, with a focus on data science, which I presume means Machine Learning using scikit-learn. Let's see how that goes. Until next time.

Inspired by this blog post : https://langui.sh/2016/12/09/data-driven-decisions/ , I wanted to play around with Google BigQuery myself. And the blog post is pretty awesome because it has sample queries. I mix and matched the examples mentioned on the blog post, intent on answering two questions -  1. How many people download the Pandas library on a daily basis? Actually, if you think about it, it's more of a question of how many times was the pandas library downloaded in a single day, because the same person could've downloaded multiple times. Or a bot could've. This was just a fun first query/question. 2. What is the adoption rate of different versions of the Pandas library? You might have come across similar graphs which show the adoption rate of various versions of Windows. Answering this question is actually important because the developers should have an idea of what the most popular versions are, see whether or not users are adopting new features/cha...

For some background, read my previous posts on the topic - simple queries to start understanding Pandas downloads ( https://rahulporuri.blogspot.in/2017/01/on-whos-downloading-pandas.html ),  building up queries incrementally to understand Pandas downloads ( https://rahulporuri.blogspot.in/2016/12/pandas-download-statistics-pypi-and.html ) and using Tableau Public to visualize Pandas downloads by version over the last 6 months ( https://rahulporuri.blogspot.in/2017/01/visualizing-pypi-pandas-download.html) Having visualized the total number of downloads per version per month of Pandas in the last post, we now come to the total number of downloads per month by location. The relevant query is SELECT STRFTIME_UTC_USEC( timestamp , "%Y-%m" ) AS yyyymm, country_code, COUNT ( * ) as total_downloads, FROM TABLE_DATE_RANGE( [the - psf:pypi.downloads], DATE_ADD( CURRENT_TIMESTAMP (), - 6 , "month" ), CURRENT_TIMESTAMP () ) WHERE file.pro...

For some background, read my previous posts on the topic - https://rahulporuri.blogspot.in/2017/01/on-whos-downloading-pandas.html and https://rahulporuri.blogspot.in/2016/12/pandas-download-statistics-pypi-and.html . Having looked at the total number of downloads for all versions of Pandas and downloads by month in the last post, we now come to the total number of downloads by month by version. The relevant query is SELECT STRFTIME_UTC_USEC( timestamp , "%Y-%m" ) AS yyyymm, file. version , COUNT ( * ) as total_downloads, FROM TABLE_DATE_RANGE( [the - psf:pypi.downloads], DATE_ADD( CURRENT_TIMESTAMP (), - 6 , "month" ), CURRENT_TIMESTAMP () ) WHERE file.project = 'pandas' GROUP BY file. version , yyyymm ORDER BY total_downloads DESC which returns a data set, that can be downloaded as a CSV file. The file is available at https://drive.google.com/file/d/0BxwQdgnuTo6JYzR1dUI0Zm5jbWs/view?usp=sharing . I v...

For those of you who don't know, Pandas ( http://pandas.pydata.org/pandas-docs/stable/ ) is a data analysis/manipulation library in Python. Most people download it using pip ( https://pip.pypa.io/en/stable/ ) which is the PyPA (Python Packaging Authority) recommended tool for installing Python packages. pip downloads the library from PyPI ( https://pypi.python.org/pypi ), which is the Python Package Index. Now, having introduced you to the jargon, let me get to the point. Because most people install Pandas using pip, PyPI has a count on the total number of Pandas downloads. Well, not just Pandas downloads but pretty much every Python library installed using pip. And, you know what, all of the data is available publicly via Google BigQuery ( https://bigquery.cloud.google.com/table/the-psf:pypi.downloads ). Think about all the data. Think about all the questions. For now, I'm going to ask a few questions, specific to the Pandas library. 1. How many people have downloaded ...

I started to find networks interesting, especially because of the insights they can provide into the system. Earlier, I worked on making a network of Universities based on co-authorship on publications. Studying such network and their evolution can be helpful. For example, if an ongoing multi-university collaboration is successful without the knowledge and support of the host universities, such analysis can be a way to lobby for official support. On similar terms, I created a new network of physicists at the Department of Physics at the Indian Institute of Technology at Madras (which is my almamater). They revamped the department's website, specifically the Recent Publications page, which is updated with publications of the faculty in the department. As you can see from the table, each row/paper contains a list of authors. By collecting such lists, we can make a network which shows who collaborates with who and without prior knowledge, take a guess at which labs collaborates wi...