Skip to main content

Stack Overflow dev survey - Cleaning up numbers

If you haven't read my last post, I am taking a look a Stack Overflow's yearly developer survey. After the survey is complete, the release the data collected publicly, which can be found here.

At first, I was only looking at numbers from the 2017 survey but there were a few questions I wanted to ask using survey numbers from 2011-2017, such as the ratio of Male/Female developers, among those responded to the developer survey and the change in salary of developers over the last 7 years.

Which is when I hit a roadblock - the actual survey data. Real life data is a mess, starting from the weird formats it is saved in to the weird data saved in them. For example, the developer's country/region has been a consistent question over the years but the possible answers have changed e.g. from "United States of America" to "United States". The column name for this data has changed as well, from "country" to "Country" to "What country or region do you live in?" etc.

This is why data analysts in the real world spend a lot more time cleaning and massaging the data into the right form/format before actually getting to do any analysis. And this is exactly what I ended up doing this weekend.

The Python scripts I wrote to cleanup the datasets can be found on GitHub. As you can see, I use the Pandas library to load and cleanup the dataset, finally saving the cleaned dataset. I also got a chance to get a broad overview of the kind of questions I have answers for from the survey.

A preliminary look at the datasets an be found in this jupyter notebook. I briefly and superficially look at the distribution of devs across the world. what their salaries are and what their gender is, among those devs who responded to the survey.

Tell me if have any interesting questions that can be answered using these datasets. Until next time ...

Popular posts from this blog

Animation using GNUPlot

Animation using GNUPlotI've been trying to create an animation depicting a quasar spectrum moving across the 5 SDSS pass bands with respect to redshift. It is important to visualise what emission lines are moving in and out of bands to be able to understand the color-redshift plots and the changes in it.
I've tried doing this using the animate function in matplotlib, python but i wasn't able to make it work - meaning i worked on it for a couple of days and then i gave up, not having found solutions for my problems on the internet.
And then i came across this site, where the gunn-peterson trough and the lyman alpha forest have been depicted - in a beautiful manner. And this got me interested in using js and d3 to do the animations and make it dynamic - using sliders etc.
In the meanwhile, i thought i'd look up and see if there was a way to create animations in gnuplot and whoopdedoo, what do i find but nirvana!

In the image, you see 5 static curves and one dynam…

on MOOCs.

For those of you who don't know, MOOC stands for Massively Open Online Course.

The internet is an awesome thing. It's making education free for all. Well, mostly free. But it's surprising at the width and depth of courses being offered online. And it looks like they are also having an impact on students, especially those from universities that are not top ranked. Students in all parts of the world can now get a first class education experience, thanks to courses offered by Stanford, MIT, Caltech, etc.

I'm talking about MOOCs because one of my new year resolutions is to take online courses, atleast 2 per semester (6 months). And I've chosen the following two courses on edX - Analyzing Big Data with Microsoft R Server and Data Science Essentials for now. I looked at courses on Coursera but I couldn't find any which was worthy and free. There are a lot more MOOC providers out there but let's start here. And I feel like the two courses are relevant to where I …

On programmers.

I just watched this brilliant keynote today. It's a commentary on Programmers and the software development industry/ecosystem as a whole.

I am not going to give you a tl;dr version of the talk because it is a talk that I believe everyone should watch, that everyone should learn from. Instead, I am going to give my own parallel-ish views on programmers and programming.
As pointed out in the talk, there are mythical creatures in the software development industry who are revered as gods. Guido Van Rossum, the creator of Python, was given the title Benevolent Dictator For Life (BDFL). People flock around the creators of popular languages or libraries. They are god-like to most programmers and are treated like gods. By which, I mean to say, we assume they don't have flaws. That they are infallible. That they are perfect.
And alongside this belief in the infallibility of these Gods, we believe that they were born programmers. That programming is something that people are born wit…