How to mine UCE results of the whole country using Python

Disclaimer: This was written solely for Educational Purposes!

UNEB recently released 2018 UCE results, and to my surprise, they were just thrown in the open. If you come up with any sensible index number, I bet you will be able to return its results at https://ereg.uneb.ac.ug/results

From https://datasciencedojo.com

It gets interesting: For a data scientist like me, this can be a great trove of information to do a hobby project on. I can easily find out the most popular names for babies born in (2002–2003)ish. I can also verify which subject was performed best against media records. If I’m a school looking to make some new hires, this data can come in handy to aid my poaching by identifying which schools performed particular subjects best and then net their teachers.

Project setup:

We shall use Python 3.7, requests and Beautiful soup libraries.

I have tried as much as possible to document everything and add explanatory comments where neccesary regardless of the fact that this code was written in 30-ish minutes.

Meanwhiling:
- Visit the results website in chrome.
- Open dev tools by right-clicking in the browser window and choosing inspect.
- Switch to the network tab of Dev tools
- make a request for an index number
- Copy the curl request from the browser as shown below
- Copy the HTTP params from the request and use them in your code

Screenshot from Chrome dev tools

The copied and pasted Curl request will look like this:

Alternatively, You can convert your copied CURL request into python/node.js code using this service: https://curl.trillworks.com/

And the HTML (we call it soup! 😍) response will look like:

Methodology:
- We make an http post request through python, the data field takes in an index number
- We get back HTML
- We parse the html using BeautifulSoup and split the contents to a dictionary

I tried to change the HTTP headers so that I receive back json in vain.

After scraping, a dictionary result of a single student looks like this:

The full code is available in a gist here:

Any corrections and modifications are fully welcome!

If you want to learn how to do this, These could be Helpful Resources: Requests: http://docs.python-requests.org/en/master/
BS4: https://www.crummy.com/software/BeautifulSoup/doc
Python data structures: https://www.datacamp.com/community/tutorials/data-structures-python

Legal Disclaimer: The estate of Edison Abahurire is not responsible for any evil doings that individuals may derive out of this project. Stay safe!

I write myself out. Code Chef | Athlete | I Dance and Love travelling. I’m diving into Data Science.

I write myself out. Code Chef | Athlete | I Dance and Love travelling. I’m diving into Data Science.