COVID Data Tracking

-
The worldwide COVID-19 pandemic has hit every corner of the world in unprecedented ways. There exist many data reports, analytics tools, and projections about the virus's current and future state. I, however, wanted to create an easily accessible way to track how the virus is spreading in my area, using the number of new cases every day as a guide.

Importing Data

It turned out the Virginia Department of Health (VDH) used data management software called Socrata to publish and manage data on COVID-19, which conveniently had a continuously updated .csv file hosted on their servers. I copied older data (that wasn't on the hosted .csv file) into a Google Sheets spreadsheet, and used the powerful IMPORTRANGE() function to pull all new data into the sheet. 

Creating the Graph

The data import seemed simple enough, so I then combined the static and imported data. After aggregating it a bit (again using Sheets functions to get the number of new cases, average over 7 days, etc.), the chart I wanted was easy to create. 

However, a day later, a problem surfaced. It turned out that the url which hosted VDH data only held a rolling 7 days of data. This meant when new data was published, data from a week ago was deleted. Since I was using the IMPORTRANGE() function in Sheets, this meant that deletion would reflect on my chart as well. The result was that days of data were missing. 

Recording Data Permanently: Creating a Google App Script

It quickly became apparent that I needed a more complex function in order to record the data I was importing from VDH, not simply display the data fetched from the url. 

Luckily, Google had some really great scripting functionality that allowed me to do just that. Google Apps Script is a developer utility that links into almost every Google Product, and allows developers like me to utilize the full power of G Suite. I set up a Google Apps Script program to retrieve and record all data published by VDH each day. This enabled my automatically-updating graph to display both the most current data along and historical data. The graph updates to this day and is available at leohorwitz.com/covid!

Comments

Post a Comment

Popular updates

Half Moon Project Part 2 - More Accuracy

-
Image
This time around, we used the Neuroph library to create a neural network  that can divide the double half-moons more accurately. Neural Networks A Neural Network is a combination of Neurons that feed into each other. Using some really complex math, they can weight each input (from a previous Neuron) differently. Then, after training, they can use backpropagation to calculate which weights should be edited. What this looks like can be seen below. More Accuracy for Half-Moons Since, previously, we were only using one neuron to calculate the line to separate the two moons, we could only draw a linear line. This time, with a more complex neural network, it can separate the two with a function that's super complicated. Theoretically, a human could also derive the equation, but it's so complex that at this point, it's easier to get the Neural Network to do it for us.
Read more

NumberVision – Machine Learning Application

-
Image
The next way we used our Neuron class was to classify digits. Using a UI, our program was assigned to take an input digit (a 5x7 grid, each box has a shade of grey between black and white) and classify the digit as a number between 0 and 9. A Smarter Neuron Graphical depiction of a Neuron receiving more than 2 inputs. Previously, I had only been using a Neuron with two inputs: an X and Y coordinate. Previously, this allowed me to plot my points on a plane like I did with the Half Moon project. However, due to the 5x7 grid of inputs I was getting, I had to alter my Neuron class to be able to take in 35 inputs. Mostly, this involved changing a few numbers.  Applying the Neuron to NumberVision The digit '1', with progressively more noise added from left to right. Good algorithms should be able to generate a result even if the input is a little distorted. I applied the same goal to NumberVision. The flow chart above shows various levels of added distortio
Read more

Machine Learning & Half Moon Project

-
Image
This was our first Machine Learning assignment. Machine Learning is something that I was (and still am) super excited about. Basically, the algorithm 'learns' something, and can then apply it in other situations. Learning & Neurons This was the toughest concept to learn (ha). We started with a Neuron, similar to the neurons in our brains. The neuron can trigger (yes/no) based on a series of inputs, in our case two inputs. The equation for whether or not the neuron triggers is SUMMATION(Weight(n) * Input(n)) > threshold . Each input is weighted differently, and the weights are what we can change to 'train' the neuron. We use this equation:  ΔWeight = learningSpeed * (target - output) * input . For example, if we start with a speed of 0.2, an input of 1, a weight of 0.5, and a target of 0, the weight would change by -0.2 (0.2 * (1 - 0) * 1). Now, with a weight of 0.3, repeat the process. Once each input returns the desired output, the best weights have been fo
Read more