Detecting Language by Letter Frequency (Python)

-
This was our first assignment in the Python language. We had to detect language based on the frequency of letters in the text.

Python...

My biggest challenge here was learning the new syntax of python. Many of the computer science concepts I learned in Java are still applicable in Python (if statements, loops, errors, etc.). However, the syntax (grammar) of the language was super different than anything I'd seen before.

Reading files is astonishingly easy in python! I just had to use the open(fileName) method to read the file. As opposed to Java, where normally I would create a File, a BufferedReader, etc.

Processing Data

Loading in and sorting the data was the toughest part of this challenge for me. I started off by converting a spreadsheet of letter frequencies (taken from the link above) and converting it to a .csv  (comma-separated values) file. This way, each cell could be found just by splitting each line by the commas' locations.

Next, for each line, I created a dictionary (a map that stores values based on unique keys) for the language. Then for each letter, I entered the frequency into the dictionary.

The Algorithm

This algorithm is pretty simple. It determines the frequency of each letter (# occurrences/total # letters) and then compares the frequency to each registered language (from the .csv file). To compare the languages, it finds the sum of the differences between each individual letter in the language.

Comments

Post a Comment

Popular updates

Half Moon Project Part 2 - More Accuracy

-
Image
This time around, we used the Neuroph library to create a neural network  that can divide the double half-moons more accurately. Neural Networks A Neural Network is a combination of Neurons that feed into each other. Using some really complex math, they can weight each input (from a previous Neuron) differently. Then, after training, they can use backpropagation to calculate which weights should be edited. What this looks like can be seen below. More Accuracy for Half-Moons Since, previously, we were only using one neuron to calculate the line to separate the two moons, we could only draw a linear line. This time, with a more complex neural network, it can separate the two with a function that's super complicated. Theoretically, a human could also derive the equation, but it's so complex that at this point, it's easier to get the Neural Network to do it for us.
Read more

NumberVision – Machine Learning Application

-
Image
The next way we used our Neuron class was to classify digits. Using a UI, our program was assigned to take an input digit (a 5x7 grid, each box has a shade of grey between black and white) and classify the digit as a number between 0 and 9. A Smarter Neuron Graphical depiction of a Neuron receiving more than 2 inputs. Previously, I had only been using a Neuron with two inputs: an X and Y coordinate. Previously, this allowed me to plot my points on a plane like I did with the Half Moon project. However, due to the 5x7 grid of inputs I was getting, I had to alter my Neuron class to be able to take in 35 inputs. Mostly, this involved changing a few numbers.  Applying the Neuron to NumberVision The digit '1', with progressively more noise added from left to right. Good algorithms should be able to generate a result even if the input is a little distorted. I applied the same goal to NumberVision. The flow chart above shows various levels of added distortio
Read more

Machine Learning & Half Moon Project

-
Image
This was our first Machine Learning assignment. Machine Learning is something that I was (and still am) super excited about. Basically, the algorithm 'learns' something, and can then apply it in other situations. Learning & Neurons This was the toughest concept to learn (ha). We started with a Neuron, similar to the neurons in our brains. The neuron can trigger (yes/no) based on a series of inputs, in our case two inputs. The equation for whether or not the neuron triggers is SUMMATION(Weight(n) * Input(n)) > threshold . Each input is weighted differently, and the weights are what we can change to 'train' the neuron. We use this equation:  ΔWeight = learningSpeed * (target - output) * input . For example, if we start with a speed of 0.2, an input of 1, a weight of 0.5, and a target of 0, the weight would change by -0.2 (0.2 * (1 - 0) * 1). Now, with a weight of 0.3, repeat the process. Once each input returns the desired output, the best weights have been fo
Read more