I was going through this machine learning course right as ChatGPT took off, so naturally I wanted to learn more about LLMs and how they work.
The Plan
The plan was simple:
- Created transcripts of our professor’s lectures.
- Train an LLM with that data.
- Have our very own “NikhilGPT”, an expert in machine learning.
- Learn about transformers and LLMs while doing it.
The Results
Turns out that when people talk, they use a lot of filler words and don’t always speak in grammatically correct ways. We tried cleaning up the data, removing “umms” and “uhhs”, but it wasn’t enough. Still, I’m pleasantly suprised that the LLM at least somewhat spoke in reasonable ways, even if it was nonsensical:
matrix multiplication is the gradient of the actual values. And we can see so if I want that gradient of the same thing that I want to see is now just looking at this, the test set of the data, I can see that it’s the probability of the line is a 10% than zero. So I’m going to be the set of those, the value to be able to predict different weights, they actually get a single vector. So let’s create the first one thing. And so that is the