How Does AI Work?
People ask me this question a fair bit. I mumble something about huge amounts of data and lots of weights. I got the question last night from someone who knew that AIs generate one token after another, but wanted to know How do AIs work? They didn’t understand how that simple process could possibly create an image or answer a complex question.
This morning I woke up with a germ of an idea: DNA. How does DNA work? I understand the sperm and the egg form a zygote, each donating a half strand with some copying errors. Then, lots more copying, and a few months later out pops a baby. Really?!?
How do ants (who all share the same exact DNA) build this?
Do the ants sit down with an architect first?
We can imagine how an ant builds something out of soil, because we know how to build sandcastles. Yet, that doesn’t capture how an ant thinks, or how an entire colony of ants (clones) can produce something so varied and complex.
When I see an ant colony or a baby, it explains how AI works. Yes, really. After decades of working with computers, it seems self-evident to me that lots of simple steps can produce complex things. It’s definitely not obvious to the people who ask me how AI works so I’m going to try to connect the dots.
Space and Time
Two factors shared by ants, cells, and LLMs are time and space. A zygote takes about a day to split into two cells. That’s a slow process, and a zygote doubles in size, or takes up twice as much space. Two cells can do twice the work of one so there are four cells at the end of day two. Four times as much work! Rinse repeat for 275 days, and a baby pops out with 2,500,000,000,000 cells. Working together, space and time can create wondrous things.
Cells are slow. Artificial neurons are very fast. A computer running an LLM can process a single instruction 86,000,000,000,000 times faster than a cell can divide. Moreover, a leading-edge chatbot uses 330,000 cores (computing units) so it can actually process 28,380,000,000,000,000,000 (10¹⁹) instructions per day.
For comparison, human brains have about 80,000,000,000 (10¹¹) neurons, of which about a billion (10⁹) are active at any one time. Current research suggests neurons fire around 10 to 100 times per second, putting total brain activity at roughly a maximum of 100,000,000,000 (10¹¹) spikes (or “instructions”) per second. For rhetorical (“don’t quote me”) purposes, let’s say an LLM processes about 10,000 times faster than a human brain.
The point is that ants, cells, neurons, and modern computers leverage both time and space – in differing amounts – to produce really big things. These numbers are so large that they are difficult for us to comprehend, and this leads to us not being able to understand how they work. We may know how a cell divides, but no one knows exactly when a trophectoderm cell will form. That happens randomly.
Emergent Behavior
How do jazz musicians solo? One note at a time. And, lots of listening and practice. Why can the same musician have a monumental solo followed by an ordinary one? Their listening and practice hasn’t changed. It seems random, because it is. More concretely, a jazz solo is the product of a stochastic generative process, which is how an LLM does its thing.
Stochastic just means there are random parts to the process. Generative means the next step in the process depends on the previous state of the system. A jazz musician chooses the next note(s) at random, but it fits in with the previous one(s) based on years of training, which is itself a stochastic generative process.
An LLM is trained in the same way: a stochastic generative process. The numbers behind training an LLM are staggering (1,000s of computers over many months). An LLM uses an enormous amount of data as input, just like a musician listens to music to learn from, except that it learns by reading (not by practicing, subject for another article). The learning process is similar: it keeps trying until it hits a “good enough” threshold (determined by the programmers).
The process is similar to Darwinian evolution where complex behavior (life) emerged from inanimate chemicals over a very long space-time period. DNA emerged, then simple cells, and more complex organisms, until finally, we have Spinal Tap.
Focus, Focus, Focus
Busy brains make mistakes. Today I was thinking about this article when I hit the end of the pool. Ouch! That happens a lot to me. I try not to multitask, but focusing on exactly one thing is very hard for me. And, I’m guessing, for you.
Once an LLM is trained (per above), it executes on exactly one thing: the text you give it. It doesn’t worry about another article it might want to write, its posture when standing at a desk, or if it’s time for a snack. Its only job is generating one token after another based on a (relatively) small amount of information with a powerful computer.
This ability to “focus” on exactly one thing is another “big number” advantage. Our brains operate unconsciously 90% of the time. Our thoughts are many and simultaneous. What we say/do is just a tiny fraction of what we think. An LLM uses massive compute power and a “google” of data to say exactly what it thinks and nothing more or less.
Intelligent behavior emerges from simple operations iterating over very long space-time periods. The individual parts make sense on their own, and the sum of these parts creates something utterly amazing that seems more complicated than it really is. Or as Thelonious Monk said, simple ain’t easy.
AI Disclaimer
I would like to thank ChatGPT and Claude for improving the quality of the reasoning, internal consistency, grammar, and breadth of this article. All hallucinations are my own.
I was originally going to have an LLM write the article, since I have it write most of my code these days. The text just started flowing, and I wrote all of it myself, one character at a time.