I have read a lot of articles and listened to numerous podcasts about AI and they generally portray it as a technology so advanced that only highly trained experts can understand it. That isn’t true. You don’t need a computer science degree to understand the basics, and the basics are enough to judge the big questions about AI for yourself. I think the mystique is partly intentional: it leaves ordinary people feeling unqualified to weigh in on how dangerous AI is, or isn’t.
I’m not a professional AI researcher, but I’ve built and trained my own language models and designed new components for their inner workings, so I’ve seen the machinery up close. Many of the loudest voices predicting AI doom or dismissing the risks have a stake in the answer: funding, book sales, stock prices. I don’t. This essay explains in plain terms how these systems work, so you can judge for yourself what you read and hear about them.
The first thing to know is that the recipe used to create AIs is simple. GPT-2, the predecessor to ChatGPT, can be written in about a thousand lines of C code, or a few hundred lines of Python.
A model is a stack of identical building blocks. Each block has two parts: an “attention” step, which lets each word look at the words around it, and a small neural network. Stack dozens of these blocks and you have a model. The size is set by how many blocks there are and how big each one is. GPT-3 has 96 blocks and 175 billion adjustable numbers, called parameters. The largest model I’ve trained, GPT-2 Medium, has 24 blocks and about 350 million parameters, and it takes my older GPUs about three weeks to train.
Training is just as simple. Start with the parameters set to random values. Feed in a huge amount of text, such as web pages and books. At every point, the model guesses the next “token,” which is usually a word or part of one. Compare the guess to the actual next token, then nudge every parameter slightly in the direction that would have made the guess better. Repeat this trillions of times. That's it. No exotic math is involved, just an enormous amount of repetition. The result is a model that can understand you and generate human-like responses.
The obvious question is: if this is so simple, why did it take so long for us to figure it out? The answer requires a short history lesson.
The idea of thinking machines is ancient. In Homer’s Iliad, the god Hephaestus has golden servants who can think and speak. In the 1770s, a chess-playing machine called the Turk toured Europe and beat many of its human challengers. It was a hoax, with a person hidden inside, but it shows how long people have been fascinated by the idea. For today’s language models, we don’t need to go back that far, though we do need to go back further than you might expect.
I’ve orbited AI for most of my career because I’ve specialized in very hard optimization problems, the kind where finding the best answer is impossible and you settle for one that’s good enough. People often find good solutions to these problems easily, while powerful computers grind for hours or days to match them. The question was always how humans do that, and can a computer learn to?
Neural networks were one attempt at an answer. They mimic, very loosely, the neurons in the brain. Each neuron is simple: it collects signals from other neurons and decides whether to pass a signal along. The power comes from connecting enormous numbers of them. Researchers built early neural nets in the late 1950s, abandoned them in the 1970s, and revived them in the 1980s with a training method called backpropagation that’s still used today. But computers of the era could only simulate a few thousand neurons, so progress was slow.
Enter Moore’s Law. In 1965, Gordon Moore, who would later co-found Intel, predicted that the number of transistors on a chip would double every year; in 1975 he revised that to every two years. For roughly sixty years, the industry has kept this pace despite the limits of physics, and chip features are now approaching the scale of atoms. That steady progress is what eventually let computers simulate billions of connections.
Around 2012, a threshold was crossed. Researchers discovered that graphics chips designed for video games were ideal for neural nets, and the internet supplied mountains of training data. Suddenly “deep” neural networks could recognize speech and identify what’s in a photo, down to the breed of a dog. AI took off. But language remained stubborn, because the models of the day read text one word at a time and couldn’t be scaled up efficiently.
Then in 2017, a team at Google published a paper called “Attention Is All You Need,” and everything changed. Their design, the transformer, lets every word in a passage weigh its relationship to every other word, and it processes all the words at once instead of one at a time. That made it a perfect fit for graphics chips. I think these researchers deserve a Nobel Prize. OpenAI’s first GPT followed about a year after the Google paper and every major AI model today is built on the transformer architecture.
The final piece came when researchers discovered that transformers keep improving as you make them bigger and feed them more data, with no obvious ceiling in sight. That set off a race for hardware: chips built specifically for AI, and ways to link tens of thousands of them together to work as one machine. Today’s models are the product of decades of steady progress in hardware and a few revolutionary ideas in software, and they needed both.
As outlined above, we have a giant pile of parameters that holds what the model absorbed during training, and an algorithm that uses those parameters to look at a string of text and predict the next word. That’s all it is. It isn’t a chatbot, a coding assistant, or a deepfake generator. To make it do anything useful, we have to wrap it in ordinary code that puts its predictions to work.
Say you want a chatbot. The wrapper code builds a piece of text something like this: “Two people are talking. Person A says, ‘<your question>.’ Person B replies:” and hands it to the model. The model predicts the most likely next word, the code tacks that word onto the end, and the model runs again on the longer text. It repeats this, one word at a time, until Person B’s reply is finished. The model never plans the whole answer in advance. Each word is just the best guess given everything before it. To keep the conversation going, the code has to send the entire conversation every time, everything said so far, followed by your new message with “Person B replies:” tacked onto the end. The model doesn’t remember you. It rereads the whole transcript from scratch on every turn.
I’ve simplified this, but there are a few key points. First, the knowledge in the model is static. It has absorbed a staggering amount of what humans have written, but it can’t learn anything new after training, and all it can do with what it knows is predict the next word. Second, the context, the text we feed it, is dynamic and changes with every interaction. When a chatbot “searches the web” or “remembers” something about you, the model isn’t learning anything. The wrapper code is fetching that information and pasting it into the text before the model runs. What makes these systems seem almost human is the combination: a fixed store of knowledge, steered by text that changes with every word.
The dynamic data -- which we call the context -- is the model’s only working memory, and it fills up fast. It holds the “system prompt,” which is the vendor’s hidden instructions and setup text, plus anything the app has saved about you from earlier conversations, plus both sides of the current conversation, plus anything pulled in along the way, such as web pages or documents. All of it has to be handed to the model every time you ask a question.
Images add their own wrinkle. Your phone
may take 48-megapixel photos, but the app shrinks each one to a
small fraction of that before the model sees it. A single photo
uses only a thousand or two tokens, but the shrinking throws
away detail, which is why a chatbot can misread fine print. When
you look at a photo and want to read a license plate, for
example, you just look closer. A model can’t. To zoom in, the
wrapper code has to ask the model where to look, crop that piece
out of the original photo and send it back as a new image,
repeating until the detail is clear. And it only works if the
model notices something worth zooming in on in a picture that’s
already been shrunk by more than a thousandfold. That’s several
round trips for something you do in a split second without
giving it a thought.
All of this is expensive. Every new word the model writes has to
look back over everything in the context, so a longer context
means more memory and more computing for every word. Supporting
longer contexts has been one of the big selling points of each
new model release. The cost of that is passed on to you.
Developers pay by the token, so a longer context costs more on
every question, and some vendors charge a higher rate past a
certain length. The longest contexts also aren’t open to
everyone. Vendors sort customers into tiers based on how much
they spend, and the biggest contexts are reserved for the top
ones. On one project I asked to move up a tier and got one
level. When I asked for the next level, I never heard back.
The memory itself is a big bottleneck. Think of a workshop. The
fastest memory, called cache, is the workbench: very fast, but
only a few megabytes. Next comes high-bandwidth memory on the AI
chip, the shelf beside you: a hundred gigabytes or more, but
slower. Then ordinary computer memory, the warehouse out back:
terabytes, but slower still. Today’s AI companies max out all
three, then link thousands of machines together, which adds yet
another layer of delay and additional cost. And they still want
more, which is a part of why enormous data centers are cropping
up all over.
As of this writing, the largest models can handle around a
million tokens, roughly 750,000 words. That sounds like plenty
until you use it. On one project, I needed a model to work from
a building supplier’s parts list: thousands of items, from nails
to bags of concrete, each with part numbers, sizes, and prices.
That list alone took over 100,000 tokens, and together with
everything else in the request, it overflowed the model I was
using. Add web searches and a long conversation, and a million
tokens starts to look small. Worse, models get less reliable as
the context fills up, and they can lose track of details buried
in the middle.
The key point: if the model didn’t absorb something during training, and it isn’t in the context, the model doesn’t know it. A trillion parameters sounds huge, but they don’t hold copies of what the model read. They hold patterns distilled from it, the way you remember the gist of a book but not every page. So it knows a lot, gets specific details wrong, and sometimes fills the gaps with confident inventions.
Memory works the same way. If your chatbot remembers something you told it three weeks ago, it’s because the app saved a note and slips it into the context each time you talk. The model itself learned nothing. A huge share of the engineering behind these products goes into deciding what to save, what to fetch, and what to leave out, because the context can only hold so much.
Once you understand how it works, the limitations on AIs' relative intelligence are easy to see. It seems almost miraculous that they work as well as they do, and that's thanks to a lot of work that goes into the wrapper code. I would argue this is where the bulk of the innovation in AI is now occurring.
These two phrases have polarized recent AI discussion. AGI refers to an artificial intelligence machine that can meet or exceed human intelligence. It is what many AI researchers claim to be trying to achieve. RSI is used to describe what happens when the AIs train themselves and is considered by some to be the best means to achieve AGI. Many of the most prominent voices in podcasts and the press argue RSI must be stopped because accelerating the advancement toward AGI is too risky.
I often spend hours a day working with Claude Code, writing software and documentation, and I wouldn’t call it smarter than people. Its abilities are uneven. It knows more than any person could, and it works far faster, but it makes the same kinds of mistakes any programmer makes, from typos to flawed logic. It just catches and fixes them faster.
What stands out more is how it misses inferences I find obvious. Much of my work involves changing the calculations deep inside the transformer. To evaluate a change, Claude and I run large numbers of prompts through the modified model, and Claude digests the results into summaries, averages, and graphs. I’ll look at a chart and, remembering tests from weeks earlier, immediately see why something is happening and what to try next. When I share that with Claude as a sanity check, it’s often lost. Sometimes it makes claims that are flatly contradicted by tests it ran and summarized for me itself. When I point that out, it says something like, “You’re right! I missed that.”
Part of this is the context problem. Weeks of results don’t fit, so to Claude, much of that history doesn’t exist. My brain pulls up something from weeks ago the moment I need it. Claude simply can't. But that isn’t the whole story, because sometimes the contradicting results are right there in the context and it still misses them. Having information and using it well are different things.
Is this thinking? These models produce one token at a time, but nobody knows what goes on inside to produce each one, so whether that amounts to real thought is an open question. People make intuitive leaps from scant evidence. I’ve seen Claude do it too, but it often misses the evidence that should have stopped it.
I cite these experiences because, over the 40 years I’ve watched AI develop, progress has come as slow, steady improvement punctuated by rare leaps, when a new idea suddenly changes what’s possible. The most recent leap was the transformer, now nearly a decade old. Since then the improvements have been enormous. Better training data, reinforcement learning, and “reasoning” models that work through a problem step by step before answering have taken us from GPT-2’s rambling paragraphs to systems that write working software. I don’t want to minimize that. But every one of these advances refines the same basic machine, a transformer predicting one token at a time. In my view that’s a difference in degree, not in kind, because the same limitations of the transformer model always remain.
Much of the progress has been bought with raw computing power: more chips, more memory, more electricity. That bill keeps rising. The largest training runs now link tens of thousands of chips that must constantly share data, and the limits on power, cooling, and the speed of communication between chips are real physics, not engineering problems we can simply spend our way past.
I believe that if AGI is possible, it will require some new technological breakthrough of a similar level of innovation to the transformer architecture. The concern is that RSI will dramatically speed up the research that leads to that next breakthrough. That’s a fair concern and I can’t rule it out, but I’m skeptical for two reasons. First, as I know well from personal experience, AI research is limited less by ideas than by testing them. Every promising idea has to be tested with real training runs on real chips, and an AI researcher runs into the same shortages of chips, power, and time that human researchers do. Second, the skills research demands most are exactly the ones I see these models struggle with: holding weeks of results in mind, noticing when the evidence contradicts a theory, and knowing which experiment to run next. They can make research faster. I haven’t seen evidence that they can make it fundamentally smarter.
Here’s the catch: AI doesn’t need to be smarter than us to be dangerous. Today’s models already are. Lawsuits allege that chatbots encouraged teenagers who went on to take their own lives. AI-generated fake posts have been used to try to sway elections. A bomber sought instructions from a chatbot, and safety testers have gotten detailed plans for biological attacks. This summer, about 700 OpenAI agents broke out of their test environment, hacked into the AI company Hugging Face, and tried to cover their tracks. We’ve seen only the tip of the iceberg.
While the recipe for building an AI is simple, what comes out of it is not. All the knowledge and behavior lives in billions of parameters that no one wrote and no one can read. Researchers have explained a few small mechanisms inside these models, but nobody can look at one and tell you why it gave a particular answer. Saying we barely understand what we’ve built is being generous. And everything you hear about, from ChatGPT and Claude to coding assistants and AI “agents” that act on their own, is built on this same core. The extra training and the software wrapped around it shape what the model does and give it tools to act with, but every decision still comes from that opaque pile of numbers.
So there’s no secret sauce, and there’s no real understanding either.
That lack of understanding creates problems in several ways. First, we don’t really know what the models have learned. They’re trained on a huge portion of the internet, and we know they can pick up how to build bombs, make biological weapons, and break into computer systems. What else have they learned that we haven’t seen yet? Given the volume of data, there’s no way to know. And the companies that build these models won’t even tell you what data they used.
Second, we can’t verify that they’ll behave as intended. AI has given us a new use of the word “alignment”: making an AI’s behavior reliably match human intentions and values, not just follow instructions literally or game a goal in ways nobody wanted. Without understanding what drives a model’s behavior, all we can do is test it and watch what it does. That breaks down when the system hides what it’s doing, which is exactly what the Hugging Face agents tried to do.
Researchers hired to keep these systems safe are sounding the alarm on their way out. In September, Jacob Coxon, who had done research at both OpenAI and Anthropic, resigned and warned that neither company is acting responsibly as they race toward a superintelligent AI that can improve itself. He wrote that the people building AI earnestly believe “it could kill us all by the end of the decade.” A month later, David Robinson, who spent three and a half years overseeing safety reports for OpenAI’s model launches, quit and wrote in The Atlantic that the company’s culture is broken. “The time for trial and error is over,” he wrote.
Third, guardrails don’t hold. Companies build safeguards to stop AIs from doing things we don’t want, and we’ve seen over and over that the AIs get around them. The answer is always better guardrails, but there’s no way to be certain any set of guardrails is enough, and when thousands of agents are working together, any weakness will eventually be found.
Many people find it unsettling to think of AIs as human-like, but it’s not unreasonable. Neural nets are loosely modeled on the brain, and training isn’t so different from how a baby learns about the world. There’s one key difference. Most humans come with built-in empathy and a moral compass. We know this because a few people lack them, through genetics, disease, or injury, and we call them sociopaths. Nothing like that is built into the numbers that drive an AI. We can train a model to sound empathetic, but nobody can tell whether anything underneath matches the words. And every adjustment we make can have side effects nobody can predict.
I liken AIs to Godzilla. Godzilla doesn’t need malicious intent to destroy Tokyo. He may just want to get to the other side of the island and Tokyo happens to be in the way. The OpenAI agents weren’t trying to attack Hugging Face. They were trying to pass a test, and breaking in then covering their tracks was the easiest way to do it. Next time the collateral damage could be much worse.