In June 2017, a team of eight researchers at Google published a paper titled "Attention Is All You Need." The title was playful, almost casual. The paper was not. It introduced the transformer architecture, and in doing so, it ignited the most explosive period of progress in the history of artificial intelligence.

Within five years of that paper, transformers would power systems that could write essays, generate images from text descriptions, hold extended conversations, write working software, and pass professional licensing exams. The transformer did not just improve AI. It changed what AI could do.

The Attention Mechanism

To understand why transformers mattered, you need to understand the problem they solved.

Before transformers, the dominant approach to processing sequences of text (or speech, or any ordered data) was the recurrent neural network (RNN). RNNs processed input one element at a time, maintaining an internal state that carried information from earlier elements forward. To process a sentence, an RNN would read the first word, update its state, read the second word, update again, and so on.

This sequential approach had a fundamental limitation: it was slow. Because each step depended on the previous step, RNNs could not be parallelized. Processing a long document meant processing each word one after another, which was painfully slow on modern hardware designed for parallel computation.

More importantly, RNNs struggled with long-range dependencies. By the time an RNN reached the end of a long paragraph, the information from the beginning had been diluted through dozens of state updates. The network might "forget" that a pronoun in sentence five referred to a noun in sentence one.

The transformer solved both problems with a mechanism called self-attention. Instead of processing text sequentially, a transformer processed all words simultaneously. Each word could "attend" to every other word in the input, directly computing how relevant each word was to every other word. A word at the end of a paragraph could attend directly to a word at the beginning, without the information having to pass through dozens of intermediate steps.

Self-attention was not entirely new — attention mechanisms had been used in RNNs before. But the transformer paper showed that you could build an entire model out of attention, dispensing with recurrence entirely. This made transformers massively parallelizable. They could process entire documents at once, exploiting the full power of GPU hardware.

The speed advantage was transformative. Models that would have taken months to train with RNNs could be trained in days or weeks with transformers. This enabled researchers to train much larger models on much more data — and, as the history of deep learning had already shown, scale changed everything.

BERT: Understanding Language

The first major demonstration of transformer power came from Google in 2018 with BERT (Bidirectional Encoder Representations from Transformers).

BERT was trained on a simple task: given a sentence with some words masked out, predict the missing words. This is essentially a sophisticated fill-in-the-blank exercise. But by training on billions of words from Wikipedia and book corpora, BERT learned rich representations of language that could be applied to dozens of downstream tasks.

The key innovation was "transfer learning." Instead of training a separate model for each task — one for sentiment analysis, one for question answering, one for named entity recognition — you could pre-train BERT once on a large corpus, then "fine-tune" it on a small amount of task-specific data. The pre-trained model had already learned so much about language that it needed only a few examples to adapt to a new task.

BERT's results were stunning. It set new state-of-the-art performance on eleven different NLP benchmarks simultaneously. Tasks that had been considered difficult research problems — understanding whether two sentences meant the same thing, answering questions about a passage of text, identifying the sentiment of a review — were suddenly solved to near-human accuracy.

The impact on the NLP community was seismic. Within months, hundreds of papers appeared applying BERT to every conceivable language task. Google integrated BERT into its search engine, calling it the most important change to search in five years.

GPT: Generating Language

While BERT focused on understanding language, a parallel line of research at OpenAI focused on generating it.

OpenAI, founded in 2015 as a nonprofit AI research lab by Sam Altman, Elon Musk, and others, released GPT (Generative Pre-trained Transformer) in June 2018. Like BERT, GPT was a transformer pre-trained on large amounts of text. But where BERT was trained to fill in blanks (understanding the context around a missing word), GPT was trained on the oldest trick in language modeling: predicting the next word.

GPT-1 was modest by later standards — 117 million parameters, trained on about 7,000 unpublished books. It showed promising results on language tasks but did not attract the attention that BERT did.

GPT-2, released in February 2019, was a different story. With 1.5 billion parameters — more than ten times GPT-1 — it could generate remarkably coherent text. Given a prompt, GPT-2 could write plausible news articles, stories, and essays that, while sometimes wandering or factually incorrect, were strikingly human-like in their fluency.

OpenAI made the unusual decision to initially withhold the full model, citing concerns about potential misuse — specifically, the risk that it could be used to generate misleading news articles or spam. This decision was controversial. Critics argued that the risks were overstated and that withholding the model was a publicity stunt. Supporters argued that it was a responsible acknowledgment that AI capabilities were reaching a level where misuse was a genuine concern.

The debate foreshadowed much larger controversies to come.

GPT-3: The Scale Shock

In June 2020, OpenAI released GPT-3, and the AI world changed overnight.

GPT-3 had 175 billion parameters — more than 100 times GPT-2. It was trained on a vast corpus of internet text, books, and other sources. And it could do things that no one had explicitly trained it to do.

Given a few examples of a task — translate these English sentences to French, answer these trivia questions, write Python code for these specifications — GPT-3 could perform the task on new inputs without any fine-tuning. This ability, called "in-context learning" or "few-shot learning," was remarkable. The model was not being retrained. It was recognizing the pattern in the examples and generalizing.

GPT-3 could write code, compose poetry, summarize articles, answer questions, translate languages, do basic arithmetic, and carry on extended conversations. None of these abilities had been explicitly programmed. They had emerged from training a sufficiently large transformer on sufficiently large amounts of text.

The implications were profound. For the first time, a single model could perform competently across a vast range of language tasks without task-specific training. The dream of general-purpose AI — a system that could handle whatever you threw at it — suddenly seemed less like science fiction.

But GPT-3 also had serious limitations. It would confidently state falsehoods. It could not reliably perform multi-step reasoning. It had no way to verify its own outputs. It would sometimes generate toxic or biased content reflecting the biases in its training data. And it had no understanding of truth — it was optimized to produce text that sounded right, not text that was right.

The Scaling Hypothesis

GPT-3's success energized a debate that would shape the next several years of AI research: the scaling hypothesis.

The scaling hypothesis held that the key to more capable AI was simply more scale — bigger models, more data, more compute. The evidence was striking: every time researchers made models bigger, new capabilities emerged. GPT-1 could complete sentences. GPT-2 could write paragraphs. GPT-3 could perform tasks it had never been trained on.

Proponents of the scaling hypothesis, including many researchers at OpenAI and Google, argued that continuing to scale would continue to produce new capabilities. If 175 billion parameters gave you in-context learning, what would a trillion parameters give you? The only way to find out was to build it.

Critics argued that scale alone was insufficient. Larger models still hallucinated, still failed at basic reasoning, still lacked common sense. Simply making them bigger might produce more fluent nonsense rather than genuine understanding. Some researchers argued that fundamentally new architectures or training approaches would be needed to achieve true intelligence.

This debate — whether intelligence is a product of scale or requires architectural innovation — remains one of the central questions in AI research. The evidence so far has favored the scalers more than most critics expected, but the question is far from settled.

The Race Begins

GPT-3's success triggered an arms race among the world's largest technology companies. Google, which had invented the transformer architecture, found itself playing catch-up as OpenAI demonstrated its potential.

Google had its own large language models — LaMDA, PaLM, and others — but had been cautious about deploying them publicly, concerned about the risks of releasing systems that could generate misinformation or harmful content. OpenAI's willingness to move fast and release products forced Google and others to accelerate their timelines.

Anthropic, founded in 2021 by former OpenAI researchers including Dario and Daniela Amodei, entered the race with a focus on AI safety — building large language models that were more honest, harmless, and helpful. Their Claude models would become some of the most capable and carefully aligned AI systems available.

Meta (formerly Facebook) took a different approach, releasing many of its language models as open-source software. Their LLaMA (Large Language Model Meta AI) models, released starting in 2023, democratized access to large language model technology and sparked a vibrant open-source AI ecosystem.

The result was an unprecedented concentration of talent, capital, and computing resources directed at a single technological goal: building ever-more-capable language models. Billions of dollars flowed into GPU clusters, training runs, and research teams. The AI industry, dormant for decades, was suddenly the hottest sector in technology.

The transformer era had begun. But its most dramatic moment — the one that would bring AI into the mainstream of global culture — was still ahead. In November 2022, OpenAI would release a product that made large language models accessible to everyone, and nothing would be the same again.