This chapter explains what happened when deep learning found enough scale: more data, more compute and architectures designed to exploit both. By the end, you will understand why the Transformer reorganized the entire discipline by making training radically more parallelizable, what scaling laws are and why they turned scaling into a scientific methodology with testable predictions, and how massive pretraining produced foundation models—a reusable general base for hundreds of different tasks. The chapter is accessible with a basic understanding of machine learning and is especially useful for understanding the logic behind systems such as GPT-3, BERT and Gemini.
The previous three chapters assembled the necessary pieces: representing the world with symbols, mechanizing procedures and learning from data. After 2012, those pieces stopped advancing independently. Data, compute, optimization and architecture began reinforcing one another at unprecedented scale.
This chapter follows that regime change. It is not only about the rise of deep learning. It is about the point at which progress became increasingly driven by a systematic combination of scale, reuse and transfer, eventually producing models that no longer solved one task but entire families of tasks.
AlexNet won ILSVRC 2012 with a result that changed the field's perception: 15.3% top-5 error versus 26.2% for the runner-up. The system was trained on 1.2 million images using two GTX 580 GPUs for six days, combining a deeper-than-usual network with ReLU, dropout, data augmentation and an efficient GPU implementation.
It is important to understand what this result actually demonstrated. AlexNet did not invent convolutional networks from scratch, nor did it single-handedly establish mature scaling laws. What it clearly showed was that when depth, data, regularization and compute reach sufficient critical mass, performance can improve in a way that no longer looks like a small incremental refinement.
The lesson of 2012 was not that architecture had stopped mattering. It was that a good architecture can remain below its potential for years and suddenly take off when hardware and data volume stop being the bottleneck.
2012 changed the regime
AlexNet's jump was not incremental. It was a break: more data, more compute, and GPUs as the training engine broke a trajectory of slow improvement that had been stuck for decades.
Before the jump, ImageNet error was improving slowly. AlexNet reduced it almost in half in a single year.
ImageNet top-5 error
Best 2010
28.2%
Best 2011
25.8%
AlexNet 2012
15.3%
GoogLeNet 2014
6.7%
Human level ref.
~5%
Axis: ImageNet top-5 error (%). Lower is better. AlexNet cut the error by about 10 percentage points at once, after years of 1–2 point improvements.
Three pieces converged at the same time. No single one would have been sufficient.
DATA
1.2M
ImageNet: 1.2 million labeled images. The scale of the training data was incomparably larger than what had been available in earlier years.
COMPUTE
GPU
Two GTX 580 GPUs. The GPUs parallelized thousands of operations simultaneously, turning weeks of CPU work into days of training.
ARCHITECTURE
CNN
A deep convolutional network with ReLU, dropout, and max-pooling. Each layer learned more abstract representations than the previous one.
The message was not "AlexNet won ImageNet." It was "more data + more compute + better representations produce predictable improvements."
BEFORE
Computer vision = manual feature engineering (SIFT, HOG, LBP…). Each domain required different experts.
→
AFTER
The network learns the features by itself from raw data. The same paradigm works in vision, audio, text, and more.
The regime change was not technical: it was epistemological. Data and compute became the dominant levers.
The next major shift arrived with Attention Is All You Need in 2017. The Transformer was not merely another language architecture. It reorganized the problem around attention mechanisms, removing recurrence from the model's core and making training far more parallelizable.
Two especially influential trajectories followed. BERT demonstrated the strength of bidirectional pretraining followed by fine-tuning on specific tasks. GPT-2, and later GPT-3, showed that a sufficiently large autoregressive model could transfer to new tasks directly from context—first with striking zero-shot behavior and then with much stronger few-shot and in-context learning capabilities.
Attention unlocks reuse
The Transformer was not only more accurate. It was the first architecture that made it possible to train a general base and reuse it across dozens of different tasks without retraining from scratch.
RNNs processed tokens sequentially. That prevented parallelization and limited how much context the model could integrate at once.
BEFORE · RNN/LSTM
t₁
→
t₂
→
t₃
→
t₄
Sequential. Token 4 does not see token 1 directly. Long context = gradient degradation.
AFTER · Transformer
t₁
t₂
t₃
t₄
Each token attends to all the others at once. Parallelizable. Full context without degradation.
Self-attention lets any token consult any other token directly. The model learns which relationships matter, without explicit instructions.
1
Pretraining at scale
Billions of tokens. The objective is simple: predict the next one. The network learns the structure of language and the world as a side effect.
2
Transferable representations
Each layer encodes patterns at a different level of abstraction. Those representations are not specific to one task: they are general.
3
Downstream reuse
The same base works for translation, summarization, code, image analysis, and more. The cost of pretraining is amortized across each use.
A single investment in pretraining becomes the base for many different applications. Each downstream adaptation costs a fraction of the original.
PRETRAINING
$10M – $100M
Base model
A one-time investment. General learning of language, code, and the world.
ADAPTATION
Conversational assistant
$1K – $100K
ADAPTATION
Code assistant
$1K – $100K
ADAPTATION
Medicine / Legal / Science
$1K – $100K
ADAPTATION
Vision + language
$1K – $100K
The decisive change was not only better benchmark performance. It changed the economics of AI development: one base, many applications.
This changed the logic of progress. For a long time, each important task required its own model, pipeline and annotated dataset. Large-scale pretraining increasingly made it more effective to train a general base once and then adapt, fine-tune or condition it for specific uses.
The same family of ideas spread beyond text. Vision Transformer carried the paradigm into vision. CLIP aligned images and language at scale. DALL·E and latent diffusion models demonstrated new ways to generate images from language. Later multimodal systems such as Gemini 1.5 reinforced the idea of a single model capable of working with text, images, audio and video within one system.
Scale leaves the lab and enters the product
GPT-3 showed that scaling worked. ChatGPT showed that scaling could reach one hundred million people. The step between them was not more compute: it was alignment.
175 billion parameters. Scale revealed capabilities that smaller models did not have: translation, summarization, code, simple reasoning, without task-specific fine-tuning.
PARAMETERS
175B
×1,590 vs GPT-2 in 2 years
TRAINING TOKENS
300B
CommonCrawl, books, Wikipedia, code
ACCESS
API
Only selected researchers and companies
GPT-3 was impressive but unstable. Its responses depended heavily on how the instruction was phrased. It was not usable by the general public.
There is a gulf between a good base model and a good assistant. Alignment was the missing piece: teaching the model which response humans prefer, not only which response is probable.
BASE MODEL
Text predictor
Completes sequences with high probability. It does not distinguish between useful, harmful, or simply strange responses.
"How do I do X?" → continues according to language probabilities and can generate anything plausible
RLHF
Reinforcement Learning from Human Feedback
ALIGNED MODEL
Assistant
Learns human preferences through comparisons. Follows instructions, avoids harmful responses, and maintains a conversational format.
"How do I do X?" → a structured, safe response in the appropriate tone
ChatGPT was not the most powerful model of its time. It was the first that anyone could use naturally. Its adoption speed was unprecedented.
Time to reach 100 million active users
Netflix
3.5 years
Instagram
2.5 years
TikTok
9 months
ChatGPT
2 months
Source: UBS/Similarweb as cited by Reuters, January 2023. The exact metric varies by source, but the order of magnitude is clear.
Scale stopped being a result in a paper. It became a public interface.
Before scaling laws, no one knew whether doubling compute would produce a better model. Scaling laws showed that it would, and how much better: improvement follows a consistent power-law curve across parameters, data, and compute.
BEFORE · 2019
"Is investing 10× more compute worth it?"
Unknown. You tried it and saw what happened.
→
AFTER · Kaplan et al. 2020
"Is investing 10× more compute worth it?"
Yes. Loss drops by a predictable factor. You can plan for it before training.
Explore what changes as you scale
Small scale
Loss: ~3.5–4.0
Question: What is the capital of France?
Model response: "France is a country. Countries have capitals. A capital is a city. Cities such as London are capitals."
Incoherent · does not answer
Task: Translate "Hello" into Spanish
Model response: "Hola. It is a word. Greeting people is important. Words change across languages."
Does not follow the instruction
Medium scale
Loss: ~2.5–3.0
Question: What is the capital of France?
Model response: "The capital of France is Paris, an important city in Europe."
Correct · follows the instruction
Task: Write Python code that adds two numbers
Model response: "def add(a, b): result = a + b. return result if a is greater than b."
Partial code · syntax errors
Large scale
Loss: ~1.8–2.2
Task: Explain in simple terms why airplanes fly
Model response: "Airplanes fly because their wings are designed so air moves faster over the top than underneath, creating a pressure difference that generates upward lift."
Correct and clear explanation
Task: Write a function that reverses a string in Python
Model response: "def reverse(s): return s[::-1]"
Correct and efficient code
Current frontier
Loss: ~1.2–1.5
Task: Reason step by step: if Maria has 3 brothers and each brother has 2 children, how many nieces and nephews does Maria have?
Model response: "Each of the 3 brothers has 2 children, so the total number of their children is 3 × 2 = 6. All of those children are Maria's nieces and nephews. Maria has 6 nieces and nephews."
Correct multi-step reasoning
Task: Write a unit test for the reverse(s) function above
L(x) ~ x−α The exponent α varies with what you scale: parameters (≈0.076), data (≈0.095), compute (≈0.050). This is an empirical fit, not a universal law. Scaling laws predict average loss, not which specific capabilities will appear as scale increases.
This does not mean scale explains everything. It means that once an architecture and training objective are good enough, increasing resources stops being a secondary implementation choice and becomes part of the system's practical theory.
An important caution belongs here. The literature on emergent abilities has been influential because it describes abrupt performance jumps on some tasks once models cross certain sizes. But later work, such as Are Emergent Abilities of Large Language Models a Mirage?, argues that some of this apparent abruptness can depend on the metric or evaluation method. The prudent conclusion is therefore not that every new capability mysteriously appears at a threshold, but that scale has produced new or much more robust capabilities while the strong interpretation of emergence remains debated.
Real emergence or measurement artifact?
Some capabilities seem to appear suddenly when a model reaches a certain size. But whether that abrupt threshold is real or depends on how we measure it is an open question with important consequences.
One specific task at different scales
Question: "How many r letters are in the word strawberry?"
~1B parameters
"The word strawberry has 2 r letters."
✗ Incorrect
The model does not count characters; it predicts the most likely response from the text distribution.
~7B parameters
"Strawberry has 3 r letters: s-t-r-a-w-b-e-r-r-y."
△ Correct, but without clear traceability
The model gets the count right, but the response does not provide solid evidence that it is operating over character positions.
~70B parameters
"There are 3 r letters in strawberry: at positions 3, 8 and 9."
✓ Correct with reasoning
At this scale, the ability to reason about the internal structure of words appears.
The debate: how should you interpret that improvement?
Strong reading · Wei et al. 2022
Real emergence: the capability does not exist below the threshold and appears suddenly
Task performance
1B
7B
13B
70B
175B
Metric: correct / incorrect (binary)
With a binary metric (correct or not), the jump looks abrupt. The model "cannot" and suddenly "can".
Implication: there are real capability thresholds that cannot be extrapolated from smaller scales.
Cautious reading · Schaeffer et al. 2023
Metric artifact: improvement is gradual but the binary metric makes it look abrupt
Task performance
1B
7B
13B
70B
175B
Metric: partial credit (how many r letters it detects correctly)
With partial credit, progress is continuous. The model improves at each scale, although it does not reach 100% until larger scales.
Implication: there is no "magic" in large-scale thresholds. Emergence may be an artifact of choosing metrics that do not capture partial improvements.
This leads to the framework of foundation models. The central idea is not simply that a model is large, but that it is trained on broad data—typically through large-scale self-supervision—and can then be adapted to a wide variety of downstream tasks.
The technical and economic consequence is enormous. The same base model can serve as reusable infrastructure for writing, summarization, translation, classification, information extraction, code generation, knowledge retrieval and multimodal work, with relatively small adaptations compared with training a new system for every task.
That is the deeper shift of this period. For decades, AI progressed as a collection of specialized systems. With foundation models, the center of gravity moved toward general pretrained bases that are subsequently adapted, aligned or composed for particular uses.
From raw data to product: three stages
A foundation model is not a finished product. Between pretraining and real-world use there are two more stages, each with its own logic, cost, and limitations.
$10M – $100M · weeks of compute
Objective: predict the next token
Input
📚 Books
🌐 Web
💻 Code
📰 Articles
Output
🧠
Base model
Trillions of parameters. General knowledge of language and the world. Powerful, but not reliably steerable through instructions.
$100K – $10M · human interaction
Objective: follow instructions and avoid harm
Input
👤 Human instructions
⚖ Compared preferences
🔄 RLHF / DPO
Output
🤝
Aligned model
Follows instructions, maintains tone, avoids harmful responses. Ready for direct human interaction.
Variable · per call or deployment
The aligned model orchestrated with context and tools
Runtime input
🔧 External tools
📄 Documents / RAG
🧩 Temporary memory
🎯 System instruction
Output
🚀
Product
The aligned model orchestrated with context, tools, and memory. This is what the user experiences.
By 2024, the field had changed structurally. AI was no longer conceived primarily as a set of isolated solutions; it increasingly organized itself around larger, reusable and multimodal base models.
That change prepares the next stage. Scale is no longer only a matter of adding parameters or data. It becomes the foundation from which new problems emerge: more effective memory, better tool use, more active search and a richer relationship with the world beyond text.
Next chapter
Chapter 5 — Beyond the Transformer → — Which limits pure scaling exposed and which directions the field is opening: inference-time memory, active search, world models and robotics.
Why is AlexNet's 2012 success considered a regime change rather than just another milestone?
Before AlexNet, computer vision relied heavily on features hand-designed by human experts: the model learned from those representations but did not learn the representation itself. AlexNet showed that a deep network trained directly on pixels could outperform those systems by such a large margin—15.3% versus 26.2% top-5 error on ImageNet—that it changed the field's perception and demonstrated the leverage of data and compute alongside architecture.
What architectural advantage does the Transformer have over recurrent networks?
Recurrent networks process a sequence step by step, which limits training parallelism and makes long-range dependencies harder to preserve because each step depends on the previous one. With self-attention, the Transformer lets each token directly attend to other positions in the context while training can be massively parallelized. This made it practical to train on data volumes that would previously have been much harder to exploit.
What do scaling laws imply for investment in AI?
They make part of the progress more predictable. In suitable regimes, loss follows approximate power-law relationships with model size, data and compute. This lets teams estimate expected improvements before committing to very expensive training runs and reason about how resources should be allocated, rather than depending only on one-off architectural intuition.
Why is a foundation model considered infrastructure rather than only a technical advance?
Because it is trained once as a general base and then reused across many tasks with comparatively small adaptations, amortizing the high cost of pretraining across many applications. Earlier models were often trained for one narrow task and reused much less broadly. The shift is both technical and economic: the same base can support writing, translation, classification, code generation and multiple modalities, changing the logic of software development.