A complete history of AI

A logical neuron

McCulloch and Pitts describe networks of simplified neurons as logical machinery.

Leads to The perceptron

Can machines think?

Alan Turing replaces an argument about definitions with an observable test of conversation.

Leads to A field gets its name

The perceptron

Rosenblatt turns a learning rule into a machine and a public promise about pattern recognition.

Leads to Learning reaches the hidden layers

Learning reaches the hidden layers

Backpropagation makes multi-layer neural networks practically trainable, reviving connectionism.

Leads to The lines converge at AlexNet

Networks learn what to remember

Long short-term memory gives recurrent networks gates for retaining and discarding information.

A dataset becomes infrastructure

ImageNet supplies the scale and shared benchmark needed to compare visual recognition systems.

Leads to The lines converge at AlexNet

Words acquire geometry

word2vec makes relationships between words visible as directions in a learned vector space.

Leads to Models learn where to look

Models learn where to look

Attention lets a model weigh the parts of an input that matter for each output instead of compressing everything equally.

Leads to Attention is enough

Attention is enough

The transformer removes recurrence and makes sequence learning vastly more parallel.

Leads to Scale becomes an interface · The second convergence

137 billion parameters, sparsely used

Sparsely gated mixture-of-experts models grow capacity without activating every parameter for every token.

Leads to The second convergence

Scale becomes an interface

GPT-3 shows that one large model can perform many tasks from instructions and examples supplied in text.

Leads to People shape the model’s behaviour · AI enters the editor · The second convergence

People shape the model’s behaviour

InstructGPT demonstrates how human preferences can turn raw language modelling into instruction following.

Leads to The second convergence

Compute begins compounding

Moore observes the rapid growth of components on integrated circuits; the substrate starts its long acceleration.

Leads to The GPU becomes programmable

NVIDIA is founded

A company built around computer graphics enters a market whose parallel hardware will later reshape AI.

Leads to The GPU becomes programmable · Parallel compute gets a language

The GPU becomes programmable

Graphics processors move from fixed pipelines toward programmable parallel machines.

Leads to Parallel compute gets a language · The lines converge at AlexNet

Parallel compute gets a language

CUDA lets researchers program NVIDIA GPUs without disguising general computation as graphics.

Leads to The lines converge at AlexNet

Silicon specialises for learning

Tensor cores put matrix multiplication for neural networks directly into high-end GPUs.

Leads to Scale becomes an interface

A field gets its name

A funding proposal from McCarthy, Minsky, Rochester and Shannon coins “artificial intelligence” and sets out a research programme; the workshop it asks for runs at Dartmouth the following summer.

Leads to ELIZA feels like somebody · The first retreat

ELIZA feels like somebody

A small script exposes how readily people attribute understanding to convincing language behaviour.

Leads to The second convergence

The first retreat

The ALPAC report finds machine translation disappointing and helps close a major stream of funding.

The winter deepens

The Lighthill report attacks AI’s failure to scale beyond narrow demonstrations; support contracts again.

Deep Blue defeats Kasparov

A machine beats the reigning world chess champion under match conditions and becomes front-page news.

The lines converge at AlexNet

Old neural ideas, a huge labelled dataset, and two gaming GPUs combine in an ImageNet victory too large to ignore.

Leads to Models learn where to look · AlphaGo’s move 37

AlphaGo’s move 37

The match against Lee Sedol turns machine learning progress into a global cultural event.

AI enters the editor

GitHub Copilot puts generative models into an everyday professional workflow.

Leads to The second convergence

The second convergence

ChatGPT packages language modelling, scale, instruction tuning, compute and a conversational interface for everyone.