Neural Network

From PRIMO.ai
Jump to navigation Jump to search

YouTube ... Quora ...Google search ...Google News ...Bing News


Neural Networks (NNs), also referred to as Artificial neural networks (ANNs) or neural nets, are a mathematical system modeled on the human brain that learns skills by finding patterns in data through layers of artificial neurons, outputting predictions or classifications. Neural Networks are computing systems inspired by the biological neural networks that constitute animal brains. Neural networks can be hardware- (neurons are represented by physical components) or software-based (computer models), and can use a variety of topologies and learning algorithms. A Neural Network is based on a collection of connected units or nodes called artificial neurons, which loosely model the neurons in a biological brain. Each connection, like the synapses in a biological brain, can transmit a signal to other neurons. An artificial neuron receives signals then processes them and can signal neurons connected to it. The "signal" at a connection is a real number, and the output of each neuron is computed by some non-linear function of the sum of its inputs. The connections are called edges. Neurons and edges typically have a weight that adjusts as learning proceeds. The weight increases or decreases the strength of the signal at a connection. Neurons may have a threshold such that a signal is sent only if the aggregate signal crosses that threshold. Typically, neurons are aggregated into layers. Different layers may perform different transformations on their inputs. Signals travel from the first layer (the input layer), to the last layer (the output layer), possibly after traversing the layers multiple times. Artificial neural network | Wikipedia

The phrase "modeled on the human brain" deserves a caveat. An artificial neuron is a drastic simplification: it adds up weighted inputs and passes the total through a simple function. A real cortical neuron is far more complex; a 2021 study in Neuron found that a deep network with five to eight layers was needed just to reproduce the input-output behavior of a single detailed model of a cortical pyramidal neuron. Artificial networks borrow the brain's broad idea, many simple units whose connection strengths change with experience, rather than its detailed biology.

The idea's architects span eight decades. Warren McCulloch and Walter Pitts proposed the first mathematical neuron in 1943; Frank Rosenblatt built the learning perceptron in 1958; John Hopfield (1982) and Geoffrey Hinton with Terrence Sejnowski (1985) connected networks to statistical physics through Hopfield networks and Boltzmann machines; David Rumelhart, Hinton, and Ronald Williams popularized training by backpropagation in 1986; Yann LeCun developed trainable convolutional networks; and Yoshua Bengio advanced sequence and language modeling. Hopfield and Hinton received the 2024 Nobel Prize in Physics for this foundational work.

Neural networks address a basic problem: how can a machine learn a complicated input-output relationship, such as recognizing a face, transcribing speech, or translating a sentence, from examples, when nobody can write the rules down? Instead of programming rules, you define a flexible network and adjust its weights until its outputs match the examples. This "connectionist" approach differs from the symbolic AI that dominated early research, which represented knowledge as explicit logical rules. Neural networks store knowledge implicitly, spread across millions or billions of weights, which makes them powerful at perception and pattern recognition but hard to inspect. Today they are the engine of deep learning, large language models, and generative AI.


A neural network learns by adjusting the strengths of connections between many simple units until the whole system reproduces patterns in its data. Its knowledge lives in those weights rather than in written rules, which makes neural networks extraordinarily capable and, at the same time, difficult to explain.


The Mechanism

The artificial neuron. Each neuron takes inputs x1…xn, multiplies each by a weight, adds a bias, and passes the sum through an activation function such as ReLU, sigmoid, or tanh. The weights say how much each input matters; the bias shifts the threshold; the activation adds the nonlinearity that lets networks model curved, complex relationships.

Layers. Neurons are grouped into layers. The input layer receives raw data, hidden layers transform it, and the output layer produces a prediction, often through a softmax that turns scores into probabilities. A network with many hidden layers is a deep neural network (see below).

Learning. Training repeats a simple loop:

  1. Forward pass: compute the network's output for a batch of examples.
  2. Loss: measure the error with a loss function.
  3. Backward pass: use backpropagation to compute how each weight contributed to the error.
  4. Update: nudge each weight to reduce the error using gradient descent.

After thousands to billions of such updates, the weights encode the patterns in the data. Choices such as learning rate, network size, and number of layers are hyperparameters set by the designer and tuned by experiment, and data is split into training, validation, and test sets (Train, Validate, and Test) to check that the network generalizes rather than memorizes.

Major network types

Type How it works Typical uses
Perceptron A single neuron with a threshold; learns a straight-line boundary Historical foundation; linear classification
Feedforward network (multilayer perceptron) Signals flow one way through fully connected layers Tabular data, classification, building blocks of larger models
Convolutional network Small shared filters scan across local patches Images, video, medical scans, audio spectrograms
Recurrent network and LSTM Feedback loops carry a memory through a sequence Speech, time series, earlier translation systems
Transformer Attention lets every element weigh every other element Language models, vision, protein and multimodal models
Graph neural network Neurons exchange messages along the edges of a graph Molecules, materials, social and transport networks
Autoencoder and VAE Compress data to a small code, then reconstruct it Denoising, anomaly detection, generative modeling
GAN A generator and a discriminator compete Image synthesis, data augmentation
Hopfield network and Boltzmann machine Network settles into low-energy states that act as memories Associative memory; historical basis of deep belief networks
Spiking neural network Neurons communicate with discrete timed spikes Neuromorphic hardware, low-power sensing
Reservoir computing A fixed random recurrent network; only the output layer is trained Fast learning of chaotic and temporal signals, physical computing

Dependencies and conditions. Neural networks need representative data, enough capacity for the task, an architecture whose built-in assumptions suit the data (convolution for images, attention for language), and careful tuning. They can fail silently on inputs unlike their training data (Out-of-Distribution (OOD) Generalization).

Where uncertainty remains. We can build and train networks reliably, but explaining what a trained network has learned, and proving when it will generalize, remain active research problems (see Opening the Black Box and Explainable AI).

The Physics

The neuron as an equation. A neuron computes

<math>y = \varphi\!\left(\sum_{i=1}^{n} w_i x_i + b\right)</math>

where wi are weights, b is the bias, and φ is the activation function. Without φ, any stack of layers would collapse into a single linear transformation; the nonlinearity is what gives depth its power.

Learning rules. Donald Hebb's 1949 idea that "cells that fire together wire together" corresponds to the rule

<math>\Delta w_{ij} = \eta\, x_i\, x_j</math>

which strengthens connections between co-active neurons. Modern networks instead use gradient descent, which moves every weight against the gradient of the loss:

<math>w \leftarrow w - \eta\, \nabla_w \mathcal{L}</math>

with learning rate η. Backpropagation computes all of these gradients efficiently by applying the chain rule layer by layer.

Energy and statistical physics. Hopfield's 1982 network assigns an energy to every pattern of neuron states si = ±1:

<math>E(\mathbf{s}) = -\frac{1}{2}\sum_{i \neq j} w_{ij}\, s_i\, s_j - \sum_i \theta_i\, s_i</math>

As neurons update, the energy can only fall, so the network slides into a nearby minimum, and stored memories are those minima. This is why a Hopfield network can complete a partial or noisy pattern, much like recalling a whole song from a few notes. A classic Hopfield network stores only about 0.14N patterns reliably with N neurons. The Boltzmann machine adds temperature T and makes states probabilistic,

<math>P(\mathbf{s}) = \frac{e^{-E(\mathbf{s})/T}}{Z}</math>

borrowing the Boltzmann distribution from thermodynamics; learning adjusts weights so the network's equilibrium distribution matches the data. These physics links are why the 2024 Nobel Prize came in physics. Modern "dense" Hopfield networks store exponentially more patterns, and their update rule turns out to be mathematically equivalent to transformer attention.

Approximation and the curse of dimensionality. Universal approximation theorems show that even one hidden layer can approximate any continuous function, but the number of neurons needed can explode with input dimension d. For a smooth function with m degrees of smoothness approximated to accuracy ε, a shallow network may need on the order of

<math>N_{\text{shallow}} = O\!\left(\varepsilon^{-d/m}\right)</math>

units, exponential in d. Tomaso Poggio and colleagues showed that when a function is compositional (built from small functions of a few variables, arranged hierarchically), a deep network whose structure matches the hierarchy needs only roughly

<math>N_{\text{deep}} = O\!\left((d-1)\,\varepsilon^{-2/m}\right)</math>

units for constituent functions of two variables, which grows linearly rather than exponentially in d. Images, speech, and language are plausibly compositional in this sense, which helps explain why deep networks beat shallow ones on them.

Generalization and implicit bias. Networks with far more parameters than training examples should overfit by classical reasoning, yet they often generalize well. One explanation is that gradient descent prefers particular solutions among the many that fit the data: for linearly separable classification with logistic or exponential-type losses, it converges in direction to the maximum-margin solution, and Poggio's group extended related results to deep networks (see the DNN section below).

Information theory. The information bottleneck view describes a good hidden representation T of input X as one that compresses X while preserving information about the label Y:

<math>\min_{p(t \mid x)} \; I(X;T) - \beta\, I(T;Y)</math>

Whether trained networks actually follow this principle is disputed (see Opening the Black Box).

Physical constraints of hardware. Most of a network's computation is matrix multiplication, and on conventional chips moving data between memory and processors costs more energy than the arithmetic itself. The human brain, with roughly 86 billion neurons and on the order of 100 trillion synapses, runs on about 20 watts by computing where memory lives, sparsely and in parallel. Neuromorphic and photonic hardware try to capture some of that efficiency.

Why This Matters

  • Ubiquity. Neural networks underpin image and speech recognition, translation, recommendation systems, fraud detection, driver assistance, medical image analysis, and the large language models behind ChatGPT and similar assistants.
  • Science. Networks predict protein structures, emulate expensive physics simulations, and, as the Polymathic AI program shows, may help reveal new scientific laws when their learned knowledge is extracted and interpreted.
  • Medicine and neuroscience. Neural decoders in brain-computer interfaces now translate brain activity into text and speech for people with paralysis, and artificial networks serve as testable models of parts of the brain (Neuroscience).
  • Engineering and energy. The computational demands of large networks drive the design of chips, data centers, and new computing paradigms, including neuromorphic and photonic processors.
  • Limitations. Networks need large amounts of data, can be fooled by small adversarial changes, may learn shortcuts or biases in their data, are hard to interpret, and can be confidently wrong.
  • What would change if theory catches up with practice. A full theory of why networks generalize would let engineers predict failures before deployment, design networks with fewer trial-and-error experiments, and certify behavior in safety-critical uses.
  • What remains uncertain. How closely networks should imitate the brain, whether interpretability can scale to the largest models, and how far new hardware can reduce the energy cost of neural computation.

Recent Developments

Neural networks win the Nobel Prize

In October 2024, John Hopfield and Geoffrey Hinton received the Nobel Prize in Physics for foundational discoveries and inventions that enable machine learning with artificial neural networks, specifically Hopfield's associative memory network and Hinton's Boltzmann machine, both rooted in statistical physics. The award recognized that ideas about energy landscapes and collective behavior in physical systems shaped the networks now used throughout AI. Both laureates' Nobel lectures are listed under Additional Viewing.

Opening the box: circuit-level interpretability

After years of mostly qualitative visualization, interpretability research has begun producing testable maps of network internals. In March 2025, Anthropic published "circuit tracing" methods that replace parts of a language model with more interpretable components and build attribution graphs showing how internal features combine to produce an output; the study found evidence of multi-step internal reasoning and of planning ahead, such as choosing a rhyming word before writing the line that leads to it. In May 2025 the tools were open-sourced for open-weight models. These methods explain only part of any given computation and require substantial human analysis, but they mark a shift from speculating about what networks do to measuring it.

Decoding inner speech

Neural networks are central to brain-computer interfaces that translate neural activity into language. In a study published in Cell in August 2025, Erin Kunz, Frank Willett, and colleagues at Stanford and the BrainGate consortium recorded from electrode arrays in the motor cortex of four people with severe paralysis. They found that imagined ("inner") speech produced patterns similar to, but weaker than, attempted speech, and decoded inner-speech sentences from a 125,000-word vocabulary with up to 74 percent accuracy. Because inner speech could leak into a decoder unintentionally, the team tested safeguards, including a mental "password" phrase that unlocked decoding and was recognized more than 98 percent of the time. The work builds on 2023 and 2024 studies that decoded attempted speech at conversational speeds; it involved few participants and implanted electrodes, so it is a proof of principle rather than a clinical product.

New hardware for neural networks

  • Neuromorphic scale. Intel's Hala Point system, delivered to Sandia National Laboratories in 2024, packs 1,152 Loihi 2 chips supporting up to 1.15 billion spiking neurons and 128 billion synapses across 140,544 cores, using at most 2,600 watts. Neuromorphic chips compute with sparse, event-driven spikes and keep memory next to computation, which can save large amounts of energy on suitable workloads; the open question is whether mainstream deep learning models can be adapted to run on them without losing accuracy.
  • Photonic processors. In April 2025, Nature published two photonic computing results. A Lightmatter team described a six-chip processor combining photonic tensor cores with electronic control that ran unmodified ResNet, BERT, and Atari reinforcement-learning networks at accuracy comparable to standard 32-bit electronic systems, reporting 65.5 trillion operations per second on 78 watts of electrical and 1.6 watts of optical power. A Lightelligence team reported a large integrated photonic accelerator with very low latency. Light performs matrix multiplication efficiently, but nonlinear activations, analog noise, and conversions between optical and electronic signals remain hurdles.

Physics foundation models

Polymathic AI, launched in 2023 by the Simons Foundation's Flatiron Institute with partners including the University of Cambridge, Princeton, and New York University, is building neural networks trained on scientific data rather than text. In December 2024 it released The Well (about 15 terabytes of physics simulations) and the Multimodal Universe (about 100 terabytes of astronomical data). In 2025 the team released Walrus, a transformer-based foundation model for fluid and continuum dynamics trained on The Well, and AION-1, a multimodal astronomy model presented at NeurIPS in December 2025. Early results suggest that pretraining across many physical systems helps when data for a new system is scarce; how well such models generalize to genuinely new physics remains to be established. See The Next Great Scientific Theory below.

Modern Hopfield networks and attention

A 2020 analysis by Hubert Ramsauer, Sepp Hochreiter, and colleagues showed that "modern" continuous Hopfield networks, which can store exponentially many patterns, perform an update mathematically equivalent to the attention mechanism in transformers. This link has renewed interest in associative-memory views of how large models store and retrieve information, connecting the 1982 Hopfield network directly to today's language models.

Open questions

  • Can a complete theory explain when and why deep networks generalize?
  • Will circuit-level interpretability scale to frontier models with hundreds of billions of parameters?
  • Can neuromorphic or photonic hardware run large modern networks with competitive accuracy and much lower energy?
  • How much more brain-like do networks need to become, and are learning rules more plausible than backpropagation (such as Forward-Forward) competitive at scale?
  • What privacy and consent protections should govern devices that decode thoughts?

Deep Neural Network (DNN)

A deep neural network is simply a neural network with many hidden layers. Depth lets the network build a hierarchy of features: in an image model, early layers respond to edges, middle layers to textures and parts, and later layers to whole objects. Deep networks became practical after 2006, when Hinton's group showed how to pretrain them layer by layer, and dominant after 2012, when GPU-trained convolutional networks won the ImageNet challenge. Residual connections (2015) made networks with more than a hundred layers trainable, and transformers (2017) extended depth to language. See Deep Learning for architectures and training methods.

Why does depth help? The MIT theory. Researchers' study | The MIT research trio of Tomaso Poggio, Andrzej Banburski, and Quianli Liao - Center for Brains, Minds, and Machines) compared deep and shallow networks in which both used identical sets of procedures such as pooling, convolution, linear combinations, a fixed nonlinear function of one variable, and dot products. Why do deep networks have great approximation powers, and tend to achieve better results than shallow networks given they are both universal approximators?

The scientists observed that with convolutional deep neural networks with hierarchical locality, this exponential cost vanishes and becomes more linear again. Then they demonstrated that dimensionality can be avoided for deep networks of the convolutional type for certain types of compositional functions. The implications are that for problems with hierarchical locality, such as image classification, deep networks are exponentially more powerful than shallow networks.

“In approximation theory, both shallow and deep networks are known to approximate any continuous functions at an exponential cost,” the researchers wrote. “However, we proved that for certain types of compositional functions, deep networks of the convolutional type (even without weight sharing) can avoid the curse of dimensionality.”

The team then set out to explain why deep networks, which tend to be over-parameterized, perform well on out-of-sample data. The researchers demonstrated that for classification problems, given a standard deep network, trained with gradient descent algorithms, it is the direction in the parameter space that matters, rather than the norms or the size of the weights.

The implications are that the dynamics of gradient descent on deep networks are equivalent to those with explicit constraints on both the norm and size of the parameters: gradient descent converges to the max-margin solution. This mirrors a known result for linear models. Gradient descent on least-squares regression, started from zero, converges to the minimum-norm (pseudoinverse) solution among all solutions that fit the data; and on linearly separable classification with logistic or exponential loss, it converges in direction to the maximum-margin solution, the same boundary a support vector machine finds (Daniel Soudry and colleagues, 2018).

In effect, the team posit that the act of training deep networks serves to provide implicit regularization and norm control. The scientists attribute the ability for deep networks to generalize, sans explicit capacity controls of a regularization term or constraint on the norm of the weights, to the mathematical computation that shows the unit vector (computed from the solution of gradient descent) remains the same, whether or not the constraint is enforced during gradient descent. In other words, deep networks select minimum norm solutions, hence the gradient flow of deep networks with an exponential-type loss locally minimizes the expected error. A New AI Study May Explain Why Deep Learning Works MIT researchers’ new theory illuminates machine learning’s black box.| Cami Rosso - Psychology Today ..PNAS (Proceedings of the National Academy of Sciences of the United States of America) | T. Poggio, A. Banburski, and Q. Liao

The study appeared in PNAS in 2020 as "Theoretical issues in deep networks." Its approximation result is a mathematical theorem. Its generalization argument is a theoretical account for a specific setting (classification with exponential-type losses under gradient flow), and it is one of several competing explanations alongside implicit-bias, flat-minima, and double-descent analyses. No single theory yet explains generalization across all architectures and training practices.

Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube

Opening the Black Box

In 2017, Ravid Shwartz-Ziv and Naftali Tishby proposed that training a deep network proceeds in two phases: a short "fitting" phase in which layers gain information about the labels, followed by a long "compression" phase in which layers discard information about the input that is irrelevant to the label. They tracked this by plotting each layer's mutual information with the input and the output (the "information plane"), and argued that compression explains generalization. The idea attracted wide attention.

It also drew a strong response. In 2018, Andrew Saxe and colleagues reported that the compression phase appeared with saturating activation functions such as tanh but not with the ReLU activations used in most modern networks, and that networks could generalize well without compressing. Part of the dispute also concerns how mutual information is estimated in networks with continuous values. Today the information bottleneck is regarded as an influential lens on representation learning rather than a settled explanation of why deep networks generalize. Tishby died in 2021; Shwartz-Ziv has continued related work on representation learning.

Since then, the "black box" problem has been attacked more directly. Mechanistic interpretability tries to identify the specific features and circuits a network uses, and 2024 and 2025 work on sparse features and circuit tracing in large language models has made some of those internal computations measurable (see Recent Developments and Explainable AI).

Load video
YouTube
Load video
YouTube

Decoding the Human Mind

Neural networks run in both directions between brains and machines. As decoders, they translate recorded brain activity into words, movements, or images; as models, they help neuroscientists test theories of how the brain processes information. See Neuroscience and Connecting Brains.

  • Attempted speech. In 2023 and 2024, teams at Stanford, UC San Francisco, and UC Davis used implanted electrodes and neural network decoders to turn attempted speech into text at conversational speeds for people with ALS or brainstem stroke, with the best systems reporting word accuracy above 97 percent after training.
  • Inner speech. The 2025 Stanford study described in Recent Developments decoded imagined speech, raising the possibility of faster, more comfortable communication along with new mental-privacy questions.
  • Non-invasive decoding. Studies using fMRI have reconstructed the gist of what a person hears or imagines, though such systems need hours of individual training data and the person's cooperation, and they produce paraphrases rather than exact words.
  • Networks as models of the brain. Networks trained to recognize objects predict neural responses in the primate visual cortex better than earlier models, and language models trained to predict the next word predict activity in human language areas. These parallels are partial: brains learn from far less data and use mechanisms that standard backpropagation does not capture.

Load video
YouTube

Neural Network History

The history of neural networks is long and complex, but it can be traced back to the early days of artificial intelligence research. In 1943, Warren McCulloch and Walter Pitts published a paper in which they proposed a mathematical model of the neuron, treating it as a simple threshold unit that combines inputs and either fires or does not, and showed that networks of such units could compute logical functions. This model, which was based on the idea that neurons in the brain are connected to each other, became the foundation for modern neural networks. In 1949, psychologist Donald Hebb proposed that connections between neurons strengthen when they are active together, an idea that inspired later learning rules.

The inventors of the neural network Walter Pitts and Warren McCulloch pictured here in 1949. Semantic Scholar

In the 1950s, neural networks began to be used to solve real-world problems. One of the most famous early applications was the development of the Perceptron by Frank Rosenblatt in 1958. The Perceptron was a simple neural network that could be used to classify data. It was initially very successful, but it was later shown to be limited in its capabilities: in their 1969 book Perceptrons, Marvin Minsky and Seymour Papert proved that a single-layer perceptron cannot learn functions such as XOR, which are not linearly separable.

In the 1960s and 1970s, neural networks fell out of favor as researchers focused on other approaches to artificial intelligence, partly because of those limitations and because no practical way to train multilayer networks was widely known. The mathematics needed, reverse-mode differentiation, was described by Seppo Linnainmaa in 1970, and Paul Werbos proposed applying it to neural networks in 1974, but the work drew little attention at the time. In 1980, Kunihiko Fukushima's Neocognitron introduced the layered, locally connected design that later inspired convolutional networks.

In the 1980s, there was a resurgence of interest in neural networks. This was due in part to the development of new algorithms that made neural networks more powerful. John Hopfield's 1982 network linked neural computation to the physics of energy landscapes; the Boltzmann machine (David Ackley, Geoffrey Hinton, and Terrence Sejnowski, 1985) introduced a probabilistic learning algorithm; and in 1986 David Rumelhart, Hinton, and Ronald Williams showed in Nature that backpropagation could train hidden layers to learn useful internal representations. The same year, Rumelhart, James McClelland, and the PDP Research Group published the two-volume Parallel Distributed Processing, which made connectionism a major movement in cognitive science.

In the 1990s, neural networks found real applications, including handwriting recognition (LeCun's convolutional networks read a significant share of US bank checks by the late 1990s), speech recognition systems, and early language modeling. The 1997 LSTM by Sepp Hochreiter and Jürgen Schmidhuber solved much of the problem of learning long-range dependencies in sequences. Even so, from the late 1990s through the mid-2000s many researchers favored support vector machines and other kernel methods, which had stronger theory and often matched neural networks on the small datasets of the time. The large datasets and fast GPUs that neural networks needed arrived mainly in the late 2000s. There were many major contributors to neural networks in this period. Some of the most notable include:

  • Geoffrey Hinton, co-authored the 1986 paper that popularized backpropagation, a technique for training neural networks that is still used today (the underlying method had earlier roots in work by Linnainmaa and Werbos); co-invented the Boltzmann machine; and later led the 2006 deep belief network work and the 2012 AlexNet breakthrough with his students.
  • Yann LeCun, developed convolutional neural networks trained with backpropagation (1989) and the LeNet family, which are now widely used in image recognition and were also applied to speech and text before transformers took over most language tasks.
  • Yoshua Bengio, analyzed why recurrent neural networks are hard to train over long sequences (1994), introduced neural probabilistic language models (2003), and co-developed attention for neural machine translation (2014); recurrent networks themselves predate his work, with early forms by Hopfield, Michael Jordan, and Jeffrey Elman.
  • Michael Jordan, designed an early recurrent architecture (the "Jordan network," 1986) and made foundational theoretical contributions connecting neural networks with probabilistic graphical models, variational inference, and statistics.
  • Terrence Sejnowski, co-invented the Boltzmann machine with Hinton, built NETtalk (1987), a network that learned to pronounce English text, developed new learning algorithms and models, including independent component analysis methods, and applied neural networks to a variety of problems in machine learning and neuroscience.

In the 2000s and 2010s, neural networks became far more powerful through deep learning, which uses many layers of artificial neurons. Key steps included Hinton's layer-wise pretraining of deep belief networks (2006), the ImageNet dataset (2009), GPU training, and AlexNet's decisive ImageNet win in 2012. Generative adversarial networks (2014), residual networks (2015), DeepMind's AlphaGo (2016), and the transformer (2017) followed, and Bengio, Hinton, and LeCun shared the 2018 Turing Award. Deep learning has been used to achieve state-of-the-art results on a wide range of problems, including image recognition, speech recognition, and natural language processing.

Today, neural networks are one of the most important tools in artificial intelligence. They are used in a wide range of applications, including self-driving cars, facial recognition, and fraud detection, as well as the large language models behind modern AI assistants (from 2022 onward) and scientific tools such as AlphaFold. The 2024 Nobel Prizes in Physics (Hopfield and Hinton) and Chemistry (in part for AlphaFold) marked their arrival as a core instrument of science.

Year Milestone
1943 McCulloch and Pitts publish a mathematical model of the neuron.
1949 Hebb proposes that co-active neurons strengthen their connections.
1958 Rosenblatt's perceptron learns from examples.
1969 Minsky and Papert show the limits of single-layer perceptrons.
1970 to 1974 Linnainmaa describes reverse-mode differentiation; Werbos proposes it for training networks.
1980 Fukushima's Neocognitron introduces a hierarchical, convolution-like design.
1982 Hopfield network links memory to energy minima.
1985 Ackley, Hinton, and Sejnowski introduce the Boltzmann machine.
1986 Rumelhart, Hinton, and Williams popularize backpropagation; the PDP volumes are published.
1989 LeCun trains convolutional networks with backpropagation for digit recognition.
1997 Hochreiter and Schmidhuber introduce the LSTM.
2006 Hinton and colleagues pretrain deep belief networks layer by layer.
2012 AlexNet wins ImageNet, launching the deep learning era.
2017 The transformer architecture is introduced.
2018 Bengio, Hinton, and LeCun receive the Turing Award.
2022 ChatGPT brings large neural language models to the public.
2024 Hopfield and Hinton receive the Nobel Prize in Physics.

____________

AI will soon become impossible for humans to comprehend – the story of neural networks tells us why | David Beer - The Conversation

David Rumelhart, who had a background in psychology and was a co-author of a set of books published in 1986 that would later drive attention back again towards neural networks, found himself collaborating on the development of neural networks with his colleague Jay McClelland. As well as being colleagues they had also recently encountered each other at a conference in Minnesota where Rumelhart’s talk on “story understanding” had provoked some discussion among the delegates. Following that conference McClelland returned with a thought about how to develop a neural network that might combine models to be more interactive. What matters here is Rumelhart’s recollection of the “hours and hours and hours of tinkering on the computer”...

We sat down and did all this in the computer and built these computer models, and we just didn’t understand them. We didn’t understand why they worked or why they didn’t work or what was critical about them.

Rumelhart's admission still describes much of the field. Engineering success has repeatedly run ahead of explanation, which is why theory (such as the MIT work above) and interpretability research remain so important.

Load video
YouTube

The Next Great Scientific Theory

The Next Great Scientific Theory is Hiding Inside a Neural Network: Machine learning methods such as neural networks are quickly finding uses in everything from text generation to construction cranes. Excitingly, those same tools also promise a new paradigm for scientific discovery. In this Presidential Lecture, Miles Cranmer will outline an innovative approach that leverages neural networks in the scientific process.


Rather than directly modeling data, the approach interprets neural networks trained using the data. Through training, the neural networks can capture the physics underlying the system being studied. By extracting what the neural networks have learned, scientists can improve their theories.


He will also discuss the Polymathic AI initiative, a collaboration between researchers at the Flatiron Institute and scientists around the world. Polymathic AI is designed to spur scientific discovery using similar technology to that powering ChatGPT. Using Polymathic AI, scientists will be able to model a broad range of physical systems across different scales. The Next Great Scientific Theory is Hiding Inside a Neural Network | Miles Cranmer - Simons Foundation

How the approach works. Cranmer and colleagues train a neural network with built-in structure that mirrors the physics, for example a graph network in which particles exchange "messages" analogous to forces. They then use symbolic regression, a search over mathematical expressions (implemented in Cranmer's open-source PySR library), to find compact equations that reproduce what each part of the network learned. In a 2020 demonstration, this recovered known force laws from simulated particle data and proposed a new, more accurate formula for predicting dark matter concentrations in cosmological simulations. The approach works best when the network's structure matches the problem; an equation extracted this way is a hypothesis that still needs testing against new data.

Polymathic AI since the lecture. Cranmer, now at the University of Cambridge, remains part of Polymathic AI, led by Shirley Ho at the Flatiron Institute. The collaboration released The Well and the Multimodal Universe datasets in December 2024 and, in 2025, the Walrus foundation model for fluid-like dynamics and the AION-1 astronomy model (see Recent Developments). Its bet is that, just as language models learn grammar shared across languages, physics models pretrained across many systems will learn structure shared across scientific fields. Whether such models will yield genuinely new theories, rather than fast emulators of known physics, is the question the program is designed to answer.

Load video
YouTube

Resources

Additional Viewing

Load video
YouTube

Nobel Prize Lecture: John J. Hopfield, "Physics is a point of view" (December 2024). Hopfield describes how a career in condensed matter physics led him to neuroscience and to the Hopfield network, in which memories are stable low-energy states of a collective system. The lecture explains the physics thinking behind associative memory covered in The Physics section and the Nobel Prize discussed in Recent Developments. Watch on YouTube

Load video
YouTube

Nobel Prize Lecture: Geoffrey Hinton, "Boltzmann Machines" (December 2024). Hinton explains Hopfield networks, energy-based learning, and Boltzmann machines without equations, and traces how these ideas helped make deep networks trainable. It complements the history section and the statistical physics material on this page. Watch on YouTube

Load video
YouTube

MIT 6.S191: Introduction to Deep Learning, Lecture 1 (2026 edition). Alexander Amini builds up from the single perceptron to multilayer networks, covering activation functions, loss functions, gradient descent, backpropagation, and regularization. It is a clear, current walkthrough of The Mechanism section, with free slides and labs at introtodeeplearning.com. Watch on YouTube

References

Foundations and history

Theory and interpretability

Brains, hardware, and science

Retrieved from "https://primo.ai/w/index.php?title=Neural_Network&oldid=46678"