Technical Report IDSIA-77-21 (v1), IDSIA, 24 Sep 2021
Scientific Integrity, the 2021 Turing Lecture, and the 2018 Turing Award for Deep Learning
This is a point-for-point critique of ACM's justification of the ACM A. M. Turing Award for deep learning, as well as a critique of the Turing Lecture given by the awardees (published by ACM in July 2021).
In brief, three Europeans went to North America, where they republished
methods and concepts first published by other Europeans whom they did
not cite—not even in later surveys. Instead, they credited each other at
the expense of the field's pioneers. Apparently, the ACM was not aware
of this. This work can also be seen as a short history of deep learning,
at least as far as ACM's erroneous laudation and the Turing Lecture are
Note of September 2021:
Following the great success of massive open online peer review (MOOR) for my
2015 survey of deep learning[DL1]
(now the most cited article ever published in
the journal Neural Networks), this extended version of a
June 2020 article[T20a][R12]
is currently undergoing MOOR as well.
Please send suggestions for improvements and additional relevant
references to firstname.lastname@example.org.
To cite this article in publications use: J. Schmidhuber. Scientific
Integrity, the 2021 Turing Lecture, and the 2018 Turing Award for Deep
Learning. Technical Report IDSIA-77-21 (v1), IDSIA, Lugano, Switzerland,
24 Sep 2021.
Abstract. ACM's 2018 A. M. Turing Award was about
deep learning in artificial neural networks.
ACM lauds Dr. LeCun, Dr. Bengio, and Dr. Hinton (LBH) for work based on algorithms
and conceptual foundations first published by other researchers
whom the awardees failed to cite
(see Executive Summary
ACM explicitly mentions "astonishing" deep learning breakthroughs in 4 fields:
(A) speech recognition,
(B) natural language processing,
(D) computer vision,
as well as "powerful" new deep learning tools in 3 fields:
(VII) medicine, astronomy, materials science.
Most of these breakthroughs and tools, however, were direct consequences of
the breakthroughs of my lab and other labs in the past 3 decades
I correct ACM's distortions of deep learning history (see Sec.
mention no fewer than 10 of our direct priority disputes
with Dr. Bengio & Dr. Hinton (see Sec. XVII, I).
I respond to LBH's recent ACM article (July 2021).
This document (over 20,000 words)
expands material in my Critique of the 2019 Honda Prize[HIN] (~3,000 words).
It has several layers of hierarchical abstraction:
Abstract & Outline (~300 words),
Introduction (~300 words),
Critique of LBH's ACM article (Turing Lecture) of July 2021[DL3a]
(~800 words: a compact list of many issues—hurried readers may start
here, then follow the links to the details in later sections),
Executive summary of what's wrong with ACM's laudation (~1,000 words),
21 comments on 21 claims by ACM (~8,000 words),
Conclusion and Acknowledgments (~2,000 words).
All backed up by over 250 references (~9,000 words).
The text contains numerous hyperlinks to relevant overview sites from the AI Blog.
We must stop crediting the wrong people for inventions made by others.
Instead let's heed the recent call in the journal Nature: let us value "those who ensure that
science is self-correcting."[SV20]
Like those who know me can testify, finding and citing original sources
of scientific and technological innovations is important to me, whether
they are mine or other people's.[DL1-2][HIN][NASC1-9] The present page is offered as a resource for all good computer scientists who share this inclination.
By grounding research in its true intellectual foundations and crediting the original inventors,
I am not diminishing important contributions made by popularizers
of those inventions.
My goal is to encourage the entire community to be more scholarly in its
efforts, to recognize the foundational work that sometimes gets lost in
the frenzy of modern AI and machine learning,
and to fight plagiarism, collusion rings,[LIT21] and systemic academic corruption in all of their more and less subtle forms.[FAKE]
I am also inviting others to contribute additional relevant
references (please send any and all directly to me at email@example.com).
will start with a critique of
LBH's 2021 ACM article[DL3a] which necessitated an extension of the
of this post.[T20a][R12]
Subsequent sections will focus on
contributions praised by
ACM's official justification[T19] of the
2018 A.M. Turing Award[R1]
published in 2019.
After the Executive Summary in Sec. 3, Sec. 4 will split
ACM's full text[T19]
into 21 parts
labeled by "ACM:"
Each part is marked by a blue bar and followed by a critique.
Most of the critiques are based on references to original papers and material from the AI Blog.[AIB][MIR][DEC][HIN]
They'll point out
that highly cited publications of the awardees ignored fundamental
relevant prior work—this may be the reason for some of ACM's misattributions.
As recently as of July 2021, Dr. LeCun, Dr. Bengio, Dr. Hinton (LBH) and the ACM
have continued to promulgate their revisionist "history" of deep learning by
publishing yet another misleading overview of the field, this time based on LBH's Turing Lecture.[DL3a]
In the new piece, LBH credit again themselves
for fundamental work first done by others, and fail to correct
LBH's well-known earlier omissions.[DLC][HIN][T20a]
LBH claim to "briefly describe the origins of deep learning"[DL3a] without even mentioning the world's first working deep learning nets by
Ivakhnenko and Lapa in 1965[DEEP1-2][R8] (see Sec. II).
dedicate an extra section to
their unsupervised pre-training of deep neural networks (NNs) around 2006, without mentioning that
this class of methods was pioneered in 1991[UN-UN2] (see Sec. II, III).
LBH mention the "most popular class of convolutional net architecture
for computer vision," the "ResNet family," without clarifying that
ResNet is just an (open-gated)
the first really deep feedforward NN.[HW1-3]
The so-called "ResNet family" is actually the "Highway Net family"
(see Sec. D, VI).
In this context, LBH devote an extra section to the importance of NN depth,
without mentioning that the relevant breakthroughs emphasized by LBH
were all driven by my lab:[MOST] In 1991, I had the
first very deep NNs based on unsupervised pre-training;[UN-UN2]
soon afterwards our
brought essentially unlimited depth to gradient-based supervised recurrent NNs;[LSTM0-17]
later our Highway Nets[HW1-3] brought it to feedforward NNs.
LBH cite Hinton et al.'s work on speech recognition since 2009 without mentioning our earlier and superior methods
based on LSTM[LSTM0-6] (1990s-2005) and CTC (2006).[CTC]
By the time the Turing Award was handed out,
our CTC-LSTM-based speech recognition (not that of Hinton) had been on most smartphones for years[GSR][GSR15-19][DL4] (see Sec. A, VI, XI, XV). Similarly for machine translation (see Sec. B).
LBH cite Hinton (2012) for "dropout" without mentioning that dropout is just a variant of Hanson's 1990 stochastic delta rule[Drop1-2] (see Sec. XIV).
Several times, LBH mention backpropagation—and LBH's papers on applications of this method—but neither its inventor Linnainmaa (1970)[BP1-5][BPA-C] nor Werbos, who first applied it to NNs in 1982 (see Sec. XII, XIX, XXI).
LBH devote an extra section to rectified linear units (ReLUs), citing
papers of the 2000s by Hinton and his former students, without citing
von der Malsburg who introduced ReLUs in 1973[CMB] (see Sec. XIV).
LBH claim ReLUs enabled deep learning to outperform previous methods for
object recognition, referring to their GPU-based ImageNet 2012 winner
called AlexNet,[GPUCNN4] without mentioning that our earlier groundbreaking deep GPU-based DanNet[GPUCNN1-3,5-8][DAN] did not need ReLUs at all to win 4 earlier object recognition competitions and to achieve superhuman results already in 2011[GPUCNN1-8][R5-6] (see Sec. XIV).
LBH refer to LeCun's work on CNNs, citing neither Fukushima—who created the basic CNN architecture in the 1970s[CNN1-4]—nor Waibel, who in 1987 was the first to combine NNs with convolutions with backpropagation[BP1-6] and weight sharing (see Sec. D,
LBH cite Hinton (1981) for multiplicative gating, without mentioning
Ivakhnenko and Lapa who had multiplicative gating in deep networks
already in 1965[DEEP1-2][R8] (see Sec. II).
LBH cite the "fast weights" of Hinton (1987) without mentioning the
earlier fast weights of von der Malsburg (1981) and Feldman (1982).[FAST,FASTa-b][FWP]
LBH refer to Hinton's 2014 paper on "a high-capacity, short-term
memory" through fast weights without clarifying that this was first
described in the 1991-93 papers on Fast Weight Programmers and linear Transformers[FWP0-1,6] (see Sec. XVI, XVII-2).
dedicate an extra section to attention-based Transformers,[TR1-6] citing Bengio's team (2014) for "soft attention"[ATT14] without citing the much earlier original work of 1991-1993 on soft attention and linear Transformers[FWP,FWP0-2,6][ATT] (see Sec. XVII-1, XVI).
LBH claim that Bengio's team[NPM]
first showed in 2002 on real sentences that "activity vectors can be
used to model the structure inherent in a set of symbol strings by
learning appropriate activity vectors for each symbol and learning
non-linear transformations that allow the activity vectors that
correspond to missing elements of a symbol string to be filled in."
However, this was shown on real sentences already in 1995 in the context
of text compression[SNT] (see Sec. XVI, XVII-1).
LBH cite Bengio's 2014 paper on Generative Adversarial Networks (GANs)[GAN0-1] without mentioning that
GANs are instances
of the Adversarial Curiosity Principle of 1990[AC90-20][MIR](Sec. 5) (see Sec. XVII).
In summation, LBH have repeatedly chosen to ignore the previous well-known critiques[DLC][HIN][T20a] and deep learning surveys,[DL1-2]
and ACM's peer review process failed to catch this. The repetitive
nature of LBH's and ACM's failures to uphold basic scientific standards
represents a serious attack on the integrity of the field of Artificial
If we, in turn, choose to ignore this, then we will be committing a
grievous sin against ourselves and our scientific predecessors.
It is clear from the dilligence with which Turing cited his
such as Gödel and Church[GOD][CHU][TUR] (see Sec. IV),
that he would have never approved of being associated with something like this.
While Dr. LeCun, Dr. Bengio, and Dr. Hinton (LBH for short)
have made useful improvements of algorithms for
artificial neural networks (NNs)
and deep learning (e.g., Sec. I), ACM lauds
them for more visible
work based on fundamental methods whose inventors they did not cite—not even in later surveys
(this may actually explain some of ACM's misattributions). I correct ACM's distortions of deep learning history.
Numerous references can be found under the relevant section links I-XXI
which adhere to the sequential order of ACM's text[T19]
(while this summary groups related sections together).
In contrast to ACM's claims,
NNs for pattern recognition etc. were introduced long before the 1980s.
Deep learning with multilayer perceptrons started in 1965 through Ivakhnenko & Lapa
long before LBH who have never cited them—not even in recent work.
In the 1980s, "modern" gradient-based learning
worked only for rather shallow NNs,
it became really deep in 1991 in my lab,
unsupervised pre-training of NNs,
then through the
Sec. I contains 4 subsections
A, B, C, D
on the 4 deep learning "breakthroughs" explicitly
mentioned by ACM. ACM does not mention that they were
mostly based on the deep learning techniques developed by my team:
A: Speech Recognition (see also Sec. VI & XI & XV): The first superior end-to-end neural speech recognition
combines two methods from my lab: LSTM (1990s-2005) and CTC (2006), which were
applied to speech in 2007.
Hinton (2012) and Bengio (XV)
still used an old hybrid approach of the 1980s and 90s:
Hinton et al. (2012) did not compare it to
our revolutionary CTC-LSTM which was soon on most smartphones.
Sec. B: Natural Language Processing (see also Sec. VI & XI & XVI):
The first superior end-to-end neural machine translation
(soon used for several billions of
translations each day by the big platform companies)
was also based on our LSTM.
Sec. C: Robotics.
Our LSTM trained by Reinforcement Learning (RL) was also the core of the
most visible breakthroughs
in robotics and video games.
Sec. D: Computer Vision
(see also Sec.
XVIII & XIV & XI & VI)
was revolutionized by convolutional NNs (CNNs).
The basic CNN architecture is due to Fukushima (1979).
NNs with convolutions were later (1987) combined by Waibel with backpropagation and weight sharing,
and applied to speech. All before LeCun's CNN work (XVIII).
We showed twice (1991-95 and 2006-10) that
don't need unsupervised
pre-training (in contrast to Hinton's claims). Our DanNet was the first CNN fast & deep enough for
superior computer vision in 2011,
winning 4 image recognition contests in a row
before Hinton's team won one. ResNet (ImageNet 2015 winner)
is an open-gated version of our earlier Highway Nets.
Again ACM recognizes work that failed to cite the pioneers.
Long before Hinton (2012), Hanson (1990) had a variant of dropout,
and v. d. Malsburg (1973) had rectified linear neurons; Hinton did not cite them.
Already in 2011,
deep & fast CNN
more than "halved the error rate for object recognition" (ACM's wording)
in a computer vision contest
(where LeCun participated),
long before Hinton's similar CNN (2012).
Sec. XI: ACM mentions GPU-accelerated NNs
pioneered by Jung & Oh (2004). LBH
did not cite them.
deep GPU-NN of 2010
debunked unsupervised pre-training (introduced by myself in 1991 and later championed by Hinton),
and our GPU-CNN of 2011 (DanNet) was the first
to win contests in computer
vision (explicitly mentioned by ACM).
ACM credits LeCun for developing CNNs. However, the foundations of CNNs were laid earlier by
Fukushima and Waibel (see Sec. D).
ACM also explicitly mentions autonomous driving and medical image analysis.
The first team to win relevant international contests in these fields
through deep CNNs was ours (2011, 2012, 2013).
VII: ACM explicitly mentions medicine and
materials science. Our deep NNs were the
first to win medical imaging competitions
in 2012 and 2013, and the first to apply deep NNs to material defect detection in industry (since 2010).
Sec. XII & XIX & XXI: Modern
was first published by Linnainmaa (1970),
not by LeCun or Hinton or their collaborators (1985) who did not cite Linnainmaa,
not even in later surveys.
Ivakhnenko's deep feedforward nets (since 1965) learned
internal representations long before Hinton's shallower ones (1980s).
Hinton has never cited him.
Sec. XX: ACM credits LeCun for work on
hierarchical feature representation which did not cite Ivakhnenko's much earlier work
on this (since 1965).
Sec. XXI: ACM credits LeCun for work on
automatic differentiation which did not cite its inventor Linnainmaa (1970).
And also for work on
deep learning for graphs that failed to cite
the earlier work by Sperduti & Goller & Küchler & Pollack.
XV: ACM credits Bengio for hybrids of NNs and probabilistic models of sequences.
was not the first on this topic, and is
not important for modern deep learning speech recognition systems (mentioned by ACM) based on our
credits Bengio for neural probabilistic language models.
Our 1995 neural probabilistic text model greatly predates Bengio's.
ACM mentions NNs that learn
We started this in 1990-93
long before LBH
who did not cite the relevant prior work.
Generative Adversarial Networks (GANs, 2010-14) of Bengio's team, an instance of
(1990) which he did not cite.
I list 9 of
our additional priority disputes with Bengio & Hinton (many more than can be explained by chance),
vanishing gradients (1991),
unsupervised pre-training (1991),
compressing or distilling one NN into another (1991),
learning sequential attention with NNs (1990),
fast weight programmers using
outer products (1991),
and other topics.[R2-R6]
Sec. IV is on Turing (1936) and his predecessors
Gödel (1931) and Church (1935).
The 21 comments also contain details of issues raised in the
Critique of LBH's ACM article (Turing Lecture) of July 2021.
In the recent decade of deep learning,
most major AI applications mentioned by ACM
(speech recognition, language translation, etc.) on billions of devices (also healthcare applications)
heavily depended on my lab's deep learning techniques and conceptual foundations,
while LBH's most visible work ignored
essential prior art since the 1960s—see, e.g.,
Sec. II &
But in science, by definition, the facts will always win in the end.
As long as the facts have not yet won it's not yet the end.
In what follows, ACM's full text [T19] is split into 21 parts
labeled by "ACM:"
Each part is marked by a blue bar and followed by a critique.
ACM named Yoshua Bengio, Geoffrey Hinton, and Yann LeCun recipients of
the 2018 ACM A.M. Turing Award for conceptual and engineering
breakthroughs that have made deep neural networks a critical component
of computing. ...
Working independently and together, Hinton, LeCun and Bengio developed
conceptual foundations for the field, identified surprising phenomena
through experiments, and contributed engineering advances that
demonstrated the practical advantages of deep neural networks. In recent
years, deep learning methods have been responsible for astonishing
breakthroughs in computer vision, speech recognition, natural language
processing, and robotics—among other applications.
LBH and their co-workers have contributed certain useful improvements of existing deep learning methods.[CNN2,4][CDI][LAN][RMSP][XAV][ATT14][CAPS]
However, the essential
"conceptual foundations" of deep learning (mentioned by ACM)
were laid by others, e.g., deep learning multilayer perceptrons
that learn internal representations
architectures of recurrent NNs (1943-56)[MC43][K56]
and convolutional NNs (1979),[CNN1]
principles of generative adversarial NNs and artificial curiosity (1990),[AC90,90b][AC20]
unsupervised pre-training for deep NNs (1991),[UN1-2]
vanishing gradients (1991)[VAN1] &
Long Short-Term Memory or LSTM (Sec. A),
GPU-accelerated NNs (2004),[GPUNN][DAN][DAN1][GPUCNN5]
NNs with over 100 layers (2015),[HW1-3][R5]
fast weight programmers (1991).[FWP0-2,6]
Often LBH failed to cite essential prior work, even in their later surveys.[DL3,DL3a][DLC][HIN][MIR](Sec. 21)[R2-R5, R7-R8]
This may explain some of ACM's misattributions.[T19]
Although ACM does not literally claim
that LBH were somehow responsible for the
"astonishing breakthroughs in computer vision, speech recognition, natural language processing, and robotics,"
ACM's wording seems to suggest this.
ACM does not mention that these breakthroughs were
fundamentally derived from
three decades of research that came out of other deep learning groups—including my own (e.g., A & B & C).
The deep NNs
of our team, for example, revolutionised Pattern Recognition and Machine Learning.
By the 2010s,[DEC] they were
heavily used in
academia and industry,[DL4]
in particular, by Microsoft, Google & Facebook, former employers of Hinton & LeCun.
I will focus on the 4 fields explicitly
mentioned by ACM (labeled as A, B, C, D) below:
A. Speech recognition. The first superior end-to-end neural speech recogniser that outperformed the
state of the art was based on two methods from my lab:
Long Short-Term Memory
or LSTM (1990s-2005)[LSTM0-6]
which overcomes the famous
vanishing gradient problem
first analysed by my
student Sepp Hochreiter in 1991.[VAN1]
This happened long before the similar work of Bengio (see Sec. XVII).[MIR]
(Sec. 3,Sec. 4)
LSTM was refined with my student Felix Gers[LSTM2]
through "forget gates" based on end-to-end-differentiable fast weights.[MIR](Sec. 8)[FWP,FWP0-1]
(A2) Connectionist Temporal Classification by my student Alex Graves et al. (2006).[CTC] Our team successfully applied CTC-trained LSTM to speech in 2007[LSTM4] (also with hierarchical LSTM stacks[LSTM14]).
This was very different from previous hybrid methods since the late
1980s which combined NNs and traditional approaches such as hidden
Markov models (HMMs)[BW][BRI][BOU] (Sec. XV). Hinton et al. (2012) still used the old hybrid approach[HYB12] and did not compare it to CTC-LSTM.
In 2009, through the efforts of Alex, CTC-trained LSTM
became the first recurrent NN (RNN) to win international competitions.
He later reused our end-to-end neural speech recognizer[LSTM4][LSTM14] as a postdoc in Hinton's lab.[LSTM8]
By 2015, when compute had become cheap enough,
CTC-LSTM dramatically improved Google's speech recognition.[GSR][GSR15][DL4]
By the time the Turing Award was handed out,
this had been on most smartphones for years;
on-device speech recognition[GSR19]
(not any longer on the server)
is still based on
(see Sec. VI & XI & XV).
B. Natural Language Processing (NLP). The first superior end-to-end neural machine translation was also based on our LSTM.
In 1995, we already had excellent neural probabilistic models
of text[SNT] (see Sec. XVI).
In 2001, we showed that LSTM can learn languages unlearnable by traditional models such as HMMs,[LSTM13]
i.e., a neural "subsymbolic" model suddenly excelled at learning
"symbolic" tasks. Compute still had to get 1000 times cheaper, but by
2016-17, both Google Translate[GT16]—whose whitepaper[WU] mentions LSTM over 50 times—and Facebook Translate[FB17] were based on two connected LSTMs,[S2S] one for incoming texts, and one for outgoing translations—much better than what existed before.[DL4] By 2017, Facebook's users made 30 billion LSTM-based translations per week[FB17][DL4]
(the most popular youtube video needed 2 years to achieve only 6 billion clicks).
See also Sec. VI & XI & XV.
It should be mentioned that further improvements were due to an
tailored by Bengio's team.[ATT14][FWP]
However, such attention mechanisms also
have their roots in my lab (1991);[FWP][FWP0-2,6]
see Sec. XVI.
C. Robotics & RL etc. Since 2003, our team has used LSTM for Reinforcement Learning (RL) and robotics.[LSTM-RL][RPG][LSTMPG]
In the 2010s,
combinations of RL and LSTM have become standard,
in particular, our
LSTM trained by policy gradients—or PGs (2007).[RPG07][RPG][LSTMPG]
For example, in 2018, a PG-trained LSTM was the core of OpenAI's famous Dactyl which learned to control a dextrous robot hand without a teacher.[OAI1][OAI1a]
Similar for Video Games: In 2019, DeepMind (co-founded by a student from my lab) famously
beat a pro player in the game of Starcraft, which is theoretically harder than Chess or Go[DM2] in many ways, using
Alphastar whose brain has a deep LSTM core trained by PG.[DM3]
An RL LSTM (with 84% of the model's total parameter count) also was the core of the famous
which learned to defeat human experts in the
Dota 2 video game (2018).[OAI2]
Bill Gates called this a "huge milestone in advancing artificial intelligence".[OAI2a][MIR](Sec. 4)[LSTMPG]
Apart from A, B, C above,
the 2010s saw many additional LSTM applications, e.g.,
chemistry, molecular design, lip reading, speech synthesis,[AM16]
stock market prediction, self-driving cars,
mapping brain signals to speech,
predicting what's going on in nuclear fusion reactors, and so on.[DEC][DL4]
By 2016, more than a quarter of the power of all the
Tensor Processing Units in Google's data centers
was being used for LSTM (only 5% for the CNNs of Sec. D).[JOU17]
Apparently the first LSTM journal paper[LSTM1][R5] is now the most frequently cited
computer science paper of the 20th century—though citations are a highly questionable measure of true impact.[NAT1]
D. Computer Vision was revolutionized in the 2010s by
a particular feedforward NN called the convolutional NN (CNN).[CNN1-4]
The basic CNN architecture with convolutional and downsampling layers is due to Fukushima (1979).[CNN1] The popular downsampling variant
called max-pooling was introduced by Weng et al. (1993).[CNN3]
In 1987, NNs with convolutions were combined by Waibel with weight sharing and backpropagation.[CNN1a] Waibel did not call this CNNs but TDNNs.
LeCun's team later contributed improvements of CNNs, especially for images[CNN2,4] (see Sec. XVIII).
Finally, my own team showed in 2010[MLP1]
unsupervised pre-training is not necessary
to train deep NNs, contrary to claims by Hinton[VID1] who said that "nobody in their right mind would ever suggest" this. Then we
greatly sped up the training of deep
CNNs (Dan Ciresan et al. 2011).
Our fast GPU-based CNN of 2011[GPUCNN1] known as DanNet[DAN,DAN1][R6]
was a practical breakthrough. It was much deeper and faster than earlier GPU-accelerated
CNNs of 2006.[GPUCNN]
In 2011, DanNet was the first pure deep CNN
to win computer vision contests.
For a while, it enjoyed a monopoly.
From 2011 to 2012 it won every contest it entered,
winning four of them
in a row (15 May 2011, 6 Aug 2011, 1 Mar 2012, 10 Sep 2012).[GPUCNN5]
at IJCNN 2011 in Silicon Valley, DanNet blew away the competition and achieved the first superhuman visual pattern recognition[DAN1] in an international contest (where LeCun's team took a distant second place, with
three times worse performance).
Even the NY Times mentioned this.
DanNet was also the first deep CNN to win:
a Chinese handwriting contest (ICDAR 2011),
an image segmentation contest (ISBI, May 2012),
a contest on object detection in large images (ICPR, 10 Sept 2012), and—
at the same time—a medical imaging contest on cancer detection.[GPUCNN8]
In July 2012, our
CVPR paper on DanNet[GPUCNN3]
hit the computer vision community.
All of this happened before
the similar GPU-accelerated AlexNet (Dec 2012)
of Hinton's student Krizhevsky won the ImageNet[IM09] 2012 contest[GPUCNN4-5][R6] (now also without unsupervised pre-training, citing DanNet).
Our CNN image scanners were 1000 times faster than previous methods.[SCAN]
This attracted tremendous interest from the healthcare industry. Today
IBM, Siemens, Google and many startups are pursuing this approach.
The VGG network (ImageNet 2014 winner)[GPUCNN9]
and other highly cited CNNs[RCNN1-3]
further extended the work of 2011.[MIR](Sec. 19)
ResNet, the ImageNet 2015 winner[HW2] (Dec 2015) which currently gets
more citations per year[MOST]
than any other machine learning paper, is a version (with open gates) of our earlier
Highway Net (May 2015).[HW1-3][R5] The Highway Net is actually the feedforward net version of vanilla LSTM.[LSTM2] It was the first working, really deep feedforward NN with hundreds of layers (previous NNs had at most a few tens of layers).
See also Sec. XVIII & XIV & XI & VI.
While the use of artificial neural networks as a tool to help
computers recognize patterns and simulate human intelligence had been
introduced in the 1980s, ...
Perhaps ACM's lack of knowledge about NN history is the reason why
they praise works by LBH that failed to cite the original work.
In fact, NNs of the kind mentioned by ACM
appeared long before the 1980s.
The most powerful NN architectures (recurrent NNs)
were proposed already in the 1940s/50s[MC43][K56]
(but don't forget prior work in physics since the 1920s[L20][I25][K41][W45]).
Fukushima's now widely used
deep convolutional NN architecture was proposed in the 1970s.[CNN1]
Minsky's simple neural SNARC computer dates back to 1951.
NNs without hidden layers learned in 1958[R58]
started around 1800 when Gauss & Legendre introduced linear
regression and the method of least squares[DL1-2]).
In the early 1960s, interesting ideas
about deeper adaptive NNs[R61,R62]
did not get very far.
Successful learning in deep architectures started in 1965 when
Ivakhnenko & Lapa published the first general, working learning
algorithms for deep multilayer perceptrons with arbitrarily many hidden
layers (already containing the now popular multiplicative gates).[DEEP1-2][DL1-2] A paper of 1971[DEEP2]
already described a deep learning net with 8 layers, trained by their
highly cited method which was still popular in the new millennium,[DL2] especially in Eastern Europe, where much of Machine Learning was born. Ivakhnenko did not call it an NN, but that's what it was.[MIR](Sec. 1)[R8] LBH failed to cite this.
See also Sec.
ACM seems to be influenced by a misleading "history of deep learning" propagated by
LBH & co-authors, e.g., Sejnowski[S20] (see Sec. XIII). It goes more or less like this: "In 1969, Minsky & Papert[M69]
showed that shallow NNs without hidden layers are very limited and the
field was abandoned until a new generation of neural network
researchers took a fresh look at the problem in the 1980s."[S20] However, as mentioned above, the 1969 book[M69] addressed a "problem" of Gauss & Legendre's shallow learning (~1800)[DL1-2] that had already been solved 4 years prior by Ivakhnenko & Lapa's popular deep learning method.[DEEP1-2][DL2]
Minsky was apparently unaware of this and failed to correct it later.[HIN](Sec. I)
In the 1980s, "modern" gradient-based learning worked only for rather shallow NNs
(but see a 1989 paper[MOZ]).
However, it became really deep in 1991 in my lab,[UN-UN3] which has
always focused on the depth in deep learning.
See Sec. 1 of the overview:[MIR]
First Very Deep NNs, Based on Unsupervised Pre-Training (1991).
By 1993, my unsupervised pre-training helped to solve previously unsolvable
"Very Deep Learning" tasks of depth > 1000.[UN2][DL1][UN]
Then, however, we replaced it by the even better, purely supervised LSTM—see Sec. A.[MIR](Sec. 4)
(By 2003, LSTM variants successfully dealt with language problems of depth up to 30,000[LSTM17]
twice my lab
drove the shift
from unsupervised pre-training to purely supervised learning (1991-95; 2006-10).[HIN](Sec. II)[MIR]
Also see Sec.
III. Note that
brought essentially unlimited depth to supervised recurrent NNs; Highway Nets[HW1-3] brought it to feedforward NNs.[MOST]
... by the early 2000s, LeCun, Hinton and Bengio were among a small
group who remained committed to this approach. Though their efforts to
rekindle the AI community's interest in neural networks were initially
met with skepticism, their ideas recently resulted in major
technological advances, and their methodology is now the dominant
paradigm in the field.
Comment: However, it isn't "their" methodology because it
was introduced much earlier
by others (Sec.
As mentioned above, others introduced
deep learning multilayer perceptrons (1965),[DEEP1-2][R8]
architectures of recurrent NNs (1943-56)[MC43][K56]
and convolutional NNs (1979),[CNN1]
principles of generative adversarial NNs and artificial curiosity (1990),[AC90,90b][AC20]
unsupervised pre-training for deep NNs,[UN1-2]
the vanishing gradient problem (1991)[VAN1] &
solutions to it (Sec. A),
GPU-accelerated NNs (2004),[GPUNN][GPUCNN5]
and other foundations.[DL1-2][R2-R8]
Often LBH failed to cite essential prior work.[DLC][HIN][MIR](Sec. 21)
ACM may have been misled by LBH's web site
deeplearning.net which until 2019 advertised
deep learning as "moving beyond shallow machine learning since 2006",[DL7]
referring to Hinton's[UN4] and Bengio's[UN5]
unsupervised layer-wise pre-training for deep NNs
we had this type of deep learning already in 1991;[UN][UN1-2] see Sec.
II & XVII (5).
Not to mention Ivakhnenko's even earlier supervised layer-wise training of deep NNs[DEEP1-2]
which Hinton,[UN4] Bengio,[UN5] and
LBH[DL3,DL3a] did not cite either.
See Sec. X.
The ACM A.M. Turing Award, often referred to as the "Nobel Prize of
Computing," carries a $1 million prize, with financial support provided
by Google, Inc. It is named for Alan M. Turing, the British
mathematician who articulated the mathematical foundation and limits of
Skip this comment if you are not interested in deviating from the topic of
LBH—this comment appears here only because
my comments systematically track the sequential order of ACM's claims.[T19]
ACM's statement on Turing is greatly misleading, like some of its other statements.[T19]
It is correct that Turing "articulated the mathematical foundation and limits of computing." However, many
have done this over the decades, and when it comes to credit assignment in science,
the important question is: Who did it first? It wasn't Turing.
Turing published five years after the groundbreaking work of
the Austrian mathematician Kurt Gödel (1931)[GOD][GOD21,21a] and one year after the American Alonzo Church (1935),[CHU] Turing's PhD advisor. Of course, he cited both of them in his 1936 paper.[TUR]
With that in mind, let us look more closely at the birth of modern computer science.
In the early 1930s, Gödel founded modern theoretical computer science.[GOD][GOD34][LEI21,21a] He introduced a universal coding language (1931-34).[GOD][GOD34-21a] It was
based on the integers,
and allows for formalizing the operations of any digital computer in axiomatic form.
Gödel used it to represent both data (such as axioms and theorems) and programs[VAR13]
(such as proof-generating sequences of operations on the data).
He famously constructed formal statements that talk about the
computation of other formal statements—especially self-referential
statements which imply that they are not decidable, given a
computational theorem prover that systematically enumerates all possible
theorems from an enumerable set of axioms. Thus he identified
fundamental limits of algorithmic theorem proving, computing, and
any type of computation-based AI.[GOD][BIB3][MIR](Sec. 18)[GOD21,21a]
Much of early AI in the 1940s-70s was actually about theorem proving[ZU48][NS56]
and deduction in Gödel style through expert systems and logic programming.
In 1935, Church derived a corollary / extension of Gödel's result by demonstrating that Hilbert & Ackermann's Entscheidungsproblem (decision problem) does not have a general solution.[CHU] To do this, he used his alternative universal coding language called Untyped Lambda Calculus, which forms the basis of the
highly influential programming language LISP.
In 1936, Turing
introduced yet another universal model: the
Turing Machine.[TUR] He rederived the above-mentioned result,[CHU][TUR][HIN][GOD21,21a][TUR21][LEI21,21a]
citing both Gödel and Church.[TUR]
In the same year of 1936, Emil Post published yet another independent universal model of computing,[POS]
also citing Gödel and Church.
Today we know many such models.
Nevertheless, although he was standing on the shoulders of others, Turing
was certainly an important computer science pioneer.
my reply to Hinton
who criticized my website on Turing
without suggesting any fact-based corrections.[HIN])
The Gödel Prize for theoretical computer science is named after Gödel.
The currently more lucrative ACM A. M. Turing Award was created in 1966 for
contributions "of lasting and major technical importance to the computer field."
It is funny—and at the same time embarrassing—that Gödel
(1906-1978) never got one, although he not only laid the foundations of
the "modern" version of the field, but also identified its most famous
open problem "P=NP?" in his famous letter to John von Neumann (1956).[GOD56][URQ10]
Neither did Church (1903-1995). There would have been plenty of time
though—these pioneers died years after the award was introduced.
Likewise, Konrad Zuse (1910-1995)
never got a Turing award despite having
created the world's first working programmable general-purpose computer 1935-41.
His patent application of 1936[ZU36-38][Z36][RO98][ZUS21]
described the digital circuits required by programmable physical hardware,
predating Claude Shannon's 1937 thesis on digital circuit design.[SHA37]
Zuse also created the first high-level programming language in the early 1940s.[BAU][KNU]
Zuse's Z3 computer of 1941 was a working practical device, not just a
theoretical and impractical pen & paper construct like
those of Gödel (1931-34), Church (1935), Turing (1936), and Post (1936).
Ignoring the inevitable storage limitations of any physical computer,
the physical hardware of Z3 was indeed
universal in the modern sense of the
theory papers above—simple arithmetic tricks
can compensate for its lack of an explicit
conditional jump instruction.[RO98]
(BTW, programming a Turing machine or Post machine is much more awkward than that.)
In sum, the two founders of the theory and practice of modern computing never got Turing awards.
"Artificial intelligence is now one of the fastest-growing areas in
all of science and one of the most talked-about topics in society,"
said ACM President Cherri M. Pancake. "The growth of and interest in AI
is due, in no small part, to the recent advances in deep learning for
which Bengio, Hinton and LeCun laid the foundation."
The foundations of deep learning
were actually laid by others much earlier, e.g.,
deep learning multilayer perceptrons
that learn internal representations (1965),[DEEP1-2][R8]
architectures of recurrent NNs (1943-56)[MC43][K56]
and convolutional NNs (1979),[CNN1]
principles of generative adversarial NNs and artificial curiosity (1990),[AC][AC90,90b][AC10][AC20]
unsupervised pre-training for deep NNs (1991),[UN1-2][UN]
vanishing gradients (1991)[VAN1] &
solutions to it (Sec. A),[LSTM0-17][CTC]
supervised GPU-accelerated NNs
record-breaking deep supervised NNs
and contest-winning deep CNNs (2011),[DAN][DAN1][GPUCNN5]
NNs with over 100 layers (2015),[HW1-3][R5]
fast weight programmers (1991),[FWP0-2,6]
Often LBH failed to cite essential prior work.[DL3,DL3a][DLC][HIN][MIR](Sec. 21)[R2-R5,R7,R8,R11]
These technologies are used by billions of people. Anyone who has a
smartphone in their pocket can tangibly experience advances in natural
language processing and computer vision that were not possible just 10
"advances in natural language processing" and in speech
in the past 10 years
came mainly through
the LSTM and CTC developed outside of LBH's groups. They were developed instead by our group[LSTM1-6][CTC] (1991-2007)—see Sec. B & Sec. A. And even the "advances in computer vision" were possible only through the speedups of
supervised NNs and
achieved by our group 2010-2011[MLP1-2][DAN][DAN1][GPUCNN5][R6]
and through Highway Net-like NNs (2015),[HW1-3][R5] although the principles of CNNs were invented and developed by others since the 1970s.[CNN1-4] See Sec. D & XVIII & XIV
as well as Sec. 4 & Sec. 19 of the overview.[MIR]
In addition to the products we use every day, new advances in deep
learning have given scientists powerful new tools—in areas ranging from
medicine, to astronomy, to materials science."
Comment: But who really started this?
ACM explicitly mentions medicine. Our
the first NN to win a medical imaging contest through deep learning
(Sept 2012, on cancer detection).[GPUCNN5,8]
ACM also explicitly mentions materials science. In 2010, we introduced our
deep and fast GPU-based NNs to Arcelor Mittal, the world's largest steel producer,
and were able to greatly improve steel defect detection.[ST]
To the best of my knowledge, this was the first deep learning breakthrough in heavy industry.
All of this happened before the similar GPU-accelerated AlexNet of Hinton's student Krizhevsky won ImageNet 2012.[GPUCNN5][R6]
One year later, our team also won the MICCAI Grand Challenge on
has transformed medical imaging, and
many major companies are using it now (see Sec.
And of course,
our LSTM (see Sec. A & B & C) is also massively used in healthcare and medical diagnosis—a simple Google Scholar search turns up thousands of such articles.
"Deep neural networks are responsible for some of the greatest
advances in modern computer science, helping make substantial progress
on long-standing problems in computer vision, speech recognition, and
natural language understanding," said Jeff Dean, Google Senior Fellow
and SVP, Google AI.
"At the heart of this progress are fundamental techniques developed
starting more than 30 years ago by this year's Turing Award winners,
Yoshua Bengio, Geoffrey Hinton, and Yann LeCun."
As pointed out above,
LBH actually used the "fundamental techniques" invented by others, including our team, often
without citing them.[DL1][DLC][HIN][R2-R4][R7-R8]
By dramatically improving the ability of computers to make sense of
the world, deep neural networks are changing not just the field of
computing, but nearly every field of science and human endeavor."
Machine Learning, Neural Networks and Deep Learning
In traditional computing, a computer program directs the computer with
explicit step-by-step instructions. In deep learning, a subfield of AI
research, the computer is not explicitly told how to solve a particular
task such as object classification. Instead, it uses a learning
algorithm to extract patterns in the data that relate the input data,
such as the pixels of an image, to the desired output such as the label
"cat." The challenge for researchers has been to develop effective
learning algorithms that can modify the weights on the connections in an
artificial neural network so that these weights capture the relevant
patterns in the data.
Geoffrey Hinton, who has been advocating for a machine learning approach
to artificial intelligence since the early 1980s, looked to how the
human brain functions to suggest ways in which machine learning systems
might be developed. Inspired by the brain, he and others proposed
"artificial neural networks" as a cornerstone of their machine learning
However, as mentioned above, those "others" mentioned by ACM
proposed such systems decades before Hinton
who failed to cite them, even in later
work.[HIN][DLC][DL1-2][DEEP1-2][CMB][R7-R8] See Sec.
In computer science, the term "neural networks" refers to systems
composed of layers of relatively simple computing elements called
"neurons" that are simulated in a computer. These "neurons," which only
loosely resemble the neurons in the human brain, influence one another
via weighted connections. By changing the weights on the connections, it
is possible to change the computation performed by the neural network.
Hinton, LeCun and Bengio recognized the importance of building deep
networks using many layers—hence the term "deep learning."
The ancient term "deep learning" (explicitly mentioned by ACM) was actually
first introduced to Machine Learning by Dechter (1986), and to NNs by Aizenberg et al (2000).[DL2] To my knowledge, LBH have never cited them.
(Margin note: our 2005 paper on deep RL[DL6,6a] was
the first machine learning
publication with the word combination "learn deep" in the title.)
LBH started talking about "deep learning ... moving beyond shallow machine learning since 2006",[DL7] referring to their unsupervised pre-training methods of 2006.
See Sec. III.
It is true though that LBH "recognized the importance of building deep networks using many layers." However,
others built careers on this notion long before LBH recognized this.[DEEP1-2][CNN1][HIN][R8][DL1][DLC] Even deep learning through unsupervised pre-training was introduced by others.[UN1-3][R4][HIN](Sec. II)
See also Sec.
The conceptual foundations and engineering advances laid by LeCun,
Bengio and Hinton over a 30-year period were significantly advanced by
the prevalence of powerful graphics processing unit (GPU) computers, as
well as access to massive datasets. In recent years, these and other
factors led to leap-frog advances in technologies such as computer
vision, speech recognition and machine translation.
Again ACM lauds work that failed to cite the pioneers.
As mentioned above,
the essential "conceptual foundations" of deep learning were laid by others
ignored by LBH's papers[HIN][R7-R8][R2-R5] (see Sec.
XII & XIX &
X & XVII).
ACM correctly mentions advancements through GPUs. The first to use GPUs for NNs were Jung & Oh (2004),[GPUNN][GPUCNN5]
apparently never cited by LBH.
our team (Dan Ciresan et al.)
was the one that
made GPU-based NNs fast and deep enough
an important benchmark record,[MLP1-2]
unsupervised pre-training (pioneered by myself in 1991)
is not necessary
to train deep NNs, contrary to Hinton's claims.[VID1]
our CNNs were deep and fast enough[DAN][DAN1][GPUCNN5]
to win competitions in computer
vision (explicitly mentioned by ACM) for the first time[R6] (see Sec. D).
Furthermore, by the mid 2010s, speech recognition and machine translation
(explicitly mentioned by ACM) were actually dominated by LSTM and CTC of our team.[LSTM1-4][CTC]
In particular, as mentioned in Sec. A,
the CTC-LSTM combination (2006-2007) was the first superior end-to-end
neural speech recogniser, while previous methods since the late 1980s
(including Bengio's and Hinton's) combined NNs with traditional models
such as HMMs.[BW][BOU][BRI][HYB12]
As mentioned in Sec. B and XVI, the first superior end-to-end neural machine translation was also based on LSTM.
... Select Technical Accomplishments ...
Backpropagation: In a 1986 paper, "Learning Internal Representations by
Error Propagation," co-authored with David Rumelhart and Ronald
Williams, Hinton demonstrated that the backpropagation algorithm allowed
neural nets to discover their own internal representations of data,
making it possible to use neural nets to solve problems that had
previously been thought to be beyond their reach. The backpropagation
algorithm is standard in most neural networks today.
ACM credits Hinton for work that failed to cite the origins of the backpropagation algorithm.
ACM's statement is "less wrong" than Honda's[HIN](Sec. I) but still
very misleading since non-experts
(and apparently even other award committees[HIN](Sec. I)
are left with the impression that
Hinton and colleagues created this method. They didn't. In fact,
Hinton was co-author of an article on
backpropagation by Rumelhart et al. (1985-86)[RUM]
which did not state that 3 years earlier, Werbos proposed to train NNs in this way
And the article[RUM] even failed to mention Linnainmaa, the inventor of this famous algorithm for credit assignment in networks (1970),[BP1]
also known as "reverse mode of automatic differentiation." In 1960,
Kelley already had a precursor thereof in the field of control theory;[BPA] see also later work of the early 1960s.[BPB][BPC][R7]
By 1985, compute had become about 1,000 times cheaper than in 1970, and
the first desktop computers
had just become accessible in wealthier academic labs. Computational
experiments then demonstrated that backpropagation can yield useful
internal representations in hidden layers of NNs.[RUM] But this was essentially just an experimental analysis of a known method.[BP1-2] And
the authors did not cite the prior art—not even in later surveys.[DL3,DL3a][DLC]
More on the
history of backpropagation
can be found at Scholarpedia[DL2] and in my award-winning survey.[DL1]
Also see Sec. XIX, II.
Some claim that "backpropagation is just the chain rule of Leibniz (1676) & L'Hopital (1696)." No, it is the efficient way of applying the chain rule to big networks with differentiable nodes (there are also many inefficient ways of doing this). It was not published until 1970.[BP1]
recent debate:[HIN] It is true that in 2018,
did not credit himself but his co-author
Rumelhart[RUM] with the "invention" of backpropagation.
Nevertheless, he accepted the Honda Prize
for "creating" the method and for other things he didn't do.[HIN]
Neither in a popular book[AOI]
nor in other recent work[DL3,DL3a] did he
cite Linnainmaa (1970),[BP1] the true creator.[BP4-5]
It should be mentioned
that his 2015 survey[DL3] does cite Werbos (1974) who however described the method correctly only
later in 1982[BP2] and
also failed to cite Linnainmaa[BP1]
(compare Amari's work of 1977[BP6]).
Linnainmaa's method was well-known.[BP5][DL1-2][DLC]
It wasn't created by "lots of different people" as Hinton suggested[AOI][HIN][R11]
but by exactly
one person who published first[BP1] and therefore should get the credit.
Boltzmann Machines: In 1983, with Terrence Sejnowski, Hinton
invented Boltzmann Machines, one of the first neural networks capable of
learning internal representations in neurons that were not part of the
input or output.
Again ACM credits work that failed to cite the pioneers.
I have once called the
Boltzmann Machine (BM)[BM] a
significant contribution to deep
Recently, however, I learnt through a reader that even the BM paper[BM] did not cite prior relevant work
by Sherrington & Kirkpatrick[SK75] and Glauber.[G63]
(Compare related work.[H86][H88][S93])
The BM paper should also have mentioned
that already two decades earlier, in 1965, Ivakhnenko & Lapa
published the first general, working learning algorithms for deep
multilayer perceptrons with arbitrarily many layers.[DEEP1-2][HIN]
networks were fully "capable of learning internal representations in
neurons that were not part of the input or output." The BM paper[BM] did not cite this. LBH have never cited this—not even in recent work. See also
As mentioned in Sec. II, Sejnowski's rather self-serving "history of deep learning" [S20] claims: In 1969, Minsky & Papert[M69]
showed that shallow NNs are very limited "and the field was abandoned
until a new generation of neural network researchers took a fresh look
at the problem in the 1980s."[S20] However, the 1969 book[M69] addressed a "deep learning problem"
(a limitation of Gauss & Legendre's shallow learning around 1800[DL1-2]) that had already been solved four years prior (see Sec. II),
and deep learning research was alive and kicking
also in the 1970s, especially outside of the Anglosphere.[DEEP2][BP6][CNN1][DL1-2]
Improvements to convolutional neural networks: In 2012, with his
students, Alex Krizhevsky and Ilya Sutskever, Hinton improved
convolutional neural networks using rectified linear neurons and dropout
regularization. In the prominent ImageNet competition, Hinton and his
students almost halved the error rate for object recognition and
reshaped the computer vision field.
Comment: Again ACM recognizes work that failed to cite the pioneers.
Rectified linear neurons (ReLUs) were actually known much earlier—see v. d. Malsburg's work (1973).[CMB] Hinton's 2012 paper[GPUCNN4]
did not cite their origins. Instead, it cited another paper
by Hinton which also did not cite the original work.
Dropout is actually a variant of Hanson's much earlier stochastic delta rule (1990).[Drop1-2] Hinton's 2012 paper and his later patent did not cite this either.
Apart from this,
as we showed already in 2011 in a contest where LeCun's team participated as well,[DAN1]
neither dropout nor ReLUs are necessary
to win computer vision competitions and achieve
Sec. D above. Back then, the only really
important CNN-related task was to greatly accelerate the training
of deep CNNs through GPUs.[GPUCNN1,3,5][R6]
Already before ImageNet 2012,[R6]
fast deep CNN called DanNet
neither ReLUs nor dropout / Hanson's rule) had
a monopoly on winning computer vision competitions.[GPUCNN5] It more than "halved the error rate for object recognition" (ACM's wording) in a contest already in 2011[GPUCNN2][DAN,DAN1][R6] long before the similar system of Hinton's student.
See Sec. D
as well as Sec. 19 of the overview.[MIR]
Probabilistic models of sequences: In the 1990s, Bengio combined neural
networks with probabilistic models of sequences, such as hidden Markov
models. These ideas were incorporated into a system used by AT&T/NCR
for reading handwritten checks, were considered a pinnacle of neural
network research in the 1990s, and modern deep learning speech
recognition systems are extending these concepts.
However, such hybrids of NNs and hidden Markov models (HMMs) etc.
since the late 1980s.[BW][BRI][BOU]
It is not true that
"modern deep learning speech recognition systems are extending these concepts"
because they basically abandon HMMs and are based on
two methods from my lab:
and CTC[CTC] (2006), which were applied to speech
CTC-LSTM is end-to-end-neural and thus very different from (and superior to) the hybrid methods since the late 1980s.[BW][BRI][BOU][HYB12]
By the time the 2018 Turing Award was handed out, our
CTC-LSTM-based speech recognition was on most smartphones.
See also Sec. A.
High-dimensional word embeddings and attention: In 2000, Bengio
authored the landmark paper, "A Neural Probabilistic Language Model,"
that introduced high-dimension word embeddings as a representation of
word meaning. Bengio's insights had a huge and lasting impact on natural
language processing tasks including language translation, question
answering, and visual question answering. His group also introduced a
form of attention mechanism which led to breakthroughs in machine
translation and form a key component of sequential processing with deep
5 years earlier, in 1995, we already had a similar, excellent neural probabilistic text model.[SNT] Bengio[NPM] characterizes it only briefly as "related"
(see also Pollack's earlier work on embeddings of words and other structures[PO87][PO90]).
In the 2010s,
the central method in
the mentioned fields of "language translation, question answering, and visual question answering"
was actually the LSTM of our team,[LSTM0-6] which Bloomberg called the "arguably the most commercial AI achievement."[AV1][MIR](Sec. 4) See Sec. B.
A particular attention mechanism tailored to NLP by
Bengio's team[ATT14] has indeed become important.
For example, it helped to further improve Facebook's LSTM-based translation (see Sec. B).
already in 1990-93, we had both of the now common types of
adaptive neural sequential attention: end-to-end-differentiable
"soft" attention in the latent space of Fast Weight Programmers (FWPs),[FWP2][FWP] and "hard" attention (in observation space) in the context of RL[ATT][ATT0-1] (1990).
In fact, the now widely used
attention-based Transformers[TR1-6] are
closely related to my
FWPs of 1991[FWP0-1]
which have become a popular alternative to RNNs.
A traditional slow neural net (NN) learns by gradient descent to program the changes of
the fast weights of
Like RNNs, FWPs can learn to memorize past data.
My FWP of 1991[FWP0-1]
computed its fast weight changes through
additive outer products of self-invented activation patterns
(now often called keys and values for self-attention).[TR1-6][FWP]
Transformers combine this with projections
and softmax. Towards the end of
despite their limited time windows,
started to excel at Natural Language Processing,
a traditional LSTM domain (see Sec. B).
Nevertheless, there are still many language tasks that LSTM can
rapidly learn to solve quickly[LSTM13,17]
(in time proportional to sentence length)
while plain Transformers can't—see[TR3-4]
for additional limitations of Transformers.
For long input sequences, the efficiency of Transformers was improved through
linear Transformers or Performers[TR5-6]
which are formally equivalent to my 1991 FWPs (apart from normalization).[FWP6][FWP]
In 1993, I introduced
the attention terminology[FWP2] now used
in this context,[ATT] and
extended the approach to
RNNs that program themselves.
See[MIR](Sec. 9)[R4] for my related priority dispute on attention with Hinton.
He was the reviewer of my 1990 paper[ATT2]
which summarised in its Section 5 our early work on attention, to my
knowledge the first implemented neural system for combining glimpses
that jointly trains a recognition & prediction component
with an attentional component (the fixation controller).
Two decades later Hinton wrote about
his own work:[ATT3]
"To our knowledge, this is the first implemented system for
combining glimpses that jointly trains a recognition component ... with
an attentional component (the fixation controller)."
Generative adversarial networks: Since 2010, Bengio's papers on
generative deep learning, in particular the Generative Adversarial
Networks (GANs) developed with Ian Goodfellow, have spawned a revolution
in computer vision and computer graphics. In one fascinating
application of this work, computers can actually create original images,
reminiscent of the creativity that is considered a hallmark of human
Again ACM lauds Bengio for work that failed to cite the original work.
GANs[GAN0-1] (2010-2014) are actually
a simple application[AC]
of the adversarial curiosity (AC) principle
from 1990[AC90,90b][AC20] (see also surveys[AC09-10]). This principle
is now widely used for exploration in RL (e.g., Sec. C) and
for image synthesis[GAN1] (also mentioned by ACM in Sec. XVIII).
works as follows. One NN—the controller—probabilistically generates
Another NN—the world model—sees the outputs of the controller and
predicts environmental reactions to them. Using gradient descent, the
predictor NN minimizes its error, while the generator NN tries to make outputs that maximize this error: one net's loss is the other net's gain. 4 years before the GAN paper,[GAN1] a well-known 2010 survey[AC10] summarised the generative adversarial NNs of 1990 as follows: a
"neural network as a predictive world model is used to maximize the
controller's intrinsic reward, which is proportional to the model's
prediction errors" (which are minimized).
GANs are a version of this where the trials are very short (like in
bandit problems) and the environment simply returns 1 or 0 depending on
whether the controller's (or generator's) output is in a given set.[AC20][AC]
early adversarial machine learning settings[S59][H90]
were very different—they
neither involved unsupervised NNs nor were about modeling data nor used gradient descent.[AC20]) Bengio et al. neither cited the original work[AC90,90b][AC20] nor corrected
their erroneous claims[GAN1] about
adversarial NNs using "predictability minimization" (PM) for creating disentangled representations
The priority dispute above was picked up by the popular press, e.g.,
after a particularly notable encounter between me and Bengio's student Dr. Goodfellow at a N(eur)IPS conference.
He gave a talk on GANs, encouraging people to ask questions.
I did, addressing problems in
their NIPS 2014 paper[GAN1]
and some of the erroneous claims it made about my prior work.[AC20]
Subsequent efforts to correct these issues in a common paper didn't work out.
Goodfellow eventually admitted that PM is adversarial (his paper[GAN1] still claims the opposite), but emphasized that it's not generative. However, the even earlier AC[AC90,90b][AC10][AC20] is both adversarial and generative (its generator contains probabilistic units[AC90] like in StyleGANs[GAN2]).
When the authors[GAN1]
did not produce an erratum,
I published one myself in the hopes of correcting the annals of history.[AC20]
Remarkably, Bengio was backed by LeCun who called GANs
"the coolest idea in machine learning in the last twenty years" without mentioning
that they are instances of my earlier work.[R2][AC20]
XVII-1. Additional priority disputes with Bengio's group, mostly going back 3 decades and more
(B2) 3 years after my student Sepp Hochreiter had published his analysis of
vanishing gradient problem,[MIR](Sec. 3)[VAN1] Bengio published his own,[VAN2] without citing Sepp.
At the 1996 N(eur)IPS conference, this dispute
was settled in favor of Sepp.[VAN1]
However, even after a common publication,[VAN3] Bengio published papers[VAN4][XAV]
that cited only his own 1994 paper but not Sepp's original work (1991).
Disturbingly, this has apparently helped him to get more citations for vanishing gradients
than Sepp—another sign that citation counts
are poor indicators of truly pioneering work.[NAT1]
(Margin note: Bengio states[YB20] that in 2018 he
"ranked as the most cited computer scientist worldwide"—the above illustrates what such citation counts are really worth.)
The deontology of science requires:
If one "re-invents" something that was already known,
and only becomes aware of it later,
one must at least clarify it later,[DLC]
and correctly give credit
in all follow-up papers and presentations.
Bengio also claims[YB20] that in 1995
he "introduced the use of a hierarchy of time scales to combat the vanishing gradients issue"
my publications on exactly this topic
date back to 1991-93.[UN0-2][UN]
(B4) Another dispute was on
metalearning (learning to learn—now a hot topic)
which I started in 1987[META1][META] long before Bengio
who suggested in public at N(eur)IPS 2019
that he did it before me.[R3]
Bengio also writes[YB20] that in
1999 he "introduced, for the first time, auto-regressive neural networks for density estimation"
although we used a very similar set-up for text compression
in 1995[SNT]—see Sec. XVI.
Regarding attention-based Transformers,[TR1-6] Bengio[DL3a] cites his own team (2014) for "soft attention" without citing my much earlier original work of 1991-1993 on soft attention and linear Transformers.[FWP,FWP0-2,6]
There is more. For example,
Bengio has also heavily used our LSTM (see Sec. A-C),
but for some reason he introduced in 2014 what he called
"gated recurrent units (GRU)"[LSTMGRU]
for a variant of our vanilla LSTM architecture[LSTM2] (2000) which he did not cite
although our work[LSTM2] was the one that introduced gated recurrent units.
In addition, our team automatically evolved lots of additional LSTM variants and topologies already in 2009[LSTM7] without changing the name of the basic method.
(Margin note: GRU cells lack an important gate and can neither
learn to count[LSTMGRU2] nor learn simple non-regular
also do not work as well for challenging translation tasks,
according to Google Brain.[LSTMGRU3])
XVII-2. Additional priority disputes with Hinton's group, going back 3 decades and more
(H1) The dispute on
for deep NNs.[UN0-4][HIN](Sec. II)[MIR](Sec. 1)
Hinton's paper[UN4] (2006) appeared long after my earlier
work on this[UN0-2]
the first NNs shown to solve very deep problems
(see Sec. II above).[UN]
It was published in 1991-92[UN1] when compute was about 1000 times more expensive than in 2006.
did not mention it—not even in LBH's later
although he and Bengio knew it well (also from discussions by email).
See also Sec. II & III.
(H2) The dispute on
compressing or distilling
one NN into another.[UN0-2][DIST1-2][MIR](Sec. 2)
Hinton[DIST2] (2006) did not cite my much earlier original
work on this (1991),[UN1][UN] not even in his later patent application
(H3) The dispute on
fast weight programmers[FWP][FWP0-4a]
through tensor-like outer products (1991-2016) and their motivation[FWP2][FWP4a][MIR](Sec. 8) (see also Sec. XVI above).
(H4) The dispute on
learning sequential attention
with NNs.[MIR](Sec. 9)
did not mention
our much earlier work on this[ATT1][ATT] although
he was both reviewer and editor of my summary[ATT2] (1990; see Sec. XVI above).
The ten priority disputes mentioned in the present Sec. XVII are not on the only ones.[R4] Remarkably, three of them
are related to the 1991 paper[UN1][UN] which in many ways started what people now call deep learning, going beyond
Ivakhnenko's "early" deep learning[DEEP1-2] (which LBH did not cite either[DLC]—see Sec. II & III).
Most of them go back to work of 1990-91.[MIR]
See Sec. I for additional related issues of credit assignment.
For decades, it seems like much of the more prominent work of Dr. Hinton
and Dr. Bengio has simply been repackaged versions of earlier work
that they produced without proper citation.
Convolutional neural networks: In the 1980s, LeCun developed
convolutional neural networks, a foundational principle in the field,
which, among other advantages, have been essential in making deep
learning more efficient.
In the late 1980s, while
working at the University of Toronto and Bell Labs, LeCun was the first
to train a convolutional neural network system on images of handwritten
digits. Today, convolutional neural networks are an industry standard in
computer vision, as well as in speech recognition, speech synthesis,
image synthesis, and natural language processing. They are used in a
wide variety of applications, including autonomous driving, medical
image analysis, voice-activated assistants, and information filtering.
LeCun's team has made important contributions to CNNs since 1989.[CNN2,4]
However, the basic CNN architecture with convolutional and downsampling layers is actually due to Fukushima (1979).[CNN1] NNs with convolutions were later (1987) combined by Waibel with weight sharing and backpropagation.[CNN1a] Waibel called this TDNN and
also was the first to apply this to speech (explicitly mentioned by ACM).
All of this happened before LeCun's work on CNNs. See Sec. D above and Sec. 21 of the overview of our Annus Mirabilis 1990-1991.[MIR]
ACM explicitly mentions autonomous driving.
The first team to win a relevant international contest through deep CNNs was ours:
at IJCNN 2011 in Silicon Valley, our DanNet[DAN][GPUCNN1-3] won the
traffic sign recognition competition with
while LeCun's team took a distant second place (with
three times worse performance).[DAN1] Again see Sec. D.
ACM explicitly mentions medical image analysis.
The first team to win a medical image analysis competition through deep CNNs was again ours:
at ICPR 2012, our DanNet[GPUCNN1-3] won the
medical imaging contest
(Sept 2012, on detection of mitosis/cancer)[GPUCNN5,7,8]
(before the similar AlexNet won ImageNet 2012[GPUCNN5][R6] and the similar VGG network[GPUCNN9] won ImageNet 2014).
One year later, our team also won the MICCAI Grand Challenge on
This approach has transformed medical imaging.
Many major companies are using it now. See Sec. D & VII.
ACM also addresses image synthesis—see Sec. XVII.
ACM also explicitly mentions speech recognition, speech synthesis,[AM16][DL1]
natural language processing,
voice-activated assistants, and information filtering.
All of these fields were heavily shaped in the 2010s by our non-CNN methods.[DL1][DL4][AM16][GSR][GSR15][GT16][WU][FB17] See
Sec. A, B, VI, XI.
Improving backpropagation algorithms: LeCun proposed an early version of
the backpropagation algorithm (backprop), and gave a clean derivation
of it based on variational principles. His work to speed up
backpropagation algorithms included describing two simple methods to
accelerate learning time.
Comment: ACM recognizes
LeCun for work that did not cite the pioneers of this method.
As mentioned in Sec. XII, backpropagation was actually proposed earlier as a learning method for NNs by Werbos (1982)[BP2-4] (see also Amari's work of 1977[BP6]).
And already in 1970, the modern backpropagation algorithm itself—the
real centerpiece of all this later applied work, also known as the
reverse mode of automatic differentiation—was published by Linnainmaa[BP1,4][R7] (with a "clean derivation," of course).
LeCun has never cited this—not even in
In 1960, Kelley already had a precursor of the algorithm.[BPA] Furthermore, many
besides LeCun have worked "to speed up backpropagation algorithms"[DL1] (ACM's wording). More on the history of backpropagation can be found at Scholarpedia.[DL2][BP4]
Broadening the vision of neural networks: LeCun is also credited
with developing a broader vision for neural networks as a computational
model for a wide range of tasks, introducing in early work a number of
concepts now fundamental in AI. For example, in the context of
recognizing images, he studied how hierarchical feature representation
can be learned in neural networks—a concept that is now routinely used
in many recognition tasks.
However, "hierarchical feature representation" in deep learning networks is what Ivakhnenko & Lapa (1965)[DEEP1-2] (and also Fukushima[CNN1][DL2]) had long before LeCun.
ACM may have been misled by the fact that LeCun has never cited Ivakhnenko—not even in his later survey.[DL3][DLC]
See Sec. D &
Together with Leon Bottou, he proposed the idea, used in every
modern deep learning software, that learning systems can be built as
complex networks of modules where backpropagation is performed through
automatic differentiation. They also proposed deep learning
architectures that can manipulate structured data, such as graphs.
What does ACM mean by "modules"? Neuron-like elements? Bigger modules? Anyway,
LeCun et al. neither cited the origins[BP1] (1970) of this
widely used type of automatic differentiation for differentiable networks of modules[DL2][BP4-5][DLC]
nor a computer program (1980) for automatically deriving and implementing backpropagation
for such systems.[S80] See also
Sec. XIX & XII.
And "deep learning architectures that can manipulate structured data, such as graphs" were
proposed by Sperduti, Goller, and Küchler in the 1990s[GOL][KU][SP93-97]
before LeCun who did not cite them. See also Pollack's even earlier relevant work.[PO87-90]
(Furthermore, "complex networks of modules where backpropagation is performed" were the central theme of my much earlier habilitation thesis (1993).[UN2] For example, our
adaptive subgoal generators
(1991)[HRL0-2] were trained through end-to-end-differentiable chains of such modules.[MIR](Sec. 10)
Same for my
planning and reinforcement learning with recurrent neural world models
(1990).[PLAN][MIR](Sec. 11) Same for my linear transformer-like
fast weight programmers[FWP0-2][FWP][ATT][MIR](Sec. 8) since 1991 (see Sec. XVI)
consisting of chains of several modules.)
In the hard sciences, the only things that
count are the facts. Science is not democratic. If 100 persons claim
one thing, and only one person claims the opposite, but he/she can back
it up through facts, then he/she wins. If you haven't already read it,
see "100 Authors against Einstein."[AH1]
The deontology of science enforces proper scientific standards
and behavior when it comes to identifying prior art and assigning
Unlike politics, science is immune to
ad hominem attacks[AH2-3][HIN]
true to the motto:
"If you cannot dispute a fact-based message, attack the messenger himself."[HIN]
Science has a well-established way of dealing with plagiarism and
priority disputes, based on facts such as time stamps of publications
and patents. Sometimes it may take a while to settle disputes, but in
the end, the facts must always win.
As long as the facts have not yet won it's not yet the end. No fancy
award can ever change that.[HIN]
Dr. Hinton, Dr. LeCun, Dr. Bengio,
and their co-workers have contributed useful improvements of deep learning methods.[CNN2,4][CDI][LAN][RMSP][XAV][ATT14][CAPS]
But their most visible work (praised by ACM) mainly helped to
popularize methods created by other researchers
whom they did not cite
(see, e.g., Sec.
XX, and 2).
My lab is especially affected by ACM's misleading statements (see, e.g.,
Sec. I, A, B, C, D, XVII, VI, and XVI).
As emphasized earlier:[DLC][HIN]
"The inventor of an important method should get credit for inventing
it. They may not always be the one who popularizes it. Then the
popularizer should get credit for popularizing it—but not for inventing
If one "re-invents" something that was already known,
and only becomes aware of it later,
one must at least clarify it later,
and correctly give credit
in follow-up papers and presentations.
It is a sign of our field's immaturity that popularizers
are sometimes still credited for the creations of other researchers whom
ACM (or anyone for that matter) is free to hand out awards to anybody,
but one should not decorate anybody for work based on unmentioned
contributions of others.
To fulfill its mandate,
ACM should revise its statements so that it can preserve the reputation
of the Turing award
and its significance to computer science—others will. Similar for
scientific journals, which "need to make clearer and firmer commitments
to self-correction,"[SV20] as is already the standard in other scientific fields.
Could it be that seemingly unbiased award committees are actually affected by PR efforts
in popular science venues without peer review? For example, the narrator of a popular 2018 Bloomberg video[VID2]
is thanking Hinton for speech recognition and machine translation,
although both were actually done (at production time of the video) on
billions of smartphones by deep learning methods developed in my labs in
Germany and Switzerland (LSTM & CTC; see Sec. A) long before Hinton's methods. Similarly, in 2016, the NY Times published an article[NYT3]
about the new, greatly improved, LSTM-based Google Translate without
even mentioning our LSTM (instead featuring Hinton who had little to do
with it), although
Google's original 2016 paper
on Google Translate[WU] mentions LSTM over 50 times (see Sec. B).
In ad hominem style,[AH2-3]
LeCun stated in the NY Times that "Jürgen ... keeps
claiming credit he doesn't deserve for many, many things",[NYT1] without
providing a single example.
LeCun also called the GANs of Bengio's team[GAN1]
"the coolest idea in machine learning in the last twenty years" without mentioning
GANs are variations
of my work in 1990.[AC90,90b][AC20][R2] According to Bloomberg,[AV2] Bengio has simply "denied my claims" without
backing up his denial by any facts; see Sec. XVII.
It has been requested that
"scientists must be willing to speak out when they see false
information being presented in social media, traditional print or
broadcast press" and "must speak out against false information and fake science in circulation
and forcefully contradict public figures who promote it."[FAKE]
LBH, who called themselves the deep learning conspiracy,[DLC]
and otherwise supported each other through interviews and other PR
at the expense of the true pioneers.
Apparently this has earned them many citations, which
is just another sign that citation counts
are poor indicators of truly pioneering work—see Sec. XVII. As I pointed out in Nature (2011):[NAT1]
like the less-than-worthless collateralized debt obligations that
drove the 2008 financial bubble, citations are easy to print and
inflate, providing an incentive for professors to maximize citation
counts instead of scientific progress—witness how relatively unknown
scientists can now collect more citations than the most influential
founders of their fields.
In fact, many of my critical comments above do address highly cited work.
Our LSTM paper[LSTM1] has got more citations
than any paper by Bengio or LeCun,[R5]
and more per year than any other computer science paper of the 20th century.
Hinton's most cited paper (2012) is the one on GPU-based CNNs.[GPUCNN4][R5] It follows our earlier work on supervised
deep NNs (2010)[MLP1]
(which abandoned the
unsupervised pre-training for deep NNs
[UN][UN0-3] and later
championed by Hinton;[UN4][VID1] see Sec. D).
Hinton (2012)[GPUCNN4] characterizes
our deep and fast DanNet (2011)[GPUCNN1-3] as
"somewhat similar"—DanNet won 4 computer vision contests before Hinton's
AlexNet won one;[R6]
see Sec. D, XIV.
The highly cited VGG network (2014)[GPUCNN9]
further extended our work.
Hinton's 2nd most cited paper[RUM][R5]
is the one on experiments with backpropagation (note that in 2019
his Google Scholar page greatly exaggerated the citation count
of Hinton's paper,[RUM] adding citations
for a book by Rumelhart & McClelland[R5]).
Backpropagation is a previously invented
method[BP1] whose origins
Hinton did not cite—not even in later surveys;[R7] see Sec. XII.
His nets learned internal representations two decades after the nets
of Ivakhnenko whom he has never cited;[DEEP1-2][R7-R8] see Sec. II, XIII.
Bengio's 2nd most cited research paper is the one on GANs (2014),[GAN1] instances of my
(1990)[AC90,90b][AC20][R2] which he did not cite;
see Sec. XVII.
As of 2020, the machine learning paper with the most citations per year
is the one on ResNet (2015),[HW2][R5] a version of our earlier Highway Net,[HW1-3] which was the first working feedforward NN with over 100 layers—see Sec. D (in fact, ResNets are just Highway Nets whose gates remain open).
Hinton's highly cited papers on unsupervised pre-training for deep NNs (2006-)[UN4]
(1991-) by 15 years, but he did not cite them[UN0-2][R4][HIN](Sec. II)—see Sec. II & III.
His papers/patents on dropout and rectified neurons
were preceded by Hanson's[Drop1-2]
and v. d. Malsburg's[CMB] by decades, but he did not cite them—see Sec. Sec. XIV.
As recently as of 2021, ACM published yet another misleading deep learning "survey" by LBH,[DL3a] again heavily citing LBH without
correcting the previous omissions.
Consult the Executive Summary and Sec. I-XXI of this critique for more.
So virtually all the algorithms that have attracted
many citations in the recent deep learning revolution
have their conceptual and technical roots in my labs in Munich and Lugano,[MOST]
apart from the old basic principles
of deep learning MLPs since 1965[DEEP1-2] (see Sec. II, XX)
and backpropagation (1960-70)[BPA][BP1] (see Sec. XIX, XII)
and convolutional NNs since 1979[CNN1-4] (see Sec. XVIII, D).
Here is an overview of our relevant work compressed into a few lines that link to
subsections of the present article, where "A→B" indicates that A conceptually led to B:
(1990s, see Sec. A, B; also for RL, 2003-, see Sec. C)
→ our Highway Net (May 2015) → ResNet (Dec 2015, see Sec. D).
Our adversarial Artificial Curiosity (1990) → GANs (2010s, see Sec. XVII).
our own unsupervised pre-training of deep NNs
(1991, see Sec. II & III)
for recurrent NNs in the 1990s → our LSTM (see Sec. A-C) and
for feedforward NNs in 2010 → our DanNet (2011) → AlexNet (2012); VGG Net (2014) (see Sec. D).
As mentioned earlier,
brought essentially unlimited depth to supervised recurrent NNs in the 1990s; our Highway Nets[HW1-3] brought it to feedforward NNs in May 2015.[MOST]
Even earlier, our DanNet brought
superior computer vision (2011, see Sec. D, XVIII),
medical diagnosis (2012, see Sec. VII, XVIII), and many other applications.[DEC]
Our LSTM brought superior
speech recognition (with our CTC, 2007-15, see Sec. A),
machine translation (2016, see Sec. B),
robotics & video game players (2018-19, see Sec. C),
and many other applications.[DEC]
Fast Weight Programmers (1991, see Sec. XVI) are formally equivalent to linear Transformers (now popular in NLP).
In fact, our methods and conceptual foundations shaped most
of the application areas mentioned by ACM—see, e.g.,
I, A, B, C, D, VII, XVIII.
As mentioned earlier,[MIR](Sec. 21)
when only consulting surveys from the Anglosphere,
it is not always clear[DLC]
that Deep Learning was first conceived outside of it. It started in 1965
in the Ukraine (back then the USSR) with the first nets of arbitrary
depth that really learned.[DEEP1-2][R8] Five years later, modern
was published "next door" in Finland (1970).[BP1] The basic deep convolutional NN architecture (now widely used) was invented in the 1970s in Japan[CNN1] where NNs with convolutions were later (1987) also combined with "weight sharing" and backpropagation.[CNN1a] We are standing on the shoulders of these authors and many others—see 888 references in the survey.[DL1]
Our own work since the 1980s mostly took place in Germany and Switzerland.
Unfortunately, LBH's frequent failures to credit essential prior work by others
cannot serve as a role model for PhD students who are told by their advisors
to perform meticulous research on prior art, and to avoid at all costs
the slightest hint of plagiarism.
It is worrisome that the 2018 Turing award seems to reward LBH for this behavior.
Yes, this critique is also an implicit critique of certain other awards to LBH.[HIN]
It is also related to some of the most popular posts and comments of 2019 at
reddit.com/r/MachineLearning[R1-R12] (the largest machine learning forum with back then over 800k subscribers),
many of them influenced by my overview.[MIR]
Dr. LeCun himself is well aware of the challenges to scientific integrity in our field:[LECP] "...
citing an obscure paper, rather than an accepted paper by a prominent
author is dangerous, and has zero benefits. Sure, author A might be
upset, but who cares about upsetting some guy from the university of
Oriental Syldavia that you will never have to confront at a conference
and who will never be asked to write a letter for your tenure case? On
the other hand, author B might be asked to write a review for your next
paper, your next grant application, or your tenure case. So, voicing the
fact that he doesn't deserve all the credit for the idea is very
dangerous. Hence, you don't cite what's right. You cite what everybody
Note that I am insisting on proper credit assignment not only in my own research field but also in quite disconnected areas,[HIN] as demonstrated by my numerous letters in this regard published in Science and Nature, e.g., on the history of aviation,[NASC1-2] the telephone,[NASC3] the computer,[NASC4-7] resilient robots,[NASC8] and scientists of the 19th century.[NASC9]
As Elvis Presley put it, "Truth is like the sun. You can shut it out for a time, but it ain't goin' away." It is fun to speculate how future supersmart
AI scientists and AI historians
equipped with artificial curiosity[SA17][AC90-AC20][PP-PP2]
will be fascinated by their own roots, and how
will rummage through all available data (old papers, email messages,
videos, etc) to fully understand every little detail of their origins in
human civilization. However, today's scientists won't have to wait for
AI historians to establish proper credit assignment. It is easy enough
to do the right thing right now.
Thanks to many expert reviewers for useful comments. Since science is about self-correction, let me know under firstname.lastname@example.org if you can spot any remaining error. Many additional relevant publications can be found in my
publication page and my
arXiv page. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
J. Schmidhuber (AI Blog, 2021). 3 decades of artificial curiosity & creativity. Our
artificial scientists not only answer given questions but also invent
new questions. They achieve curiosity through: (1990) the principle of
generative adversarial networks, (1991) neural nets that maximise
learning progress, (1995) neural nets that maximise information gain
(optimally since 2011), (1997) adversarial design of surprising
computational experiments, (2006) maximizing compression progress like
scientists/artists/comedians do, (2011) PowerPlay... Since 2012:
applications to real robots.
Making the world differentiable: On using fully recurrent
self-supervised neural networks for dynamic reinforcement learning and
planning in non-stationary environments.
Technical Report FKI-126-90, TUM, Feb 1990, revised Nov 1990.
The first paper on planning with reinforcement learning recurrent neural networks (NNs) (more) and on generative adversarial networks
where a generator NN is fighting a predictor NN in a minimax game
A possibility for implementing curiosity and boredom in
model-building neural controllers.
In J. A. Meyer and S. W. Wilson, editors, Proc. of the
International Conference on Simulation
of Adaptive Behavior: From Animals to
Animats, pages 222-227. MIT Press/Bradford Books, 1991.
J. Schmidhuber. Adaptive confidence and adaptive curiosity. Technical
Report FKI-149-91, Inst. f. Informatik, Tech. Univ. Munich, April 1991.
Curious model-building control systems.
Proc. International Joint Conference on Neural Networks,
Singapore, volume 2, pages 1458-1463. IEEE, 1991.
Optimal Artificial Curiosity, Creativity, Music, and the Fine Arts.
Connection Science, 18(2): 173-187, 2006.
J. Schmidhuber. Art & science as by-products of the search for novel
patterns, or data compressible in unknown yet learnable ways. In M.
Botta (ed.), Et al. Edizioni, 2009, pp. 98-112.
PDF. (More on
artificial scientists and artists.)
J. Schmidhuber. Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990-2010). IEEE Transactions on Autonomous Mental Development, 2(3):230-247, 2010.
With a brief summary of the generative adversarial neural networks of 1990[AC90,90b][AC20]
where a generator NN is fighting a predictor NN in a minimax game
J. Schmidhuber. Generative Adversarial Networks are Special Cases of
Artificial Curiosity (1990) and also Closely Related to Predictability
Neural Networks, Volume 127, p 58-66, 2020.
Hentschel K. (1996) A. v. Brunn: Review of "100 Authors against
Einstein" [March 13, 1931]. In: Hentschel K. (eds) Physics and National
Socialism. Science Networks—Historical Studies, vol 18. Birkhaeuser
F. H. van Eemeren, B. Garssen & B. Meuffels.
The disguised abusive ad hominem empirically investigated: Strategic manoeuvring with direct personal attacks.
Journal Thinking & Reasoning, Vol. 18, 2012, Issue 3, p. 344-364.
D. Walton (PhD Univ. Toronto, 1972), 1998. Ad hominem arguments. University of Alabama Press.
[AIB] J. Schmidhuber. AI Blog.
Includes variants of chapters of the AI Book.
Blog of Werner Vogels, CTO of Amazon (Nov 2016):
"takes advantage of bidirectional long short-term memory (LSTM)
networks using a massive amount of data to train models that convert
letters to sounds and predict the intonation contour. This technology
enables high naturalness, consistent intonation, and accurate processing
[AOI] M. Ford. Architects of Intelligence: The truth about AI from the people building it. Packt Publishing, 2018.
Preface to German edition by J. Schmidhuber.
[ATT] J. Schmidhuber (AI Blog, 2020). 30-year anniversary of end-to-end differentiable sequential neural attention. Plus goal-conditional reinforcement learning. We had both hard attention (1990) and soft attention (1991-93).[FWP] Today, both types are very popular.
[ATT0] J. Schmidhuber and R. Huber.
Learning to generate focus trajectories for attentive vision.
Technical Report FKI-128-90, Institut für Informatik, Technische
Universität München, 1990.
[ATT1] J. Schmidhuber and R. Huber. Learning to generate artificial
fovea trajectories for target detection. International Journal of Neural
Systems, 2(1 & 2):135-141, 1991. Based on TR FKI-128-90, TUM, 1990.
Learning algorithms for networks with internal and external feedback.
In D. S. Touretzky, J. L. Elman, T. J. Sejnowski, and G. E. Hinton,
editors, Proc. of the 1990 Connectionist Models Summer School, pages
52-61. San Mateo, CA: Morgan Kaufmann, 1990.
H. Larochelle, G. E. Hinton. Learning to combine foveal glimpses with a third-order Boltzmann machine. NIPS 2010. Very similar to [ATT0-2].
D. Bahdanau, K. Cho, Y. Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. 2014-16.
[AV1] A. Vance. Google Amazon and Facebook Owe Jürgen Schmidhuber a
Fortune—This Man Is the Godfather the AI Community Wants to Forget.
Bloomberg, May 15, 2018.
[AV2] A. Vance. Apple and Its Rivals Bet Their Futures on These Men's Dreams. Business Week,
Bloomberg, May 17, 2018.
F. L. Bauer, H. Woessner (1972). The "Plankalkül" of Konrad Zuse: A Forerunner of Today's Programming Languages.
A local learning algorithm for dynamic feedforward and
Connection Science, 1(4):403-412, 1989.
(The Neural Bucket Brigade—figures omitted!).
Compare TR FKI-124-90, TUM, 1990.
W. Bibel (2003).
Mosaiksteine einer Wissenschaft vom Geiste. Invited talk at
the conference on AI and Gödel, Arnoldsheim, 4-6 April 2003.
D. Ackley, G. Hinton, T. Sejnowski (1985). A Learning Algorithm for Boltzmann Machines. Cognitive Science, 9(1):147-169.
[BOU] H Bourlard, N Morgan (1993). Connectionist speech recognition. Kluwer, 1993.
H. J. Kelley. Gradient Theory of Optimal Flight Paths. ARS Journal, Vol. 30, No. 10, pp. 947-954, 1960.
Precursor of modern backpropagation.[BP1-4]
A. E. Bryson. A gradient method for optimizing multi-stage allocation
processes. Proc. Harvard Univ. Symposium on digital computers and their
S. E. Dreyfus. The numerical solution of variational problems. Journal
of Mathematical Analysis and Applications, 5(1): 30-45, 1962.
[BP1] S. Linnainmaa. The representation of the cumulative rounding error
of an algorithm as a Taylor expansion of the local rounding errors.
Master's Thesis (in Finnish), Univ. Helsinki, 1970.
See chapters 6-7 and FORTRAN code on pages 58-60.
See also BIT 16, 146-160, 1976.
The first publication on "modern" backpropagation, also known as the reverse mode of automatic differentiation.
[BP2] P. J. Werbos. Applications of advances in nonlinear sensitivity
analysis. In R. Drenick, F. Kozin, (eds): System Modeling and
Optimization: Proc. IFIP,
First application of backpropagation[BP1] to NNs (concretizing thoughts in his 1974 thesis).
[BP4] J. Schmidhuber (AI Blog, 2014; updated 2020).
Who invented backpropagation?
A. Griewank (2012). Who invented the reverse mode of differentiation?
Documenta Mathematica, Extra Volume ISMP (2012): 389-400.
S. I. Amari (1977).
Neural Theory of Association and Concept Formation.
Biological Cybernetics, vol. 26, p. 175-185, 1977.
See Section 3.1 on using gradient descent for learning in multilayer networks.
P. J. Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE 78.10, 1550-1560, 1990.
R. J. Williams and D. Zipser. Gradient-based learning algorithms for
recurrent networks. In: Backpropagation: Theory, architectures, and
applications, p 433, 1995.
[BRI] Bridle, J.S. (1990). Alpha-Nets: A Recurrent "Neural" Network
Architecture with a Hidden Markov Model Interpretation, Speech
Communication, vol. 9, no. 1, pp. 83-92.
[BW] H. Bourlard, C. J. Wellekens (1989).
Links between Markov models and multilayer perceptrons. NIPS 1989, p. 502-510.
S. Sabour, N. Frosst, G. E. Hinton (2017).
Dynamic routing between capsules. Proc. NIPS 2017, pp. 3856-3866.
G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural computation 14.8 (2002): 1771-1800.
A. Church (1935). An unsolvable problem of elementary number theory.
Bulletin of the American Mathematical Society, 41: 332-333. Abstract of a
talk given on 19 April 1935, to the American Mathematical Society.
Also in American Journal of Mathematics, 58(2), 345-363 (1 Apr 1936).
First explicit proof that the Entscheidungsproblem (decision problem) does not have a general solution.
C. v. d. Malsburg (1973).
Self-Organization of Orientation Sensitive Cells in the Striate Cortex. Kybernetik, 14:85-100, 1973. See
Table 1 for rectified linear units or ReLUs. Possibly this was also the
first work on applying an EM algorithm to neural nets.
[CNN1] K. Fukushima: Neural network model for a mechanism of pattern
recognition unaffected by shift in position—Neocognitron.
Trans. IECE, vol. J62-A, no. 10, pp. 658-665, 1979.
The first deep convolutional neural network architecture, with
alternating convolutional layers and downsampling layers. In Japanese.
English version: [CNN1+]. More in Scholarpedia.
K. Fukushima: Neocognitron: a self-organizing neural network model for a
mechanism of pattern recognition unaffected by shift in position.
Biological Cybernetics, vol. 36, no. 4, pp. 193-202 (April 1980).
[CNN1a] A. Waibel. Phoneme Recognition Using Time-Delay Neural Networks. Meeting of IEICE, Tokyo, Japan, 1987. First application of backpropagation[BP1][BP2] and weight-sharing
to a convolutional architecture.
[CNN1b] A. Waibel, T. Hanazawa, G. Hinton, K. Shikano and K. J. Lang.
Phoneme recognition using time-delay neural networks. IEEE Transactions
on Acoustics, Speech, and Signal Processing, vol. 37, no. 3, pp.
328-339, March 1989.
[CNN2] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W.
Hubbard, L. D. Jackel: Backpropagation Applied to Handwritten Zip Code
Recognition, Neural Computation, 1(4):541-551, 1989.
[CNN3] Weng, J.,
Ahuja, N., and Huang, T. S. (1993). Learning recognition and
segmentation of 3-D objects from 2-D images. Proc. 4th Intl. Conf.
Computer Vision, Berlin, Germany, pp. 121-128. A CNN whose downsampling layers use Max-Pooling
(which has become very popular) instead of Fukushima's
[CNN4] M. A. Ranzato, Y. LeCun: A Sparse and Locally Shift Invariant
Feature Extractor Applied to Document Images. Proc. ICDAR, 2007
J. Koutnik, F. Gomez, J. Schmidhuber (2010). Evolving Neural Networks in Compressed Weight Space. Proceedings of the Genetic and Evolutionary Computation Conference
(GECCO-2010), Portland, 2010.
J. Koutnik, G. Cuccu, J. Schmidhuber, F. Gomez.
Evolving Large-Scale Neural Networks for Vision-Based Reinforcement Learning.
Proceedings of the Genetic and Evolutionary
Computation Conference (GECCO), Amsterdam, July 2013.
R. K. Srivastava, J. Schmidhuber, F. Gomez.
Generalized Compressed Network Search.
Proc. GECCO 2012.
[CTC] A. Graves, S. Fernandez, F. Gomez, J. Schmidhuber. Connectionist
Temporal Classification: Labelling Unsegmented Sequence Data with
Recurrent Neural Networks. ICML 06, Pittsburgh, 2006.
R. J. Williams.
Complexity of exact gradient computation algorithms for recurrent
neural networks. Technical Report NU-CCS-89-27, Northeastern University,
College of Computer Science, 1989.
J. Koutnik, K. Greff, F. Gomez, J. Schmidhuber. A Clockwork RNN. Proc.
31st International Conference on Machine Learning (ICML), p. 1845-1853,
Beijing, 2014. Preprint arXiv:1402.3511 [cs.NE].
J. Schmidhuber (AI Blog, 2021).
10-year anniversary. In 2011, DanNet triggered the deep convolutional neural network (CNN) revolution. Named
after my outstanding postdoc Dan Ciresan, it was the first deep and
fast CNN to win international computer vision contests, and had a
temporary monopoly on winning them, driven by a very fast implementation
based on graphics processing units (GPUs).
1st superhuman result in 2011.[DAN1]
Now everybody is using this approach.
J. Schmidhuber (AI Blog, 2011; updated 2021 for 10th birthday of DanNet): First superhuman visual pattern recognition.
At the IJCNN 2011 computer vision competition in Silicon Valley,
our artificial neural network called DanNet
performed twice better than humans, three times better than the closest
artificial competitor, and six times better than the best non-neural
[DEC] J. Schmidhuber (AI Blog, 02/20/2020, revised 2021). The 2010s: Our Decade of Deep Learning / Outlook on the 2020s. The
recent decade's most important developments and industrial applications
based on our AI, with an outlook on the 2020s, also addressing privacy
and data markets.
Ivakhnenko, A. G. and Lapa, V. G. (1965). Cybernetic Predicting Devices. CCM Information Corporation. First working Deep Learners with many layers, learning internal representations.
Ivakhnenko, Alexey Grigorevich. The group method of data of handling; a
rival of the method of stochastic approximation. Soviet Automatic
Control 13 (1968): 43-55.
Ivakhnenko, A. G. (1971). Polynomial theory of complex systems. IEEE Transactions on Systems, Man and Cybernetics, (4):364-378.
[DIST1] J. Schmidhuber, 1991.[UN-UN2]
O. Vinyals, J. A. Dean, G. E. Hinton.
Distilling the Knowledge in a Neural Network.
Preprint arXiv:1503.02531 [stat.ML], 2015.
[DL1] J. Schmidhuber, 2015.
Deep learning in neural networks: An overview. Neural Networks, 61, 85-117.
Got the first Best Paper Award ever issued by the journal Neural Networks, founded in 1988.
[DL2] J. Schmidhuber, 2015.
[DL3] Y. LeCun, Y. Bengio, G. Hinton (2015). Deep Learning. Nature 521, 436-444.
A "survey" of deep learning that does not mention the pioneering works of deep learning.
[DL3a] Y. Bengio, Y. LeCun, G. Hinton (2021). Turing Lecture: Deep Learning for AI. Communications of the ACM, July 2021. HTML.
Local copy (HTML only).
Another "survey" of deep learning that does not mention the pioneering works of deep learning.
[DL4] J. Schmidhuber (AI Blog, 2017).
Our impact on the world's most valuable public companies: Apple, Google, Microsoft, Facebook, Amazon... By
2015-17, neural nets developed in my labs were on over 3 billion
devices such as smartphones, and used many billions of times per day,
consuming a significant fraction of the world's compute. Examples:
greatly improved (CTC-based)
speech recognition on all Android phones, greatly improved machine
translation through Google Translate and Facebook (over 4 billion
LSTM-based translations per day), Apple's Siri and Quicktype on all
iPhones, the answers of Amazon's Alexa, etc. Google's 2019
on-device speech recognition
(on the phone, not the server)
is still based on
F. Gomez and J. Schmidhuber.
Co-evolving recurrent neurons learn deep memory POMDPs.
In Proc. GECCO'05, Washington, D. C.,
pp. 1795-1802, ACM Press, New York, NY, USA, 2005.
J. Schmidhuber (AI Blog, Nov 2020). 15-year anniversary: 1st paper with "learn deep" in the title (2005). Our deep reinforcement learning & neuroevolution solved problems of depth 1000 and more.[DL6] Soon after its publication, everybody started talking about "deep learning." Causality or correlation?
"Deep Learning ... moving beyond shallow machine learning since 2006!"
Web site deeplearning.net of Y. Bengio's MILA (2015, retrieved May 2020; compare the version in the
referring to Hinton's[UN4] and Bengio's[UN5]
unsupervised pre-training for deep NNs[UN4] (2006) although
this type of deep learning dates back to 1991.[UN1-2][UN]
[DLC] J. Schmidhuber (AI Blog, June 2015).
Critique of Paper by "Deep Learning Conspiracy" (Nature 521 p 436).
The inventor of an important method should get credit for inventing
it. She may not always be the one who popularizes it. Then the
popularizer should get credit for popularizing it (but not for inventing
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D.
Wierstra, M. Riedmiller. Playing Atari with Deep Reinforcement Learning.
Tech Report, 19 Dec. 2013,
[DM2] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G.
Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S.
Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D.
Wierstra, S. Legg, D. Hassabis. Human-level control through deep
reinforcement learning. Nature, vol. 518, p 1529, 26 Feb. 2015.
S. Stanford. DeepMind's AI, AlphaStar Showcases Significant Progress Towards AGI. Medium ML Memoirs, 2019.
Alphastar has a "deep LSTM core."
[DNC] Hybrid computing using a neural network with dynamic external
A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A.
Grabska-Barwinska, S. G. Colmenarejo, E. Grefenstette, T. Ramalho, J.
Agapiou, A. P. Badia, K. M. Hermann, Y. Zwols, G. Ostrovski, A. Cain, H.
King, C. Summerfield, P. Blunsom, K. Kavukcuoglu, D. Hassabis.
Nature, 538:7626, p 471, 2016.
[Drop1] S. J. Hanson (1990). A Stochastic Version of the Delta Rule, PHYSICA D,42, 265-272.
Dropout is a variation of the stochastic delta rule—compare preprint
N. Frazier-Logue, S. J. Hanson (2020). The Stochastic Delta Rule: Faster
and More Accurate Deep Learning Through Adaptive Weight Noise. Neural
H. Hopf, A. Krief, G. Mehta, S. A. Matlin.
Fake science and the knowledge crisis: ignorance can be fatal.
Royal Society Open Science, May 2019.
Quote: "Scientists must be willing to speak out when they see false
information being presented in social media, traditional print or
broadcast press" and "must speak out against false information and fake
science in circulation
and forcefully contradict public figures who promote it."
[FAST] C. v.d. Malsburg. Tech Report 81-2, Abteilung f. Neurobiologie,
Max-Planck Institut f. Biophysik und Chemie, Goettingen, 1981.
First paper on fast weights or dynamic links.
J. A. Feldman. Dynamic connections in neural networks.
Biological Cybernetics, 46(1):27-39, 1982.
2nd paper on fast weights.
G. E. Hinton, D. C. Plaut. Using fast weights to deblur old memories.
Proc. 9th annual conference of the Cognitive Science Society (pp.
Two types of weights with different learning rates.
By 2017, Facebook
over 4 billion automatic translations per day (The Verge, August 4, 2017);
Facebook blog by J.M. Pino, A. Sidorov, N.F. Ayan (August 3, 2017)
S. Hochreiter and J. Schmidhuber.
Flat minimum search finds simple nets.
Technical Report FKI-200-94, Fakultät für Informatik,
Technische Universität München, December 1994.
J. Schmidhuber (AI Blog, 26 March 2021).
26 March 1991: Neural nets learn to program neural nets with fast weights—like Transformer variants. 2021: New stuff!
30-year anniversary of a now popular
alternative[FWP0-1] to recurrent NNs.
A slow feedforward NN learns by gradient descent to program the changes of
the fast weights[FAST,FASTa] of
Such Fast Weight Programmers[FWP0-6,FWPMETA1-7] can learn to memorize past data, e.g.,
by computing fast weight changes through additive outer products of self-invented activation patterns[FWP0-1]
(now often called keys and values for self-attention[TR1-6]).
The similar Transformers[TR1-2] combine this with projections
and softmax and
are now widely used in natural language processing.
For long input sequences, their efficiency was improved through
linear Transformers or Performers[TR5-6]
which are formally equivalent to the 1991 Fast Weight Programmers (apart from normalization).
In 1993, I introduced
the attention terminology[FWP2] now used
in this context,[ATT] and
extended the approach to
RNNs that program themselves.
Learning to control fast-weight memories: An alternative to recurrent nets.
Technical Report FKI-147-91, Institut für Informatik, Technische
Universität München, 26 March 1991.
First paper on fast weight programmers: a slow net learns by gradient descent to compute weight changes of a fast net.
[FWP1] J. Schmidhuber. Learning to control fast-weight memories: An
alternative to recurrent nets. Neural Computation, 4(1):131-139, 1992.
[FWP2] J. Schmidhuber. Reducing the ratio between learning complexity
and number of time-varying variables in fully recurrent nets. In
Proceedings of the International Conference on Artificial Neural
Networks, Amsterdam, pages 460-463. Springer, 1993.
First recurrent fast weight programmer based on outer products.
Introduced the terminology of learning "internal spotlights of
[FWP3] I. Schlag, J. Schmidhuber. Gated Fast Weights for On-The-Fly
Neural Program Generation. Workshop on Meta-Learning, @N(eur)IPS 2017,
Long Beach, CA, USA.
[FWP3a] I. Schlag, J. Schmidhuber. Learning to Reason with Third Order
Tensor Products. Advances in Neural Information Processing Systems
(N(eur)IPS), Montreal, 2018.
Preprint: arXiv:1811.12143. PDF.
[FWP4a] J. Ba, G. Hinton, V. Mnih, J. Z. Leibo, C. Ionescu. Using Fast Weights to Attend to the Recent Past. NIPS 2016.
PDF. Like [FWP0-2].
D. Bahdanau, K. Cho, Y. Bengio (2014).
Neural Machine Translation by Jointly Learning to Align and Translate. Preprint arXiv:1409.0473 [cs.CL].
Y. Tang, D. Nguyen, D. Ha (2020).
Neuroevolution of Self-Interpretable Agents.
F. J. Gomez and J. Schmidhuber.
Evolving modular fast-weight networks for control.
In W. Duch et al. (Eds.):
LNCS 3697, pp. 383-389, Springer-Verlag Berlin Heidelberg, 2005.
Reinforcement-learning fast weight programmer.
[FWP6] I. Schlag, K. Irie, J. Schmidhuber.
Linear Transformers Are Secretly Fast Weight Programmers. ICML 2021. Preprint: arXiv:2102.11174.
[FWP7] K. Irie, I. Schlag, R. Csordas, J. Schmidhuber.
Going Beyond Linear Transformers with Recurrent Fast Weight Programmers.
Preprint: arXiv:2106.06295 (June 2021).
[FWPMETA1] J. Schmidhuber. Steps towards `self-referential' learning.
Technical Report CU-CS-627-92, Dept. of Comp. Sci., University of
Colorado at Boulder, November 1992.
First recurrent fast weight programmer that can learn
to run a learning algorithm or weight change algorithm on itself.
[FWPMETA2] J. Schmidhuber. A self-referential weight matrix.
In Proceedings of the International Conference on Artificial
Neural Networks, Amsterdam, pages 446-451. Springer, 1993.
[FWPMETA3] J. Schmidhuber.
An introspective network that can learn to run its own weight change algorithm. In Proc. of the Intl. Conf. on Artificial Neural Networks,
Brighton, pages 191-195. IEE, 1993.
A neural network that embeds its own meta-levels.
In Proc. of the International Conference on Neural Networks '93,
San Francisco. IEEE, 1993.
J. Schmidhuber. Habilitation thesis, TUM, 1993. PDF.
A recurrent neural net with a self-referential, self-reading, self-modifying weight matrix
can be found here.
L. Kirsch and J. Schmidhuber. Meta Learning Backpropagation & Improving It. Metalearning Workshop at NeurIPS, 2020.
Preprint arXiv:2012.14905 [cs.LG], 2020.
I. Schlag, T. Munkhdalai, J. Schmidhuber.
Learning Associative Inference Using Fast Weight Memory.
To appear at ICLR 2021.
Report arXiv:2011.07831 [cs.AI], 2020.
[G63] R. J Glauber (1963). Time-dependent statistics of the Ising model.
Journal of Mathematical Physics, 4(2):294-307, 1963.
H. Sak, A. Senior, K. Rao, F. Beaufays, J. Schalkwyk—Google Speech Team.
Google voice search: faster and more accurate.
Google Research Blog, Sep 2015, see also
Aug 2015 Google's speech recognition based on CTC and LSTM.
improvement of Google's speech recognition through LSTM:
Alphr Technology, Jul 2015, or 9to5google, Jul 2015
Y. He, T. N. Sainath, R. Prabhavalkar, I. McGraw, R. Alvarez, D. Zhao,
D. Rybach, A. Kannan, Y. Wu, R. Pang, Q. Liang, D. Bhatia, Y. Shangguan,
B. Li, G. Pundak, K. Chai Sim, T. Bagby, S. Chang, K. Rao, A.
Streaming end-to-end speech recognition for mobile devices. ICASSP
2019-2019 IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP). IEEE, 2019.
dramatically improved Google Translate of 2016 is based on LSTM, e.g.,
WIRED, Sep 2016,
siliconANGLE, Sep 2016
O. Niemitalo. A method for training artificial neural networks to generate missing data within a variable context.
Blog post, Internet Archive, 2010.
A blog post describing the basic ideas[AC][AC90, AC90b][AC20] of GANs.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair,
A. Courville, Y. Bengio.
Generative adversarial nets. NIPS 2014, 2672-2680, Dec 2014.
Description of GANs that does not cite the original work of 1990[AC][AC90,AC90b][AC20][R2] (also containing wrong claims about
T. Karras, S. Laine, T. Aila. A style-based generator architecture for generative adversarial
networks. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages
K. Gödel. Über formal unentscheidbare Sätze der Principia Mathematica
und verwandter Systeme I. Monatshefte für Mathematik und Physik,
In the early 1930s,
Gödel founded theoretical computer science. He identified fundamental limits of mathematics and theorem proving and computing and Artificial Intelligence.
K. Gödel (1934).
On undecidable propositions of formal mathematical
systems. Notes by S. C. Kleene and J. B. Rosser on lectures
at the Institute for Advanced Study, Princeton, New Jersey, 1934, 30
pp. (Reprinted in M. Davis, (ed.), The Undecidable. Basic Papers on Undecidable
Propositions, Unsolvable Problems, and Computable Functions,
Raven Press, Hewlett, New York, 1965.)
Gödel introduced the first universal coding language.
R. J. Lipton and K. W. Regan.
Gödel's lost letter and P=NP.
Collected works Volume I: Publications 1929-36,
S. Feferman et. al., editors, Oxford Univ. Press, Oxford, 1986.
[GOD21] J. Schmidhuber (2021). 90th anniversary celebrations: 1931: Kurt Gödel, founder of theoretical computer science,
shows limits of math, logic, computing, and artificial intelligence.
This was number 1 on Hacker News.
J. Schmidhuber (2021). Als Kurt Gödel die Grenzen des Berechenbaren entdeckte.
(When Kurt Gödel discovered the limits of computability.)
Frankfurter Allgemeine Zeitung, 16/6/2021.
C. Goller & A. Küchler (1996). Learning task-dependent distributed
representations by backpropagation through structure. Proceedings of
International Conference on Neural Networks (ICNN'96). Vol. 1, p.
347-352 IEEE, 1996.
Based on TR AR-95-02, TU Munich, 1995.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A.
Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A.
Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M.
Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S.
Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I.
Sutskever, D. Amodei.
Language Models are Few-Shot Learners (2020).
Oh, K.-S. and Jung, K. (2004). GPU implementation of neural networks. Pattern Recognition, 37(6):1311-1314. Speeding up traditional NNs on GPU by a factor of 20.
K. Chellapilla, S. Puri, P. Simard. High performance convolutional
neural networks for document processing. International Workshop on
Frontiers in Handwriting Recognition, 2006. Speeding up shallow CNNs on GPU by a factor of 4.
[GPUCNN1] D. C. Ciresan, U. Meier, J. Masci, L. M. Gambardella, J.
Schmidhuber. Flexible, High Performance Convolutional Neural Networks
for Image Classification. International Joint Conference on Artificial Intelligence (IJCAI-2011, Barcelona), 2011. PDF. ArXiv preprint.
Speeding up deep CNNs on GPU by a factor of 60.
win four important computer vision competitions 2011-2012 before others won any
with similar approaches.
[GPUCNN2] D. C. Ciresan, U. Meier, J. Masci, J. Schmidhuber.
A Committee of Neural Networks for Traffic Sign Classification.
International Joint Conference on Neural Networks (IJCNN-2011, San Francisco), 2011.
First superhuman performance in a computer vision contest, with half
the error rate of humans, and one third the error rate of the closest
competitor.[DAN1] This led to massive interest from industry.
[GPUCNN3] D. C. Ciresan, U. Meier, J. Schmidhuber. Multi-column Deep Neural Networks for Image Classification. Proc. IEEE Conf. on Computer Vision and Pattern Recognition CVPR 2012, p 3642-3649, July 2012. PDF. Longer TR of Feb 2012: arXiv:1202.2745v1 [cs.CV]. More.
[GPUCNN4] A. Krizhevsky, I. Sutskever, G. E. Hinton. ImageNet
Classification with Deep Convolutional Neural Networks. NIPS 25, MIT
Press, Dec 2012.
J. Schmidhuber (AI Blog, 2017; updated 2021 for 10th birthday of DanNet): History of computer vision contests won by deep CNNs since 2011. DanNet won 4 of them in a row before the similar AlexNet/VGG Net and the Resnet (a Highway Net with open gates) joined the party. Today, deep CNNs are standard in computer vision.
[GPUCNN6] J. Schmidhuber, D. Ciresan, U. Meier, J. Masci, A. Graves. On
Fast Deep Nets for AGI Vision. In Proc. Fourth Conference on Artificial
General Intelligence (AGI-11), Google, Mountain View, California, 2011.
[GPUCNN7] D. C. Ciresan, A. Giusti, L. M. Gambardella, J. Schmidhuber.
Mitosis Detection in Breast Cancer Histology Images using Deep Neural
Networks. MICCAI 2013.
[GPUCNN8] J. Schmidhuber. First deep learner to win a contest on object detection in large images—
first deep learner to win a medical imaging contest (2012). HTML.
How IDSIA used GPU-based CNNs to win the
ICPR 2012 Contest on Mitosis Detection
MICCAI 2013 Grand Challenge.
K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale image recognition. Preprint arXiv:1409.1556 (2014).
[H86] J. L. van Hemmen (1986). Spin-glass models of a neural network.
Phys. Rev. A 34, 3435, 1 Oct 1986.
H. Sompolinsky (1988). Statistical Mechanics of Neural Networks.
Physics Today 41, 12, 70, 1988.
W. D. Hillis.
Co-evolving parasites improve simulated evolution as an optimization
Physica D: Nonlinear Phenomena, 42(1-3):228-234, 1990.
S. El Hihi, Y. Bengio. Hierarchical recurrent neural networks for long-term dependencies. NIPS, 1996.
P. Dayan, G. E. Hinton, R. M. Neal, and R. S. Zemel.
The Helmholtz machine.
Neural Computation, 7:889-904, 1995.
[HIN] J. Schmidhuber (AI Blog, 2020). Critique of Honda Prize for Dr. Hinton. Science must not allow corporate PR to distort the academic record.
Towards compositional learning with dynamic neural networks.
Technical Report FKI-129-90, Institut für Informatik, Technische
Universität München, 1990.
J. Schmidhuber. Learning to generate sub-goals for action sequences. In
T. Kohonen, K. Mäkisara, O. Simula, and J. Kangas, editors, Artificial
Neural Networks, pages 967-972. Elsevier Science Publishers B.V.,
North-Holland, 1991. PDF. Extending TR FKI-129-90, TUM, 1990.
HTML & images in German.
J. Schmidhuber and R. Wahnsiedler.
Planning simple trajectories using neural subgoal generators.
In J. A. Meyer, H. L. Roitblat, and S. W. Wilson, editors, Proc.
of the 2nd International Conference on Simulation of Adaptive Behavior,
pages 196-202. MIT Press, 1992.
HTML & images in German.
P. Dayan and G. E. Hinton.
Feudal Reinforcement Learning.
Advances in Neural Information Processing Systems 5, NIPS, 1992.
M. Wiering and J. Schmidhuber. HQ-Learning. Adaptive Behavior 6(2):219-246, 1997.
[HW1] R. K. Srivastava, K. Greff, J. Schmidhuber. Highway networks.
Preprints arXiv:1505.00387 (May 2015) and arXiv:1507.06228 (July 2015). Also at NIPS 2015. The
first working very deep feedforward nets with over 100 layers (previous
NNs had at most a few tens of layers). Let g, t, h, denote non-linear
differentiable functions. Each non-input layer of a highway net computes
g(x)x + t(x)h(x), where x is the data from the previous layer. (Like
LSTM with forget gates[LSTM2] for RNNs.) Resnets[HW2] are a version of this where the gates are always open: g(x)=t(x)=const=1.
Highway Nets perform roughly as well as ResNets[HW2] on ImageNet.[HW3] Highway layers are also often used for natural language processing, where the simpler residual layers do not work as well.[HW3]
R. K. Srivastava, K. Greff, J. Schmidhuber. Highway networks.
Presentation at the Deep Learning Workshop, ICML'15, July 10-11, 2015.
[HW2] He, K., Zhang,
X., Ren, S., Sun, J. Deep residual learning for image recognition. Preprint
(Dec 2015). Residual nets are a version of Highway Nets[HW1]
where the gates are always open:
g(x)=1 (a typical highway net initialization) and t(x)=1.
K. Greff, R. K. Srivastava, J. Schmidhuber. Highway and Residual Networks learn Unrolled Iterative Estimation. Preprint
arxiv:1612.07771 (2016). Also at ICLR 2017.
Hinton, G. E., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V.,
Nguyen, P., Sainath, T. N., and Kingsbury, B. (2012). Deep neural networks for acoustic modeling
in speech recognition: The shared views of four research groups. IEEE Signal Process. Mag.,
E. Ising (1925). Beitrag zur Theorie des Ferromagnetismus. Z. Phys., 31 (1): 253-258, 1925.
J. Deng, R. Socher, L.J. Li, K. Li, L. Fei-Fei (2009). Imagenet: A
large-scale hierarchical image database. In 2009 IEEE conference on
computer vision and pattern recognition, pp. 248-255). IEEE, 2009.
[JOU17] Jouppi et al. (2017). In-Datacenter Performance Analysis of a Tensor Processing Unit.
H. A. Kramers and G. H. Wannier (1941). Statistics of the Two-Dimensional Ferromagnet. Phys. Rev. 60, 252 and 263, 1941.
S.C. Kleene. Representation of Events in Nerve Nets and Finite Automata.
Automata Studies, Editors: C.E. Shannon and J. McCarthy, Princeton
University Press, p. 3-42, Princeton, N.J., 1956.
D. E. Knuth, L. T. Pardo (1976). The Early Development of Programming
Languages. Stanford University, Computer Science Department.
Discovering neural nets with low Kolmogorov complexity
and high generalization capability.
Neural Networks, 10(5):857-873, 1997.
[KU] A. Küchler & C. Goller (1996). Inductive learning in symbolic
domains using structure-driven recurrent neural networks. Lecture Notes
in Artificial Intelligence, vol 1137. Springer, Berlin, Heidelberg.
W. Lenz (1920). Beitraege zum Verständnis der magnetischen
Eigenschaften in festen Körpern. Physikalische Zeitschrift, 21:
J. L. Ba, J. R.Kiros, G. E. Hinton. Layer Normalization.
A New Publishing Model in Computer Science.
Local copy (HTML only).
[LEI21] J. Schmidhuber (AI Blog, 2021). 375th birthday of Leibniz, founder of computer science.
J. Schmidhuber (2021). Der erste Informatiker. Wie Gottfried Wilhelm Leibniz den Computer erdachte.
(The first computer scientist. How Gottfried Wilhelm Leibniz conceived the computer.)
Frankfurter Allgemeine Zeitung (FAZ), 17/5/2021. FAZ online:
M. L. Littman (2021).
Collusion Rings Threaten the Integrity of Computer Science Research.
Communications of the ACM, Vol. 64 No. 6, p. 43-44, June 2021.
S. Hochreiter and J. Schmidhuber.
Long Short-Term Memory.
TR FKI-207-95, TUM, August 1995.
[LSTM1] S. Hochreiter, J. Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735-1780, 1997. PDF.
Based on [LSTM0]. More.
[LSTM2] F. A. Gers, J. Schmidhuber, F. Cummins. Learning to Forget:
Continual Prediction with LSTM. Neural Computation, 12(10):2451-2471,
The "vanilla LSTM architecture" with forget gates
that everybody is using today, e.g., in Google's Tensorflow.
[LSTM3] A. Graves, J. Schmidhuber. Framewise phoneme classification with
bidirectional LSTM and other neural network architectures. Neural
Networks, 18:5-6, pp. 602-610, 2005.
S. Fernandez, A. Graves, J. Schmidhuber. An application of
recurrent neural networks to discriminative keyword
Intl. Conf. on Artificial Neural Networks ICANN'07,
[LSTM5] A. Graves, M. Liwicki, S. Fernandez, R. Bertolami, H. Bunke, J.
Schmidhuber. A Novel Connectionist System for Improved Unconstrained
Handwriting Recognition. IEEE Transactions on Pattern Analysis and
Machine Intelligence, vol. 31, no. 5, 2009.
[LSTM6] A. Graves, J. Schmidhuber. Offline Handwriting Recognition with
Multidimensional Recurrent Neural Networks. NIPS'22, p 545-552,
Vancouver, MIT Press, 2009.
[LSTM7] J. Bayer, D. Wierstra, J. Togelius, J. Schmidhuber.
Evolving memory cell structures for sequence learning.
Proc. ICANN-09, Cyprus, 2009.
[LSTM8] A. Graves, A. Mohamed, G. E. Hinton. Speech Recognition with
Deep Recurrent Neural Networks. ICASSP 2013, Vancouver, 2013.
O. Vinyals, L. Kaiser, T. Koo, S. Petrov, I. Sutskever, G. Hinton.
Grammar as a Foreign Language. Preprint arXiv:1412.7449 [cs.CL].
A. Graves, D. Eck and N. Beringer, J. Schmidhuber. Biologically
Plausible Speech Recognition with LSTM Neural Nets. In J. Ijspeert
(Ed.), First Intl. Workshop on Biologically Inspired Approaches to
Advanced Information Technology, Bio-ADIT 2004, Lausanne, Switzerland,
p. 175-184, 2004.
N. Beringer and A. Graves and F. Schiel and J. Schmidhuber. Classifying
unprompted speech by retraining LSTM Nets. In W. Duch et al. (Eds.):
Proc. Intl. Conf. on Artificial Neural Networks ICANN'05, LNCS 3696, pp.
575-581, Springer-Verlag Berlin Heidelberg, 2005.
D. Wierstra, F. Gomez, J. Schmidhuber. Modeling systems with internal
state using Evolino. In Proc. of the 2005 conference on genetic and
evolutionary computation (GECCO), Washington, D. C., pp. 1795-1802, ACM
Press, New York, NY, USA, 2005. Got a GECCO best paper award.
F. A. Gers and J. Schmidhuber.
LSTM Recurrent Networks Learn Simple Context Free and
Context Sensitive Languages.
IEEE Transactions on Neural Networks 12(6):1333-1340, 2001.
S. Fernandez, A. Graves, J. Schmidhuber.
Sequence labelling in structured domains with
hierarchical recurrent neural networks. In Proc.
IJCAI 07, p. 774-779, Hyderabad, India, 2007 (talk).
A. Graves, J. Schmidhuber.
Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks.
Advances in Neural Information Processing Systems 22, NIPS'22, p 545-552,
Vancouver, MIT Press, 2009.
M. Stollenga, W. Byeon, M. Liwicki, J. Schmidhuber. Parallel
Multi-Dimensional LSTM, With Application to Fast Biomedical Volumetric
Image Segmentation. Advances in Neural Information Processing Systems
J. A. Perez-Ortiz, F. A. Gers, D. Eck, J. Schmidhuber.
Kalman filters improve LSTM network performance in
problems unsolvable by traditional recurrent nets.
Neural Networks 16(2):241-250, 2003.
J. Schmidhuber (AI Blog, Dec 2020). 10-year anniversary of our journal paper on deep reinforcement learning with policy gradients for LSTM (2007-2010). Recent
famous applications: DeepMind's Starcraft player (2019) and OpenAI's
dextrous robot hand & Dota player (2018)—Bill Gates called this a
huge milestone in advancing AI.
B. Bakker, F. Linaker, J. Schmidhuber.
Reinforcement Learning in Partially Observable Mobile Robot
Domains Using Unsupervised Event Extraction.
In Proceedings of the 2002
IEEE/RSJ International Conference on
Intelligent Robots and Systems (IROS 2002), Lausanne, 2002.
[LSTMGRU] J. Chung, C. Gulcehre, K. Cho, Y. Bengio (2014). Empirical
Evaluation of Gated Recurrent Neural Networks on Sequence Modeling.
Preprint arXiv:1412.3555 [cs.NE].
[LSTMGRU2] G. Weiss, Y. Goldberg, E. Yahav. On the Practical
Computational Power of Finite Precision RNNs for Language Recognition.
[LSTMGRU3] D. Britz et al. (2017). Massive Exploration of Neural Machine Translation
Architectures. Preprint arXiv:1703.03906
[M69] M. Minsky, S. Papert. Perceptrons (MIT Press, Cambridge, MA, 1969).
W. S. McCulloch, W. Pitts. A Logical Calculus of Ideas Immanent in Nervous Activity.
Bulletin of Mathematical Biophysics, Vol. 5, p. 115-133, 1943.
J. Schmidhuber (AI Blog, 2020). 1/3 century anniversary of
first publication on metalearning machines that learn to learn (1987).
For its cover I drew a robot that bootstraps itself.
1992-: gradient descent-based neural metalearning. 1994-:
Meta-Reinforcement Learning with self-modifying policies. 1997: Meta-RL
plus artificial curiosity and intrinsic motivation.
2002-: asymptotically optimal metalearning for curriculum learning.
2003-: mathematically optimal Gödel Machine. 2020: new stuff!
Evolutionary principles in self-referential learning, or on learning
how to learn: The meta-meta-... hook. Diploma thesis,
Institut für Informatik, Technische Universität München, 1987.
Searchable PDF scan (created by OCRmypdf which uses
(GP) is applied to itself, to recursively evolve
better GP methods through Meta-Evolution. More.
[MGC] MICCAI 2013 Grand Challenge on Mitosis Detection, organised by M.
Veta, M.A. Viergever, J.P.W. Pluim, N. Stathonikos, P. J. van Diest of
University Medical Center Utrecht.
[MIR] J. Schmidhuber (AI Blog, 2019). Deep Learning: Our Miraculous Year 1990-1991. Preprint
The deep learning neural networks of our team have revolutionised
pattern recognition and machine learning, and are now heavily used in
academia and industry. In 2020-21, we celebrate that many of the basic
ideas behind this revolution were published within fewer than 12 months
in our "Annus Mirabilis" 1990-1991 at TU Munich.
[MLP1] D. C. Ciresan, U. Meier, L. M. Gambardella, J. Schmidhuber. Deep
Big Simple Neural Nets For Handwritten Digit Recognition. Neural
Computation 22(12): 3207-3220, 2010. ArXiv Preprint.
Showed that plain backprop for deep standard NNs is sufficient to
break benchmark records, without any unsupervised pre-training.
[MLP2] J. Schmidhuber
(AI Blog, Sep 2020). 10-year anniversary of supervised deep learning breakthrough (2010). No unsupervised pre-training.
By 2010, when compute was 100 times more expensive than today, both our feedforward NNs[MLP1]
and our earlier recurrent NNs were able to beat all competing
algorithms on important problems of that time. This deep learning
revolution quickly spread from Europe to North America and Asia. The
rest is history.
J. Schmidhuber (AI Blog, 2021). The most cited neural networks all build on work done in my labs. Foundations of the most popular NNs originated in my labs at TU Munich and IDSIA. Here I mention: (1) Long Short-Term Memory (LSTM), (2) ResNet (which is our earlier Highway Net with open gates), (3) AlexNet and VGG Net (both citing our similar earlier DanNet: the first deep convolutional NN to win
image recognition competitions),
(4) Generative Adversarial Networks (an instance of my earlier
Adversarial Artificial Curiosity), and (5) variants of Transformers (linear Transformers are formally equivalent to my earlier Fast Weight Programmers).
Most of this started with our
Annus Mirabilis of 1990-1991.[MIR]
M. Mozer. A Focused Backpropagation Algorithm for Temporal Pattern Recognition.
Complex Systems, 1989.
[NAS] B. Zoph, Q. V. Le. Neural Architecture Search with Reinforcement Learning.
Preprint arXiv:1611.01578 (PDF), 2017.
[NASC1] J. Schmidhuber. First Pow(d)ered flight / plane truth. Correspondence, Nature, 421 p 689, Feb 2003.
J. Schmidhuber. Zooming in on aviation history.
Correspondence, Nature, vol 566, p 39, 7 Feb 2019.
[NASC3] J. Schmidhuber. The last inventor of the telephone. Letter, Science, 319, no. 5871, p. 1759, March 2008.
[NASC4] J. Schmidhuber. Turing: Keep his work in perspective.
Correspondence, Nature, vol 483, p 541, March 2012, doi:10.1038/483541b.
[NASC5] J. Schmidhuber. Turing in Context.
Letter, Science, vol 336, p 1639, June 2012.
(On Gödel, Zuse, Turing.)
See also comment on response by A. Hodges (DOI:10.1126/science.336.6089.1639-a)
[NASC6] J. Schmidhuber. Colossus was the first electronic digital computer. Correspondence, Nature, 441 p 25, May 2006.
[NASC7] J. Schmidhuber. Turing's impact. Correspondence, Nature, 429 p 501, June 2004
[NASC8] J. Schmidhuber. Prototype resilient, self-modeling robots. Correspondence, Science, 316, no. 5825 p 688, May 2007.
[NASC9] J. Schmidhuber. Comparing the legacies of Gauss, Pasteur, Darwin. Correspondence, Nature, vol 452, p 530, April 2008.
[NAT1] J. Schmidhuber. Citation bubble about to burst? Nature, vol. 469, p. 34, 6 January 2011.
[NHE] J. Schmidhuber. The Neural Heat Exchanger.
Oral presentations since 1990 at various universities including TUM and
University of Colorado at Boulder. Also in In S. Amari, L. Xu, L. Chan,
I. King, K. Leung, eds., Proceedings of the Intl. Conference on Neural
Information Processing (1996), pages 194-197, Springer, Hongkong.
Y. Bengio, R. Ducharme, P. Vincent, C. Jauvin (2003).
A Neural Probabilistic Language Model.
Journal of Machine Learning Research 3, p 1137-1155, 2003.
A. Newell and H. Simon.
The logic theory machine—A complex information processing system.
IRE Transactions on Information Theory 2.3 (1956):61-79.
NY Times article
by J. Markoff, Nov. 27, 2016: When A.I. Matures, It May Call Jürgen Schmidhuber 'Dad'
NY Times article
by G. Lewis-Kraus, Dec. 14, 2016: The Great A.I. Awakening
G. Powell, J. Schneider, J. Tobin, W. Zaremba, A. Petron, M. Chociej, L.
Weng, B. McGrew, S. Sidor, A. Ray, P. Welinder, R. Jozefowicz, M.
Plappert, J. Pachocki, M. Andrychowicz, B. Baker.
Learning Dexterity. OpenAI Blog, 2018.
OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew,
J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider,
S. Sidor, J. Tobin, P. Welinder, L. Weng, W. Zaremba.
Learning Dexterous In-Hand Manipulation. arxiv:1312.5602 (PDF).
C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D.
Farhi, Q. Fischer, S. Hashme, C. Hesse, R. Jozefowicz, S. Gray, C.
Olsson, J. Pachocki, M. Petrov, H. P. de Oliveira Pinto, J. Raiman, T.
Salimans, J. Schlatter, J. Schneider, S. Sidor, I. Sutskever, J. Tang,
F. Wolski, S. Zhang (Dec 2019).
Dota 2 with Large Scale Deep Reinforcement Learning.
An LSTM composes 84% of the model's total parameter count.
J. Rodriguez. The Science Behind OpenAI Five that just Produced One of
the Greatest Breakthrough in the History of AI. Towards Data Science,
2018. An LSTM with 84% of the model's total parameter count was the core of OpenAI Five.
G.Z. Sun, H.H. Chen, C.L. Giles, Y.C. Lee, D. Chen. Neural Networks with
External Memory Stack that Learn Context—Free Grammars from Examples.
Proceedings of the 1990 Conference on Information Science and Systems,
Vol.II, pp. 649-653, Princeton University, Princeton, NJ, 1990.
M. Mozer, S. Das. A connectionist symbol manipulator that discovers the structure of context-free languages. Proc. NIPS 1993.
R. J. Williams. Simple statistical gradient-following algorithms for
connectionist reinforcement learning. Machine Learning 8.3-4: 229-256,
Dynamische neuronale Netze und das fundamentale raumzeitliche
(Dynamic neural nets and the fundamental spatio-temporal
credit assignment problem).
Institut für Informatik, Technische
Universität München, 1990.
J. Schmidhuber (AI Blog, 2020). 30-year anniversary of planning & reinforcement learning with recurrent world models and artificial curiosity (1990). This work also introduced high-dimensional reward signals, deterministic policy gradients for RNNs,
the GAN principle
(widely used today). Agents with adaptive recurrent world models even
suggest a simple explanation of consciousness & self-awareness.
An on-line algorithm for dynamic reinforcement learning and planning
in reactive environments.
Proc. IEEE/INNS International Joint Conference on Neural
Networks, San Diego, volume 2, pages 253-258, 1990.
Based on TR FKI-126-90 (1990).[AC90]
Reinforcement learning in Markovian and non-Markovian environments.
In D. S. Lippman, J. E. Moody, and D. S. Touretzky, editors,
Advances in Neural Information Processing Systems 3, NIPS'3, pages 500-506. San
Mateo, CA: Morgan Kaufmann, 1991.
Partially based on TR FKI-126-90 (1990).[AC90]
On Learning to Think: Algorithmic Information Theory for Novel
Combinations of Reinforcement Learning Controllers and Recurrent Neural
Report arXiv:1210.0118 [cs.AI], 2015.
One Big Net For Everything. Preprint arXiv:1802.08864 [cs.AI], Feb 2018.
D. Ha, J. Schmidhuber. Recurrent World Models Facilitate Policy
Evolution. Advances in Neural Information Processing Systems (NIPS),
Montreal, 2018. (Talk.)
Github: World Models.
[PM0] J. Schmidhuber. Learning factorial codes by predictability
minimization. TR CU-CS-565-91, Univ. Colorado at Boulder, 1991. PDF.
[PM1] J. Schmidhuber. Learning factorial codes by predictability
minimization. Neural Computation, 4(6):863-879, 1992. Based on [PM0],
[PM2] J. Schmidhuber, M. Eldracher, B. Foltin. Semilinear predictability
minimzation produces well-known feature detectors. Neural Computation,
[PO87] J. B. Pollack. On Connectionist Models of Natural Language Processing.
PhD thesis, Computer Science Department, University of Illinois, Urbana, 1987.
[PO90] J. B. Pollack. Recursive Distributed Representations. Artificial Intelligence,
[PP] J. Schmidhuber.
POWERPLAY: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem.
Frontiers in Cognitive Science, 2013.
ArXiv preprint (2011):
[PP1] R. K. Srivastava, B. Steunebrink, J. Schmidhuber.
First Experiments with PowerPlay.
Neural Networks, 2013.
ArXiv preprint (2012):
[PP2] V. Kompella, M. Stollenga, M. Luciw, J. Schmidhuber. Continual
curiosity-driven skill acquisition from high-dimensional video inputs
for humanoid robots. Artificial Intelligence, 2015.
Relevant threads with many comments at reddit.com/r/MachineLearning, the largest machine learning forum with over 800k subscribers in 2019 (note that my name is often misspelled):
[R1] Reddit/ML, 2019. Hinton, LeCun, Bengio receive ACM Turing Award.
[R2] Reddit/ML, 2019. J. Schmidhuber really had GANs in 1990.
[R3] Reddit/ML, 2019. NeurIPS 2019 Bengio Schmidhuber Meta-Learning Fiasco.
[R4] Reddit/ML, 2019. Five major deep learning papers by G. Hinton did not cite similar earlier work by J. Schmidhuber.
[R5] Reddit/ML, 2019. The 1997 LSTM paper by Hochreiter & Schmidhuber has become the most cited deep learning research paper of the 20th century.
[R6] Reddit/ML, 2019. DanNet, the CUDA CNN of Dan Ciresan in J. Schmidhuber's team, won 4 image recognition challenges prior to AlexNet.
[R7] Reddit/ML, 2019. J. Schmidhuber on Seppo Linnainmaa, inventor of backpropagation in 1970.
[R8] Reddit/ML, 2019. J. Schmidhuber on Alexey Ivakhnenko, godfather of deep learning 1965.
[R9] Reddit/ML, 2019. We
find it extremely unfair that Schmidhuber did not get the Turing award.
That is why we dedicate this song to Juergen to cheer him up.
[R11] Reddit/ML, 2020. Schmidhuber: Critique of Honda Prize for Dr. Hinton
[R12] Reddit/ML, 2020. J. Schmidhuber: Critique of Turing Award for Drs. Bengio & Hinton & LeCun
[R15] Reddit/ML, 2021. J. Schmidhuber's work on fast weights from 1991 is similar to linearized variants of Transformers
Rosenblatt, F. (1958). The perceptron: a probabilistic model for information storage and organization
in the brain. Psychological review, 65(6):386.
Joseph, R. D. (1961). Contributions to perceptron theory. PhD thesis, Cornell Univ.
Rosenblatt, F. (1962). Principles of Neurodynamics. Spartan, New York.
R. Girshick, J. Donahue, T. Darrell, J. Malik.
Rich feature hierarchies for accurate object detection and semantic segmentation.
Preprint arXiv/1311.2524, Nov 2013.
Fast R-CNN. Proc. of the IEEE international conference on computer vision, p. 1440-1448, 2015.
K. He, G. Gkioxari, P. Dollar, R. Girshick.
Preprint arXiv/1703.06870, 2017.
R. Rojas (1998). How to make Zuse's Z3 a universal computer. IEEE Annals of Computing, vol. 19:3, 1998.
T. Tieleman, G. E. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a
running average of its recent magnitude. COURSERA: Neural networks for
machine learning 4.2 (2012): 26-31.
A. J. Robinson and F. Fallside.
The utility driven dynamic error propagation network.
Technical Report CUED/F-INFENG/TR.1, Cambridge University Engineering Department, 1987.
D. Wierstra, A. Foerster, J. Peters, J. Schmidhuber (2010). Recurrent
policy gradients. Logic Journal of the IGPL, 18(5), 620-634.
D. Wierstra, A. Foerster, J. Peters, J. Schmidhuber. Solving Deep Memory POMDPs
with Recurrent Policy Gradients.
Intl. Conf. on Artificial Neural Networks ICANN'07,
[RUM] DE Rumelhart, GE Hinton, RJ Williams (1985). Learning Internal
Representations by Error Propagation. TR No. ICS-8506, California Univ
San Diego La Jolla Inst for Cognitive Science. Later version published
Learning representations by back-propagating errors. Nature, 323, p.
This experimental analysis of backpropagation did not cite the origin of the method,[BP1-4] also known as the reverse mode of automatic differentiation.
D. Sherrington (1993).
Neural networks: the spin glass approach.
North-Holland Mathematical Library,
vol 51, 1993, p. 261-291.
T. Sejnowski. The unreasonable effectiveness of deep learning in artificial intelligence. PNAS, January 28, 2020.
B. Speelpenning (1980). Compiling Fast Partial Derivatives of Functions Given by Algorithms. PhD
thesis, Department of Computer Science, University of Illinois, Urbana-Champaign.
I. Sutskever, O. Vinyals, Quoc V. Le. Sequence to sequence learning with
neural networks. In: Advances in Neural Information Processing Systems
(NIPS), 2014, 3104-3112.
A. L. Samuel.
Some studies in machine learning using the game of checkers.
IBM Journal on Research and Development, 3:210-229, 1959.
[SA17] J. Schmidhuber.
The Past, Present and Future of Artificial Intelligence.
Scientific American, Observations, Nov 2017.
[SCAN] J. Masci,
A. Giusti, D. Ciresan, G. Fricout, J. Schmidhuber. A Fast Learning
Algorithm for Image Segmentation with Max-Pooling Convolutional
Networks. ICIP 2013. Preprint arXiv:1302.1690.
C. E. Shannon (1938). A Symbolic Analysis of Relay and Switching
Circuits. Trans. AIEE. 57 (12): 713-723. Based on his thesis, MIT, 1937.
J. Schmidhuber, S. Heil (1996).
Sequential neural text compression.
IEEE Trans. Neural Networks, 1996.
(An earlier version appeared at NIPS 1995.)
D. Sherrington, S. Kirkpatrick (1975).
Solvable Model of a Spin-Glass.
Phys. Rev. Lett. 35, 1792, 1975.
J. Masci, U. Meier, D. Ciresan, G. Fricout, J. Schmidhuber
Steel Defect Classification with Max-Pooling Convolutional Neural Networks.
Proc. IJCNN 2012.
[SP93] A. Sperduti (1993).
Encoding Labeled Graphs by Labeling RAAM. NIPS 1993: 1125-1132
[SP94] A. Sperduti (1994).
Labelling Recursive Auto-associative Memory. Connect. Sci. 6(4): 429-459 (1994)
[SP95] A. Sperduti (1995).
Stability properties of labeling recursive auto-associative memory. IEEE Trans. Neural Networks 6(6): 1452-1460 (1995)
[SPG95] A. Sperduti, A. Starita, C. Goller (1995).
Learning Distributed Representations for the Classification of Terms. IJCAI 1995: 509-517
[SPG96] A. Sperduti, D. Majidi, A. Starita (1996).
Extended Cascade-Correlation for Syntactic and Structural Pattern Recognition. SSPR 1996: 90-99
[SPG97] A. Sperduti, A. Starita (1997).
Supervised neural networks for the classification of structures.
IEEE Trans. Neural Networks 8(3): 714-735, 1997.
[SV20] S. Vazire (2020). A toast to the error detectors. Let 2020 be the
year in which we value those who ensure that science is
self-correcting. Nature, vol 577, p 9, 2/2/2020.
ACM's justification of the 2018 A.M. Turing Award (announced in 2019). WWW link.
Local copy 1 (HTML only).
Local copy 2 (HTML only).
[T20a] J. Schmidhuber (AI Blog, 25 June 2020). Critique of 2018 Turing Award for Drs. Bengio & Hinton & LeCun. The first version of the present critique.
[THE17] S. Baker (2017). Which countries and universities are leading on
AI research? Times Higher Education World University Rankings, 2017.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
L. Kaiser, I. Polosukhin (2017). Attention is all you need. NIPS 2017,
J. Devlin, M. W. Chang, K. Lee, K. Toutanova (2018). Bert: Pre-training
of deep bidirectional transformers for language understanding. Preprint
[TR3] K. Tran, A. Bisazza, C. Monz. The Importance of Being Recurrent
for Modeling Hierarchical Structure. EMNLP 2018, p 4731-4736. ArXiv
M. Hahn. Theoretical Limitations of Self-Attention in Neural Sequence
Models. Transactions of the Association for Computational Linguistics,
Volume 8, p.156-171, 2020.
A. Katharopoulos, A. Vyas, N. Pappas, F. Fleuret.
Transformers are RNNs: Fast autoregressive Transformers
with linear attention. In Proc. Int. Conf. on Machine
Learning (ICML), July 2020.
K. Choromanski, V. Likhosherstov, D. Dohan, X. Song,
A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin,
L. Kaiser, et al. Rethinking attention with Performers.
In Int. Conf. on Learning Representations (ICLR), 2021.
A. M. Turing. On computable numbers, with an application to the
Entscheidungsproblem. Proceedings of the London Mathematical Society,
Series 2, 41:230-267. Received 28 May 1936. Errata appeared in Series 2,
43, pp 544-546 (1937).
2nd explicit proof that the Entscheidungsproblem (decision problem) does not have a general solution.
[TUR21] J. Schmidhuber (AI Blog, Sep 2021). Turing Oversold. It's not Turing's fault, though.
J. Schmidhuber (AI Blog, 2021). 30-year anniversary. 1991: First very deep learning with unsupervised pre-training. Unsupervised
hierarchical predictive coding finds compact internal representations
of sequential data to facilitate downstream learning. The hierarchy can
be distilled into a single deep neural network (suggesting a simple
model of conscious and subconscious information processing). 1993:
solving problems of depth >1000.
Neural sequence chunkers.
Technical Report FKI-148-91, Institut für Informatik, Technische
Universität München, April 1991.
[UN1] J. Schmidhuber. Learning complex, extended sequences using the
principle of history compression. Neural Computation, 4(2):234-242,
1992. Based on TR FKI-148-91, TUM, 1991.[UN0] PDF.
First working Deep Learner based on a deep RNN hierarchy (with
different self-organising time scales),
overcoming the vanishing gradient problem through unsupervised
pre-training and predictive coding.
Also: compressing or distilling a teacher net (the chunker) into a
student net (the automatizer) that does not forget its old skills—such
approaches are now widely used. More.
[UN2] J. Schmidhuber. Habilitation thesis, TUM, 1993. PDF.
An ancient experiment on "Very Deep Learning" with credit assignment
across 1200 time steps or virtual layers and unsupervised pre-training
for a stack of recurrent NN
can be found here (depth > 1000).
J. Schmidhuber, M. C. Mozer, and D. Prelinger.
Continuous history compression.
In H. Hüning, S. Neuhauser, M. Raus, and W. Ritschel, editors,
Proc. of Intl. Workshop on Neural Networks, RWTH Aachen, pages 87-95.
[UN4] G. E. Hinton, R. R. Salakhutdinov. Reducing the dimensionality of
data with neural networks. Science, Vol. 313. no. 5786, pp. 504—507,
Y. Bengio, P. Lamblin, D. Popovici, H. Larochelle.
Greedy layer-wise training of deep networks.
Proc. NIPS 06, pages 153-160, Dec. 2006.
A. Urquhart. Von Neumann, Gödel and complexity theory. Bulletin of Symbolic Logic 16.4 (2010): 516-530.
[VAN1] S. Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, TUM, 1991 (advisor J. Schmidhuber). PDF.
More on the Fundamental Deep Learning Problem.
[VAN2] Y. Bengio, P. Simard, P. Frasconi. Learning long-term
dependencies with gradient descent is difficult. IEEE TNN 5(2), p
[VAN3] S. Hochreiter, Y. Bengio, P. Frasconi, J. Schmidhuber. Gradient
flow in recurrent nets: the difficulty of learning long-term
dependencies. In S. C. Kremer and J. F. Kolen, eds., A Field Guide to
Dynamical Recurrent Neural Networks. IEEE press, 2001.
[VAN4] Y. Bengio. Neural net language models. Scholarpedia, 3(1):3881, 2008. Link.
M. Y. Vardi (2013). Who begat computing? Communications of the ACM, Vol. 56(1):5, Jan 2013.
[VID1] G. Hinton.
The Next Generation of Neural Networks.
Youtube video [see 28:16].
Quote: "Nobody in their right mind would ever suggest"
to use plain backpropagation for training deep networks.
But in 2010, our team showed[MLP1-2]
unsupervised pre-training is not necessary
to train deep NNs.
[VID2] Bloomberg Hello World.
The Rise of AI.
Youtube video, 2018.
G. H. Wannier (1945).
The Statistical Problem in Cooperative Phenomena.
Rev. Mod. Phys. 17, 50.
[WU] Y. Wu et al. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation.
Preprint arXiv:1609.08144 (PDF), 2016. Based on LSTM which it mentions at least 50 times.
X. Glorot, Y. Bengio.
Understanding the difficulty of training deep feedforward neural networks.
Proc. 13th Intl. Conference on Artificial Intelligence and Statistics,
PMLR 9:249-256, 2010.
Y. Bengio. Notable Past Research.
WWW link (retrieved 15 May 2020).
Local copy (plain HTML only).
K. Zuse (1936).
Verfahren zur selbsttätigen Durchführung von Rechnungen mit Hilfe von
Rechenmaschinen. Patent application Z 23 139 / GMD Nr. 005/021, 1936.
First patent application describing
a general, practical, program-controlled computer.
K. Zuse (1948). Über den Plankalkül als Mittel zur Formulierung schematisch kombinativer Aufgaben.
Archiv der Mathematik 1(6), 441-449 (1948).
Apparently the first practical design of an automatic theorem prover
(based on Zuse's high-level programming language Plankalkül).
J. Schmidhuber (AI Blog, 2021). 80th anniversary celebrations: 1941: Konrad Zuse completes the first working general computer, based on his 1936 patent application.