ADALINE (an acronym for "Adaptive Linear Neuron," later also interpreted as "Adaptive Linear Element") is an early single-layer artificial neural network model and the name of the physical hardware device that implemented it. It was developed by professor Bernard Widrow and his doctoral student Marcian Hoff at Stanford University in 1960.
Historical Context
ADALINE was among the earliest trainable neural network architectures, developed shortly after Frank Rosenblatt's perceptron. It was implemented as a physical device whose weights and biases were adjusted using rheostats (referred to as the "knobby ADALINE") and, in later versions, memistors. ADALINE found extensive use in adaptive signal processing, particularly in adaptive noise filtering.
Architecture and Definition
ADALINE is a single-layer neural network with multiple nodes, where each node accepts multiple inputs and generates one output. Given an input vector x, a weight vector w, a bias b, and N inputs, the output o is computed as:
o = Σ(xₙwₙ) + b
If x₀ = 1 and w₀ = b, the output simplifies to a weighted sum of the inputs and bias.
Relationship to the Perceptron
The principal difference between ADALINE and the standard (Rosenblatt) perceptron lies in the learning method. In ADALINE, unit weights are adjusted to match a teacher signal before applying the (Heaviside) threshold function. In the standard perceptron, weights are adjusted to match the correct output after applying the Heaviside function.
Learning Rule
ADALINE uses the Least Mean Squares (LMS) algorithm, also known as the Widrow–Hoff rule or the delta rule, which is a special case of gradient descent. Given a learning rate η, the model output o, the desired target y, and the squared error E = (y − o)², the weights are updated as:
w ← w + η (y − o) x
This update rule minimizes the squared error E and is equivalent to the stochastic gradient descent update used in linear regression. Training is based on minimizing a continuous cost function.
MADALINE
A multilayer network composed of ADALINE units is known as MADALINE (Many ADALINE). MADALINE is a three-layer, fully connected, feedforward neural network for classification that uses ADALINE units in its hidden and output layers, employing the sign function as its activation. Because the sign function is non-differentiable, standard backpropagation could not be used, leading to the development of three alternative training algorithms: Rule I, Rule II, and Rule III.
The largest MADALINE machine built had 1,000 weights, each implemented by a memistor, and was constructed in 1963. Some MADALINE machines demonstrated tasks including inverted pendulum balancing, weather forecasting, and speech recognition.
Significance and Legacy
ADALINE is recognized as a foundational model in the history of artificial neural networks and machine learning. Its continuous-cost-function approach laid groundwork for later algorithms, and its associated LMS learning rule became fundamental to numerous adaptive signal processing and machine learning systems, including adaptive antennas, adaptive noise canceling, and adaptive equalization in high-speed modems. The model is documented extensively in academic literature and historical accounts of neural network research, confirming its status as an established and widely recognized concept.
