Principles of Artificial Neural Networks

The following sections are included:

• Dedication
• Acknowledgments
• Preface to the Fourth Edition
• Preface to the First Edition
• Contents

Artificial neural networks are, as their name indicates, computational networks which attempt to simulate, in a gross manner, the decision process in networks of nerve cell (neurons) of the biological (human or animal) central nervous system. This simulation is a gross cell-by-cell (neuron-by-neuron, element-by-element) simulation. It borrows from the neurophysiological knowledge of biological neurons and of networks of such biological neurons. It thus differs from conventional (digital or analog) computing machines that serve to replace, enhance or speed-up human brain computation without regard to organization of the computing elements and of their networking. Still, we emphasize that the simulation afforded by neural networks is very gross…

The biological neural network consists of nerve cells (neurons) as in Fig. 2.1, which are interconnected as in Fig. 2.2. The cell body of the neuron, which includes the neuron’s nucleus is where most of the neural “computation” takes place. Neural activity passes from one neuron to another in terms of electrical triggers which travel from one cell to the other down the neuron’s axon, by means of an electrochemical process of voltage-gated ion exchange along the axon and of diffusion of neurotransmitter molecules through the membrane over the synaptic gap (Fig. 2.3). The axon can be viewed as a connection wire. However, the mechanism of signal flow is not via electrical conduction but via charge exchange that is transported by diffusion of ions. This transportation process moves along the neuron’s cell, down the axon and then through synaptic junctions at the end of the axon via a very narrow synaptic space to the dendrites and/or soma of the next neuron at an average rate of 3 m/sec., as in Fig. 2.3…

The following sections are included:

• Basic Principles of ANN Design
• Basic Neural Structures
• The Perceptron’s Input-Output Principles
• The Adaline (ALC)

The following sections are included:

• The Basic Structure
• The Single-Layer Representation Problem
• The Limitations of the Single-Layer Perceptron
• Many-Layer Perceptrons
• Perceptron Case Study: Identifying Autoregressive Parameters of a Signal (AR Time Series Identification)

The Madaline (Many Adaline) is a multilayer extension of the single-neuron bipolar Adaline to a network. It is also due to B. Widrow (1988). Since the Madaline network is a direct multi-layer extension of the Adaline of Sec. 3, we present it before discussing the Back Propagation network that is historically earlier (see our discussion in Sec. 4.4 above). Its weight adjustment methodology is more intuitive than in Back Propagation and provides understanding into the difficulty of adjusting weights in a multi-layer network, though it is less efficient. Its basic structure is given in Fig. 5.1 which is in terms of two layers of Adalines, plus an input layer which merely serves as a network’s input distributor (see Fig. 5.2).

The following sections are included:

• The Back Propagation Learning Procedure
• Derivation of the BP Algorithm
• Modified BP Algorithms
• Back Propagation Case Study: Character Recognition
• Back Propagation Case Study: The Exclusive-OR (XOR) Problem (2-Layer BP)
• Back Propagation Case Study: The XOR Problem — 3 Layer BP Network
• Average Monthly High and Low Temperature Prediction Using Backpropagation Neural Networks

The following sections are included:

• Introduction
• Binary Hopfield Networks
• Setting of Weights in Hopfield Nets — Bidirectional Associative Memory (BAM) Principle
• Walsh Functions
• Network Stability
• Summary of the Procedure for Implementing the Hopfield Network
• Continuous Hopfield Models
• The Continuous Energy (Lyapunov) Function
• Hopfield Network Case Study: Character Recognition
• Hopfield Network Case Study: Traveling Salesman Problem
• Cell Shape Detection Using Neural Networks

The following sections are included:

• Introduction
• Kohonen Self-Organizing Map (SOM) Layer
• Grossberg Layer
• Training of the Kohonen Layer
• Training of Grossberg Layers
• The Combined Counter Propagation Network
• Counter Propagation Network Case Study: Character Recognition

The following sections are included:

• Motivation
• The ART Network Structure
• Setting-Up of the ART Network
• Network Operation
• Properties of ART
• Discussion and General Comments on ART-I and ART-II
• ART-I Network Case Study: Character Recognition
• ART-I Case Study: Speech Recognition

The following sections are included:

• Introduction
• The Basic Principles of the Cognitron
• Network Operation
• Cognitron’s Network Training
• The Neocognitron

The following sections are included:

• Fundamental Philosophy
• Annealing Methods
• Simulated Annealing by Boltzmann Training of Weights
• Stochastic Determination of Magnitude of Weight Change
• Temperature-Equivalent Setting
• Cauchy Training of Neural Network
• Statistical Training Case Study: A Stochastic Hopfield Network for Character Recognition
• Statistical Training Case Study: Identifying AR Signal Parameters with a Stochastic Perceptron Model

The following sections are included:

• Recurrent/Discrete Time Networks
• Fully Recurrent Networks
• Continuously Recurrent Back Propagation Networks
• Recurrent Back Propagation Case Study: Character Recognition

The following sections are included:

• Definition
• Brief History of DNN and of Its Applications
• The Scope of DLNN
• Introduction to Specific DLNN Algorithms

The following sections are included:

• Introduction
• Feed-Forward Loop
• The Convolution Layer
• Back Propagation
• ReLu Layers
• Pooling Layers
• Dropout
• Output FC Layer
• Parameter (Weight) Sharing
• Applications

The following sections are included:

• LAMSTAR Principles
• LAMSTAR-1 (LNN-1)
• LAMSTAR-2 (LNN-2)
• Data Analysis LAMSTAR
• Comments and Applications

This chapter discusses the performance of the Deep-Learning neural networks and specifically the networks of Chaps. 14 and 15. It will do so using 20 case studies over a wide range of applications, which were performed by graduate students in the author’s class. The chapter will also compare the computational speeds of these networks, as is most important for any real-time online application (as a part of a controller, or in many applications in medical devices, medical diagnosis, and in many traffic of financial applications. The chapter also aims to provide better understanding of how these networks are applied in a wide range of applications in the above areas and beyond. Conclusion drawn from these 20 Case Studies are certainly not representative of the huge literature on deep learning neural networks. Especially, they may not be applicable to their applications to huge data bases (some of which is mentioned in Chap. 14, with respect to CNN applications). However, hardly any comparison is available in the literature, at least, not for comparing CNN and LAMSTAR, and certainly not for the same input data. And no comparison is of value if the data are not the same for the networks being compared. In contrast, the present case studies were specially designed to do just this, namely to apply several DLNN networks with the exact same data to a broad range of applications and with the same computing device (as is essential when evaluating computing speed). Further comparisons on very large databases are certainly called for.

The following sections are included:

• Problems
• References
• Author Index
• Subject Index