Principles of Artificial Neural Networks

The following sections are included:

• Acknowledgments

• Preface to the Third Edition

• Preface to the Second Edition

• Preface to the First Edition

• Contents

Artificial neural networks are, as their name indicates, computational networks which attempt to simulate, in a gross manner, the decision process in networks of nerve cell (neurons) of the biological (human or animal) central nervous system. This simulation is a gross cell-by-cell (neuron-by-neuron, element-by-element) simulation. It borrows from the neurophysiological knowledge of biological neurons and of networks of such biological neurons. It thus differs from conventional (digital or analog) computing machines that serve to replace, enhance or speed-up human brain computation without regard to organization of the computing elements and of their networking. Still, we emphasize that the simulation afforded by neural networks is very gross…

The biological neural network consists of nerve cells (neurons) as in Fig. 2.1, which are interconnected as in Fig. 2.2. The cell body of the neuron, which includes the neuron’s nucleus is where most of the neural “computation” takes place. Neural activity passes from one neuron to another in terms of electrical triggers which travel from one cell to the other down the neuron’s axon, by means of an electrochemical process of voltage-gated ion exchange along the axon and of diffusion of neurotransmitter molecules through the membrane over the synaptic gap (Fig. 2.3). The axon can be viewed as a connection wire. However, the mechanism of signal flow is not via electrical conduction but via charge exchange that is transported by diffusion of ions. This transportation process moves along the neuron’s cell, down the axon and then through synaptic junctions at the end of the axon via a very narrow synaptic space to the dendrites and/or soma of the next neuron at an average rate of 3 m/sec., as in Fig. 2.3.…

The following sections are included:

• Basic Principles of ANN Design

• Basic Network Structures

• The Perceptron’s Input-Output Principles

• The Adaline (ALC)
  • LMS training of ALC
  • Steepest descent training of ALC

The following sections are included:

• The Basic Structure
  • Perceptron’s activation functions

• The Single-Layer Representation Problem

• The Limitations of the Single-Layer Perceptron

• Many-Layer Perceptrons

• Perceptron Case Study: Identifying Autoregressive Parameters of a Signal (AR Time Series Identification)

The following sections are included:

• Madaline Training

• Madaline Case Study: Character Recognition
  • Problem statement
  • Design of network
  • Training of the network
  • Results
  • Conclusions and observations
  • MATLAB source code for implementing MADALINE network:

The following sections are included:

• The Back Propagation Learning Procedure

• Derivation of the BP Algorithm

• Modified BP Algorithms
  • Introduction of bias into NN
  • Incorporating momentum or smoothing to weight adjustment
  • Other modification concerning convergence

• Back Propagation Case Study: Character Recognition
  • Introduction
  • Network design
  • Results
  • Discussion and conclusions
  • Source Code (C++)

• Back Propagation Case Study: The Exclusive-OR (XOR) Problem (2-Layer BP)

• Back Propagation Case Study: The XOR Problem — 3 Layer BP Network

• Average Monthly High and Low Temperature Prediction Using Backpropagation Neural Networks
  • Introduction
  • Design
  • Results
  • Conclusion
  • Source Code (Matlab)

The following sections are included:

• Introduction

• Binary Hopfield Networks

• Setting of Weights in Hopfield Nets — Bidirectional Associative Memory (BAM) Principle

• Walsh Functions

• Network Stability

• Summary of the Procedure for Implementing the Hopfield Network

• Continuous Hopfield Models

• The Continuous Energy (Lyapunov) Function

• Hopfield Network Case Study: Character Recognition
  • Introduction
  • Network design
  • Setting of weights
  • Testing
  • Results and conclusions
  • MATALAB source codes

• Hopfield Network Case Study: Traveling Salesman Problem
  • Introduction
  • Hopfield neural network design
  • Input selection
  • Implementation details
  • Output results
  • Concluding discussion

• Cell Shape Detection Using Neural Networks
  • Introduction
  • NN design
  • Data setup
  • Results
  • Backpropagation-based preprocessor and its effect
  • Conclusions
  • Source Code (Matlab)

The following sections are included:

• Introduction

• Kohonen Self-Organizing Map (SOM) Layer

• Grossberg Layer

• Training of the Kohonen Layer
  • Preprocessing of Kohonen layer’s inputs
  • Initializing the weights of the Kohonen layer
  • Interpolative mode layer

• Interpolative mode layer

• The Combined Counter Propagation Network

• Counter Propagation Network Case Study: Character Recognition
  • Introduction
  • Network structure
  • Network training
  • Test mode
  • Results and conclusions
  • Source codes (MATLAB)

The following sections are included:

• Motivation

• Basic Principles of the LAMSTAR Neural Network

• Detailed Outline of the LAMSTAR Network
  • Basic structural elements
  • Setting of input-storage weights and determination of winning neurons
  • Adjustment of resolution in SOM modules
  • Links between SOM modules and from SOM modules to output modules
  • Determination of winning decision via link weights
  • N_j weights (not implemented in most applications)
  • Initialization and local minima

• Forgetting Feature

• Training vs. Operational Runs
  • INPUT WORD for training and for information retrieval

• Operation in Face of Missing Data

• Advanced Data Analysis Capabilities
  • Feature extraction and reduction in the LAMSTAR NN
  • Correlation, Interpolation, Extrapolation and Innovation-Detection
  • Innovation detection in the LAMSTAR NN

• Modified Version: Normalized Weights

• Concluding Comments and Discussion of Applicability

• LAMSTAR Network Case Study: Character Recognition
  • Introduction
  • Design of the network
  • Fundamental principles
  • Training algorithm
  • Testing procedure
  • Results and their analysis
  • Summary and concluding observations
  • LAMSTAR SOURCE CODE (MATLAB)

• Application to Medical Diagnosis Problems
  • Application to ESWL Medical Diagnosis Problem
  • Application to Renal Cancer Diagnosis Problem
  • Application to Diagnosis of Drug Abuse for Emergency Cases
  • Detection of Epilepsy
  • Application to Assessing of Fetal Well-Being
  • Predicting Onset of Sleep Apnea Events using Modified LAMSTAR Network

• Predicting Price Movement in Market Microstructure via LAMSTAR
  • Abstract
  • Introduction
  • Input Data
  • Setup of LAMSTAR Neural Network
  • Results
  • Conclusions
  • Computer Codes – Main Outputs (LAMSTAR NN, SVM machine, BP NN, RBF NN)
  • LAMSTAR Source Code (Matlab)

• Constellation Recognition
  • Introduction
  • Approach
  • Recognition
  • System Architecture
  • Results
  • Conclusions
  • Source Code (Java)

The following sections are included:

• Motivation

• The ART Network Structure
  • The comparison layer (CL)
  • The recognition layer (RL)
  • Gain and reset elements

• Setting-Up of the ART Network
  • Initialization of weights
  • Training

• Network Operation

• Properties of ART

• Discussion and General Comments on ART-I and ART-II

• ART-I Network Case Study: Character Recognition
  • Introduction
  • The data set
  • Network design
  • Performance results and conclusions
  • Source Code for ART neural network (Java)

• ART-I Case Study: Speech Recognition
  • Input matrix set-up for spoken words
  • Simulation programs Set-Up
  • Source Code – simulation of ART program (C-language)
  • Simulation results

The following sections are included:

• Background of the Cognitron

• The Basic Principles of the Cognitron

• Network Operation

• Cognitron’s Network Training

• The Neocognitron

The following sections are included:

• Fundamental Philosophy

• Annealing Methods

• Simulated Annealing by Boltzman Training of Weights

• Stochastic Determination of Magnitude of Weight Change

• Temperature-Equivalent Setting

• Cauchy Training of Neural Network

• Statistical Training Case Study: A Stochastic Hopfield Network for Character Recognition
  • Motivation
  • Problem statement
  • Algorithm set-up
  • Computed results

• Statistical Training Case Study: Identifying AR Signal Parameters with a Stochastic Perceptron Model
  • Problem set-up
  • Source Code (written in MATLAB® – see also Sec. 4.A)
  • Estimated parameter set at each iteration (using stochastic training)

The following sections are included:

• Recurrent/Discrete Time Networks

• Fully Recurrent Networks

• Continuously Recurrent Back Propagation Networks

• Recurrent Back Propagation Case Study: Character Recognition
  • Introduction
  • Design of neural network
  • Results
  • Discussion and conclusions
  • Source Code (C++)

The following sections are included:

• Problems

• References

• Author Index

• Subject Index