Complex-Valued Neural Networks

The following sections are included:

• Foreword

• Preface

• Contents

Please refer to full text.

This chapter presents some results of an analysis on the decision boundaries of the complex-valued neural networks whose weights, threshold values, input and output signals are all complex numbers. The main results can be summarized as follows. (a) Decision boundary of a single complex-valued neuron consists of two hypersurfaces which intersect orthogonally, and divides a decision region into four equal sections. Decision boundary of a three-layered complex-valued neural network has this as a basic structure, and its two hypersurfaces intersect orthogonally if net inputs to each hidden neuron are all sufficiently large. (b) Most of the decision boundaries in the 3-layered complex-valued neural network intersect orthogonally when the network is trained using the Complex-BP algorithm. As a result, the orthogonality of the decision boundaries improves its generalization ability. (c) Furthermore, the average of the learning speed of the Complex-BP is several times faster than that of the Real-BP. The standard deviation of the learning speed of the Complex-BP is smaller than that of the Real-BP. It seems that the complex-valued neural network and the related algorithm are natural for learning of complex-valued patterns for the above reasons.

An associative memory model called complex-valued neural network (CVNN) is presented in this chapter. In a CVNN states are represented by quantization values defined on the unit circle of the complex plane. Such a network is able to perform the task of storing and recalling gray-scale images. The stability properties under different updating modes are investigated by using the energy function approach. It is proved that the model will be globally convergent to a fixed point when operating in a asynchronous mode and to a cycle of length at most 2 when operating in a synchronous mode. Then, some existing learning methods for this model are reviewed and discussed. Finally, simulation results are presented to illustrate the performance of this model.

This chapter presents a model of associative memories using complex-valued neural networks and studies its qualitative behavior theoretically. The model is a direct extension of the conventional real-valued associative memories of self-correlation type. One of the most familiar models of associative memories is self-correlation type. We are interested in what will become of the conventional real-valued associative memories when they are directly extended in the complex domain. We investigate the structures and asymptotic behavior of solution orbits near each memory pattern. We also discuss a recalling condition of each memory pattern, that is, a condition which assures that each memory pattern is correctly recalled.

This chapter introduces a class of feed-forward neural networks which have Clifford valued weight and activation values. Clifford Algebras generalise the Complex and Quaternion algebras to higher-dimensions, thus Clifford networks are natural generalisations of complex valued networks. In this chapter the back-propagation algorithm is derived for Clifford valued networks and an approximation theorem is proved. Because Clifford Algebras are a generalisation of the Complex and Quaternion algebras the approximation results also shows that Complex networks are universal functional approximators.

In this chapter, we present two different applications of complex neuron models. One is a complex associative memory and the other is a complex version of the Nagumo-Sato model of a single neuron. Although these two applications are on the opposite extremes of scale, the one with many (desirably infinite in the limit) units and the other with a single or at most a few neurons, the motivation for the use of complex numbers is the same; we want to treat the timing or the phase of impulse trains with as simple a method as possible. Therefore, the properties of the model so far found should be attributed to the fact that the phase of impulse trains are being taken into consideration. It is our future work to use this model to explain (to some degree of accuracy) those phenonema in the brain in which timing of impulses or chaotic behaviros are important.

A class of data-reusing (DR) learning algorithms for complex-valued linear and nonlinear adaptive filters is analyzed. This class of algorithms has an improved convergence over the standard algorithms by virtue of re-using of the external input data while performing iterations on weight adaptation. The class of algorithms are introduced starting from the case of linear adaptive filters trained with the complex-valued least mean square (CLMS) algorithm, through to the case of feedforward and recurrent neural networks employed as nonlinear adaptive filters trained with a complex-valued gradient descent (CGD) learning algorithm and a complex-valued real time recurrent (CRTRL) learning algorithm, respectively. Both the error bounds and convergence conditions are provided for the case of contractive and expansive complex activation functions. The improved local performance of the complex-valued data-reusing algorithm over the standard algorithms is verified by simulations on the prediction of linear and nonlinear complex-valued signals.

Neural network architectures that can handle complex inputs, such as backpropagation networks, perceptrons or generalized Hopfield networks, require a large amount of time and resources for the training process. Here we adapt the time-efficient corner classification approach to train feedforward neural networks to handle complex inputs and present a new algorithm called the 3C algorithm. This algorithm uses prescriptive learning, where the network weights are assigned simply upon examining the inputs. The performance of the algorithm is tested using the pattern classification experiment and the time series prediction experiment with the Mackey-Glass time series. An input encoding called quaternary encoding is used for both experiments since it reduces the network size significantly by cutting down on the number of neurons that are required at the input layer.

We consider a complex-valued neuron model which can take K (K>=2) states on the unit circle in the complex plane as the extension of a well-known binary neuron which can take only two values (1,-1). In the network in which all the complex-valued neurons are fully connected with each other, we determine the state transition rule of each neuron under the condition that the state transition always deceases the network energy by a maximum amount. Next discussed is how a grayscale image can be expressed using the complex-valued neural network obtained above. We present a method for image representation using only phase. This is accomplished through embedding of the amplitude data into the phase data after the 2-dimensional discrete Fourier transform of an image. The embedding of the amplitude data into the phase data can be realized by a phase shift operation. We show that adjusting the phase shift quantity enables us to control the arbitrary frequency component of an input image.

We demonstrate two examples of image processing using complex-valued neural networks applying the aforementioned. They are an image filtering, and a grayscale image associative memory which can memorize grayscale images and recall one from a noisy or imperfect version of the memorized image. Note that we propose two novel types of neuron when implementing these networks. One is a model in which a phase of complex-valued output signal is shifted by an amount corresponding to the amplitude of a complex-valued input signal. The other is a model, in which the output signal is generated as the projection to the real or imaginary axis of a complex-valued input signal.

To deal with temporal sequences is very important and difficult problem for applications of neural networks. In this chapter, we aim at constructing novel recurrent neural network which can process temporal sequences. The memorization ability of temporal sequences can be used in a lot of fields e.g. control, information processing, thinking support systems, and so on.

In section 1, the background and the purposes of this chapter is described.

In section 2, a Multilayer Network using Complex neurons with local Feedback (MNCF) is explained. A complex neuron model can keep previous information more easily than a conventional neuron models because of the phase component. A simple learning algorithm based on the back-propagation for temporal sequences which is named Complex Back-Propagation for Temporal sequences (CBPT) is derived. It can be considered as a generalized original backpropagation. It is shown in some computer simulations that the network has better ability than the conventional ones, including Elman's network.

In sections 3 and 4, a music retrieval system using a complex-valued recurrent neural network which is named MUSIC (MUltilayer network for Sequential Inputs using Complex neurons) is described. In the system, melodies are treated as temporal sequences. In the conventional associative memory models, melodies should be given at a time and they are treated as static patterns. MUSIC can treat a number of melodies by some smaller networks. Such architecture has an advantage that the pattern matching process is not required. MUSIC uses a part of the melodies as a key instead of the text information.

In section 5, the results mentioned in this chapter are concluded.

Principal component extraction is an efficient statistical tool that is applied to feature extraction, data compression, and signal processing. The Generalized Hebbian Algorithm (GHA) (Sanger 1992) can be used to iteratively extract principal eigenvectors in the real domain. In some scenarios such as sensor array signal processing, we encounter complex data. The Complex-valued Generalized Hebbian Algorithm (CGHA) (Zhang et al. 1997) is presented in this chapter. Convergence of CGHA is proved. Like GHA, CGHA can be implemented by a single-layer linear neural network. An application of CGHA to sensor array signal processing is demonstrated through Direction of Arrival (DOA) estimation.

In this chapter, we introduce a phasor model of neural networks where the state of each neuron possibly takes the value at the origin as well as on the unit circle and show some important properties concerning the stability of an equilibrium. Moreover, an application of the phasor model to multiuser detection in code-division multiple-access (CDMA) communication is considered. In the CDMA system considered, transmitted data take complex values, and users are allowed to be in the inactive mode as well as the active mode. Simulation results show that a detector using the phasor model can outperform a conventional detector.

This Chapter presents an application of the complex-valued neural network (CVNN) for interferometric radar (InSAR-Interferometric Synthetic Aperture Radar) image processing. The InSAR image, whose pixels are naturally represented by complex numbers, is modeled by CMRF (Complex-valued Markov Random Field). Then the InSAR image that is corrupted by noise-induced singular points (SP) is mapped to a CMRF lattice neural network (CMRF-LNN). By evolving the states of the CMRF-LNN toward a minimum energy value, the SP number is eventually reduced. The advantages for phase unwrapping are demonstrated for a simulated as well as real InSAR images.

This chapter reviews our early papers on a complex phase-conjugate neural network model with a Hopfield-like energy function. The complex neural network of the proposed model can change both the amplitude and the phase, and their dynamics has a close analogy to the dynamics of self-oscillation generated in an optical phase-conjugate resonator. It is shown that the optical gain medium should have a phase conjugate property in order for the generated complex optical fields to have an energy function that decreases monotonically with the time evolution of the fields. The results of experiments and computer simulations are presented that demonstrate the behaviors of the complex neural fields predicted by the theory.

The spatial coherence of lightwave extends the ability of optical information-processing systems based on the spatial parallelism. Orthogonally to space, the temporal coherence also raises their potential by make good use of the vast optical frequency domain, i.e., the frequency domain multiplexing. The frequency domain attracts neural network systems because many of the neural specific dynamics originate from the distributed and parallel architecture. Furthermore, the optical carrier frequency can become the key information for modulation of the network behavior such as learning, self-organization and adaptive processing. In this chapter, we treat the complex-valued neural networks from the viewpoint of frequency-sensitive coherent lightwave information processing. We describe the theory and experimental results where the carrier frequency is found useful to control the learning and processing behavior of the neural networks.

The following sections are included:

• Index

• Contributors