Narrow Results

The advent of the Connection Machine profoundly changes the world of supercomputers. Its highly nontraditional architecture makes possible the exploration of algorithms that were impractical for standard Von Neumann architectures. Kanerva’s sparse distributed memory (SDM) is an example of such an algorithm.

Sparse distributed memory is a particularly simple and elegant formulation for an associative memory. In this paper I describe the foundations for sparse distributed memory, and give some simple examples of using the memory. I continue by showing the relationship of sparse distributed memory to random-access memory. Finally, I discuss the implementation of the algorithm for sparse distributed memory on the Connection Machine.

We study a neural network with asymmetric connections used as an associative memory. Asymmetry allows the nominal patterns to be stored in cycles. We apply an unlearning procedure, which modifies the synaptic connections. We analyze the global performance, including the network capacity, the attraction basin's size and also the relaxation time distribution. The latter shows a convenient bimodality that is used for discriminating between spurious and stored memory attractors. We show that unlearning in asymmetric networks allows enhancing the global performance of retrieval including retrieval of a sequence of correlated patterns.

This paper defines the truncated normalized max product operation for the transformation ofstates of a network and provides a method for solving a set of equations based on this operation. The operation serves as the transformation for the set of fully connected units in a recurrent network that otherwise might consist of linear threshold units. Component values of the state vector and ouputs of the units take on the values in the set {0, 0.1, …, 0.9, 1}. The result is a much larger state space given a particular number of units and size of connection matrix than for a network based on threshold units. Since the operation defined here can form the basis of transformations in a recurrent network with a finite number of states, fixed points or cycles are possible and the network based on this operation for transformations can be used as an associative memory or pattern classifier with fixed points taking on the role of prototypes. Discrete fully recurrent networks have proven themselves to be very useful as associative memories and as classifiers. However they are often based on units that have binary states. The effect of this is that the data to be processed consisting of vectors in ℜⁿ have to be converted to vectors in {0, 1}^m with m much larger than n since binary encoding based on positional notation is not feasible. This implies a large increase in the number of components. The effect can be lessened by allowing more states for each unit in our network. The network proposed demonstrates those properties that are desirable in an associative memory very well as the simulations show.

This paper is concerned with large scale associative memory design. A serious problem with neural associative memories is the quadratic growth of the number of interconnections with the problem size. An overlapping decomposition algorithm is proposed to attack this problem. Specifically, a pattern to be processed is decomposed into overlapping sub-patterns. Then, neural sub-networks are constructed that process the sub-patterns. An error correction algorithm operates on the outputs of each sub-network in order to correct the mismatches between sub-patterns that are obtained from the independent recall processes of individual sub-networks. The performance of the proposed large scale associative memory is illustrated using two-dimensional images. It is shown that the proposed method reduces the computing cost of the design of the associative memories compared with non-interconnected associative memories.

We investigate the memory structure and retrieval of the brain and propose a hybrid neural network of addressable and content-addressable memory which is a special database model and can memorize and retrieve any piece of information (a binary pattern) both addressably and content-addressably. The architecture of this hybrid neural network is hierarchical and takes the form of a tree of slabs which consist of binary neurons with the same array. Simplex memory neural networks are considered as the slabs of basic memory units, being distributed on the terminal vertexes of the tree. It is shown by theoretical analysis that the hybrid neural network is able to be constructed with Hebbian and competitive learning rules, and some other important characteristics of its learning and memory behavior are also consistent with those of the brain. Moreover, we demonstrate the hybrid neural network on a set of ten binary numeral patters.

A class of interconnected neural networks composed of generalized Brain-State-in-a-Box (gBSB) neural subnetworks is considered. Interconnected gBSB neural network architectures are proposed along with their stability conditions. The design of the interconnected neural networks is reduced to the problem of solving linear matrix inequalities (LMIs) to determine the interconnection parameters. A method for solving LMIs is devised generating the solutions that, in general, are further away from zero than the corresponding solutions obtained using MATLAB's LMI toolbox, thus resulting in stronger interconnections between the subnetworks. The proposed architectures are then used to construct neural associative memories. Simulations are performed to illustrate the results obtained.

Associative memory networks based on quaternionic Hopfield neural network are investigated in this paper. These networks are composed of quaternionic neurons, and input, output, threshold, and connection weights are represented in quaternions, which is a class of hypercomplex number systems. The energy function of the network and the Hebbian rule for embedding patterns are introduced. The stable states and their basins are explored for the networks with three neurons and four neurons. It is clarified that there exist at most 16 stable states, called multiplet components, as the degenerated stored patterns, and each of these states has its basin in the quaternionic networks.

Synchronization of neural activity in the gamma band, modulated by a slower theta rhythm, is assumed to play a significant role in binding and segmentation of multiple objects. In the present work, a recent neural mass model of a single cortical column is used to analyze the synaptic mechanisms which can warrant synchronization and desynchronization of cortical columns, during an autoassociation memory task. The model considers two distinct layers communicating via feedforward connections. The first layer receives the external input and works as an autoassociative network in the theta band, to recover a previously memorized object from incomplete information. The second realizes segmentation of different objects in the gamma band. To this end, units within both layers are connected with synapses trained on the basis of previous experience to store objects. The main model assumptions are: (i) recovery of incomplete objects is realized by excitatory synapses from pyramidal to pyramidal neurons in the same object; (ii) binding in the gamma range is realized by excitatory synapses from pyramidal neurons to fast inhibitory interneurons in the same object. These synapses (both at points i and ii) have a few ms dynamics and are trained with a Hebbian mechanism. (iii) Segmentation is realized with faster AMPA synapses, with rise times smaller than 1 ms, trained with an anti-Hebbian mechanism. Results show that the model, with the previous assumptions, can correctly reconstruct and segment three simultaneous objects, starting from incomplete knowledge. Segmentation of more objects is possible but requires an increased ratio between the theta and gamma periods.

We discuss stochastic resonance in associative memory with a canonical neural network model that describes the generic behavior of a large family of dynamical systems near bifurcation. Our result shows that stochastic resonance helps memory association. The relationship between stochastic resonance, associative memory, storage load, history of memory and initial states are studied. In intelligent systems like neural networks, it is likely that stochastic resonance combined with synaptic information enhances memory recalls.

Synchronization is introduced into a chaotic neural network model to discuss its associative memory. The relative time of synchronization of trajectories is used as a measure of pattern recognition by chaotic neural networks. The retrievability of memory is shown to be connected to synapses, initial conditions and storage capacity. The technique is simple and easy to apply to neural systems.

Among the several findings deriving from the application of complex network formalism to the investigation of natural phenomena, the fact that linguistic constructions follow power laws presents special interest for its potential implications for psychology and brain science. By corresponding to one of the most essentially human manifestations, such language-related properties suggest that similar dynamics may also be inherent to the brain areas related to language and associative memory, and perhaps even consciousness. The present work reports a preliminary experimental investigation aimed at characterizing and modeling the flow of sequentially induced associations between words from the English language in terms of complex networks. The data is produced through a psychophysical experiment where a word is presented to the subject, who is requested to associate another word. Complex network and graph theory formalism and measurements are applied in order to characterize the experimental data. Several interesting results are identified, including the characterization of attraction basins, association asymmetries, context biasing, as well as a possible power-law underlying word associations, which could be explained by the appearance of strange loops along the hierarchical structure underlying word categories.

Calculating the exact number of fixed points and attractors of an arbitrary Hopfield neural network is a non-deterministic polynomial (NP)-hard problem. In this paper, we first calculate the average number of fixed points in such networks versus their size and threshold of neurons, in terms of a statistical method, which has been applied to the calculation of the average number of metastable states in spin glass systems. Then the same method is expanded to study the average number of attractors in such networks. The results of the calculation qualitatively agree well with the numerical calculation. The discrepancies between them are also well explained.

The unique characteristic of quantum mechanics may be used in the near future to create a quantum associative memory with a capacity exponential in the number of neurons. In this paper we discuss some quantum computational ideas and algorithms necessary to develop a quantum associative memory.

Traditional biological neural networks lack the capability of reflecting variable synaptic weights when simulating associative memory of human brains. In this paper, we investigate the existence and exponential stability of a novel memristive multidirectional associative memory neural networks (MAMNNs) model, which includes the time-varying delays. In the proposed approach, the time-varying delays are set to be bounded, and it is not necessary for their derivative to be differentiable. With removal of certain conditions, less conservative results are generated. Sufficient criteria guaranteeing the stability of the memristive MAMNNs are derived based on the Lyapunov function and some inequality techniques. To illustrate the performance of the proposed criteria, a procedure is designed to realize information storage. Meanwhile, the effectiveness of the proposed theories is validated with numerical experiments.

Fully recurrent networks have proven themselves to be very useful as associative memories and as classifiers. However, they are generally based on units that have binary states. The effect of this is that data to be processed consisting of vectors in have to be converted to vectors in {0, 1}^m with m much larger than n since binary encoding based on positional notation is not feasible. This implies a large increase in number of components. This effect can be lessened by allowing more states for each unit in our network.

This paper describes two effective learning algorithms for a network whose units take the dot product of the input with a weight vector, followed by a tanh transformation and a discretization transformation in the form of rounding or truncation. The units have states that are in {0, 0.1, 0.2, …, 0.9, 1} rather than in {0, 1} or {-1, 1}. The result is a much larger state space given a particular number of units and size of connection matrix. Two convergent learning algorithms for training such a network to store fixed points or attractors are proposed. The network exhibits those properties that are desirable in an associative memory such as limit cycles of 1, attraction to the closest attractor and few transitions required to reach attractors. Since memories that are stored can be used to represent prototypes of patterns the network is useful for pattern classification. A pattern to be classified would be entered and its class would be the same as the class of the prototype to which it is attracted to which it is.

The Hopfield model for associative recall in a massively connected binary network is reviewed. The problems involved in representation are pointed out. Learning of cross-correlations is introduced, and results of computer simulations are displayed. The relationship with least mean squares learning in feed-forward layered networks is pointed out, leading to the analogue of unlearning. Simulation results show the performance of such a system.

A connectionist expert system model is proposed in this paper. The system combines the capability of production systems and associative memory. Rules are explicitly represented within a network structure. The Perceptron algorithm is generalized to include samples with uncertain components. A gradually-augmented-node learning algorithm is used to guarantee fast memorization of all rules. The convergence of the learning algorithms is analyzed. The system has a dynamic knowledge organization and adapts in real time to acquire new knowledge, or to relearn existing knowledge, through interaction with the user. This allows the system to be built incrementally. An example system is presented.

A pattern recognition system mainly contains two functional parts, i.e. feature extraction and pattern classification. The success of such a system depends on not only the effectiveness of each of them, but also their operation in concert. The feature extraction process in a traditional recognition system has two major tasks, namely, to extract deformation invariant signals and to reduce data. When a neural network is used as a pattern classifier, however, an alteration in these basic objectives is needed. In particular, the consideration of data reduction will be replaced by that of the suitability of feature vectors to the neural network. In this paper, feature extraction algorithms in character recognition have been designed based on these principles. The improvements made by these algorithms have been demonstrated in a series of experiments which justify such a change in the fundamental objectives of the feature extraction process when an associative memory classifier is used.

In this paper the Bipolar Random Network is described, which constitutes an extension of the Random Neural Network model and exhibits autoassociative memory capabilities. This model is characterized by the existence of positive and negative nodes and symmetrical behavior of positive and negative signals circulating in the network. The network's ability of acting as autoassociative memory is examined and several techniques are developed concerning storage and reconstruction of patterns. These approaches are either based on properties of the network or constitute adaptations of existing neural network techniques. The performance of the network under the proposed schemes has been investigated through experiments showing very good storage and reconstruction capabilities. Moreover, the scheme exhibiting the best behavior seems to outperform other well-known associative neural network models, achieving capacities that exceed 0.5n where n is the size of the network.

The paper describes a parallel implementation of a vision system based on associative memories. The proposed real-time image-recognition system is based on the associative 'noise-like coding' model and is implemented on transputer-based tree structures. A high-performance device, the 'complex node' (CN), is introduced. The CN integrates two transputers by a dual-port memory and supports a total of eight links. Tree structures increase their throughput performance when CNs are included. A CN-including tree architecture is compared with a standard transputer-based tree structure having the same computational power. A comparative performance analysis shows the improvement in efficiency obtained when the novel device is used. In addition, theoretical derivations lead to a formula for the system's efficiency, and one demonstrates that expected values fit with measured ones, thus confirming the validity of the overall approach.