International Journal of Pattern Recognition and Artificial Intelligence

The task discussed in this paper is that of learning to map input sequences to output sequences. In particular, problems of phoneme recognition in continuous speech are considered, but most of the discussed techniques could be applied to other tasks, such as the recognition of sequences of handwritten characters. The systems considered in this paper are based on connectionist models, or artificial neural networks, sometimes combined with statistical techniques for recognition of sequences of patterns, stressing the integration of prior knowledge and learning. Different architectures for sequence and speech recognition are reviewed, including recurrent networks as well as hybrid systems involving hidden Markov models.

This paper describes the development of an algorithm for verification of signatures written on a touch-sensitive pad. The signature verification algorithm is based on an artificial neural network. The novel network presented here, called a “Siamese” time delay neural network, consists of two identical networks joined at their output. During training the network learns to measure the similarity between pairs of signatures. When used for verification, only one half of the Siamese network is evaluated. The output of this half network is the feature vector for the input signature. Verification consists of comparing this feature vector with a stored feature vector for the signer. Signatures closer than a chosen threshold to this stored representation are accepted, all other signatures are rejected as forgeries. System performance is illustrated with experiments performed in the laboratory.

We describe a method, “Shortest Path Segmentation” (SPS), which combines dynamic programming and a neural net recognizer for segmenting and recognizing character strings. We describe the application of this method to two problems: recognition of handwritten ZIP Codes, and recognition of handwritten words. For the ZIP Codes, we also used the method to automatically segment the images during training: the dynamic programming stage both performs the segmentation and provides inputs and desired outputs to the neural network. Results are reported for a test set of 2642 unsegmented handwritten 212 dpi binary ZIP Code (5- and 9-digit) images.

For handwritten word recognition, we combined SPS with a “Space Displacement Neural Network” approach, in which a single-character-recognition network is extended over the entire word image, and in which SPS techniques are then used to rank order a given lexicon. We report results on a test set of 3000 300 ppi gray scale word images, extracted from images of live mail pieces, for lexicons of size 10, 100, and 1000.

Representing the problem as a graph as proposed in this paper has advantages beyond the efficient finding of the final optimal segmentation, or the automatic segmentation of images during training. We can also easily extend the technique to generate K “runner up” answers (for example, by finding the K shortest paths). This paper will also describe applications of some of these ideas.

A boosting algorithm, based on the probably approximately correct (PAC) learning model is used to construct an ensemble of neural networks that significantly improves performance (compared to a single network) in optical character recognition (OCR) problems. The effect of boosting is reported on four handwritten image databases consisting of 12000 digits from segmented ZIP Codes from the United States Postal Service and the following from the National Institute of Standards and Technology: 220000 digits, 45000 upper case letters, and 45000 lower case letters. We use two performance measures: the raw error rate (no rejects) and the reject rate required to achieve a 1% error rate on the patterns not rejected. Boosting improved performance significantly, and, in some cases, dramatically.

In practical applications, recognition accuracy is sometimes not the only criterion; capability to reject erroneous patterns might also be needed. We show that there is a trade-off between these two properties. An efficient solution to this trade-off is brought about by the use of different algorithms implemented in various modules, i.e. multi-modular architectures.

We present a general mechanism for designing and training multi-modular architectures, integrating various neural networks into a unique pattern recognition system, which is globally trained. It is possible to realize, within the system, feature extraction and recognition in successive modules which are cooperatively trained. We discuss various rejection criteria for neural networks and multi-modular architectures.

We then give two examples of such systems, study their rejection capabilities and show how to use them for segmentation. In handwritten optical character recognition, our system achieves performances at state-of-the-art level, but is eight times faster. In human face recognition, our system is intended to work in the real world.

A multi-staged system for off-line handwritten numeral recognition is presented here. After scanning, the digitized binary bitmap image of the source document is passed through a preprocessing stage which performs segmentation, thinning and rethickening, normalization, and slant correction. The recognizer is a three-layered neural net trained with back-propagation algorithm. While a few systems that use three-layered nets for recognition have been presented in the literature, the contribution of our system is based on two aspects: elaborate preprocessing based on structural pattern recognition methods combined with a neural net based recognizer; and integration of neural net based and structural pattern recognition methods to produce high accuracies.

We make a comparision of classification ability between BPN (BackPropagation Neural Network) and k-NN (k-Nearest Neighbor) classification methods. Voice data and patellar subluxation images are used. The result was that the average recognition rate of BPN was 9.2 percent higher than that of the k-NN classification method. Although k-NN classification is simple in theory, classification time was fairly long. Therefore, it seems that real time recognition is difficult. On the other hand, the BPN method has a long learning time but a very short recognition time. Especially if the number of dimensions of the samples is large, it can be said that BPN is better than k-NN in classification ability.

Recently, much interest has been generated regarding speech recognition systems based on Hidden Markov Models (HMMs) and neural network (NN) hybrids. Such systems attempt to combine the best features of both models: the temporal structure of HMMs and the discriminative power of neural networks. In this work we establish one more relation between the HMM and the NN paradigms by introducing the time-warping network (TWN) that is a generalization of both an HMM-based recognizer and a backpropagation net. The basic element of such a network, a time- warping neuron, extends the operation of the formal neuron of a backpropagation network by warping the input pattern to match it optimally to its weights. We show that a single-layer network of TW neurons is equivalent to a Gaussian density HMM-based recognition system. This equivalent neural representation suggests ways to improve the discriminative power of this system by using backpropagation discriminative training, and/or by generalizing the structure of the recognizer to a multi-layer net. The performance of the proposed network was evaluated on a highly confusable, isolated word, multi-speaker recognition task. The results indicate that not only does the recognition performance improve, but the separation between classes is enhanced, allowing us to set up a rejection criterion to improve the confidence of the system.

Optical flow computation in dynamic image processing can be formulated as a minimization problem by a variational approach. Because solving the problem is computationally intensive, we reformulate the problem suitable for neural computing. In this paper, we propose a recurrent neural network model which may be implemented in hardware with many processing elements (neurons) operating asynchronously in parallel to achieve a possible real-time solution. We derive and prove the properties of the reformulation, as well as analyze the asymptotic stability and convergence rate of the proposed neural network. Experiments using both the test patterns and the real laboratory images are conducted.

A neural network of cascaded Restricted Coulomb Energy (RCE) nets is constructed for the recognition of two-dimensional objects. A number of RCE nets are cascaded together to form a classifier where the overlapping decision regions are progressively resolved by a set of cascaded networks. Similarities among objects which have complex decision boundaries in the feature space are resolved by this multi-net approach. The generalization ability of an RCE net recognition system, referring to the ability of the system to correctly recognize a new pattern even when the number of learning exemplars is small, is increased by the proposed coarse-to-fine learning strategy. A feature extraction technique is used to map the geometrical shape information of an object into an ordered feature vector of fixed length. This feature vector is then used as an input to the neural network. The feature vector is invariant to object changes such as positional shift, rotation, scaling, illumination variance, variation of camera setup, perspective distortion, and noise distortion. Experimental results for recognition of several objects are also presented. A correct recognition rate of 100% was achieved for both the training and the testing input patterns.

This paper advances two approaches to integrating handwritten character segmentation and recognition within one system, where the underlying function is learned by a backpropagation neural network. Integrated segmentation and recognition is necessary when characters overlap or touch, or when an individual character is broken up. The first approach exhaustively scans a field of characters, effectively creating a possible segmentation at each scan point. A neural net is trained to both identify when its input window is centered over a character, and if it is, to classify the character. This approach is similar to most recently advanced approaches to integrating segmentation and recognition, and has the common flaw of generating too many possible segmentations to be truly efficient. The second approach overcomes this weakness without reducing accuracy by training a neural network to mimic the ballistic and corrective saccades (eye movements) of human vision. A single neural net learns to jump from character to character, making corrective jumps when necessary, and to classify the centered character when properly fixated. The significant aspect of this system is that the neural net learns to both control what is in its input window as well as to recognize what is in the window. High accuracy results are reported for a standard database of handprinted digits for both approaches.

There has been much interest in increasing the computational power of neural networks. In addition there has been much interest in “designing” neural networks better suited to particular problems. Increasing the “order” of the connectivity of a neural network permits both. Though order has played a significant role in feedforward neural networks, its role in dynamically driven recurrent networks is still being understood. This work explores the effect of order in learning grammars. We present an experimental comparison of first order and second order recurrent neural networks, as applied to the task of grammatical inference. We show that for the small grammars studied these two neural net architectures have comparable learning and generalization power, and that both are reasonably capable of extracting the correct finite state automata for the language in question. However, for a larger randomly-generated ten-state grammar, second order networks significantly outperformed the first order networks, both in convergence time and generalization capability. We show that these networks learn faster the more neurons they have (our experiments used up to 10 hidden neurons), but that the solutions found by smaller networks are usually of better quality (in terms of generalization performance after training). Second order nets have the advantage that they converge more quickly to a solution and can find it more reliably than first order nets, but that the second order solutions tend to be of poorer quality than those of the first order if both architectures are trained to the same error tolerance. Despite this, second order nets can more successfully extract finite state machines using heuristic clustering techniques applied to the internal state representations. We speculate that this may be due to restrictions on the ability of first order architecture to fully make use of its internal state representation power and that this may have implications for the performance of the two architectures when scaled up to larger problems.

We report on progress in understanding how to build machines which adaptively acquire the language for their task. The generic mechanism in our research has been an information-theoretic connectionist network embedded in a feedback control system. In this paper, we investigate the capability of such a network to learn associations between messages and meaningful responses to them as a task increases in size and complexity. Specifically, we consider how one might reflect task structure in a network architecture in order to provide improved generalization capability in language acquisition. We propose a method for constructing networks from component subnetworks, namely a product network, which provides improved generalization by factoring the associations between words and actions through an intermediate layer of semantic primitives. A two-dimensional product network was evaluated in a 1000-action data retrieval system, the object of which is to answer questions about 20 attributes of the 50 states of the USA. The system was tested by 13 subjects over a two-week period, during which over 1000 natural language dialogues were recorded. The experiment was conducted using typed input with unconstrained vocabulary and syntax. During the course of performing its task, the system acquired over 500 words and retained 92% of what it learned. We provide a description of the system and details on the experimental results.

MultiLayer Perceptrons (MLP) are an effective family of algorithms for the smooth estimation of highly-dimensioned probability density functions that are useful in continuous speech recognition. Hidden Markov Models (HMM) provide a structure for the mapping of a temporal sequence of acoustic vectors to a generating sequence of states. For HMMs that are independent of phonetic context, the MLP approaches have consistently provided significant improvements (once we learned how to use them). Recently, these results have been extended to context-dependent models.

In this paper, after having reviewed the basic principles of our hybrid HMM/MLP approach, we describe a series of experiments with continuous speech recognition. The hybrid methods directly trade off computational complexity for reduced requirements of memory and memory bandwidth. Results are presented on the widely used Resource Management speech database that is distributed by the National Institute of Standards and Technology. These results demonstrate performance that is at least as good as any other reported continuous speech recognition system (for this task).

We have developed an algorithm for unsupervised adaptive classification based on a finite number of “prototype populations” with distinctly different feature distributions, each representing a typically different source population of the inspected products. Intermittently updated feature distributions, of samples collected from the currently classified products, are compared to the distributions of pre-stored prototype populations, and accordingly the system switches to the most appropriate classifier. The goal of our approach is similar to the objectives of the previously proposed “Decision Directed” adaptive classification algorithms but our solution is particularly suitable for automatic inspection and classification on a production line, when the inspected items may come from a finite number of distinctly different sources.

The recognition of prototype populations as well as the classification task proper may be implemented by conventional classifiers, however neural networks (NN) are advantageous in two respects: There is no need to develop separate mathematical models for each classifier because the NN does it automatically during the training stage. The parallel structure of NN has the potential for very fast classification in real time, if implemented by dedicated parallel hardware. This is particularly important for high speed automatic sorting on a production line.

The practical feasibility of the approach was demonstrated by two applied examples, wherein two prototype populations of apples are recognized and sorted by size and color derived by machine vision. Three “Boltzmann-Perceptron Networks” (BPN) were used, one to recognize the prototype populations, while switching between the other two, for optimally classifying apples into two size and color categories. It is shown that misclassifications by adaptive classification are reduced, in comparison to non-adaptive classification.

We present a system for the interpretation of camera images of scenes composed of several known objects with mutual occlusion. The scenes are analyzed by the recognition of the objects present and by the determination of their occlusion relations. Objects are internally represented by stored model graphs. These are formed in a semi-automatic way by showing objects against a varying background. Objects are recognized by dynamic link matching. Our experiments show that our system is very successful in analyzing cluttered scenes. The system architecture goes beyond classical neural networks by making extensive use of flexible links between units, as proposed in the dynamic link architecture. The present implementation is, however, rather algorithmic in style and is to be regarded as a pilot study that is preparing the way for a detailed implementation of the architecture.

Up till recently, state-of-the-art, large vocabulary, continuous speech recognition (CSR) had employed hidden Markov modeling (HMM) to model speech sounds. In an attempt to improve over HMM we developed a hybrid system that integrates HMM technology with neural networks.

We present the concept of a Segmental Neural Net (SNN) for phonetic modeling in CSR. By taking into account all the frames of a phonetic segment simultaneously, the SNN overcomes the well-known conditional-independence limitation of HMMs. We have developed a novel hybrid SNN/HMM system that combines the advantages of SNNs and HMMs using a multiple hypothesis (or N-best) paradigm. In this system, we generate likely phonetic segmentations from the HMM N-best list of word sequences, which are scored by the SNN. The HMM and SNN scores are then combined to optimize performance.

In several speaker-independent, 1000-word CSR tests, the error rate for the hybrid system dropped 20% from that of a state-of-the-art HMM system alone.