Advances in Image Processing and Understanding

The following sections are included:

• Contents

It is well recognized that hand programming for vision, audition, and autonomous robots is extremely challenging, especially in partially unknown or complex environments. Although machine learning techniques have been widely used in these fields, fundamental limits remain. Here, we discuss some fundamental problems of the current task-specific paradigm to building complex systems and contrast it with some recent studies in neuroscience that indicates the power of developmental mechanisms in animals that enable autonomous development of cognitive and behavioral capabilities in humans and animals. What do we mean by development? Does it lead us to a more tractable and more systematic approach to vision, audition, robotics and beyond? Motivated by human mental development from infancy to adulthood, the work presented here aims to enable robots to develop their mental skills automatically, through online, real time interactions with its environment. The SAIL developmental robot that has been built at MSU is an early prototype of such a new kind of robots. Our experiments indicated that it appears feasible to develop vision, audition and other cognitive capabilities as well as cognition based behaviors through online interactions by an autonomous robot.

Capturing real facial motions from videos enables automatic creation of dynamic models for facial animation. In this paper, we propose an explanation-based facial motion tracking algorithm based on a piecewise Bézier volume deformation model (PBVD). The PBVD is a suitable model both for synthesis and analysis of facial images. With this model, basic facial movements, or action units, are first interactively defined. Then, by linearly combining these action units, various facial movements are synthesized. The magnitudes of these action units can be estimated from real videos using a model-based tracking algorithm. The predefined PBVD action units may also be adaptively modified to customize the dynamic model for a particular face. In this paper, we first briefly introduce the PBVD model and its application in computer facial animation. Then a coarse-to-fine PBVD-based motion tracking algorithm is presented. We also describe an explanation-based tracking algorithm that takes a collection of predefined action units as the initial dynamic model and adaptively improves this model during the tracking process. Experimental results on PBVD-based animation, model-based tracking, and explanation-based tracking are demonstrated.

The following sections are included:

• Introduction

• Motivation and Previous Work

• GOES Cloud Image Sequences

• System Outline

• Local Nonrigid Motion Model

• Local Motion Model Fitting

• Global Constraints

• Experimental Results

• Validations

• Conclusions and Future Work

• Acknowledgments

• References

The following sections are included:

• Introduction

• System Framework

• Pixel-Level Processing

• Graph Representations of Maps

• Map Graph Segmentation and Re-linking

• Rolling Ball: Road Structure Vectorization

• House Data Generation

• Discussion

• Reference

The following sections are included:

• Introduction

• Learning Visual Concepts

• Discussion

• Summary

• References

The following sections are included:

• Introduction to Human Facial Feature Extraction

• Overview of Automated Facial Feature Extraction Algorithm

• Multi-resolutional Face Templates

• Face Detection by Coarse-to-fine Multi-resolution Searching

• Feature Extraction by Global-to-local Matching

• Feedback Process

• Experiments

• Results and Conclusions

• References

The following sections are included:

• Introduction

• Concepts and Notations

• Related Work

• The Proposed Approach

• Experiments, Results and Evaluations

• Discussions and Conclusions

• Acknowledgment

• References

This chapter presents a subband/wavelet video coder using an object-based spatiotemporal segmentation. The moving objects in a video are extracted by means of a joint motion estimation and segmentation algorithm based on a compound Markov random field (MRF) model. The two important features of our technique are the temporal linking of the objects, and the guidance of the motion segmentation with spatial color information. This results in spatiotemporal (3-D) objects that are stable in time, and leads to a new motion-compensated temporal updating and contour coding scheme that greatly reduces the bit-rate to transmit the object boundaries. The object interiors can be encoded by either 2-D or 3-D subband/wavelet coding. Simulations at very low bit-rates yield comparable performance in terms of reconstructed PSNR to the H.263 coder. The object-based coder produces visually more pleasing video with less blurriness and is devoid of block artifacts.

The following sections are included:

• Introduction

• Methodology

• Experimental Results

• Summary and Discussion

• References

The following sections are included:

• Introduction

• TRIRF-frame Coding

• Video Stream Packetization

• Coding Performance Comparisons

• Efficiency Analysis of TRIRF Coding

• Conclusions

• Bibliography

The following sections are included:

• Preliminary Comments

• Fundamentals of 2D AM-FM Modeling

• Isolating the Multiple Image Components

• AM-FM Demodulation

• AM-FM Image Segmentation

• AM-FM Reaction-Diffusion for Texture Completion

• Multidimensional Orthogonal FM Transforms

• Concluding Remarks

• Acknowledgement

• References

In this chapter, we describe several fixed length coding schemes based on trellis coded quantization (TCQ) that have been developed for image transmission over noisy channels such as wireless links. In general, wireless channels are time-varying, bandwidth constrained, and error pruned. Therefore, it is desired that the images are not only appropriately coded to achieve bandwidth compression, but also coded in such a way that the coded bitstream is resilient to channel errors and robust to time-varying channel distortion and fading. The proposed schemes consist of the basic algorithm of uniform threshold TCQ (UTTCQ) and two enhanced schemes with increased computational complexity. The enhanced schemes include nonuniform threshold TCQ and UTTCQ with block classification. We demonstrate that, in the absence of channel coding, the proposed fixed length coding schemes can be designed to achieve efficient compression, error resilience, and robustness. In particular, the fixed length coding schemes will not suffer from the loss of synchronization that often causes catastrophic error effects when more efficient variable length source coding is adopted. We also present a scheme of layered transmission with unequal error protection channel coding to further improve the performance of image transmission over noisy channels.

We present a review of algorithms and their performance for determining three-dimensional (3D) motion and structure of rigid objects when their corresponding features are known at different times or are viewed by different cameras. Three categories of problems are considered, depending upon whether the features are two- (2D) or three-dimensional (3D) and the type of correspondence: a) 3D to 3D (i.e., locations of corresponding features in 3D space are known at two different times), b) 2D to 3D (i.e., locations of features in 3D space and their projection on the camera plane are known), and c) 2D to 2D (i.e., projections of features on the camera plane are known at two different times). Features considered include points, straight lines, curved lines, and corners. Emphasis is on problem formulation, efficient algorithms for solution, existence and uniqueness of solutions, and sensitivity of solutions to noise in the observed data. Algorithms described have been used in a variety of applications. Some of these are: a) positioning and navigating 3D objects in a 3D world, b) camera calibration, i.e., determining location and orientation of a camera by observing 3D features whose location is known, c) estimating motion and structure of moving objects relative to a camera. We mention some of the mathematical techniques borrowed from algebraic geometry, projective geometry, and homotopy theory that are required to solve these problems, list unsolved problems, and give some directions for future research.

Recent advances in various signal-processing technologies, coupled with an explosion in the available computing power, have given rise to a number of novel human–computer interaction (HCI) modalities—speech, vision-based gesture recognition, eye tracking, electroencephalograph, etc. Successful embodiment of these modalities into an interface has the potential of easing the HCI bottleneck that has become noticeable with the advances in computing and communication. It has also become increasingly evident that the difficulties encountered in the analysis and interpretation of individual sensing modalities may be overcome by integrating them into a multimodal human–computer interface.

In this paper, we examine several promising directions toward achieving multimodal HCI. We consider some of the emerging novel input modalities for HCI and the fundamental issues in integrating them at various levels—from early "signal" level to intermediate "feature" level to late "decision" level. We discuss the different computational approaches that may be applied at the different levels of modality integration. We also briefly review several demonstrated multimodal HCI systems and applications. Despite all the recent developments, it is clear that further research is needed for interpreting and fusing multiple sensing modalities in the context of HCI. This research can benefit from many disparate fields of study that increase our understanding of the different human communication modalities and their potential role in HCI.

Image processing techniques find applications in many areas, chief among which are image enhancement, pattern recognition, and efficient picture coding. Some aspects of image processing are discussed—specifically: the mathematical operations one is likely to encounter, and ways of implementing them by optics and on digital computers; image description; and image quality evaluation. Many old results are reviewed, some new ones presented, and several open questions are posed.