Multimodal Interface for Human-Machine Communication

The following sections are included:

• List of Contributors

• Introduction to Multimodal Interface for Human–Machine Communication

• Contents

The following sections are included:

• Introduction

• Face Location in a Cluttered Background
  • Eye Hypothesis Extraction
    • Segmentation by Connected Component Analysis
    • Segmentation by Image Derivative Analysis
    • Validation by Eye Template Matching
    • Validation by Contour Correlation Technique
    • Local Maxima Filtering
  • Face Hypothesis Extraction
    • Filtering Using Geometrical Constraints
    • Filtering Using Image Measures
  • Eyeball and Eye Region Similarity
  • Face Symmetry
    • Filtering Using Image Models
  • Selecting the Best Face
  • Face Tracking

• Robust Face Recognition
  • Tangent Distance
  • A priori Knowledge Encoding of Rotation, Translation and Scaling
  • Face Prototyping for Template-Based Matching
    • Photometric Transforms
      • Mask1: Logarithmic function on grey-levels
      • Mask2: Exponential function on grey-levels
      • Mask3: Generation of Vertical and Horizontal shadow masks
    • Geometric Transform
  • Optical Flow Estimation
  • Multiresolution Optical Flow Estimation
    • Prototypes Synthesis
    • Face Template Similarity Based on Tangent Distance
  • Face Representation Using Gabor Wavelets
    • Feature point selection
    • Gabor Filter Responses
    • Feature-Based Face Similarity Distance Based on Tangent Distance
  • Template-Based and Feature-Based Face Recognition

• Results and Performances

• Conclusion

• References

Most automatic expression analysis systems attempt to recognize a small set of prototypic expressions, such as happiness, anger, surprise, and fear. Such prototypic expressions, however, occur rather infrequently. Human emotions and intentions are more often communicated by changes in one or a few discrete facial features. In this paper, we develop an Automatic Face Analysis (AFA) system to analyze facial expressions based on both permanent facial features (brows, eyes, mouth) and transient facial features (deepening of facial furrows) in a nearly frontal-view face image sequence. The AFA system recognizes fine-grained changes in facial expression into action units (AUs) of the Facial Action Coding System (FACS), instead of a few prototypic expressions. Multi-state face and facial component models are proposed for tracking and modeling the various facial features, including lips, eyes, brows, cheeks, and furrows. During tracking, detailed parametric descriptions of the facial features are extracted. With these parameters as the inputs, a group of action units (neutral expression, 6 upper face AUs, and 10 lower face AUs) are recognized whether they occur alone or in combinations. The system has achieved average recognition rates of 96.4% (95.4% if neutral expressions are excluded) for upper face AUs and 96.7% (95.6% with neutral expressions excluded) for lower face AUs. The generalizability of the system has been tested by using independent image databases collected and FACS-coded for ground-truth by different research teams.

This paper addresses the issue on generating a 2D view of a 3D object from its other 2D views. Linear Combination method is the typical approach to this problem. However, a 2D view cannot be represented by a linear combination of other 2D views under perspective projection. Instead, we have presented a solution under perspective projection. The proposed method also applies to construction of virtual frontal view face image and the results are encouraging.

The following sections are included:

• Introduction

• Previous work

• Chinese Sign Language

• Feature Extraction

• Hidden Markov Models
  • Definition of HMMs
  • The Three Basic Problems For HMMs

• Isolated Recognition
  • CHMMs with Different Numbers of States
  • State Tying
  • The algorithm for isolated sign recognition
  • System Architecture
  • Experiments

• Continuous Sign Language Recognition
  • Language Modeling
  • Search Algorithms
    • Word candidates
    • Still Frame Detecting
  • System Architecture
  • Experiments

• Sign Language Based On Phonemes
  • HMM-based Phone Model
  • Decoding
  • System Architecture
  • Experiments

• Conclusions

• References

The following sections are included:

• Benefits of Pens and Gestures

• Problems with Gestures

• A Prototype Gesture Design Tool
  • Results and Discussion
  • Lessons for gesture design tools

• Gesture Similarity

• quill: an Intelligent Gesture Design Tool
  • Goals
  • Gesture Hierarchy and Naming
  • quill example
  • Active Feedback
    • Recognizer similarity
    • Outlying category
    • Misrecognized training example
    • Outlying gesture
    • Human similarity
    • Duplicate category or group name
  • Paper Prototype

• quill evaluation
  • Participants and Equipment
  • Procedure
  • Quantitative Results and Analysis
  • Qualitative results

• Discussion
  • Summary

• Future Work and Conclusions

• References

In a dictionary-based query translation for cross-language text retrieval, transfer ambiguity is one of main causes of performance deterioration, but this problem has not received significant attention in this field. To resolve transfer ambiguity, this paper proposes a two-phase query translation based on term re-weighting, which uses a bilingual transfer dictionary, originally designed for machine translation. In general, source language query terms each show some word association with others, so that their correct translations should be more likely to co-occur in target documents. Based on this simple intuition, the first phase discriminates more relevant target documents from the others. Using statistical and ranking information from the highly relevant documents, the second phase then converts a translated query vector into re-weighted form to add an extra weight on probably correct target terms. In experiments, results were remarkable: the proposed method achieved almost the same performance as the monolingual IR system, actually contributing to an improvement of precision by about 9% over a baseline system.

The following sections are included:

• Introduction

• Overview of DCT and JPEG

• The Spatial Relationship between a Block and its Subblocks in DCT Domain
  • Problem Description
  • Derivation
  • Illustrative Example
  • Remarks

• Direct Information Extraction
  • Color Space Conversion between YC_bC_r and RGB in DCT domain
  • Pseudo Pixel Extraction
  • Performance Assessment
  • Remarks and Discussions

• Application in the Online Web Image Retrieval
  • WEB-based On-line Image Retrieval System
  • Experiments and analysis
  • Remarks and discussions

• Conclusions

• Acknowledgements

• References

The following sections are included:

• Introduction

• The QuickSet Multimodal System
  • Interface and Application
  • Gesture and Speech Signal Recognition
  • Gesture and Speech Natural Language Processing & Fusion
  • QuickSet's Multi-agent Framework
  • QuickSet's Evaluation and Integration

• Recent Empirical Studies
  • Experiment 1
  • Experiment 2

• Conclusion

• References

The following sections are included:

• Introduction
  • Related Work

• Informative Subspaces
  • Information Theoretic Fusion
  • Capacity Control
  • The Implicit Statistical Model

• Empirical Results
  • Video Localization of Speaker
  • Quantifying Consistency Between the Audio and Video
  • Eight-way Test

• Discussion and Future Work

• References

The following sections are included:

• Introduction
  • Face and body construction
  • Avatar functions
  • Motion control
  • High-level behavior
    • Speech Recognition
  • Interaction with objects
  • Intercommunication
  • Interaction with user
  • Augmented reality
    • THE VIRTUAL HUMAN DIRECTOR SYSTEM
  • Conclusion

• References