International Journal of Computer Processing of Languages

Analyzing speech act is important for understanding the intention of a speaker and the flow of dialogue in discourse analysis. This study analyzes speech act by using syntactic pattern information which is tagged in dialogue corpus. We apply bigram Hidden Markov Model(HMM) to the task of computing speech act. We compute speech act probabilities for each utterance with forward algorithm. When computing the speech act probabilities to find the best path in HMM, there arises a sparse data problem. We smoothed probabilities through class probabilities of decision trees.

The run-time context domain has much effect on the performance of practical corpus-based applications. Previous smoothing techniques, and class-based and similarity-based models cannot handle the dynamic status perfectly. In this paper, an adaptive learning algorithm is proposed for task adaptation to fit best the run-time context domain in the application of Chinese homophone disambiguation. It shows which objects are to be adjusted and how to adjust their probabilities by a neural network model. The resulting techniques are greatly simplified and robust. The experimental results demonstrate the effects of the learning algorithm from generic domain to specific domain. A methodology is also presented to show how these techniques can be extended to various language models and corpus-based applications.

A method to automatically extract translational Japanese-KATAKANA and English word pairs from bilingual corpora is proposed. The method applies all the existing back-transliteration rules to each mora unit in a KATAKANA word, and extracts the English word which matches or partially matches one of these back-transliteration candidates as translation. The mora unit is a Japanese syllable unit and one Katakana character often corresponds to one mora. For instance, if we have in the Japanese part of a bilingual corpora, we generate such back-transliteration candidates as <graf>, <graph>, <gulerph>,… and identify similar words from the English part of the corpora. The method performs reasonably well, achieving 80%–100% precision at 75% recall against the eight corpora we used for evaluation.

In this paper, we describe a proposal to derive formal language queries from natural language queries with a combinatory categorial grammar (CCG). CCGs are well known to provide a means of deriving all the levels of information for natural language, i.e., syntax, semantics and discourse, at the same time. In our proposal, we utilize an extra level of representation for formal language queries for the aforementioned derivation. The syntactic coverage is shown with various natural language queries, including compound nouns, modification markers, various types of ellipses, numerical expressions, and subordinate and coordinate constructions. The general purpose CCG lexicon is semi-automatically augmented with the database fields and entries. We also discuss the performance of an implemented natural language query processing system.

This paper presents a technique for dividing the text into field-coherent passages. The presented method is based upon extracting field-associated terms from the text measuring how the topics grow, shrink and shift from sentence to sentence. We propose measures of topic continuity and of topic transition and suggest how those could be used to find the boundaries among passages. After collecting 12,500 documents, we obtain 88% for average precision and 78% for recall in training set.

This paper introduces a mid-depth text understanding methodology. By mid-depth we mean that the methodology is: (1) based on shallow text understanding, (2) accomplished with currently available linguistic instruments, and (3) focused towards understanding texts as deeply as possible. To realize the methodology, we set up two new linguistic devices for sentence extraction — sentence abstraction and abductive chains. Sentence abstraction is processing for sentence revision with concept abstraction. Abductive chains are lexical chains with abduction. The chain consists of abductive links which connect coherent parts of abstracted sentences. It aims to identify the causal propensity described or implied in the text as well as to delineate its recurring concepts. With this methodology, a preliminary attempt at mid-depth text understanding is made to locate the topic sentences of a given text. We expect that this methodology will lead us to develop another practical text summarization system.