International Journal of Computer Processing of Languages

The work presented in this paper is motivated by the practical need for content extraction, and the available data source and evaluation benchmark from the ACE program. The Chinese entity detection and tracking task is of particular interest to us. A novel solution is proposed to alleviate the language-independent and language-dependent problems special in this task. Mention detection takes advantages of machine learning approaches and character-based models. It manipulates different types of entities being mentioned and different constitution units (i.e., extents and heads) separately. Mentions referring to the same entity are linked together by integrating most-specific-first and closest-first rule based pairwise clustering algorithms. Types of mentions and entities are determined by head-driven classification approaches. The implemented system achieves 66.1 of ACE value, which has been one of the top-tier results.

As Arabic is known for its highly inflectional morphological structure, this hybrid approach is utilizing morphological knowledge in form of consistent root-pattern relationships, and some morpho-syntactical knowledge based on affixation and morphographemic rules to specify the word recognition and non-word correction process. Furthermore this paper is proposing novel probabilistic measures for completing the task of the correction by locating, reducing and ranking of the most probable correction candidates in Arabic derivative words. In this context based on frequency of occurrence analysis, two probabilistic measures are introduced, Root-Pattern Predictive Value, RPV, and Pattern-Root Predictive Value, PPV. Moreover, keyboard effect, letter sound and similarity are considered in addition to some lexical features as a supplementary aid to improve the process of error detection and correction.

The documentary patrimony deposited in the libraries and archives represents an essential part in the collective memory; it reflects the diversity of languages, peoples and cultures. But such memory is fragile and a part of such patrimony regularly disappears by accident, war or ageing.

There exist about three million manuscripts of an Arab or Islamic origin, scattered throughout the world. They are not well preserved, that is why researchers encounter difficulty in consulting them. The digitization of such documents has become an absolute necessity for saving and preserving them for future generations.

This paper presents a contribution in the domain of digitization and processing of images of historical documents stored in the Arabic libraries. After being scanned, the raw images obtained undergo a primary pre-processing treatment in order to clean them and restore them; then they go through a second phase of image analysis which consists in segmenting the documents so as to separate out texts and graphics, and in compressing the images so as to reduce their stocking cost and their communication time. All such operations are necessary for the retro conversion of the images of historical documents for the purpose of indexing and hosting them on the internet.

This article reports about an improved modified joint source-channel model that is used to transliterate the Bengali Named Entity (NE) into English and vice versa. A number of alternatives to the improved modified joint source-channel model have also been considered. All the models of the transliteration system have been developed without and with using linguistic knowledge. The Bengali NE is divided into Transliteration Units (TU) with patterns C⁺M, where C represents a consonant or a vowel or a conjunct and M represents the vowel modifier or matra. An English NE is divided into TUs with patterns C*V*, where C represents a consonant and V represents a vowel. The system learns mappings automatically from the bilingual training set of 25,000 named entities. Aligned transliteration units along with their contexts are automatically derived from this bilingual training set to generate the collocational statistics. This alignment is automatic and implemented in two different ways: without being guided by linguistic knowledge and being guided by linguistic knowledge. The system considers the linguistic knowledge in the form of possible conjuncts and diphthongs in Bengali and their corresponding representations in English. Experimental results of the 5-fold cross validation test has demonstrated that the improved modified joint source-channel model with linguistic knowledge performs the best during Bengali to English transliteration with a Word Agreement Ratio (WAR) of 81.4% and a Transliteration Unit Agreement Ratio (TUAR) of 95.7%. The same model has also performed the best for the 5-fold cross validation test during English to Bengali transliteration with a WAR of 79.5% and a TUAR of 93.8%. Finally, the transliteration models that do not use linguistic knowledge have been evaluated for Hindi to English (H2E) and Telugu to English (T2E) transliterations with the relatively smaller datasets. This shows that the proposed transliteration algorithm works effectively for any pair of languages that share comparable orthography.