Data Quality and High-Dimensional Data Analysis

The following sections are included:

• PREFACE

• ORGANIZING COMMITTEES

• CONTENTS

The following sections are included:

• Introduction

• Genomic Information Processing

• Genomic Information Quality

• Conclusion

• References

XML, the eXtensible Markup Language, is fast evolving into the new standard for data representation and exchange on the WWW. This has resulted in a growing number of data cleaning techniques to locate "dirty" data (artifacts). In this paper, we present DEEPDETECT – an extensible system that detects attribute outliers and duplicates in XML documents. Attribute outlier detection finds objects that contain deviating values with respect to a relevant group of objects. This entails utilizing the correlation among element values in a given XML document. Duplicate detection in XML requires the identification of subtrees that correspond to real world objects. Our system architecture enables sharing of common operations that prepare XML data for the various artifact detection techniques. DEEPDETECT also provides an intuitive visual interface for the user to specify various parameters for preprocessing and detection, as well as to view results.

In this paper, we propose a comprehensive data integration and quality assurance model, called CHARIOT, for agro-meteorological data. This model comprised of two modules: an intermediary module and a data quality control module. The intermediary provides users with reliable and continuous access to heterogeneous weather databases from various sources; it also solves various compatibility issues in meteorological time series data. The data quality control tool consists of a multi-layer system spanning internal, temporal and spatial data checks. These two modules combined together provide users with clean and error-free inputs for weather-driven agricultural management decisions. When applying CHARIOT to weather data for a real-life agricultural application, CHARIOT is shown to be effective in controlling and improving data quality, which in turn leads to better and more accurate agricultural management decisions.

Data quality rating is an important issue to be considered in many scenarios such as data integration, cooperative information system(CIS). Now it is widely accepted that data quality can be measured from multiple dimensions such as accuracy, completeness etc. Most of the work focuses on how to qualitatively analyze the dimensions and the analysis will greatly depend on experts' knowledge. Seldom work is given on how to automatically quantify data quality dimensions. To solve this challenging problem, we propose a novel approach to automatically Quantify Dimensions within Context(QDC). Data quality can be gauged by discrepancy between data view and its entity's perfect representation. Since it is difficult to obtain the perfect representation of entity, we propose to approximate the perfect representation within its available context and quality dimensions can be quantified in this context scope. By naturally borrowing entropy concepts from information theory, the measurement is easily given for different types of data. In this way the two most import quality dimensions, that are accuracy and completeness, are properly quantified. Our QDC approach can not only give an objective score and ranking in a cooperative multi-source environment but also avoid human's laborious interaction. As an automatic quality rating solution our approach is distinguished, especially for large scale datasets. Theory and experiment shows our approach performs well for quality rating.

To face the challenges posed by new techno-savvy market players, the Public Sector Banks (PSB) and the old private banks in India, have introduced Core Banking Solutions (CBS) to replace disparate branch automation systems. CBS provides centralized online banking operational database which can be exploited for building Decision Support System (DSS) in key areas. While promptness of data is ensured, other data quality needs are to be appraised before implementing any such DSS. Hence an assessment of data quality in two key areas – Customer Relationship Management and Borrower Behaviour was carried out for a sample bank for data profiling, inter-field consistency, attribute value dependent constraints, domain constraints. The study has identified critical areas for data quality improvement both for legacy data that has been migrated and new data being captured by the CBS. Measures for data cleaning and implementation of additional constraints at the database or application level are proposed for improvement of data quality for implementing these DSS.

An approach to cadastral map quality evaluation is proposed, which is elaborated and implemented by State Land Service of the Republic of Latvia. The approach is based on opinion of Land Service experts about cadastral map quality that depends on its usage points. Quality parameters of cadastral map objects identified by experts and its limit values are used for evaluation. The assessment matrix is used, which allow to define cadastral map quality that depends on its usage purpose. The matrix is used to find out, of what quality a cadastral map should be in order to be used for the chosen purpose. The given approach is flexible, it gives a possibility to change sets of quality parameters and their limit values as well as to use the approach for other type data quality evaluation.

Last years the study of the influence of genetic factors in the susceptibility to some common diseases has been obtaining satisfactory results. These results contribute to the prevention of these diseases as well as to design personalized treatments. The present work introduces a technique based on 2D representations of the genetic data that can contribute to find these disease associations. A real case application is presented in which we analyze the relation between the allele pair values of 748 Single Nucleotide Polymorphisms (SNPs) and the susceptibility to seven common psychiatrical disorders.

The following sections are included:

• AUTHOR INDEX