Data Quality Management

EII Task force

As the scale and diversity of data grows in the digital arena, the complexities of data driven engineering grow multifold with it. The last several years have brought forth several new technologies to service this need such as grid systems and web services. However, a fundamental underpinning of the success of these technologies resides in the quality of data that they can provide. Often the failure of a technology is attributed to its functionality when the real problem lies in the quality of data it uses and subsequently produces. Estimates from industry (Gartner, 2007) indicated a financial loss due to poor data quality in access of $611 billion in US alone, whereas, the investment in data quality tools and products, remains under $600 million (Forrester Research, 2008), indicating a clear and evident need to create a better understanding of the issues, challenges and solutions for data quality management.

July 2008 – July 2009

Shazia Sadiq, Ke Deng

School of Information Technology and electrical Engineering, The University of Queensland

{shazia, dengke}@itee.uq.edu.au

Prof. Xiaofang Zhou

School of Information Technology and electrical Engineering, The University of Queensland

zxf@itee.uq.edu.au

Prof. Graeme Shanks

Department of Information Systems

University of Melbourne

gshanks@unimelb.edu.au

Prof. Xuemin Lin

School of Computer science and Engineering

The University of New South Wales

lxue@cse.unsw.edu.au

Dr. Marta Indulska

UQ Business School, The University of Queensland

m.indulska@business.uq.edu.au

Dr. Rosanne Price

Faculty of Information Technology

Monash University

rosanne.price@infotech.monash.edu.au

Dr. Gabriel Fung

School of Information Technology and electrical Engineering, The University of Queensland

pcfung@itee.uq.edu.au 2

The task force will put a concerted effort towards outreach with industry (see [11] below), including both the user organizations as well as data quality vendors. The task force members have already approached and identified some participants mentioned below, but this list is expected to increase.

Vendors: Informatica, DatFlux, Trillium Software, Veda Advantage

Users: Shell Australia, MBF, Medicare, Telstra

Secondly, the taskforce will engage with international experts in research and academia to develop deep understanding of issues and challenges, as well as to validate key findings solicited from outreach activities. This engagement will take place through a comprehensive survey of the existing literature as well as a series of interviews on phone or face-to-face where possible. Experts identified so far include the following, but this list is also expected to increase.

Experts: Divesh Srivastava (AT&T Labs); Wenfei Fan (University of Edinburgh); Richard Wang (Information Quality Group, MIT); Thomas Redman (Navesink Consulting); Members of the International Association for Information and Data Quality (iaidq.org)

The area of data quality is being studied from three different perspectives: Organizational: Development of data quality objectives for the organization and strategies to establish the people, processes, policies, and standards required to manage and ensure the data quality objectives are met; Architectural: The technology landscape required to deploy developed data quality management processes, standards and policies; and Computational: Effective and efficient tools & techniques required to meet data quality objectives

However, we believe that a holistic data quality management solution must bring together the three perspectives in a complementary rather than competitive manner. The EII community is characterized by diverse strengths and interests. Through the efforts of this task force we hope to bring together the three perspectives, and thereby will create synergies between the different communities, as well as create a better understanding and appreciation of holistic approaches.

Further, bringing together diverse teams to work towards a common complex problem will set an example beyond the scope and lifetime of this task force and demonstrate leadership of the network in this regard.

The objectives of the task force are summarized as below:

• Conduct a thorough review of data quality management research literature

• Engage with industry and create an understanding of real life data quality problems and approaches to data quality management through a compilation of case studies

• Survey the data quality product market, and where possible contact vendors to gain first hand knowledge of product portfolios and capabilities

• Prepare a comprehensive report on the topic based on the above findings

• Develop teaching materials from the report

• Organize a workshop in conjunction with DASFAA (April, 2009) to attract further interest in the field and provide visibility to the topic as well as the work accomplished above

SPECIFIC MILESTONES, EVENTS or ACTIVITIES