Managing the quality of data collection in large-scale assessments (PDF)

The papers in this ebook are based on presentations at a two-day international seminar on managing the quality of data collection in large-scale assessments in May 2017 at the Organisation for Economic Co-operation and Development (OECD) headquarters in Paris. The purpose of this event was to bring together psychometricians and survey methodologists to discuss issues around the identification, treatment, and prevention of errors associated with data collections in large-scale assessments as well as prospects for the evolution of data collection methods. The topic of managing quality in data collection was chosen for two main reasons: First, data collection and field operations represent major sources of potential error in any large-scale survey, particularly those such as PIAAC that administer questionnaires and tests of cognitive skills (literacy, numeracy, and problem solving) using interviewer-based methods. The behavior and skills of interviewers, the setting and conditions in which the interview or assessment takes place, the dispositions and motivation of respondents, and the capability and capacity of survey organizations collecting the data all have effects on data quality. These effects range from data falsification at the different levels of survey operations at one extreme to satisficing by both interviewers and respondents at the other. Second, developments in information and communications technologies offer an opportunity to achieve considerable improvements in data quality through the identification, treatment, and prevention of errors. New technologies are also opening up potential new avenues for data collection. The use of computer-aided personal interviewing and computer-based testing have already led to demonstrable improvements in data quality in large-scale survey assessments and other testing programs. Automatic scoring and automatic range checks for responses reduce the chance for human error, for example. The availability of process data that represents the interactions between interviewers and respondents and the interview/testing application, including timestamps, provides a rich source of information for detecting problems and potentially adjusting for them. An important challenge for those who design and manage international large-scale assessments is to apply what has been learned so far to the design of new tools and systems to facilitate a more preventative and anticipatory approach to quality control and quality assurance in data collection. In the longer term, the availability of new tools and approaches may have profound impacts on how sample surveys and large-scale assessments are conducted. The papers discussed during the event and contained in this ebook represent an important contribution to current thinking about these issues. The design and implementation of the first cycle of PIAAC in 2012 reflected best practice at the time. However, as we move toward the second cycle, and as the field of international large-scale assessments has advanced in significant ways over recent years, possible improvements to the application of TSE principles in future large-scale assessments will need to be addressed.