Effects of response format on psychometric properties and fairness of a matrices test: multiple choice versus free response

Published on Feb 20, 2020in Frontiers in Education
· DOI :10.3389/FEDUC.2020.00015
Sonja Breuer (University of Salzburg), Thomas Scherndl8
Estimated H-index: 8
(University of Salzburg),
Tuulia M. Ortner13
Estimated H-index: 13
(University of Salzburg)
Reasoning is regarded to be an essential facet of fundamental cognitive abilities. As examinee characteristics may affect performance in Reasoning tests, concern about maintaining fairness is expressed. The purpose of the current study was to examine effects of response format on psychometric properties and fairness of a matrices test according to examinee´s sex, risk propensity, and test anxiety. A total of 433 German-speaking pupils (aged 14 to 20) were randomly assigned to either a multiple choice or a free response version of the same 25-item test. Data analysis yielded Rasch-homogeneous 23-item versions, with higher reliability, but lower criterion validity for the free response test. No interactions between response format and sex, test anxiety, or risk propensity were revealed, but a significant main effect of sex: men out-performed women in reasoning irrespective of response format. Results are discussed with reference to attributes of the test situation and sample characteristics.
📖 Papers frequently viewed together
#1Pey-Yan Liou (NCU: National Central University)H-Index: 12
#2Okan Bulut (U of A: University of Alberta)H-Index: 12
The purpose of this study was to examine eighth-grade students’ science performance in terms of two test design components, item format, and cognitive domain. The portion of Taiwanese data came from the 2011 administration of the Trends in International Mathematics and Science Study (TIMSS), one of the major international large-scale assessments in science. The item difficulty analysis was initially applied to show the proportion of correct items. A regression-based cumulative link mixed modelin...
4 CitationsSource
#1Stuart Woodcock (Griffith University)H-Index: 15
#2Steven J Howard (UOW: University of Wollongong)H-Index: 15
Last. John F Ehrich (Macquarie University)H-Index: 11
view all 3 authors...
Standardized testing is ubiquitous in educational assessment, but questions have been raised about the extent to which these test scores accurately reflect students' genuine knowledge and skills. To more rigorously investigate this issue, the current study employed a within-subject experimental design to examine item format effects on primary school students' standardized assessment results in literacy, reading comprehension, and numeracy. Eighty-nine Grade 3 students (ages 8-9 years) completed ...
1 CitationsSource
Existing measures of test anxiety used with the college student population are old with old norms and old items, and they do not capture the multiple dimensions of the test anxiety construct or ass...
4 CitationsSource
#1Patrick Kyllonen (Princeton University)H-Index: 11
#2Harrison J. Kell (Princeton University)H-Index: 15
Although personality and cognitive ability are separate (sets of) constructs, we argue and demonstrate in this article that their effects are difficult to tease apart, because personality affects the performance on cognitive tests and cognitive ability affects the item responses on personality assessments. Cognitive ability is typically measured with tests of items with correct answers; personality is typically measured with rating-scale self-reports. Oftentimes conclusions regarding the persona...
4 CitationsSource
#1Johannes Schult (Saarland University)H-Index: 7
#2Jörn R. Sparfeldt (Saarland University)H-Index: 14
This research was prepared with the support of the German funds “Bund-Lander-Programm fur bessere Studienbedingungen und mehr Qualitat in der Lehre (‘Qualitatspakt Lehre’)” [the joint program of the Federal and States Government for better study conditions and the quality of teaching in higher education (“the Teaching Quality Pact”)] at Saarland University (funding code: 01PL11012). The authors developed the topic and the content of this manuscript independently from this funding. We thank the I...
2 CitationsSource
Zusammenfassung. Die Fahigkeit einer Lehrkraft, Schulerleistungen und Aufgabenanforderungen akkurat einzuschatzen, ist essenziell, um adaquate padagogische Entscheidungen zu treffen. Bislang ist un...
4 CitationsSource
Anxiety that students experience during test taking negatively influences their academic achievement. Understanding how students perceive tests and how they feel during test taking could help in taking effective preventive measures. Hence, the current study focused on assessing children’s perceptions of tests using content analysis. The sample consisted of 1143 participants (566 females and 570 males) attending 3rd (n = 320), 4th (n = 420), 5th (n = 197), and 6th (n = 206) grade classes in three...
1 CitationsSource
#1Tova Stenlund (Umeå University)H-Index: 8
#2Per-Erik Lyrén (Umeå University)H-Index: 8
Last. Hanna Eklöf (Umeå University)H-Index: 10
view all 3 authors...
To be successful in a high-stakes testing situation is desirable for any test taker. It has been found that, beside content knowledge, test-taking behavior, such as risk-taking strategies, motivation, and test anxiety, is important for test performance. The purposes of the present study were to identify and group test takers with similar patterns of test-taking behavior and to explore how these groups differ in terms of background characteristics and test performance in a high-stakes achievement...
12 CitationsSource
#1Maya A. MingoH-Index: 3
#2Hsin-Hui ChangH-Index: 1
Last. Robert L. WilliamsH-Index: 28
view all 3 authors...
Students (N = 161) in seven sections of an undergraduate educational psychology course rated ten performance-assessment options in collegiate courses. They rated in-class essay exams as their most preferred assessment and multiple-choice exams (in-class and out-of-class) as their least preferred. Also, student ratings of multiple papers and a term paper did not differ significantly from the rating for in-class essay exams. Overall, students preferred constructed forms of assessment over more obj...
2 CitationsSource
#1Sean F. Reardon (Stanford University)H-Index: 49
#2Demetra Kalogrides (Stanford University)H-Index: 17
Last. Rosalía C. Zárate (Stanford University)H-Index: 2
view all 5 authors...
Prior research suggests that males outperform females, on average, on multiple-choice items compared to their relative performance on constructed-response items. This paper characterizes the extent to which gender achievement gaps on state accountability tests across the United States are associated with those tests’ item formats. Using roughly 8 million fourth- and eighth-grade students’ scores on state assessments, we estimate state- and district-level math and reading male-female achievement ...
57 CitationsSource
Cited By0