Differential relations between facets of complex problem solving and students’ immigration background

in Journal of Educational Psychology (2014), Advance online publication

Whereas the assessment of complex problem solving (CPS) has received increasing attention in the context of international large-scale assessments, its fairness in regard to students’ cultural background ... [more ▼]

Whereas the assessment of complex problem solving (CPS) has received increasing attention in the context of international large-scale assessments, its fairness in regard to students’ cultural background has gone largely unexplored. On the basis of a student sample of ninth-graders (N = 299), including a representative number of immigrant students (N = 127), the present study evaluated (1) whether CPS can be assessed fairly among students with or without immigration background and (2) whether achievement differences between these groups exist. Results showed that fair assessment of CPS is possible using the Genetics Lab, a computer-based microworld that incorporates game-like characteristics and multilingual-friendly features. Immigrant students were generally outperformed by their non-immigrant peers, but performance differences can largely be explained by differential enrollment in lower academic tracks. Interestingly, CPS scales were less affected by students’ educational background than a traditional paper-pencil-based reasoning scale. Moreover, a fine-grained analysis of different facets of CPS showed that irrespective of the academic track, immigrant students demonstrated a more efficient task exploration behavior than their native peers (d = .26). In sum, this might point to the potential of computer-based assessment of CPS to identify otherwise hidden cognitive potential in immigrant students. [less ▲]

La mesure de la motivation et de l'effort des élèves dans le cadre des épreuves standardisées au Luxembourg

Scientific Conference (2013, August)

Students’ Complex Problem-Solving Abilities: Their Structure and Relations to Reasoning Ability and Educational Success

in Intelligence (2013), 41(5), 289-305

Complex Problem Solving (CPS) is considered to be a promising candidate for capturing higher order thinking skills that are emphasized in new educational curricula but are not adequately measured by ... [more ▼]

Complex Problem Solving (CPS) is considered to be a promising candidate for capturing higher order thinking skills that are emphasized in new educational curricula but are not adequately measured by traditional intelligence tests. However, little is known about its psychometric structure and its exact relation to intelligence and educational success—especially in student populations. This study is among the first to use a large and representative sample of secondary school students (N = 563) to examine different measurement models of CPS—that conceptualize the construct as either faceted or hierarchical—and their implications for the construct’s validity. Results showed that no matter which way it was conceptualized, CPS was substantially related to reasoning and to different indicators of educational success. Controlling for reasoning within a joint hierarchical measurement model, however, revealed that the impressive external validity was largely attributable to the variance that CPS shares with reasoning, suggesting that CPS has only negligible incremental validity over and above traditional intelligence scales. On the basis of these results, the value of assessing CPS within the educational context is discussed. [less ▲]

The Genetics Lab: A case study on computer-based assessment within a multilingual sample of „digital natives“

Scientific Conference (2013)

Computer-based assessment forms a key aspect of educational curricula and international student assessment studies like PISA. Today’s students, however, place special demands on assessment instruments ... [more ▼]

Computer-based assessment forms a key aspect of educational curricula and international student assessment studies like PISA. Today’s students, however, place special demands on assessment instruments used within this context. Multilingual samples with often considerably varying experience in technology use tend to become more and more common in most European countries. Taking the Genetics Lab—a computer-based problem solving scenario—as example, the presented case study evaluates how these challenges can be met by the adaptation of the test construction process, the development of interactive and multimedia instructions, and the integration of game-like characteristics. Psychometric properties, especially measurement invariance of the Genetics Lab’s performance scales are discussed on base of a multilingual and heterogeneous sample of 563 Luxembourgish students. As the Genetics Lab is freely available on the internet, an outlook is given on possible future applications to gather further experience with computer-based assessment within multilingual samples. [less ▲]

Luxemburger Schülerinnen und Schüler im internationalen Vergleich

in SCRIPT; EMACS (Eds.) PISA 2012. Nationaler Bericht Luxemburg (2013)

Genetics Lab scoring algorithm

Software (2012)

The Genetics Lab_Theoretical background & psychometric evaluation

Report (2012)

Intelligence as assessed by modern computer-based problem solving scenarios

Scientific Conference (2012)

Computer-based problem solving scenarios—so-called microworlds—have often been suggested as promising alternative assessment instruments of intelligence. Potential benefits compared to traditional paper ... [more ▼]

Computer-based problem solving scenarios—so-called microworlds—have often been suggested as promising alternative assessment instruments of intelligence. Potential benefits compared to traditional paper-pencil tests involve tracking of students’ mental representations of the problems as well as their problem solving strategies by means of behavioral data. Though, it is still topic of ongoing debate whether the skills assessed by such microworlds are distinct from or identical to the construct of intelligence as measured by conventional reasoning tests. To address this issue, we thoroughly investigated construct and incremental validity of a recently developed microworld, the Genetics Lab (Sonnleitner et. al, 2011). We obtained data from a multilingual and representative Luxembourgish student sample (N = 563) who completed the Genetics Lab and 3 reasoning scales of an established intelligence test battery. Results of a Confirmatory Factor Analysis suggest that the construct assessed by the Genetics Lab is largely identical to the construct of intelligence as measured by traditional reasoning scales. Incremental validity was found with respect to the performance in a national assessment of students’ competencies and performance in the PISA study of 2009. Thus, the notion of microworlds to be a valuable measure of intelligence is supported. [less ▲]

The Genetics Lab: a computer-based test to assess students' complex problem solving abilities

Software (2012)

The Genetics Lab: On the usability and psychometric properties of a newly developed microworld to assess Complex Problem Solving in student samples

Scientific Conference (2011, April 27)