Forty years on: Childhood intelligence predicts health in middle adulthood.

in Health Psychology (2014), 33(3), 292-296

HCI Design Methods in CBA: Lessons Learned and Recommendations Based on a Concept Map Use Case

in Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2014 (2014)

This paper focuses on the user-centered design of a computer-based assessment (CBA) instrument for knowledge measurement. We describe relevant design steps in general and illustrate them via the design of ... [more ▼]

This paper focuses on the user-centered design of a computer-based assessment (CBA) instrument for knowledge measurement. We describe relevant design steps in general and illustrate them via the design of our usability improved CBA concept map instrument (ISO 9241-11, 1998; ISO 9241-210, 2010). We discuss design versus evaluation methods from a human-computer interaction (HCI) perspective and emphasize important design steps relevant for constructing usability improved assessment instruments in educational assessment. As these HCI methods are still seldom applied to CBA, this paper helps alleviate the dearth of literature and guidelines in this domain. Our objective is thus to provide the reader with a series of design recommendations for CBA, as derived both from HCI research; and to summarize lessons learned from our specific CBA design use case. [less ▲]

Self-concept in adolescence: A longitudinal study on reciprocal effects of self-perceptions in academic and social domains

in Journal of Adolescence (2013), 36

Educational Assessment Meets Usability: Equal Chances for Conducting Computer-Based Concept-Maps

in Herrington, Jan; Couros, Alec; Irvine, Valerie (Eds.) EdMedia World Conference on Educational Media & Technology 2013 (2013)

This paper describes how usability affects the usage of a computer-based assessment (CBA) instrument. The aim of our current study is twofold: a) the development of a usability improved CBA instrument ... [more ▼]

This paper describes how usability affects the usage of a computer-based assessment (CBA) instrument. The aim of our current study is twofold: a) the development of a usability improved CBA instrument (ISO 9241-11, 1998; ISO 9241-210, 2010) within the framework of human-computer interaction (HCI) and b) the comparison and evaluation of two concept map applications in regards to the impact that usability has on the assessment interaction. In this study we developed and evaluated two CBA concept map applications. We assessed 60 students aged between 16 and 18 who were asked to conduct concept maps on two different versions of the same knowledge assessment instrument. We observed significant differences regarding two out of three usability criteria of the ISO standard definition: efficiency and satisfaction of the students. Our findings highlight that usability influences the assessment interaction and therefore the chances of the assessment taker to work (successfully) through the assessment. [less ▲]

Students’ Complex Problem-Solving Abilities: Their Structure and Relations to Reasoning Ability and Educational Success

in Intelligence (2013), 41(5), 289-305

Complex Problem Solving (CPS) is considered to be a promising candidate for capturing higher order thinking skills that are emphasized in new educational curricula but are not adequately measured by ... [more ▼]

Complex Problem Solving (CPS) is considered to be a promising candidate for capturing higher order thinking skills that are emphasized in new educational curricula but are not adequately measured by traditional intelligence tests. However, little is known about its psychometric structure and its exact relation to intelligence and educational success—especially in student populations. This study is among the first to use a large and representative sample of secondary school students (N = 563) to examine different measurement models of CPS—that conceptualize the construct as either faceted or hierarchical—and their implications for the construct’s validity. Results showed that no matter which way it was conceptualized, CPS was substantially related to reasoning and to different indicators of educational success. Controlling for reasoning within a joint hierarchical measurement model, however, revealed that the impressive external validity was largely attributable to the variance that CPS shares with reasoning, suggesting that CPS has only negligible incremental validity over and above traditional intelligence scales. On the basis of these results, the value of assessing CPS within the educational context is discussed. [less ▲]

Gewissenhaftigkeit und schulische Leistung

in SCRIPT; EMACS (Eds.) PISA 2012. Nationaler Bericht Luxemburg (2013)

The effect of language and mode on two-digit magnitude judgments in German-French bilingual adults

Presentation (2012, July 17)

The study examined the effect of language and presentation mode on numerical processing skills in German-French bilinguals. This bilingual combination is particularly interesting because the order of two ... [more ▼]

The study examined the effect of language and presentation mode on numerical processing skills in German-French bilinguals. This bilingual combination is particularly interesting because the order of two digit number words is inversed in both languages: decade-unit in French but unit-decade in German. Further, previous studies used written number word presentations as verbal notation to activate language, which are, however, rarely used in everyday life. The present study therefore tackled the question whether two-digit magnitude judgments are affected by the presentation mode (visual Arabic vs. auditory words) and the language (German vs. French)? Data stem from proficient German-French bilinguals (mean age: 25,3 years. The within-subject design involved two-digit number pair comparisons presented visually and auditory in German and in French. In line with previous studies, participants responded faster and more accurately on compatible than on incompatible trials in the visual Arabic presentation mode in both languages (compatibility effect): In compatible number pairs, the decades and units concord to the same magnitude decision (e.g. 23_57; decade: 2<5 and unit:3<7) whereas decade and units discord (47_82; decade: 4<8 but 7>2) for incompatible number pairs. In the auditory mode, the results showed a tendency for a regular compatibility effect in German and an inverse compatibility effect in French with participants responding faster on incompatible trials suggesting that bilinguals process numerical tasks in the language of presentation similar to monolinguals using the same language. Conclusively, the results imply differential numerical processing according to the presentation mode (visual Arabic vs. auditory words) and the language (German vs. French). [less ▲]

Intelligence as assessed by modern computer-based problem solving scenarios

Scientific Conference (2012)

Computer-based problem solving scenarios—so-called microworlds—have often been suggested as promising alternative assessment instruments of intelligence. Potential benefits compared to traditional paper ... [more ▼]

Computer-based problem solving scenarios—so-called microworlds—have often been suggested as promising alternative assessment instruments of intelligence. Potential benefits compared to traditional paper-pencil tests involve tracking of students’ mental representations of the problems as well as their problem solving strategies by means of behavioral data. Though, it is still topic of ongoing debate whether the skills assessed by such microworlds are distinct from or identical to the construct of intelligence as measured by conventional reasoning tests. To address this issue, we thoroughly investigated construct and incremental validity of a recently developed microworld, the Genetics Lab (Sonnleitner et. al, 2011). We obtained data from a multilingual and representative Luxembourgish student sample (N = 563) who completed the Genetics Lab and 3 reasoning scales of an established intelligence test battery. Results of a Confirmatory Factor Analysis suggest that the construct assessed by the Genetics Lab is largely identical to the construct of intelligence as measured by traditional reasoning scales. Incremental validity was found with respect to the performance in a national assessment of students’ competencies and performance in the PISA study of 2009. Thus, the notion of microworlds to be a valuable measure of intelligence is supported. [less ▲]

The Genetics Lab: a computer-based test to assess students' complex problem solving abilities

Software (2012)

The Genetics Lab - A new Computer-Based Problem Solving Scenario to assess Intelligence

Scientific Conference (2011)

Assessments of intelligence by means of paper-pencil tests faced several critiques that point to their lack of face validity, insufficient coverage of the definition of intelligence, their sensitivity to ... [more ▼]

Assessments of intelligence by means of paper-pencil tests faced several critiques that point to their lack of face validity, insufficient coverage of the definition of intelligence, their sensitivity to the emotional state of the test taker, and the danger of getting outdated. The present paper discusses to what extent these limitations can be overcome by computer-based problem solving scenarios–so-called microworlds. Generally speaking, microworlds are supposed to be highly accepted by test takers, to provide process measures by directly tracing problem solving behavior, and to realize game-like characteristics that may increase test motivation and reduce test anxiety. To capitalize on these potential advantages, we developed the microworld Genetics Lab that was completed by a large, heterogeneous sample of more than 600 Luxembourgish students. Performance scores were derived for students’ problem solving strategies as well as their mental problem representations–important cognitive data which are not accessible with typical paper-pencil tests. Analyses of the psychometric characteristics of the Genetics Lab empirically underscored the construct validity for the derived performance scores. For example, process oriented measures of strategy use were found to possess discriminant validity with respect to grades. Further, acceptance and induced test anxiety of the Genetics lab was explored relative to a paper-pencil measure of intelligence. Our results show that the Genetics Lab is a reliable and valid assessment instrument and emphasize the benefits of using microworlds for assessing intelligence. [less ▲]