Genetics Lab scoring algorithm

Software (2012)

The Genetics Lab: a computer-based test to assess students' complex problem solving abilities

Software (2012)

Intelligence as assessed by modern computer-based problem solving scenarios

Scientific Conference (2012)

Computer-based problem solving scenarios—so-called microworlds—have often been suggested as promising alternative assessment instruments of intelligence. Potential benefits compared to traditional paper ... [more ▼]

Computer-based problem solving scenarios—so-called microworlds—have often been suggested as promising alternative assessment instruments of intelligence. Potential benefits compared to traditional paper-pencil tests involve tracking of students’ mental representations of the problems as well as their problem solving strategies by means of behavioral data. Though, it is still topic of ongoing debate whether the skills assessed by such microworlds are distinct from or identical to the construct of intelligence as measured by conventional reasoning tests. To address this issue, we thoroughly investigated construct and incremental validity of a recently developed microworld, the Genetics Lab (Sonnleitner et. al, 2011). We obtained data from a multilingual and representative Luxembourgish student sample (N = 563) who completed the Genetics Lab and 3 reasoning scales of an established intelligence test battery. Results of a Confirmatory Factor Analysis suggest that the construct assessed by the Genetics Lab is largely identical to the construct of intelligence as measured by traditional reasoning scales. Incremental validity was found with respect to the performance in a national assessment of students’ competencies and performance in the PISA study of 2009. Thus, the notion of microworlds to be a valuable measure of intelligence is supported. [less ▲]

The Genetics Lab: On the usability and psychometric properties of a newly developed microworld to assess Complex Problem Solving in student samples

Scientific Conference (2011, April 27)

Challenges of modern Computer Based Assessment: Usability, Scoring, and “Digital Natives”

Scientific Conference (2011)

In recent years, computer based assessment has undergone substantive change. Test developers as well as test users have become aware of the fact that computers can do more than administrate traditional ... [more ▼]

In recent years, computer based assessment has undergone substantive change. Test developers as well as test users have become aware of the fact that computers can do more than administrate traditional (paper-pencil) item formats like multiple-choice. More complex computer based item types allow tracking test takers’ mental representations of the problem or even their problem solving strategies by means of behavioral data. An example for such a modern item type are microworlds – dynamically changing problem solving scenarios with which the test taker has to interact. However, with the advent of complex item types, new challenges arise. First, usability is at stake - test takers do not intuitively know what to do or how to interact with complex tasks. Second, a massive load of data is produced and it gets difficult for the test developer to decide on relevant scores. Third, today’s students - so-called “digital natives”- grew up with computers and therefore set high quality standards for software applications. They may quickly loose trust and interest in tests with old fashioned design, cumbersome handling or even malfunctioning software. On basis of the Genetics Lab – a microworld developed to assess general mental ability – these challenges of modern computer based assessment are discussed. Three consecutive small scale studies were carried out to investigate usability issues, validate scoring algorithms and to ensure acceptance among students. The results demonstrate the importance of considering usability during the test development process, particularly with regard to scoring. The modification of conventional test development procedures for modern computer based assessment is suggested. Moreover, possibilities to satisfy even a critical target population are presented. [less ▲]

Designing the test booklets for Rasch model calibration in a large-scale assessment with reference to numerous moderator variables and several ability dimensions

in Educational Research and Evaluation (2011), 17(6), 483-495

In large-scale assessments, it usually does not occur that every item of the applicable item pool is administered to every examinee. Within item response theory (IRT), in particular the Rasch model (1960 ... [more ▼]

In large-scale assessments, it usually does not occur that every item of the applicable item pool is administered to every examinee. Within item response theory (IRT), in particular the Rasch model (1960), this is not really a problem because item calibration works nevertheless. The different test booklets only need to be conceptualized according to a connected incomplete block design. Yet, connectedness of such a design should best be fulfilled severalfold, since deletion of some items in the course of the item pool’s IRT calibration may become necessary. The real challenge, however, is to meet constraints determined by numerous moderator variables such as different response formats and several topics of content – all the more so, if several ability dimensions are under consideration, the testing duration is strongly limited or individual scoring and feedback is an issue. In this article, we offer a report of how to deal with the resulting problems. Experience is based on the governmental project of the Austrian Educational Standards (Kubinger et al., 2007). [less ▲]

Assessing Intelligence for Education in the 21st Century: The Benefits of Microworlds

Scientific Conference (2011)

Computer-based problem solving scenarios—so-called microworlds—are contemporary educational assessment instruments of intelligence that offer several benefits compared to traditional paper-pencil tests ... [more ▼]

Computer-based problem solving scenarios—so-called microworlds—are contemporary educational assessment instruments of intelligence that offer several benefits compared to traditional paper-pencil tests. This involves tracking of students’ mental representations of the problems as well as their problem solving strategies by means of behavioral data which provides key information for educational interventions. Moreover, microworlds realize game-like characteristics that may increase test motivation and reduce test anxiety. In the present study, the Genetics Lab, a newly developed microworld, was completed by a representative sample of more than 800 Luxembourgish students. Students chose among three different languages (German, French and English) in which the problem content of the Genetics Lab was presented. The present paper analyzes the psychometric properties of the various performance scores derived for the Genetics Lab with respect to their relations to school grades, and measurement invariance across gender, chosen test language, and migration background. Moreover, a direct comparison with traditional measures of intelligence demonstrated construct validity of the performance scores of the Genetics Lab. In sum, the results obtained for the Genetics Lab show the benefits of behavioral data obtained for computer-based problem-solving scenarios and support the notion of microworlds to be a valuable measure of intelligence. [less ▲]

Wandel im Bildungswesen, zunehmende Komplexität und Web 2.0: Sind traditionelle Intelligenztests noch "up to date"?

Poster (2010)

Grundlegender Wandel im schulischen Bereich führt zur berechtigten Frage, inwiefern nicht Innovationen in der Intelligenzdiagnostik nötig wären, um die hohe prädiktive Validität und dadurch wesentliche ... [more ▼]

Grundlegender Wandel im schulischen Bereich führt zur berechtigten Frage, inwiefern nicht Innovationen in der Intelligenzdiagnostik nötig wären, um die hohe prädiktive Validität und dadurch wesentliche Bedeutung bei Schulplatzierungsentscheidungen weiterhin zu sichern. Die stärkere Betonung fächerübergreifender Kompetenzen, eine vermehrte Einbindung des Computers in Unterricht und Alltag sowie zunehmende Komplexität in der Arbeitswelt führen zu neuen Herausforderungen und beleben alte Kritik wieder. In diesem Kontext wurden immer wieder computerbasierte Problemlöseszenarien als vielversprechender Ansatz genannt, Schwächen traditioneller Intelligenztests zu überwinden. Vorteile werden, neben dem dynamischen Testformat und einer vollständigeren Abdeckung des Intelligenzbegriffes, unter anderem auch in erhöhter face-validity gesehen. Während bisherige Studien den Schwerpunkt auf das empirische Verhältnis zwischen Leistungsmaßen der Szenarien und traditionellen Intelligenzmaßen setzten, versucht dieser Beitrag, bewusst die Diskussion um theoretische Aspekte zu ergänzen. Einer Analyse der Anforderungen die an eine aktuelle Intelligenzdiagnostik zu stellen sind, folgt ein systematischer Vergleich traditioneller Intelligenztestformate mit computerbasierten Problemlöseszenarien. Konkrete Beispiele sowie aktuelle Daten zur Pilotierung eines derartigen Szenarios zur Intelligenzdiagnostik ergänzen die Diskussion. [less ▲]

Das Wiener Self-Assessment Psychologie mit seinen Verfahren der experimentalpsychologischen Verhaltensdiagnostik zur Messung studienfachübergreifender Soft Skills

in Rudinger, G.; Hörsch, K. (Eds.) Self-Assessment an Hochschulen: Von der Studienfachwahl zur Profilbildung (2009)

Versuch eines Konstruktionsrationals zum Leseverständnis: Entwicklung und Evaluation auf Basis kognitiver Modelle

Scientific Conference (2009)

Due to inconclusive findings concerning the components responsible for the difficulty of reading comprehension items, this paper attempts to set up an item-generating system using hypothesis-driven ... [more ▼]

Due to inconclusive findings concerning the components responsible for the difficulty of reading comprehension items, this paper attempts to set up an item-generating system using hypothesis-driven modeling of item complexity applying Fischer’s (1973) linear logistic test model (LLTM) to a German reading comprehension test. This approach guarantees an evaluation of the postulated item-generating system; moreover construct validity of the administered test is investigated. Previous findings in this field are considered; additionally, some text features are introduced to this debate and their impact on item difficulty is discussed. Results once more show a strong influence of formal components (e.g. the number of presented response options in a multiple-choice-format), but also indicate how this effect can be minimized. [less ▲]