Challenges of modern Computer Based Assessment: Usability, Scoring, and “Digital Natives”

Scientific Conference (2011)

In recent years, computer based assessment has undergone substantive change. Test developers as well as test users have become aware of the fact that computers can do more than administrate traditional ... [more ▼]

In recent years, computer based assessment has undergone substantive change. Test developers as well as test users have become aware of the fact that computers can do more than administrate traditional (paper-pencil) item formats like multiple-choice. More complex computer based item types allow tracking test takers’ mental representations of the problem or even their problem solving strategies by means of behavioral data. An example for such a modern item type are microworlds – dynamically changing problem solving scenarios with which the test taker has to interact. However, with the advent of complex item types, new challenges arise. First, usability is at stake - test takers do not intuitively know what to do or how to interact with complex tasks. Second, a massive load of data is produced and it gets difficult for the test developer to decide on relevant scores. Third, today’s students - so-called “digital natives”- grew up with computers and therefore set high quality standards for software applications. They may quickly loose trust and interest in tests with old fashioned design, cumbersome handling or even malfunctioning software. On basis of the Genetics Lab – a microworld developed to assess general mental ability – these challenges of modern computer based assessment are discussed. Three consecutive small scale studies were carried out to investigate usability issues, validate scoring algorithms and to ensure acceptance among students. The results demonstrate the importance of considering usability during the test development process, particularly with regard to scoring. The modification of conventional test development procedures for modern computer based assessment is suggested. Moreover, possibilities to satisfy even a critical target population are presented. [less ▲]

Assessing Intelligence for Education in the 21st Century: The Benefits of Microworlds

Scientific Conference (2011)

Computer-based problem solving scenarios—so-called microworlds—are contemporary educational assessment instruments of intelligence that offer several benefits compared to traditional paper-pencil tests ... [more ▼]

Computer-based problem solving scenarios—so-called microworlds—are contemporary educational assessment instruments of intelligence that offer several benefits compared to traditional paper-pencil tests. This involves tracking of students’ mental representations of the problems as well as their problem solving strategies by means of behavioral data which provides key information for educational interventions. Moreover, microworlds realize game-like characteristics that may increase test motivation and reduce test anxiety. In the present study, the Genetics Lab, a newly developed microworld, was completed by a representative sample of more than 800 Luxembourgish students. Students chose among three different languages (German, French and English) in which the problem content of the Genetics Lab was presented. The present paper analyzes the psychometric properties of the various performance scores derived for the Genetics Lab with respect to their relations to school grades, and measurement invariance across gender, chosen test language, and migration background. Moreover, a direct comparison with traditional measures of intelligence demonstrated construct validity of the performance scores of the Genetics Lab. In sum, the results obtained for the Genetics Lab show the benefits of behavioral data obtained for computer-based problem-solving scenarios and support the notion of microworlds to be a valuable measure of intelligence. [less ▲]

Geschlechtsunterschiede in Lesen und Mathematik: Ergebnisse aus 69 Ländern.

Scientific Conference (2011)

Evaluation von Plurilingualität in einer multilingualen Gesellschaft: bloße Herausforderung und/oder Chance?

Scientific Conference (2010, September)

Unterschiede zwischen Schulformen und Schulen

in SCRIPT; EMACS (Eds.) PISA 2009. Nationaler Bericht Luxemburg (2010)

Jungen und Mädchen

in SCRIPT; EMACS (Eds.) PISA 2009. Nationaler Bericht Luxemburg (2010)

The C-test, the TCF and the CEFR: a validation study.

in Grotjahn, Rüdiger (Ed.) Der C-Test: Beiträge aus der aktuellen Forschung. The C-Test: Contributions from Current Research (2010)

Luxemburger Schülerinnen und Schüler im internationalen Vergleich

in SCRIPT; EMACS (Eds.) PISA 2009. Nationaler Bericht Luxemburg (2010)

Wandel im Bildungswesen, zunehmende Komplexität und Web 2.0: Sind traditionelle Intelligenztests noch "up to date"?

Poster (2010)

Grundlegender Wandel im schulischen Bereich führt zur berechtigten Frage, inwiefern nicht Innovationen in der Intelligenzdiagnostik nötig wären, um die hohe prädiktive Validität und dadurch wesentliche ... [more ▼]

Grundlegender Wandel im schulischen Bereich führt zur berechtigten Frage, inwiefern nicht Innovationen in der Intelligenzdiagnostik nötig wären, um die hohe prädiktive Validität und dadurch wesentliche Bedeutung bei Schulplatzierungsentscheidungen weiterhin zu sichern. Die stärkere Betonung fächerübergreifender Kompetenzen, eine vermehrte Einbindung des Computers in Unterricht und Alltag sowie zunehmende Komplexität in der Arbeitswelt führen zu neuen Herausforderungen und beleben alte Kritik wieder. In diesem Kontext wurden immer wieder computerbasierte Problemlöseszenarien als vielversprechender Ansatz genannt, Schwächen traditioneller Intelligenztests zu überwinden. Vorteile werden, neben dem dynamischen Testformat und einer vollständigeren Abdeckung des Intelligenzbegriffes, unter anderem auch in erhöhter face-validity gesehen. Während bisherige Studien den Schwerpunkt auf das empirische Verhältnis zwischen Leistungsmaßen der Szenarien und traditionellen Intelligenzmaßen setzten, versucht dieser Beitrag, bewusst die Diskussion um theoretische Aspekte zu ergänzen. Einer Analyse der Anforderungen die an eine aktuelle Intelligenzdiagnostik zu stellen sind, folgt ein systematischer Vergleich traditioneller Intelligenztestformate mit computerbasierten Problemlöseszenarien. Konkrete Beispiele sowie aktuelle Daten zur Pilotierung eines derartigen Szenarios zur Intelligenzdiagnostik ergänzen die Diskussion. [less ▲]

The cross-cultural generalizability of a new structural model of academic self-concepts.

in Learning and Individual Differences (2009), 19