Teacher-Student Relationships and Student Outcomes: A Systematic Review of Meta-Analyses and Second-Order Meta-Analysis

E-print/Working paper (2023)

Teacher-student relationships (TSRs) play a vital role in establishing a positive school climate and promoting positive student outcomes. Several meta-analyses have suggested significant associations ... [more ▼]

Teacher-student relationships (TSRs) play a vital role in establishing a positive school climate and promoting positive student outcomes. Several meta-analyses have suggested significant associations between TSRs and, for example, academic achievement, a lack of disruptive behavior, school engagement, peer relationships, motivation, executive functions, and general well-being. However, these meta-analyses have differed substantially in TSR-outcome relationships, moderators, and quality, thus complicating the interpretation of these findings. In this preregistered systematic review of meta-analyses plus original second-order meta-analyses (SOMAs), we aimed to (a) synthesize the meta-analytic evidence on relationships between TSRs and student outcomes, (b) map influential moderators of these relationships, and (c) assess the methodological quality of the meta-analyses. We synthesized over 70 years of educational research in 24 meta-analyses encompassing a total of 116 effect sizes based on more than 2 million prekindergarten and K-12 students. We conducted several three-level SOMAs and found that TSRs had similar strong significant relationships with eight clusters of outcomes: academic achievement, academic emotions, appropriate student behavior, behavior problems, executive functions and self-control, motivation, school belonging and engagement, and student well-being. Age, gender, and informant (student-, peer-, or teacher-assessments) were the most frequently examined moderators in prior research, and our moderator analyses suggested student grade level and social minority status as moderators. We further found large differences in quality between the meta-analyses, and these differences were not associated with the TSR-outcome relationships. These results map the field of TSR research; present their relationships, moderators, and meta-analytic quality; and show how TSRs can contribute to improving outcomes in students via relationship building. Future research should follow meta-analytic open science procedures to improve quality and reproducibility. [less ▲]

Seeing the Connection: A Systematic Review of Meta-Analyses on the Link Between Teacher-Student Relationships and Student Outcomes

Scientific Conference (2023, August 25)

Theoretical background School climate is a key construct with great potential to impact student outcomes. The construct is multidimensional and includes, for instance, academic, community, safety, and ... [more ▼]

Theoretical background School climate is a key construct with great potential to impact student outcomes. The construct is multidimensional and includes, for instance, academic, community, safety, and institutional environment aspects (Wang & Degol, 2016). While the dimensions may vary, researchers widely agree that teacher-student relationships play a vital role in establishing a positive school climate (Wang et al., 2020). Their role can be explained by Bronfenbrenner's (1979) bioecological theory identifying the driver of human development as the interaction with the persons in our closest (proximal) environment. Thus, in a school setting, emotional warmth and closeness or conflict and dependence in teacher-student relationships should also be associated with positive/negative student outcomes. Several meta-analyses uncovered significant associations between teacher-student relationships and school engagement, good peer relationships, executive functioning, well-being, and reductions in aggressive or disruptive behaviors (Endedijk et al., 2021; Nurmi, 2012; Roorda et al., 2011; Vandenbroucke et al., 2018). However, these meta-analyses differed in their methods and substantive findings. Moreover, the extant literature is ambiguous about which moderators (e.g., age) influence these relationships. Furthermore, the reporting and quality of meta-analyses in this field vary considerably, which can compromise the reliability and validity of their findings. Aims Given these research gaps, we systematically searched and reviewed the meta-analytic literature (Cooper & Koenka, 2012) to provide an overview of correlations between teacher-student relationships and student outcomes. In doing so, we examined three research questions: 1. To what extent are academic, behavioral, socio-emotional, motivational, and cognitive student outcomes associated with teacher-student relationships in the meta-analytic literature? 2. Which moderators influence these associations? 3. What is the methodological quality of the included meta-analyses? Methodology After preregistration, a systematic literature search was conducted. During several screening rounds, we identified 24 appropriate meta-analyses that included approximately meta-analytic 130 effect sizes for over one million students. From these meta-analyses, we extracted effect sizes on the association between teacher-student relationships and academic, behavioral, socio-emotional, motivational, and general cognitive student characteristics. We summarized the results for research questions 1 and 2 and developed a narrative overview. For research question 3, we assessed the quality of the meta-analyses using the AMSTAR-2 scale (adapted to correlational studies in psychology and education research; Shea et al., 2017). Findings and their significance Looking at the teacher-student relationship aspect of school climate, a variety of outcome variables were analyzed. The strongest associations were shown for negative teacher-student relationships with student behavior problems (r = .35 bis .57; Nurmi, 2012). Positive teacher-student relationships showed the strongest association with school involvement (r = .26 bis .34; Roorda et al., 2011), prosocial, externalizing, and internalizing behaviors (r = .25; Endedijk et al., 2021), and learning motivation combined with student involvement (r = .23; Wang et al., 2020). Age and grade level were the most frequently examined moderators, with partially contradicting findings. Gender differences, on the other hand, were found less frequently. At the same time, an informant effect was frequently examined, that is, whether and in what ways teachers, student peers, or the students themselves rated the teacher-student relationship. For research question 3, we discuss differences in reporting and the quality range of meta-analyses. With this preregistered systematic review of meta-analyses, we summarize the research landscape on correlates of the teacher-student relationship aspect of school climate. Following our findings and the bioecological theory, teachers should be made aware of the impact of teacher-student relationships and how they could contribute to a positive school climate via relationship building. Some interventions to improve these important relationships have already been meta-analytically studied with promising results (Kincade et al., 2020). Next, we need experiments to causally confirm positive teacher-student relationships as an effective strategy for improving academic, behavioral, socio-emotional, motivational, and cognitive student outcomes and school climate at large. Finally, future research should structure the broad range of conceptualizations of teacher-student relationships and review the variety of theories to explain their impact on student outcomes. References Bronfenbrenner, U. (1979). The ecology of human development: Experiments by nature and design. Harvard university press. Cooper, H., & Koenka, A. C. (2012). The overview of reviews: Unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship. American Psychologist, 67(6), 446–462. https://doi.org/10.1037/a0027119 Endedijk, H. M., Breeman, L. D., van Lissa, C. J., Hendrickx, M. M. H. G., den Boer, L., & Mainhard, T. (2021). The Teacher’s Invisible Hand: A Meta-Analysis of the Relevance of Teacher–Student Relationship Quality for Peer Relationships and the Contribution of Student Behavior. Review of Educational Research, 003465432110514. https://doi.org/10.3102/00346543211051428 Kincade, L., Cook, C., & Goerdt, A. (2020). Meta-Analysis and Common Practice Elements of Universal Approaches to Improving Student-Teacher Relationships. Review of Educational Research, 90(5), 710–748. https://doi.org/10.3102/0034654320946836 Nurmi, J.-E. (2012). Students’ characteristics and teacher–child relationships in instruction: A meta-analysis. Educational Research Review, 7(3), 177–197. https://doi.org/10.1016/j.edurev.2012.03.001 Roorda, D. L., Koomen, H. M. Y., Spilt, J. L., & Oort, F. J. (2011). The Influence of Affective Teacher–Student Relationships on Students’ School Engagement and Achievement: A Meta-Analytic Approach. Review of Educational Research, 81(4), 493–529. https://doi.org/10.3102/0034654311421793 Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., & Henry, D. A. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ, j4008. https://doi.org/10.1136/bmj.j4008 Vandenbroucke, L., Spilt, J., Verschueren, K., Piccinin, C., & Baeyens, D. (2018). The Classroom as a Developmental Context for Cognitive Development: A Meta-Analysis on the Importance of Teacher–Student Interactions for Children’s Executive Functions. Review of Educational Research, 88(1), 125–164. https://doi.org/10.3102/0034654317743200 Wang, M.-T., & Degol, J. L. (2016). School Climate: A Review of the Construct, Measurement, and Impact on Student Outcomes. Educational Psychology Review, 28(2), 315–352. https://doi.org/10.1007/s10648-015-9319-1 Wang, M.-T., L. Degol, J., Amemiya, J., Parr, A., & Guo, J. (2020). Classroom climate and children’s academic and psychological wellbeing: A systematic review and meta-analysis. Developmental Review, 57, 100912. https://doi.org/10.1016/j.dr.2020.100912 [less ▲]

Utilizing Primary Study Quality in Meta-Analyses: A Step-by-Step Tutorial [Preprint]

E-print/Working paper (2023)

Evaluating the quality of primary studies is a key step in meta-analytic reviews to reduce the risk of bias and establish the validity of the meta-analytic inferences. However, the extant body of research ... [more ▼]

Evaluating the quality of primary studies is a key step in meta-analytic reviews to reduce the risk of bias and establish the validity of the meta-analytic inferences. However, the extant body of research offers little guidance on how to represent and incorporate primary study quality (PSQ) in meta-analyses, and some common procedures, such as creating sum scores from a set of quality indicators, often lack the backing from measurement models. Addressing these issues, we present a tutorial that guides meta-analysts in their analytic decisions and approaches to represent and incorporate PSQ. Specifically, we describe, review, and illustrate approaches to (a) select or create quality indicators or scores a priori or as part of the meta-analytic model; (b) examine the possible moderator effects of PSQ; and (c) test the sensitivity of moderator effects to PSQ. We illustrate these approaches with three examples and present a step-by-step tutorial with analytic code for researchers’ guidance. Overall, we argue for representing PSQ model-based if multiple quality indicators are available, the testing of moderator effects of PSQ on the effect sizes and their heterogeneity, and performing moderator sensitivity analyses. [less ▲]

Lehrer-Schüler-Beziehungen und ihre Korrelate: Ein systematisches Review von Meta-Analysen

Scientific Conference (2023, March 01)

THEORETISCHER HINTERGRUND Gute Beziehungen zur eigenen Lehrerin können sich positiv auf den Erfolg eines Schülers auswirken. Dieser Effekt kann mit Bowlby’s (1982) Bindungstheorie erklärt werden und wird ... [more ▼]

THEORETISCHER HINTERGRUND Gute Beziehungen zur eigenen Lehrerin können sich positiv auf den Erfolg eines Schülers auswirken. Dieser Effekt kann mit Bowlby’s (1982) Bindungstheorie erklärt werden und wird empirisch immer wieder gestützt (z.B. Hamre & Pianta, 2001). Positive Lehrer-Schüler-Beziehungen zeichnen sich durch emotionale Wärme und Nähe aus; negative Aspekte durch Konflikt und Abhängigkeit. So stehen positive Lehrer-Schüler-Beziehungen nicht nur mit akademischen Leistungen in Verbindung, sondern auch mit einer Vielzahl anderer, wünschenswerter Schülerentwicklungen. Zahlreiche Meta-Analysen deuten auf signifikante Zusammenhänge zwischen Lehrer-Schüler-Beziehungen und schulischem Engagement, guten Beziehungen zu Gleichaltrigen, exekutiven Funktionen, allgemeinem Wohlbefinden und der Verringerung aggressiver oder störender Verhaltensweisen hin (Endedijk et al., 2021; Nurmi, 2012; Roorda et al., 2017; Vandenbroucke et al., 2018). Diese Befunde sind jedoch weit verstreut in der Literatur, sodass Forschungslücken unentdeckt bleiben. Auch unterscheiden sich bisherige Überblicksarbeiten in ihren Methoden und den gefundenen Zusammenhängen zwischen Lehrer-Schüler-Beziehungen und Ergebnisvariablen von Schüler*innen. Darüber hinaus ist die Literatur uneindeutig, welche Moderatoren (z.B. Alter oder Geschlecht) diese Beziehungen beeinflussen. Gleichzeitig variiert die Qualität der Meta-Analysen in diesem Feld merklich, was die Interpretation ihrer Ergebnisse erschweren kann. FRAGESTELLUNG Angesichts dieser Forschungslücken haben wir die meta-analytische Literatur systematisch durchsucht und zusammengefasst (Cooper & Koenka, 2012), um einen Überblick über Korrelate von Lehrer-Schüler-Beziehungen zu schaffen. Hierbei untersuchten wir drei Forschungsfragen 1. Inwieweit hängen akademische, verhaltensbezogene, sozio-emotionale, motivationale und kognitive Schülereigenschaften mit Lehrer-Schüler-Beziehungen in der meta-analytischen Literatur zusammen? 2. Welche Moderatoren beeinflussen diese Zusammenhänge? 3. Welche methodische Qualität haben die einbezogenen Meta-Analysen? METHODE Um diese Forschungsfragen zu beantworten, analysierten wir 24 Meta-Analysen, die rund 130 Effektstärken für über eine Million Schüler*innen umfassten. Nach der Präregistrierung erfolgte eine systematische Literatursuche. Während mehrerer Runden der Überprüfung mithilfe unserer Ein- und Ausschlusskriterien identifizierten wir 24 passende Meta-Analysen. Aus diesen Meta-Analysen extrahierten wir die Effektstärken zum Zusammenhang von Lehrer-Schüler-Beziehungen und akademische, verhaltensbezogene, sozio-emotionale, motivationale und allgemeine kognitive Schülereigenschaften. Für die Forschungsfragen 1 und 2 haben wir die Ergebnisse zusammengefasst und einen narrativen Überblick erarbeitet. Für Forschungsfrage 3 bewerteten wir die Qualität der Meta-Analysen mit Hilfe der AMSTAR-2 Skala (angepasst an korrelative Studien in der Psychologie und Bildungsforschung; Shea et al., 2017). ERGEBNISSE UND IHRE BEDEUTUNG Mit Blick auf die Lehrer-Schüler-Beziehungen werden unterschiedliche Ergebnisvariablen analysiert (Forschungsfrage 1). Die stärksten Zusammenhänge zeigten sich für Konflikt und Abhängigkeit in der Lehrer-Schüler-Beziehung mit Verhaltensproblemen der Schüler*innen (r = .35 bis .57; Nurmi, 2012). Positive Lehrer-Schüler-Beziehungen zeigte die stärkste Verbindung mit der Beteiligung in der Schule (r = .26 bis .34; Roorda et al., 2011), prosozialem, externalisierendem und internalisierendem Verhalten (r = .25; Endedijk et al., 2021) sowie mit Lernmotivation in Kombination mit Beteiligung der Schüler*innen (r = .23; Wang et al., 2020). Alter oder Klassenstufe waren die am häufigsten untersuchten Moderatoren mit teilweise gegenläufigen Befunden (Forschungsfrage 2). Geschlechterunterschiede wurden dagegen seltener festgestellt. Gleichzeitig wurde der Effekt der Informationsquelle häufig untersucht, d.h., ob und auf welche Weise Lehrkräfte, Gleichaltrige oder die Schüler*innen selbst die Lehrer-Schüler-Beziehung bewerteten. Für Forschunsgfrage 3 diskutieren wir die Qualitätsunterschiede der Meta-Analysen. Mit dem systematischen Review von Meta-Analysen fassen wir die Forschungslandschaft zu Korrelaten von Lehrer-Schüler-Beziehungen zusammen und zeigen, in welchem Zusammenhang diese mit Lehrer-Schüler-Beziehungen stehen. Unseren Ergebnissen folgend sollten Lehrkräfte für die Wirkung von Lehrer-Schüler-Beziehungen und deren Zusammenhängen sensibilisiert werden. Einige Interventionen zur Verbesserung von dieser wichtigen Beziehungen wurden bereits meta-analytisch mit vielversprechende Ergebnissen untersucht (Kincade et al., 2020). Ein nächster Schritt ist nun die experimentelle Überprüfung der gefundenen Korrelate, um positive Lehrer-Schüler-Beziehungen als wirksame Strategie zur Verbesserung von akademischen, verhaltensbezogenen, sozio-emotionalen, motivationalen und kognitiven Schülereigenschaften kausal zu bestätigen. LITERATUR Bowlby, J. (1982). Attachment and loss: Vol. 1. Attachment. (2nd ed., Vol. 1). Basic Books. Cooper, H., & Koenka, A. C. (2012). The overview of reviews: Unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship. American Psychologist, 67(6), 446–462. https://doi.org/10.1037/a0027119 Decristan, J., Kunter, M., & Fauth, B. (2022). Die Bedeutung individueller Merkmale und konstruktiver Unterstützung der Lehrkraft für die soziale Integration von Schülerinnen und Schülern im Mathematikunterricht der Sekundarstufe. Zeitschrift für Pädagogische Psychologie, 36(1–2), 85–100. https://doi.org/10.1024/1010-0652/a000329 Endedijk, H. M., Breeman, L. D., van Lissa, C. J., Hendrickx, M. M. H. G., den Boer, L., & Mainhard, T. (2021). The Teacher’s Invisible Hand: A Meta-Analysis of the Relevance of Teacher–Student Relationship Quality for Peer Relationships and the Contribution of Student Behavior. Review of Educational Research, 003465432110514. https://doi.org/10.3102/00346543211051428 Givens Rolland, R. (2012). Synthesizing the Evidence on Classroom Goal Structures in Middle and Secondary Schools: A Meta-Analysis and Narrative Review. Review of Educational Research, 82(4), 396–435. https://doi.org/10.3102/0034654312464909 Hamre, B. K., & Pianta, R. C. (2001). Early Teacher-Child Relationships and the Trajectory of Children’s School Outcomes through Eighth Grade. Child Development, 72(2), 625–638. https://doi.org/10.1111/1467-8624.00301 Kincade, L., Cook, C., & Goerdt, A. (2020). Meta-Analysis and Common Practice Elements of Universal Approaches to Improving Student-Teacher Relationships. Review of Educational Research, 90(5), 710–748. https://doi.org/10.3102/0034654320946836 Korpershoek, H., Harms, T., de Boer, H., van Kuijk, M., & Doolaard, S. (2016). A Meta-Analysis of the Effects of Classroom Management Strategies and Classroom Management Programs on Students’ Academic, Behavioral, Emotional, and Motivational Outcomes. Review of Educational Research, 86(3), 643–680. https://doi.org/10.3102/0034654315626799 Lei, H., Cui, Y., & Chiu, M. M. (2016). Affective Teacher—Student Relationships and Students’ Externalizing Behavior Problems: A Meta-Analysis. Frontiers in Psychology, 7. https://doi.org/10.3389/fpsyg.2016.01311 Nurmi, J.-E. (2012). Students’ characteristics and teacher–child relationships in instruction: A meta-analysis. Educational Research Review, 7(3), 177–197. https://doi.org/10.1016/j.edurev.2012.03.001 Roorda, D. L., Jak, S., Zee, M., Oort, F. J., & Koomen, H. M. Y. (2017). Affective Teacher–Student Relationships and Students’ Engagement and Achievement: A Meta-Analytic Update and Test of the Mediating Role of Engagement. School Psychology Review, 46(3), 239–261. https://doi.org/10.17105/SPR-2017-0035.V46-3 Roorda, D. L., Koomen, H. M. Y., Spilt, J. L., & Oort, F. J. (2011). The Influence of Affective Teacher–Student Relationships on Students’ School Engagement and Achievement: A Meta-Analytic Approach. Review of Educational Research, 81(4), 493–529. https://doi.org/10.3102/0034654311421793 Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., & Henry, D. A. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ, j4008. https://doi.org/10.1136/bmj.j4008 Vandenbroucke, L., Spilt, J., Verschueren, K., Piccinin, C., & Baeyens, D. (2018). The Classroom as a Developmental Context for Cognitive Development: A Meta-Analysis on the Importance of Teacher–Student Interactions for Children’s Executive Functions. Review of Educational Research, 88(1), 125–164. https://doi.org/10.3102/0034654317743200 Wang, M.-T., L. Degol, J., Amemiya, J., Parr, A., & Guo, J. (2020). Classroom climate and children’s academic and psychological wellbeing: A systematic review and meta-analysis. Developmental Review, 57, 100912. https://doi.org/10.1016/j.dr.2020.100912 [less ▲]

Investigating Pre-service Teachers’ Attitudes Towards LGB Students in Luxembourg

Report (2022)

This questionnaire study investigates the attitudes of pre-service teachers at the University of Luxembourg towards lesbian, gay and bisexual (LGB) students. Assessing this is crucial when it comes to ... [more ▼]

This questionnaire study investigates the attitudes of pre-service teachers at the University of Luxembourg towards lesbian, gay and bisexual (LGB) students. Assessing this is crucial when it comes to guaranteeing equality in education, as prior research, such as on the Pygmalion effect, demonstrates that teachers’ perceptions of students may have an impact on student’s educational success. This impact can be positive but also negative. Detecting trends of negative attitudes in pre-service teachers may therefore represent a first step in protecting LGB-students from educational inequality. Possible explanatory variables, such as respondents' experiences with LGB people, religiosity, gender or even political orientation will be collected. The aim of the study is to add the aspect of sexual diversity in education to the Luxembourg Education Report 2023, which is entirely devoted to the topic of diversity. We hypothesize that negative attitudes of student teachers towards LGB students are associated with 1) little social contact with LGB people; 2) male gender; 3) stronger religious beliefs; 4) heterosexual orientation; 5) a right-wing, conservative political orientation. Additionally, we examine whether hypergendering (hypermasculinity for men and hyperfemininity for women, i.e., the strict adherence to traditional gender roles and their stereotypes) influences the relationship between negative attitudes regarding LGB students. We hypothesize that 6) men with strong hypermasculinity display more negative attitudes than less hypermasculine men; we tentatively hypothesize that 7) women with strong hyperfemininity display more negative attitudes than less hyperfeminine women towards LGB students. All variables and their concrete assessment tools with respective answer options are displayed in the Survey Overview. [less ▲]

Teacher-Student-Relationships and Student Outcomes in Heterogeneous Educational Settings: A Systematic Review of Meta-Analyses

Scientific Conference (2022, November 09)

Especially in diverse educational settings, positive relationships between students and their teachers can foster students’ learning and help alleviate systematic inequalities. Characterized by emotional ... [more ▼]

Especially in diverse educational settings, positive relationships between students and their teachers can foster students’ learning and help alleviate systematic inequalities. Characterized by emotional warmth or closeness, positive teacher-student relationships (TSR) can improve several student outcomes. For instance, existing meta-analyses suggest significant links between TSR and students’ peer relations, school engagement, academic achievement, emotions, executive functions, general well-being, and reductions in aggressive or disruptive behaviors. However, the evidence on these links is scattered, and a comprehensive overview of the associations with TSR integrating academic, behavioral, socio-emotional, motivational, and general cognitive outcomes is lacking. Furthermore, researchers have been unequivocal about possible moderators, such as how these relationships vary with student age or gender. In light of these research gaps, we systematically reviewed the meta-analytic literature and examined (a) the extent to which academic, behavioral, socio-emotional, motivational, and general cognitive student outcomes are related to TSR in the meta-analytic literature; (b) which moderators influence this association; and (c) the methodological quality of the included meta-analyses. We included meta-analyses with preschool or K-12 samples in our dataset which reported some measure of the relation between TSR and student outcomes. With this dataset, we systematically mapped the evidence on (a) the TSR-outcome relationship; (b) the moderators; and (c) the methodological quality of the meta-analyses. We will present our core findings and discuss future research with this second-order, meta-analytic dataset and the impact of positive TSR in diverse and heterogeneous settings. [less ▲]

The Associations Between Teacher-Student-Relationships and Student Outcomes: A Systematic Review of Meta-Analyses (ReMA-TSR)

Report (2022)

The relationships between students and their teachers can impact students’ learning and development. Characterized by emotional warmth or closeness, positive teacher-student-relationships (TSR) can ... [more ▼]

The relationships between students and their teachers can impact students’ learning and development. Characterized by emotional warmth or closeness, positive teacher-student-relationships (TSR) can improve a variety of student outcomes. Existing meta-analyses suggest strong links between TSR and students’ peer relations, school engagement, academic achievement, emotions, executive functions, general well-being, and reductions in aggressive or disruptive behaviors. However, this evidence base is scattered, and a comprehensive overview of the TSR-outcome associations integrating academic, behavioral, socio-emotional, and general cognitive outcomes is lacking. Further, researchers have been unequivocal about possible moderators, such as how these relationships change with student age as their relationship to family, peers, and teachers change. Considering these research gaps, we aim to systematically review the meta-analytic literature and examine the following two research questions: Research Question 1: To what extent do existing meta-analyses provide evidence supporting significant relations between TSR and children’s academic, behavioral, socioemotional, motivational, and general cognitive outcomes? (Overall relationship) Research Question 2: To what extent do these relationships vary by the characteristics of the meta-analyses, such as student samples, measurement characteristics, and the quality of the meta-analyses? To address these research questions, we conduct a systematic review of existing meta-analyses, integrating the findings of eligible studies. We will include quantitative meta-analyses with preschool or K-12 samples who have no diagnosed disorder or disability. [less ▲]

Are Value-Added Scores Stable Enough for High-Stakes Decisions?

Scientific Conference (2022, March)

Theoretical Background: Can we quantify the effectiveness of a teacher or a school with a single number? Researchers in the field of value-added (VA) models may argue just that (e.g., Chetty et al., 2014 ... [more ▼]

Theoretical Background: Can we quantify the effectiveness of a teacher or a school with a single number? Researchers in the field of value-added (VA) models may argue just that (e.g., Chetty et al., 2014; Kane et al., 2013). VA models are widely used for accountability purposes in education and quantify the value a teacher or a school adds to their students’ achievement. For this purpose, these models predict achievement over time and attempt to control for factors that cannot be influenced by schools or teachers (i.e., sociodemographic & sociocultural background). Following this logic, what is left must be due to teacher or school differences (see, e.g., Braun, 2005). To utilize VA models for high-stakes decision-making (e.g., teachers’ tenure, the allocation of funding), these models would need to be highly stable over time. School-level stability over time, however, has hardly been researched at all and the resulting findings are mixed, with some studies indicating high stability of school VA scores over time (Ferrão, 2012; Thomas et al., 2007) and others reporting a lack of stability (e.g., Gorard et al., 2013; Perry, 2016). Furthermore, as there is no consensus on which variables to use as independent or dependent variables in VA models (Everson, 2017; Levy et al., 2019), the stability of VA could vary between different outcome measures (e.g., language or mathematics). If VA models lack stability over time and across outcome measures, their use as the primary information for high-stakes decision-making is in question, and the inferences drawn from them could be compromised. Questions: With these uncertainties in mind, we examine the stability of school VA model scores over time and investigate the differences between language and mathematics achievement as outcome variables. Additionally, we demonstrate the real-life implications of (in)stable VA scores for single schools and point out an alternative, more constructive use of school VA models in educational research. Method: To study the stability of VA scores on school level over time and across outcomes, we drew on a sample of 146 primary schools, using representative longitudinal data from the standardized achievement tests of the Luxembourg School Monitoring Programme (LUCET, 2021). These schools included a heterogeneous and multilingual sample of 7016 students. To determine the stability of VA scores in the subject of mathematics and in languages over time, we based our analysis on two longitudinal datasets (from 2015 to 2017 and from 2017 to 2019, respectively) and generated two VA scores per dataset, one for language and one for mathematics achievement. We further analyzed how many schools displayed stable VA scores in the respective outcomes over two years, and compared the rank correlations of VA scores between language and mathematics achievement as an outcome variable. Results and Their Significance: Only 34-38 % of the schools showed stable VA scores from grade 1 to 3 with moderate rank correlations of r = .37 with language and r = .34 with mathematics achievement. We therefore discourage using VA models as the only information for high-stakes educational decisions. Nonetheless, we argue that VA models could be employed to find genuinely effective teaching or school practices—especially in heterogeneous student populations, such as Luxembourg, in which educational disparities are an important topic already in primary school (Hoffmann et al., 2018). Consequently, we contrast the school climate and instructional quality, which might be a driver of the differences between schools with stable high vs. low VA scores. Literature Braun, H. (2005). Using student progress to evaluate teachers: A primer on value-added models. Educational Testing Service. Chetty, R., Friedman, J. N., & Rockoff, J. E. (2014). Measuring the impacts of teachers I: Evaluating bias in teacher value-added estimates. American Economic Review, 104(9), 2593–2632. https://doi.org/10.1257/aer.104.9.2593 Everson, K. C. (2017). Value-added modeling and educational accountability: Are we answering the real questions? Review of Educational Research, 87(1), 35–70. https://doi.org/10.3102/0034654316637199 Ferrão, M. E. (2012). On the stability of value added indicators. Quality & Quantity, 46(2), 627–637. https://doi.org/10.1007/s11135-010-9417-6 Gorard, S., Hordosy, R., & Siddiqui, N. (2013). How unstable are “school effects” assessed by a value-added technique? International Education Studies, 6(1), 1–9. https://doi.org/10.5539/ies.v6n1p1 Kane, T. J., McCaffrey, D. F., Miller, T., & Staiger, D. O. (2013). Have We Identified Effective Teachers? Validating Measures of Effective Teaching Using Random Assignment. Research Paper. MET Project. Bill & Melinda Gates Foundation. https://files.eric.ed.gov/fulltext/ED540959.pdf Levy, J., Brunner, M., Keller, U., & Fischbach, A. (2019). Methodological issues in value-added modeling: An international review from 26 countries. Educational Assessment, Evaluation and Accountability, 31(3), 257–287. https://doi.org/10.1007/s11092-019-09303-w LUCET. (2021). Épreuves Standardisées (ÉpStan). https://epstan.lu Perry, T. (2016). English value-added measures: Examining the limitations of school performance measurement. British Educational Research Journal, 42(6), 1056–1080. https://doi.org/10.1002/berj.3247 Thomas, S., Peng, W. J., & Gray, J. (2007). Modelling patterns of improvement over time: Value added trends in English secondary school performance across ten cohorts. Oxford Review of Education, 33(3), 261–295. https://doi.org/10.1080/03054980701366116 [less ▲]

Stability of Primary School Value-Added Scores over Time: A Comparison Between Math and Language Achievement as Outcome Variables

Scientific Conference (2021, November)

Value-added (VA) models are widely used for accountability purposes in education. Tracking a teacher’s or a school’s VA score over time forms oftentimes the basis for high-stakes decision-making and can ... [more ▼]

Value-added (VA) models are widely used for accountability purposes in education. Tracking a teacher’s or a school’s VA score over time forms oftentimes the basis for high-stakes decision-making and can determine whether teachers can keep their jobs or schools may receive certain funding. Despite their high-stakes application, the stability of VA scores over time has not yet been investigated for primary schools. Moreover, it is unclear whether different outcome measures (e.g., language and mathematics) may differ in their stability over time. In the present study, we aimed to clarify the stability of VA scores over time and investigate the differences across outcome variables. Furthermore, we wanted to showcase the real-life implications of (in)stable VA scores for single schools, with a focus on an informative use of VA scores rather than an evaluative way. The exploration of school VA scores in primary schools is especially relevant for heterogeneous student populations, for instance, in Luxembourg. Thus, we drew on representative longitudinal data from the standardized achievement tests of the Luxembourg School Monitoring Programme and examined the stability of school VA scores over two years in 146 schools (N = 7016 students). The overall stability, as measured by correlation coefficients, was moderate with r = .37 for VA scores in language and r = .34 for VA scores in mathematics from grade one to grade three. Real-life implications for schools will be discussed. [less ▲]

Meta-Analytic Structural Equation Models of Executive Functions and Math Intelligence in Preschool Children

Scientific Conference (2021, September)

BACKGROUND: Response inhibition, attention shifting, and working memory updating are the three core executive functions (EFs; Miyake et al., 2000) underlying other cognitive skills that are relevant for ... [more ▼]

BACKGROUND: Response inhibition, attention shifting, and working memory updating are the three core executive functions (EFs; Miyake et al., 2000) underlying other cognitive skills that are relevant for learning and everyday life. For example, they have shown to be differentially related to the mathematical component of intelligence (i.e., math intelligence) in school students and adults. While researchers suppose these three EFs to become more differentiated from early childhood to adulthood, neither the link of these constructs nor their structure has been conclusively established in preschool children yet. Primary studies on path models connecting EFs and math intelligence diverge in the exact relation of EFs and math intelligence. It remains unclear whether inhibition, shifting, and updating exhibit distinct but correlated constructs with respect to their relation to math intelligence. OBJECTIVES: With our meta-analysis, we aimed to (a) synthesize the relation between the three EFs and math intelligence in preschool children; and (b) compare plausible models of the effects of EFs on math intelligence. METHODS/RESULTS: Synthesizing data from 47 studies (363 effect sizes, 30,481 participants) from the last two decades via novel multilevel and multivariate meta-analytic models (Pustejovsky & Tipton, 2020), we found the three core EFs to be significantly related to math intelligence: Inhibition ("r" ̅ = .30, 95 % CI [.25, .35]), shifting ("r" ̅ = .32, 95 % CI [.25, .38]), and updating ("r" ̅ = .36, 95 % CI [.31, .40]). Looking at the three core EFs as one construct, the correlation was "r" ̅ = .34, 95 % CI [.31, .37]. Utilizing correlation-based, meta-analytic structural equation modeling (Jak & Cheung, 2020), our results exhibited significant relations of all EFs to math intelligence. These relations did not differ between the three core EFs. DISCUSSION: Our findings corroborate the positive link between EFs and math intelligence in preschool children and are similar to other age groups. From the model testing, we learned that representing EFs by a latent variable, thus capturing the covariance among the three core EFs, explained substantially more variation in math intelligence than representing them as distinct constructs. [less ▲]