How do we know that our programs, initiatives, degree programs and academic innovations help students?

This question is central to a broad range of initiatives in higher education and is an important pillar of our efforts to create equitable and inclusive spaces for student success. Yet, this question is remarkably difficult to address for many programs, units, and investigators. Access to the institutional datasets takes effort; the data itself is challenging to understand; even relatively simple-to-ask questions ‘How do students participating in X do, compared to otherwise equivalent students not involved in X?’ take real expertise to frame and code. Where these conditions exist, data has been able to illuminate important challenges and successes (e.g., Foundational Course Initiative’s equity analyses, or M-ENGIN’s assessment). Yet, we lack the capacity to assess many of our efforts in principled ways, making it difficult to understand where we are succeeding and where we could benefit from changing tack.

Our vision for the Assessment Toolkit Initiative is to reduce the challenges around engaging in this work. We aim to do this through:

  • Creating rich documentation on existing datasets and access practices
  • Sharing worked examples, notebooks, and sample code to document common analytical practices
  • Developing a community of people engaged in this work, creating opportunities to discuss goals and approaches with peers, and learn from their experiences. Community members will ideally include technically-minded people, program directors/PIs, researchers, and administrators from across the University. 

What does success look like? In a few years, one could hope that:

  • Instructors and departments want to understand what happens to their courses – They are able to perform an equity analysis and take part in a group that is concerned about the same issues and can help them understand and interpret what they learn. 
  • Directors or PIs of our initiatives can use the right computational and statistical tools to document the impact of their efforts – who is thriving, and who needs more attention? Did an initiative help anyone, compared to an otherwise similar group who was not involved? 
  • Discipline-based education researchers have a platform that dramatically lowers the entry cost towards using our rich institutional data; this helps to shape and generate the next generations of questions and insights that help us further refine our support of student learning and success.