PyBryt is an open-source Python library that aims to help the process of providing specific feedback to students without the need of instructor intervention while the student works through an assignment. PyBryt is an auto-assessment framework that examines a student's solution to some problem to determine its correctness and provide targeted messages about specific implementation details by comparing it to one or more reference implementations provided by the instructor. In this way, students can get the same helpful hints and pointers automatically rather than needing to meet with an instructor.
One of the most difficult problems that is faced by instructors is designing assignments and providing learners with feedback that will guide them toward the solution. This problem is exacerbated in computer science curricula, in which assignments can generate cryptic error messages or give the appearance of success without throwing any errors. The nature of assignments in computer science redoubles the need for targeted instructor feedback and assistance in debugging implementations.
This formulation also has another important benefit: it allows instructors to create more open-form assignments that traditional autograders would allow.
Traditional unit test-based autograders rely on students create rigidly structured objects or a specific API in their submission, which limits the students' ability to structure their solutions in unique ways. Often, one of the most time-consuming processes of grading with an autograder is to help students debug why their specific solution didn't pass the autograder tests even though the implementation is nominally correct.
PyBryt: auto-assessment and auto-grading for computational thinking
We continuously interact with computerized systems to achieve goals and perform tasks in our personal and professional lives. Therefore, the ability to program such systems is a skill needed by everyone. Consequently, computational thinking skills are essential for everyone, which creates a challenge for the educational system to teach these skills at scale and allow students to practice these skills. To address this challenge, we present a novel approach to providing formative feedback to students on programming assignments. Our approach uses dynamic evaluation to trace intermediate results generated by student's code and compares them to the reference implementation provided by their teachers. We have implemented this method as a Python library and demonstrate its use to give students relevant feedback on their work while allowing teachers to challenge their students' computational thinking skills. Paper available at [2112.02144] PyBryt: auto-assessment and auto-grading for computational thinking (arxiv.org)
With PyBryt's ability to compare student implementations againstmultiplereferences, instructors can create assignments with libraries of reference implementations that are robust to the different ways students develop of solving the same problem.