Problem

GitHub classroom is great, and it lets educators create automatic grading solution with GitHub Actions. However, one of the pitfall is that students can easily change the unit test codes and GitHub actions workflow and educators cannot trust the automatic grading result. Another problem is that students must commit and push source code themselves. Educators therefore have no idea if students are copying from other or not. Also, educators don’t know how students develop their answer, and the details of every code change. 

Solution

To solve the potential cheating problem and to move from passive to active learning, we need a source of truth to run unit test and keep all changes of source codes when students are writing codes.

First, we create .vscode/settings.json to enable auto save file with 3 seconds delay and 2 “run on save” add-on commands (one for Codespace and Visual Studio Code in Windows). 

 

 

 

{
    "python.testing.pytestArgs": [],
    "python.testing.unittestEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.analysis.extraPaths": [
        "./tests/unit_test_helper"
    ],
    "python.linting.pylintArgs": [
        "--disable=C0111"
    ],
    "emeraldwalk.runonsave": {
        "autoClearConsole": true,
        "commands": [
            {
                "match": "lab\/.+py",
                "cmd": "python ${fileDirname}/../../auto_grader.py ${file}",
            },
            {
                "match": "lab\\\\.+py",
                "cmd": "${fileDirname}\\..\\..\\.venv\\Scripts\\activate.bat && python ${fileDirname}\\..\\..\\auto_grader.py ${file}",
            }
        ]
    },
    "files.autoSave": "afterDelay",
    "files.autoSaveDelay": 3000
}

 

 

With this setup, every code change will save automatically and trigger auto_grader.py after saving the file.

Second, auto_grader.py runs specific unit test toward that the changed file.

 

If test fails, it overwrites autograding.json, commits code change with comment "[skip actions] " + sourceCodeFilePath, and push to GitHub Repo. Suffix comment with “skip actions” will suspend GitHub Actions to run for student grade, since it will keep commit and push to GitHub frequently else it will generate a lot of noise in GitHub Action.

 

If test passes, it sends a post request with header of student special API Key and body with source code file path and source code to Azure API management with Azure function API backend “pytester”. It checks the code corresponding test record from Azure table (TestResults), if pass previously, it returns "Repeated Successful Test.", else it runs unit test again in an isolated environment. If the unit test in Azure function pass as well, it overwrites autograding.json, commits code change with comment sourceCodeFilePath, push to GitHub Repo, then trigger GitHub Action for grading. Autograding.json defines how to grade students in GitHub Action. The guideline asks educators to set it one by one with web UI and it is hard to use for more than 100 tests!

We have developed a script to generate this file and it can be overwritten into student repo! 

Sample Script: https://gist.github.com/wongcyrus/69fdc3ac3b74ef9ec08931022146a4c5

How to identify student and protect unit test API?

We use Microsoft Azure API Management. CDK-TK creates user account and subscription for each student and setup rate limit. Here is the code snippet.

 

let i = 0;
    for (let student of students) {
      const apiManagementUser = new ApiManagementUser(this, "ApiManagementUser" + i, {
        userId: student.id,
        apiManagementName: apiManagement.name,
        resourceGroupName: resourceGroup.name,
        email: student.email,
        firstName: student.firstName,
        lastName: student.lastName,
        state: "active"
      })

      const apiManagementSubscription = new ApiManagementSubscription(this, "ApiManagementSubscription" + i, {
        apiManagementName: apiManagement.name,
        resourceGroupName: resourceGroup.name,
        userId: apiManagementUser.id,
        displayName: student.id + ":" + student.firstName + " " + student.lastName,
        apiId: apiManagementApi.id,
        state: "active"
      })

      new TerraformOutput(this, "SubscriptionKey_" + i, {
        sensitive: true,
        value: apiManagementSubscription.primaryKey
      });
      i++;
    }

    new ApiManagementApiPolicy(this, "ApiManagementApiPolicy", {
      apiName: apiManagementApi.name,
      apiManagementName: apiManagement.name,
      resourceGroupName: resourceGroup.name,
      xmlContent: `
<policies>
  <inbound>
    <set-header name="request-email" exists-action="override">
      <value>@(context.User.Email)</value>
    </set-header>
    <set-header name="request-id" exists-action="override">
      <value>@(context.User.Id)</value>
    </set-header>
    <rate-limit-by-key calls="10" renewal-period="60" counter-key="@(context.Request.IpAddress)" />
    <rate-limit calls="10" renewal-period="60" />
    <base />
    <set-backend-service backend-id="${apiManagementBackend.name}" />
  </inbound>
</policies>
`
    })

 

 

Azure function can extract student email from request header. Each student cannot call more than 10 times per minute.

How to run PyTest in Azure Function?

That is the hardest part! The problem is that you cannot modify and corrupt the Python code of the Azure Function. It means that you cannot let student submit code to overwrite or add into the executing Python runtime, which is not trusted, and Azure function is like a web application which has many concurrency PyTest need to run at the same time. Therefore, we need to create a separate process to run PyTest. The first trial is to install PyTest and related packages to the Azure Function runtimes and reuse the runtime. However, it cannot load packages if we make a code of source code and unit test in /tmp. Finally, we found that we can run commands to create a temporary Python Virtual Environment in /tmp, install package with pip, activate the virtual environment, extract standard unit test code, overwriten with student changed Python code and run an independent PyTest.

We are trying to speed up the process by saving the unit test virtual environment and Azure file share, then we can save time to create Virtual Environment and install packages every time execute the Azure Function.

Demo

How to work on your GitHub classroom Python Lab Exercises with Codespace?

How to work on your GitHub classroom Python Lab Exercises with VS Code in Windows?

Commit History can show how students derive their answer.

It is a step recorder and captures all code change every 3 seconds for each student. And it is possible to combine the code change and generate an animation gif. On the other hand, educators can detect student cheating based on the abnormally short commit history!

 

Azure table Test Results.

Educators can track each unit test completion time and understand the performance of your class.

Deployment Prerequisites

Setup CDK-TF

https://www.hashicorp.com/blog/building-azure-resources-with-typescript-using-the-cdk-for-terraform

https://techcommunity.microsoft.com/t5/educator-developer-blog/object-oriented-your-azure-infrastructure-with-cloud-development/ba-p/3474715

Clone the repo

 

git clone https://github.com/wongcyrus/pytest-runner-azure-function

 

 

Update the student list

Infrastructure/student_list.csv

Deploy the project

 

cd Infrastructure
npm i
cdktf deploy --auto-approve
cdktf output --outputs-file-include-sensitive-outputs --outputs-file secrets.json
npm run getstudentkey

 

 

It takes about an hour to complete as the creating Azure API management takes time.

Mail merge student_key.csv to your students with API Key.

Students need to set the API to config.py.

Visual Studio Code or GitHub CodeSpace

CodeSpace is great as students don’t need to install anything and be able to work on the lab exercises and assignment anywhere. However, GitHub classroom students can use CodeSpace without limit during the beta but after beta then it will be limited “The free allowance is estimated to be enough for a class of 50 with 5 assignments per month, on a 2 core machine with 1 codespace stored per student.”

Therefore, educators should prepare for downgrade to Visual Studio Code anytime. For my course, I request my students to use CodeSpace and Visual Studio Code alternatively. Firstly, it will be fine to downgrade anytime, and secondly, I can let my students experience both Windows and Linux development environment which is important for the future job!

Conclusion

Our settings convert the single point of assessment into the continuous change detection assessment, and educators can understand students real coding behavior. GitHub Classroom is suitable for students who are eager to learn and know how to use Git. For our case, learning Python programme is the first course for the first day of school, and it is hard to teach them use GitHub immediately. The push and grade are too passive and there is no workable solution to prevent students from changing GitHub Actions workflow and unit tests.
Our longer term goal is that GitHub Classroom auto grader will have a way to deny students changing the workflow and allows overwriting test codes before auto grading, this will allow us to get rid of running unit test in Azure Function.
Scalable solutions: API management, Azure Function in consumption plan, and Azure Storage Table are 100% server-less, so this solution is scalable, high availability, and cheap!

Thanks to the project collaborators include, Andy Lum, Jerry Lee, Fong Ho Luen, Jenny Cheung, Woodi Tsang, Eric Luan and Wina Yu from the IT114115 Higher Diploma in Cloud and Data Centre Administration. 

Resources

https://github.com/wongcyrus/pytest-runner-azure-function

https://github.com/wongcyrus/ite3101_introduction_to_programming