Microsoft Azure Automatic Grading Engine

Published Aug 25 2021 10:32 AM 1,991 Views
Established Member

Problem

One of the main challenges on teaching cloud technologies in collages is how to assess student. For other IT courses, educators can collect some files such as source code and give a grade to students. However, for cloud tech educators, it is hard to use the traditional approach to handle the assessment. Two main problems, firstly it is hard to check each student work one by one manually which is time consuming and error-pone. Secondly, it is not a good idea to get a grade based on a “snapshot”, as students may use IasC tools such as ARM template or terraform and every student gets full mark, but they know nothing!

 

In fact, for vocational training, my course IT114115 Higher Diploma in Cloud and Data Centre Administration in Department of Information Technology (IT) of the Hong Kong Institute of Vocational Education (Lee Wa.... Our Goal is to ensure our students are skilled to be highly effective cloud engineers. All our students know how to do it well and within the region employers are ready to pay students for the roles but they all want know "who is the best?".

 

Solution

Educators need to have access to the student work and see their code, as part of the Microsoft Learn Educator Programme I have been developing an opensource serverless application – Microsoft Azure Automatic Grading Engine.

The engine includes 2 main functions:

  1. Schedule Grader
  2. Class Grade Report Generator

Architecture Diagram

cyruswong_0-1629886295598.png

 

How does Schedule Grader work?

  1. Timer Trigger runs a Durable Orchestration function “Schedule Grader” every 5 minutes.
  2. “Schedule Grader” reads all json files in the “credentials” blob container and one file per class assignment. It defines the grader url and list of students and we will discuss the data structure later.
  3. All assignments process run in parallel by an independent Grade Assignment Function call.
  4. In each Grade Assignment Function call, it sends https request to Azure Grader Function in parallel. It uses Azure SDK to collect the Azure settings, the process runs a NUnit test to check the settings as expect or not and return an xml test result output file. For the fairness, it is important to grade all students at the same time. Also, using web services decouple the grader from the solution. This allows the educators to add and update the assignment grader without redeployment of the whole solution. In the long term, we can have a grader repository or hub and educators can share assignment with grader. The Grade Assignment Function saves all xml test results in test result blob container.

 

How does Class Grade Report Generator work?

  1. When Grade Report Function HttpTrigger endpoint is called, it checks query parameter “assignment”.
  2. It sets “assignment” as prefix and gets all text result xml files.
  3. By mean of xpath we extract text case full name and test result. If the test is pass, it contains 1 mark.
  4. The service then generates the accumulate mark and creates a report in an Excel file format.

How to create a class assignment?

Educators need to complete 3 tasks:

  1. Implement a NUnit test for the assignment and note down the https endpoint. You can refer AzureGraderFunction.cs the code within this C# file it is very strange forward.
  2. Collect the credentials from all students in your class. This process is simply an online form with 2 text fields: email and credentials.
  3. Prepare the assignment json and upload it to “credentials” blob container.

The format of assignment JSON file

In this example we simply want to test if two students have successfully created a vnet within their Azure Subscription.

Assignment vnet creation

If you want to define a assignment named "vnet" and the class with 2 students. graderUrl is the url of Azure Function running Nunit test and return xml result.  You just need to create and upload vnet.json as below.

 

 

 

 

 

{
  "graderUrl": "https://xxxx.azurewebsites.net/api/AzureGraderFunction",
  "students": [
    {
      "email": "xxx@vtc.edu.hk",
      "credentials": {
        "activeDirectoryEndpointUrl": "https://login.microsoftonline.com",
        "activeDirectoryGraphResourceId": "https://graph.windows.net/",
        "clientId": "fjfjlfjl;afjlafjal'fjalds;f'",
        "clientSecret": "lfakl;fkdf;kal;fkalfak;",
        "galleryEndpointUrl": "https://gallery.azure.com/",
        "managementEndpointUrl": "https://management.core.windows.net/",
        "resourceManagerEndpointUrl": "https://management.azure.com/",
        "sqlManagementEndpointUrl": "https://management.core.windows.net:8443/",
        "subscriptionId": "183966e3-8333-4764-9c74-a0eb4f883571",
        "tenantId": "7edb05ab-42e9-4348-b893-2ba625a350c6"
      }
    },
    {
      "email": "yyy@stu.vtc.edu.hk",
      "credentials": {
        "activeDirectoryEndpointUrl": "https://login.microsoftonline.com",
        "activeDirectoryGraphResourceId": "https://graph.windows.net/",
        "clientId": "xsxhskjdjksjdlsdjlksjdlsjd",
        "clientSecret": "gkjvkjv;ldjv'lvdvmdfhsdkfjsdl",
        "galleryEndpointUrl": "https://gallery.azure.com/",
        "managementEndpointUrl": "https://management.core.windows.net/",
        "resourceManagerEndpointUrl": "https://management.azure.com/",
        "sqlManagementEndpointUrl": "https://management.core.windows.net:8443/",
        "subscriptionId": "183966e3-8333-4764-9c74-a0eb4f883571",
        "tenantId": "7edb05ab-42e9-4348-b893-2ba625a350c6"
      }
    }
  ]
}

 

 

 

 

 

 

Support Environment

For grading, students we need to process that the grader has read only access credentials to the students Azure subscription, each student has to create a service principal permission for the app. 

For the engine, it is just a simple Azure Function app with consumption plan, This process has been developed so that it can even run inside Azure student subscription plan with very low cost. We have carried out a number of tests using this with Azure for Student subscription.

 

How to deploy and use it?

You need to use Visual Studio 2019 and follow the following video. How to deploy Azure Automatic Grading Engine 
Project GitHub Repo https://github.com/microsoft/AzureAutomaticGradingEngine

 

Conclusion

You can full control Microsoft Azure programmatically and educators can now spend time on preparing a better course instead of wasting on grading Azure assignment manually.

 

We have been using this kind of automatic grading tools for a few years. Students definatley benefit a lot both in technical and soft skills. Firstly, students must work honestly, seriously, and quickly. As a result, their technique is in industry standard. Also, using auto grading solutions is more fun for most students and you can treat this like a game. We have seen many students trying to get mark as fast as possible. Part of our process is that we do share their mark to employers and employers loves to refer that mark than the students GPA. 

 

Also, it is possible to use the engine for Azure Competition Game. For upcoming features we all add a game like dashboard and allow students to gamify the journey of leaning Azure. We also plan to add Chaos component and we can have trouble shooting assignment, but every student will get different bugs within certain scope.

Project collaborators include, Chan Yiu Leung (Hades), So Ka Chun, Lo Chun Hei, Ling Po Chu, Cheung Ho Shing and Pearly Law from the IT114115 Higher Diploma in Cloud and Data Centre Administration


About the Author

cyruswong_0-1629867240903.jpeg

Cyrus Wong is the senior lecturer of Department of Information Technology (IT) of the Hong Kong Institute of Vocational Education (Lee Wa... and he focuses on teaching public Cloud technologies. He is one of the Microsoft Learn for Educators Ambassador.

1 Comment
Co-Authors
Version history
Last update:
‎Aug 25 2021 11:28 PM
Updated by: