Typically Integrated microservice APIs are tested quite late in the Application lifecycle, where the application code which has already been merged into the main branch, is then deployed to an environment, after which the API tests are executed. Detection of failures at this stage means that your main branch is already in an unclean / un deployable state.
In this post we look at a Webvalidate based test harness which in many cases would enable us to validate that the changes introduced in a Pull Request (PR) do not break the Microservice API.
Our objective is to shift the microservice API tests left, and execute them as a part of PR validation itself, using Azure Pipelines. To enable this for a sample RESTful Microservice (backed by a document database) we need to build a PR pipeline / test harness which allows us to achieve these requirements:
For this post we will be looking at the Books service as the Microservice for which the APIs need to be tested during PR validation. The initial code for Books API has been taken from the dotnet samples and tweaked for purpose of this post.
It is a simple API supporting basic CRUD operations for Books (Add, Delete , Put, Get one, List etc). The datastore which this microservice relies on is a Mongodb collection.
Application configurations like mongodb connection string, database name and collection name are injected using environment variables. We later look at how these are injected as part of the PR Validation pipeline.
With a few minor modifications (which will be covered later in this post), the test harness can also work if the application fetches configurations using Default Azure Credentials, Azure App Config and Azure Key Vault.
The microsoft/webvalidate tool enables us to define tests cases and validations as simple JSON objects as shown below:
{
"path": "/api/books/60e304aac7b8d60001b2d3cd",
"verb": "Get",
"tag": "GetNonExistentBook",
"failOnValidationError": true,
"validation": {
"statusCode": 404,
"contentType": "application/problem+json;"
}
}
In this test case with tag "GetNonExistentBook" we assert that a request to get a book with id "60e304aac7b8d60001b2d3cd" would return a HTTP status code of 404 as book with that Id is not expected to exist. Complex validations as shown in the webvalidate samples can also be easily achieved.
The working code referenced in this post is available at the api-test-harness-webv github Repo . This includes the tweaked api application code as well as the dockerfile, docker-compose file, test harness scripts, test data, and webvalidate test cases.
Overview of the key folder and files in this repository:
Mongodb collection test data - /TestFiles/ApiTests/Data/BooksTestData.json
[
{
"_id": {
"$oid": "60e2e8fe7ed72f0001bf3a41"
},
"Name": "The Go Programming Language",
"Price": 20,
"Category": "Computer Programming",
"Author": "Alan Donovan"
},
{
"_id": {
"$oid": "60e2e8fe7ed72f0001bf3a46"
},
"Name": "Design Patterns",
"Price": 54.93,
"Category": "Computers",
"Author": "Ralph Johnson"
}
]
Webvalidate Test Cases - /TestFiles/ApiTests/TestCases/BooksTestCases.json
{
"requests": [
{
"path": "/api/books",
"verb": "POST",
"tag": "CreateBookValidRequest",
"failOnValidationError": true,
"body": "{\"Name\":\"Kubernetes Up and Running\",\r\n\"Price\":25,\r\n\"Category\":\"Computer Programming\",\r\n\"Author\":\"Adam Barr\"\r\n}",
"contentMediaType": "application/json-patch+json",
"validation": {
"statusCode": 201,
"contentType": "application/json",
"jsonObject": [
{
"field": "Id"
},
{
"field": "Name",
"value": "Kubernetes Up and Running"
},
{
"field": "Price",
"value": 25
}
]
}
},
{
"path": "/api/books",
"verb": "POST",
"tag": "CreateBookInvalidPrice",
"failOnValidationError": true,
"body": "{\"Name\":\"Kubernetes Up and Running\",\r\n\"Price\":\"twenty five\",\r\n\"Category\":\"Computer Programming\",\r\n\"Author\":\"Adam Barr\"\r\n}",
"contentMediaType": "application/json-patch+json",
"validation": {
"statusCode": 400,
"contentType": "application/problem+json;",
"jsonObject": [
{
"field": "errors",
"validation": {
"jsonObject": [
{ "field": "Price" }
]
}
}
]
}
},
{
"path": "/api/books/60e304aac7b8d60001b2d3cd",
"verb": "Get",
"tag": "GetNonExistentBook",
"failOnValidationError": true,
"validation": {
"statusCode": 404,
"contentType": "application/problem+json;"
}
},
{
"path": "/api/books/60e2e8fe7ed72f0001bf3a41",
"verb": "Get",
"tag": "GetExistingBook",
"failOnValidationError": true,
"validation": {
"statusCode": 200,
"contentType": "application/json",
"exactMatch": "{\"Id\":\"60e2e8fe7ed72f0001bf3a41\",\"Name\":\"The Go Programming Language\",\"Price\":20.0,\"Category\":\"Computer Programming\",\"Author\":\"Alan Donovan\"}"
}
}
]
}
PR Validation Azure Pipeline Yaml file - /ApiTestsAzurePipelines.yaml
.
.
steps:
- script: |
cd $(System.DefaultWorkingDirectory)
docker-compose -f BooksApi/dockerComposeBooksApiTest.yml build --no-cache --build-arg ENVIRONMENT=local
docker-compose -f BooksApi/dockerComposeBooksApiTest.yml up --exit-code-from webv | tee $(System.DefaultWorkingDirectory)/dc.log
displayName: "Execute API Tests"
- script: |
# Pass parameters: path to docker-compose log file, path to output Junit file, and path to scripts directory
bash $(System.DefaultWorkingDirectory)/TestFiles/scripts/webvToJunit.sh $(System.DefaultWorkingDirectory)/dc.log $(System.DefaultWorkingDirectory)/junit.xml $(System.DefaultWorkingDirectory)/TestFiles/scripts
displayName: "Convert Test Execution Log output to JUnit Format"
- task: PublishTestResults@2
displayName: 'Validate and Publish Component Test Results'
inputs:
testResultsFormat: JUnit
testResultsFiles: 'junit.xml'
searchFolder: $(System.DefaultWorkingDirectory)
testRunTitle: 'webapitestrestults'
failTaskOnFailedTests: true
There are 3 main steps in this PR validation Azure Pipeline file:
Docker Compose API Test Harness - /BooksApi/dockerComposeBooksApiTest.yml
version: '3'
services:
booksapi:
build:
context: .
dockerfile: ./Dockerfile
ports:
- '5011:80'
networks:
- books
# volumes:
# - ${HOME}/.azure:/root/.azure
environment:
- BookstoreDatabaseSettings__ConnectionString=mongodb://mongo:27017
- BookstoreDatabaseSettings__DatabaseName=BookstoreDb
- BookstoreDatabaseSettings__BooksCollectionName=Books
mongo:
container_name: books.mongo
image: mongo:4.4
networks:
- books
mongo-import:
image: mongo:4.4
depends_on:
- mongo
volumes:
- ../TestFiles:/testFiles/
- ../TestFiles/scripts/import.sh:/command/import.sh
- ../TestFiles/scripts/index.js:/command/index.js
networks:
- books
environment:
- MONGO_URI=mongodb://mongo:27017
- MONGO_DB=BookstoreDb
- MONGO_COLL=Books
- TEST_TYPE=ApiTests
entrypoint: /command/import.sh
webv:
image: retaildevcrew/webvalidate@sha256:183228cb62915e7ecac72fa0746fed4f7127a546428229291e6eeb202b2a5973
depends_on:
- mongo
- booksapi
volumes:
- ../TestFiles:/testFiles/
- ../TestFiles/scripts/executeTests.sh:/command/executeTests.sh
networks:
- books
environment:
- TEST_TYPE=ApiTests
- TEST_SVC_ENDPOINT=http://booksapi
- TEST_DATA_LOAD_DELAY=25
entrypoint: ["/bin/sh","/command/executeTests.sh"]
networks:
books:
.
Successfully built b312635a0c59
Successfully tagged booksapi_booksapi:latest
.
.
Creating books.mongo ...
Creating booksapi_booksapi_1 ...
Creating books.mongo ... done
Creating booksapi_mongo-import_1 ...
Creating booksapi_booksapi_1 ... done
Creating booksapi_webv_1 ...
Creating booksapi_mongo-import_1 ... done
Creating booksapi_webv_1 ... done
Attaching to books.mongo, booksapi_booksapi_1, booksapi_mongo-import_1, booksapi_webv_1
.
.
mongo-import_1 | 2021-07-12T07:23:29.882+0000 2 document(s) imported successfully. 0 document(s) failed to import.
.
.
webv_1 | {"date":"2021-07-12T07:23:55.3355122Z","verb":"POST","server":"http://booksapi","statusCode":201,"failed":false,"validated":true,"correlationVector":"MtKp\u002BU7CpE6VAJLRF/kGlQ.0","errorCount":0,"duration":550,"contentLength":136,"category":"","tag":"CreateBookValidRequest","path":"/api/books","errors":[]}
.
.
Stopping booksapi_mongo-import_1 ...
Stopping booksapi_booksapi_1 ...
Stopping books.mongo ...
Stopping booksapi_booksapi_1 ... done
Stopping booksapi_mongo-import_1 ... done
Stopping books.mongo ... done
Aborting on container exit...
Before testing the harness, we need to create an Azure Pipeline using out PR Validation Pipeline yaml file, and then configure this pipeline to execute as part of main branch build validation. For simplicity to cause the API test to fail let us modify the CreateBookValidRequest webvalidate test case to expect a Price of 26 in the response instead of 25, as shown in the commit
Once a Pull request is created, we should see the PR validation build kicking in, and then failing in a couple of minutes as shown:
Next we drill down into the pipeline and choose the Tests tab, which gives us an overview of the test execution. Where we see that 1 of the 4 tests has failed:
On clicking the failed test, the error details window shows us the reason of the error (Actual Price "25", Expected Price "26")
Since this is a required check for the PR, merging of the PR is blocked:
As we can see below if the PR validation is successful, the changes can be approved and merged:
There are a few limitations of this simplistic API testing during PR validation approach:
Thanks for reading this post. I hope you liked it. Please feel free to write your comments and views about the same over here or at @manisbindra
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.