Blog Post

Educator Developer Blog
8 MIN READ

DevOps and Toolsets – Teaching Performance Engineering

Lee_Stott's avatar
Lee_Stott
Icon for Microsoft rankMicrosoft
Mar 21, 2019
First published on MSDN on Feb 28, 2017

I have been working with a University over the past few weeks helping them implement a new course on Performance Engineering which encompass a lot of DevOps practices,  think the teaching and awareness of DevOps is a excellent, within Industry there are real consequences if an organisation is unable to deliver apps or services quickly or builds a completely wrong implementation from the start. So I was very excited to be involved in the course design and wanted to share some of the outcomes:

Some interesting Research stats

Over 40% of software implementations end up getting reworked because they don’t meet the users’ original requirements or these requirements have been misunderstood

The average cost of one hour downtime of a customer-facing app is calculated at 100.000 dollars per hour – and this does not take into account the damage to reputation, which can be even greater.

Fixing such production issues takes on average 200 minutes per incident

Agile Teams

But stats also indicate that over three quarters of development teams have adopted Agile methodologies today, enabling them to develop faster. But this can present a number of problems for the team, a simple lack of collaboration between dev and ops can have substantial impact on a company’s bottom line and success. DevOps is not a new concept – it’s been around for a couple of years and it grew out of the Agile movement. A converged DevOps lifecycle gives you the ability to execute on ideas quickly and iterate on feedback rapidly, with minimum friction while at the same time maintaining the highest level of quality

Key goals of DevOps

1. Shorten your development cycles and deliver better apps, more frequently

2. Achieve faster development cycles with state-of-the art DevOps tools and an agile, end-to-end, tools-to-platforms solution

3. Streamline application delivery and boost the productivity and quality of any mobile project with a full range of integrated, mobile-specific QA tools, including test management, exploratory testing, performance testing, automated UI testing on thousands of real devices, and beta testing

4. Close the loop between development and production, fix bugs faster and evolve your application over time with insights and learnings captured directly in production.

One of the key things around DevOps in the understanding of tools, services and technologies

DevOps and Tooling

There are two key views on this one from a complete Open Source Tooling perspective where organisations have the following technology and services in operation

Or those who are working around the Microsoft technology stack

Both are perfect solution to organisations and off course a hybrid approach of technologies is also suitable.

Course Structure

The academic or TA – Becoming/acting as Customer for testing of users services

The academic has a dedicated Azure subscription and implements accounts for students within the course.

The academic has a  dedicated single point for assessment/validation – i.e. checking up-time which student has the most available service

The academic has central service which can drop loads test to the students services as and when required, simulating real life loading and scenarios.

Testing of the Student provided services be Working hours i.e 9am – 5pm to allow cost/scaling/predicative modelling

Azure DNS provisioned in the academics  Subscription – This subscription will need delegated rights to students record to ensure student IP address are correct for continuous testing/pinging as each machine provisioned will get a new IP address.

Potential Test Scenarios

Availability

Geographic Response Time

Geographic Load

CDN static file web test.

SLA failure - Failed X concurrent pings – Fatal Error – Under X pings – Transient Error

Testing of End Points.

Access Web Resources Known and Not Known

Database provision of user – New Accounts i.e Database back end requirements and authentication

Semester 1

Start with VM Dev/Test to Production Environment

Implement a basic Web Service/App

Tier Application Learning – i.e Start with single tier and work up through the phases

Implement monitoring/benchmarking/Bugs in potential Apps which can be discovered through logging

Identifying issues through monitoring

Financial Modelling and Cost predictions i.e Autoscaling cost for month 1 $300 for Month 2 $200 Month 3 $100

Development of SLA – Which can be compared with availability test

AB testing – continuous service ping to replicate Live Service

Alerting and Monitoring

Semester 2

Scale out this services to multiple VMs  Scale up/out MutiTier – i.e Benefits/Risks/Costs

Implement Load Balancer

CDN, REDIS

Implement Multi Tier app with State Less app

Persistent session

Availability i.e simulated work loads i.e scale in/out/up/down

How does this effect your metric Availability/Budget/Allocations

Financial Modelling and Cost predictions

Development of SLA – Which can be compared with Ping Tests

AB testing – continuous service ping to replicate Live Service

Alerting and Monitoring

Semester 3

Scale out to SaaS or Containers..

SLA/Quality of Services

Metrics and Informed Decisions from tooling

How does this effect metrics budgets/resource

Comparison of Services offerings

Financial Modelling and Cost predictions – Scale Sets/Maximising Budget

Development of SLA – Which can be compared with availability tests

AB testing – continuous service ping to replicate Live Service

Alerting and Monitoring

Evaluation and assessment


From each of the semester evaluation and activities, there should a quantified outcome of performance from both a implementation and operations perspective and students can easily be stack ranked in terms of effectiveness of meeting the business requirements and brief.

DevOps from a organisation perspective

Every DevOps discussion in any organization has to focus on 3 elements: people/culture, process, and tools so for anyone who has used or studied ITIL this can be related simply into ITIL processes.

1. People/culture is about bringing people together in a collaborative fashion that enables multi-discipline teams to work together more easily, share common goals, and focus on identifying areas for improvement

2. Process is defined as finding ways to increase efficiency, streamline feedback and eliminate waste All software development processes should be focused on delivering business value faster - focusing on the flow of value

3. Tools are arguably the least important part of this discussion – they support processes and facilitate continuous feedback and experimentation Tools are a way of executing the strategy and vision that an organization has formulated with regards to their journey to Dev.

So What is the DevOps Lifecycle

Every project starts with an idea - and a plan how to turn this idea into reality

Once the project is under way, work needs to be managed and progress should be tracked continuously

Stage 1.

The goal of this stage is to build high-quality apps or service the users expect the best experience everywhere and do not tolerate bugs

For mobile apps each platform has different presentation styles, interaction styles and software stacks

Devices have different screen sizes, input modes and hardware capabilities

New devices and OS versions are introduced multiple times per year

Network connectivity and power levels can fluctuate widely in typical usage scenarios

New consumer applications regularly extend and revise the standards and set the bar higher for good mobile applications

Many businesses have already started their journey to mobile with consumer apps some time ago, so they are familiar with some of the biggest challenges

Stage 2.

Once a sprint starts, developers turn great ideas into reality by writing code and building functionality

Before checking in code to source control, unit tests are created and run to ensure that quality bars are met

Ideally, every check-in by a developer triggers a build and automated build verification tests ensure that the end result is a working iteration of the product

Building high-quality apps is hard – but users expect the best experience everywhere and do not tolerate bugs

Each platform has different presentation styles, interaction styles and software stacks

Devices have different screen sizes, input modes and hardware capabilities

New devices and OS versions are introduced multiple times per year

Network connectivity and power levels can fluctuate widely in typical usage scenarios

New consumer applications regularly extend and revise the standards and set the bar higher for good mobile applications

Building high-quality apps is hard – but users expect the best experience everywhere and do not tolerate bugs

Each platform has different presentation styles, interaction styles and software stacks

Devices have different screen sizes, input modes and hardware capabilities

New devices and OS versions are introduced multiple times per year

Network connectivity and power levels can fluctuate widely in typical usage scenarios

New consumer applications regularly extend and revise the standards and set the bar higher for good mobile applications

Step 3.

Complete build automation is a key requirement.

Developers want to create and manage build processes to automatically compile and test applications in the cloud or on premises, either on demand or as part of an automated continuous integration strategy. They want to link builds to test results, work items, code and releases and stop wondering if a bug was fixed in a particular build. By associating work items to code, the work items are listed in the build summary along with code changes and test results.

Step 4.

Has the app has been released to production but the project is not over?

Does the app/service provide high availability, performance and improved business results  application monitoring on services and devices and data correlation for a 360° view

Having monitoring infrastructure in place enables you to learn and understand how users use your app, how it reacts and fix issues and bugs faster

Resources and Tools

Aziure Patterns and Practices https://docs.microsoft.com/en-gb/azure/guidance/

C’#, C++, .NET coding standards http://1code.codeplex.com/releases/84683/download/357518

Using JMeter for Performance testing https://channel9.msdn.com/Events/Visual-Studio/Launch-2013/qe103

Using Application Insight and Tom Cat - https://blogs.technet.microsoft.com/random_happy_dev_thoughts/2014/03/25/application-insights-for-java-quickstart-for-apache-tomcat/

Application Insight SDK for Java Apps - https://blogs.msdn.microsoft.com/visualstudioalm/2015/03/06/introducing-the-application-insights-sdk-for-java/

Auto Scaling VMs under IaaS- https://blogs.msdn.microsoft.com/kaevans/2015/02/20/autoscaling-azurevirtual-machines/

Azure 101 Workshops - https://github.com/MSFTImagine/computerscience/tree/master/Workshop

Azure Blended learning http://mva.microsoft.com

Load Balancers

PaaS Azure Load Balancers - https://azure.microsoft.com/en-us/services/load-balancer/

IaaS Traffic Manager https://azure.microsoft.com/en-us/services/traffic-manager/

Getting started guides https://docs.microsoft.com/en-gb/azure/#howto_load_balance

Example of Docker and Containers - Azure Containers and Docker ( https://mva.microsoft.com/en-US/training-courses/azure-developer-workshop-storage-cognitive-ml-stream-analytics-containers-and-docker--17033?l=A0uXKhPND_5606218965

If you testing web apps/web services on Azure you may find you get a timeout error http status 500 or see the http request queue on Azure is very mixed load you can resolve this by turning off ARR, ARR cleverly keeps track of connecting users by giving them a special cookie (known as an affinity cookie), which allows it to know, upon subsequent requests, to which server instance they were talking to. This way, we can be sure that once a client establishes a session with a specific server instance, it will keep talking to the same server as long as his session is active. How to disable ARR https://azure.microsoft.com/pt-br/blog/disabling-arrs-instance-affinity-in-windows-azure-web-sites/

Updated Mar 21, 2019
Version 2.0
  • Joseve's avatar
    Joseve
    Copper Contributor

    Hi Lee,

     

    Can you disclose your sources for Value Delivery Challenges Slide?

    Specifically these quotes

    40% of implementations end up getting reworked because they don’t meet the users’ original requirements

    41% of development budgets for software, IT staff and external professional services will be consumed by poor requirements

    The average cost of one-hour downtime of a customer-facing app is calculated at $100K per hour

    A bug caught in production ends up costing 100x more than if the same bug was found earlier in the development cycle

     

    thank you

    Jose