I have been working with a University over the past few weeks helping them implement a new course on Performance Engineering which encompass a lot of DevOps practices, think the teaching and awareness of DevOps is a excellent, within Industry there are real consequences if an organisation is unable to deliver apps or services quickly or builds a completely wrong implementation from the start. So I was very excited to be involved in the course design and wanted to share some of the outcomes:
Some interesting Research stats
Over 40% of software implementations end up getting reworked because they don’t meet the users’ original requirements or these requirements have been misunderstood
The average cost of one hour downtime of a customer-facing app is calculated at 100.000 dollars per hour – and this does not take into account the damage to reputation, which can be even greater.
Fixing such production issues takes on average 200 minutes per incident
Agile Teams
But stats also indicate that over three quarters of development teams have adopted Agile methodologies today, enabling them to develop faster. But this can present a number of problems for the team, a simple lack of collaboration between dev and ops can have substantial impact on a company’s bottom line and success. DevOps is not a new concept – it’s been around for a couple of years and it grew out of the Agile movement. A converged DevOps lifecycle gives you the ability to execute on ideas quickly and iterate on feedback rapidly, with minimum friction while at the same time maintaining the highest level of quality
Key goals of DevOps
1. Shorten your development cycles and deliver better apps, more frequently
2. Achieve faster development cycles with state-of-the art DevOps tools and an agile, end-to-end, tools-to-platforms solution
3. Streamline application delivery and boost the productivity and quality of any mobile project with a full range of integrated, mobile-specific QA tools, including test management, exploratory testing, performance testing, automated UI testing on thousands of real devices, and beta testing
4. Close the loop between development and production, fix bugs faster and evolve your application over time with insights and learnings captured directly in production.
One of the key things around DevOps in the understanding of tools, services and technologies
DevOps and Tooling
There are two key views on this one from a complete Open Source Tooling perspective where organisations have the following technology and services in operation
Or those who are working around the Microsoft technology stack
Both are perfect solution to organisations and off course a hybrid approach of technologies is also suitable.
Course Structure
The academic or TA – Becoming/acting as Customer for testing of users services
The academic has a dedicated Azure subscription and implements accounts for students within the course.
The academic has a dedicated single point for assessment/validation – i.e. checking up-time which student has the most available service
The academic has central service which can drop loads test to the students services as and when required, simulating real life loading and scenarios.
Testing of the Student provided services be Working hours i.e 9am – 5pm to allow cost/scaling/predicative modelling
Azure DNS provisioned in the academics Subscription – This subscription will need delegated rights to students record to ensure student IP address are correct for continuous testing/pinging as each machine provisioned will get a new IP address.
Potential Test Scenarios
Availability
Geographic Response Time
Geographic Load
CDN static file web test.
SLA failure - Failed X concurrent pings – Fatal Error – Under X pings – Transient Error
Testing of End Points.
Access Web Resources Known and Not Known
Database provision of user – New Accounts i.e Database back end requirements and authentication
Semester 1
Start with VM Dev/Test to Production Environment
Implement a basic Web Service/App
Tier Application Learning – i.e Start with single tier and work up through the phases
Implement monitoring/benchmarking/Bugs in potential Apps which can be discovered through logging
Identifying issues through monitoring
Financial Modelling and Cost predictions i.e Autoscaling cost for month 1 $300 for Month 2 $200 Month 3 $100
Development of SLA – Which can be compared with availability test
AB testing – continuous service ping to replicate Live Service
Alerting and Monitoring
Semester 2
Scale out this services to multiple VMs Scale up/out MutiTier – i.e Benefits/Risks/Costs
Implement Load Balancer
CDN, REDIS
Implement Multi Tier app with State Less app
Persistent session
Availability i.e simulated work loads i.e scale in/out/up/down
How does this effect your metric Availability/Budget/Allocations
Financial Modelling and Cost predictions
Development of SLA – Which can be compared with Ping Tests
AB testing – continuous service ping to replicate Live Service
Alerting and Monitoring
Semester 3
Scale out to SaaS or Containers..
SLA/Quality of Services
Metrics and Informed Decisions from tooling
How does this effect metrics budgets/resource
Comparison of Services offerings
Financial Modelling and Cost predictions – Scale Sets/Maximising Budget
Development of SLA – Which can be compared with availability tests
AB testing – continuous service ping to replicate Live Service
Alerting and Monitoring
Evaluation and assessment
From each of the semester evaluation and activities, there should a quantified outcome of performance from both a implementation and operations perspective and students can easily be stack ranked in terms of effectiveness of meeting the business requirements and brief.
DevOps from a organisation perspective
Every DevOps discussion in any organization has to focus on 3 elements: people/culture, process, and tools so for anyone who has used or studied ITIL this can be related simply into ITIL processes.
1. People/culture is about bringing people together in a collaborative fashion that enables multi-discipline teams to work together more easily, share common goals, and focus on identifying areas for improvement
2. Process is defined as finding ways to increase efficiency, streamline feedback and eliminate waste All software development processes should be focused on delivering business value faster - focusing on the flow of value
3. Tools are arguably the least important part of this discussion – they support processes and facilitate continuous feedback and experimentation Tools are a way of executing the strategy and vision that an organization has formulated with regards to their journey to Dev.
So What is the DevOps Lifecycle
Every project starts with an idea - and a plan how to turn this idea into reality
Once the project is under way, work needs to be managed and progress should be tracked continuously
Stage 1.
The goal of this stage is to build high-quality apps or service the users expect the best experience everywhere and do not tolerate bugs
For mobile apps each platform has different presentation styles, interaction styles and software stacks
Devices have different screen sizes, input modes and hardware capabilities
New devices and OS versions are introduced multiple times per year
Network connectivity and power levels can fluctuate widely in typical usage scenarios
New consumer applications regularly extend and revise the standards and set the bar higher for good mobile applications
Many businesses have already started their journey to mobile with consumer apps some time ago, so they are familiar with some of the biggest challenges
Stage 2.
Once a sprint starts, developers turn great ideas into reality by writing code and building functionality
Before checking in code to source control, unit tests are created and run to ensure that quality bars are met
Ideally, every check-in by a developer triggers a build and automated build verification tests ensure that the end result is a working iteration of the product
Building high-quality apps is hard – but users expect the best experience everywhere and do not tolerate bugs
Each platform has different presentation styles, interaction styles and software stacks
Devices have different screen sizes, input modes and hardware capabilities
New devices and OS versions are introduced multiple times per year
Network connectivity and power levels can fluctuate widely in typical usage scenarios
New consumer applications regularly extend and revise the standards and set the bar higher for good mobile applications
Building high-quality apps is hard – but users expect the best experience everywhere and do not tolerate bugs
Each platform has different presentation styles, interaction styles and software stacks
Devices have different screen sizes, input modes and hardware capabilities
New devices and OS versions are introduced multiple times per year
Network connectivity and power levels can fluctuate widely in typical usage scenarios
New consumer applications regularly extend and revise the standards and set the bar higher for good mobile applications
Step 3.
Complete build automation is a key requirement.
Developers want to create and manage build processes to automatically compile and test applications in the cloud or on premises, either on demand or as part of an automated continuous integration strategy. They want to link builds to test results, work items, code and releases and stop wondering if a bug was fixed in a particular build. By associating work items to code, the work items are listed in the build summary along with code changes and test results.
Step 4.
Has the app has been released to production but the project is not over?
Does the app/service provide high availability, performance and improved business results application monitoring on services and devices and data correlation for a 360° view
Having monitoring infrastructure in place enables you to learn and understand how users use your app, how it reacts and fix issues and bugs faster
Resources and Tools
Aziure Patterns and Practices https://docs.microsoft.com/en-gb/azure/guidance/
C’#, C++, .NET coding standards http://1code.codeplex.com/releases/84683/download/357518
Using JMeter for Performance testing https://channel9.msdn.com/Events/Visual-Studio/Launch-2013/qe103
Using Application Insight and Tom Cat - https://blogs.technet.microsoft.com/random_happy_dev_thoughts/2014/03/25/application-insights-for-java-quickstart-for-apache-tomcat/
Application Insight SDK for Java Apps - https://blogs.msdn.microsoft.com/visualstudioalm/2015/03/06/introducing-the-application-insights-sdk-for-java/
Auto Scaling VMs under IaaS- https://blogs.msdn.microsoft.com/kaevans/2015/02/20/autoscaling-azurevirtual-machines/
Azure 101 Workshops - https://github.com/MSFTImagine/computerscience/tree/master/Workshop
Azure Blended learning http://mva.microsoft.com
Load Balancers
PaaS Azure Load Balancers - https://azure.microsoft.com/en-us/services/load-balancer/
IaaS Traffic Manager https://azure.microsoft.com/en-us/services/traffic-manager/
Getting started guides https://docs.microsoft.com/en-gb/azure/#howto_load_balance
Example of Docker and Containers - Azure Containers and Docker ( https://mva.microsoft.com/en-US/training-courses/azure-developer-workshop-storage-cognitive-ml-stream-analytics-containers-and-docker--17033?l=A0uXKhPND_5606218965
If you testing web apps/web services on Azure you may find you get a timeout error http status 500 or see the http request queue on Azure is very mixed load you can resolve this by turning off ARR, ARR cleverly keeps track of connecting users by giving them a special cookie (known as an affinity cookie), which allows it to know, upon subsequent requests, to which server instance they were talking to. This way, we can be sure that once a client establishes a session with a specific server instance, it will keep talking to the same server as long as his session is active. How to disable ARR https://azure.microsoft.com/pt-br/blog/disabling-arrs-instance-affinity-in-windows-azure-web-sites/