Frequent product releases are important to the SaaS business model and provide additional functionality to help retain customers. Since in a multi-tenant environment any changes impact all customers that use the service, all production changes and software releases need to be of the highest quality possible and any problems need to be resolved quickly. DEVOPS is often described as something that can help achieve the combination of frequent and high-quality releases. DEVOPS is commonly defined as an emerging set of principles, methods and practices for communication, collaboration and integration between software development and IT operations. (endnote 1) Although it is in use in many organizations, it is important to understand that DEVOPS is still evolving.
The problem that DEVOPS helps to solve is that three important organizations in a SaaS business have conflicting goals. Classically the operations organization is in the role of protecting production based on their responsibilities of managing availability, performance and security. Since we know that a very high percentage of production problems are caused by the introduction of software or configuration changes, this is not misplaced concern.
Software development has the responsibility of regularly introducing new features with less direct accountability for availability, performance and security. Likewise, the operations organization is not accountable for seeing that new features are introduced. The quality assurance organization (QA) is responsible for testing the newly developed software to be sure that new and old functionality works correctly and often to make sure that the performance is not adversely affected. But, they are not accountable to get new software deployed in a certain timeframe.
Given the conflicting goals, these organizations, all part of the process of deploying software functionality for customers, often are in conflict and sometimes there is significant distrust and lack of mutual respect. This in turn does not provide an optimal environment for frequent and high-quality product releases.
Agile software development, with its incremental development, rapid changes and potential for numerous releases, has provided the ability for software development to produce frequent and regular deployable software with new features. This in combination with infrastructure virtualization provides the technical and organizational backdrop for more frequent releases. However, the problem has been that the operations organization, in many cases, has not been able to keep pace with these changes. The DEVOPS philosophy and the addition of new types of configuration management tools have now made more frequent changes possible.
The path to DEVOPS
The philosophy of DEVOPS is based on many of the principles developed by the “Lean” movement in manufacturing and other operations and is focused on automation, process improvement and removing process defects. To be successful, the Lean movement had to focus not only on technical issues but also on organizational issues; this is also true of DEVOPS.
DEVOPS is a journey and, as in any operational improvement activity, it is not an end state but a continuous process of improvement. Some of the goals of DEVOPS include:
- Fast deployment of application software
- Using software to control the virtual infrastructure and configurations
- Easy and automatic rollback to previous known good states
- Lower defects
- Automated remediation of changes.
These goals are archived through tight process control and process improvement generally using automation. The general philosophy is to reduce the risk of changes by making many small and regular changes in a predictable and controlled manner.
DEVOPS requires a set of tools, processes, and a culture supportive to the initiative. Having one or two of these is not sufficient, and many people that have been directly involved in successful implementations feel that the right culture is the most important item.
Culture. A culture supportive to the DEVOPS effort needs to have shared information, respect and trust between individuals, and a problem-solving versus assessing-blame approach to problems.
Shared information comes from shared tools. Systems like source control, bug tracking, and problem tracking all need to be accessible and used by software development, QA and operations. Strict source control management is critical since software deployment is driven from there. Any private systems used only by one organization need to replaced or changed so that everyone has the same baseline information.
Respect and trust is important. Since all three of the organizations are now focused on one continuous process from development to deployment and support, everyone must work together and trust each other. Any time spent on political or other non-productive agendas can have a significant detrimental effect.
A problem-solving focus is also an important part of the culture. The days of the question “who screwed up? Operations? Or was it a software bug?” have to be over. The focus needs to be on resolving issues wherever they are in the process. Focusing on what is important to the customer is always a good way to do this. The sooner everyone is focused on solving the problem, the sooner it will be solved. Assessing blame is a tremendous waste of time and often results in future withholding of information and a loss of trust and respect.
The right incentives are an important part of the culture. Shared metrics, targets, and incentives work the best. If all three of the groups are focused and incented on metrics like customer-visible problems, availability, security and release timing and functionality as shared metrics, then they likely will work successfully together to achieve excellent results in all those areas. For example, this team of experts can work out the tradeoffs between security and functionality versus having two organizations fighting about that.
Tools. In a 2011 survey of DEVOPS trends, done by Replay Solutions, Inc., of both SaaS providers and enterprise organizations that had implemented DEVOPS, 55% felt that tools were very important, and another 37% felt that they were somewhat important.
Tools recommended in a DEVOPS environment include tools for provisioning, controlling, managing sources, and monitoring the IaaS, cloud, or virtual machine environment. Many of the tools are either likely already in use within a SaaS organization or are available as open source.
Configuration management is an area that has changed a lot with virtualized environments. The ability to control configurations using software has provided the opportunity for new tools such as Opscode Chef and PuppetLabs, both of which are tools for automating configuration management and virtual environment deployment.
Respondents to the same survey referenced above indicated that the most important tools are defect tracking (75%), source control (69%), help desk (49%) and release/configuration management (45%). Referring back to the necessary culture, it’s important that all members of the team have at least some understanding of the use of the tools and have access to the same information.
Processes. Many of the regular software development, QA, and operations processes are impacted by DEVOPS with the goal to make them highly automated, shared and defect free. Build processes, testing and QA processes, source control processes, change management, problem resolution and security and application deployment are all examples of processes that likely will and should be impacted by DEVOPS.
DEVOPS in use by SaaS companies
DEVOPS has been used by various types of SaaS businesses to make business improvements. Constant Contact, a provider of email and social media marketing tools for small and medium-sized businesses (SMBs), developed a new application for social media analysis. They did the development with DEVOPS in mind using Cassandra as a database technology. They view that this combination allowed them to reduce their infrastructure deployment time from four weeks to four hours, their development to deployment time from nine months to three months, and their cost from $millions to $150 thousand. (endnote 2)
Liveperson, a provider of online chat, call center, and Web-based engagement solutions for large enterprise customers, wanted to make operational improvements through the use of DEVOPS. They reduced the time and resource to service their applications by 85%, and had faster problem resolution and faster new feature releases. (endnote 3)
GoTime, an online city guide, was able to rebuild their entire production environment from scratch in an hour and have found that their automated infrastructure using DEVOPS concepts saves them time and money. (endnote 4) A UK-based online gaming site, 888.com, uses DEVOPS to help them manage their 85 applications, which are used by 25 million users. (endnote 5) The photo site Flickr uses DEVOPS to update their software up to 10 times per day. (endnote 6) Spotify, a music streaming service, uses DEVOPS concepts to help them with producing their frequent releases.
The point of the variety of examples is to demonstrate that this is not a concept limited to just SMBs, enterprise, gaming, or streaming businesses but is really able to be used across these segments. The common factor is the business need for frequent and high-quality software releases.
Is DEVOPS right for you?
To understand whether DEVOPS could help your business, use the following as guidelines. DEVOPS could help your business if it has at least some of the following characteristics:
- Cloud or virtual machine-based infrastructure
- Complex infrastructure
- Application releases more often than four per year
- NoSQL database
- Too much operations work
- Error-prone application updates
- Significant future growth
If you only do two releases per year and they go well, then at this point DEVOPS is probably not something that will help you.
Since DEVOPS will require an investment of time, as with any significant business project, a business case should be developed. The benefits side of the business case generally include things like less actual time, lower elapsed time and reduced errors for setting up and taking down development, QA, and production environments and deployment and rollback of application software revisions.
These benefits generally result in higher customer satisfaction, lower costs, and the ability to deal with increased complexity and growth. On the cost side of the business case, as I mentioned earlier, most of the tools are either already in place or are open source tools. The primary cost is the effort involved by the development, QA and operations staff along with the associated management support for the overall culture to implement automation and to understand and improve processes.
Assuming that you meet some of the criteria outlined above, you should consider learning more about DEVOPS. It quickly becomes a highly technical topic and the people directly involved will have to understand how it might apply to your processes and business. From that point, a business case can be developed along with the associated metrics that you want to improve.
For example, if you want to reduce the number of customer-visible production problems after a new release, then a metric to focus on that is appropriate. Or your business focus might be to be able to make software releases on a weekly basis, so you would use a metric like that. From that point. you can develop an implementation plan and regularly review the metrics you’ve developed. Don’t get discouraged if you don’t get instant results; as outlined earlier this is a process and a future way of life.
DEVOPS can be an important part of a substantially improved way to release creating and managing frequent high-quality SaaS product releases. It is in use by many SaaS companies that have seen substantial improvements. Even if your results are not as dramatic, any improvement in release quality, reduced effort and faster releases can have a substantial impact.
Paul Ressler is a consultant specializing in service delivery for SaaS, Cloud Computing, and Managed Services. As the principal of The Cirrostratus Group, Paul helps his clients improve customer satisfaction, raise service margins, introduce profitable new services, and transition to the SaaS business model.
1. DEVOPS. Retrieved from Wikipedia: http://en.wikipedia.org/wiki/DevOp
2. Connors, D. Retrieved from Slideshare: http://www.slideshare.net/daveconnors/cassandra-puppet-scaling-data-at-15-per-month
3. Retrieved from Niliosoft: http://www.noliosoft.com/customers/live-person
4. Retrieved from Opscode: http://www.opscode.com/customers/gotime/
5. Retrieved from Noliosoft: http://www.noliosoft.com/customers/gaming-888
6. Allspaw, J. Retrieved from Slideshare: http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr