Amazon’s CTO Werner Vogel recently posted about the challenge of choosing the right metrics for cloud computing. He begins with the following observation where he introduces the idea of Return on Agility:
“In the many meetings with customers in which I have done a deep dive on their architecture and applications to help them create an accurate cost picture, I have observed two common patterns: 1) It is hard for customers to come to an accurate Total Cost of Ownership (TCO) calculation of an on-premise installation and 2) they struggle with how to account for the “Return on Agility”; the fact that they are now able to pursue business opportunities much faster at much lower costs points than before.”
In our intro on Agile Cloud Development, we articulated why we think this is the future of software development. Today we’ll kick off the Dev in the Cloud series with a look at Continuous Integration (CI) and the cloud’s impact on this popular agile development practice. We’ll explain how CI is the essential building block for Continuous Deployment, the secret sauce for start-ups from Facebook to Netflix. We’ll also highlight the marketplace for the growing number of CI products hosted in the cloud. While some vendors today are offering hosted solutions for just CI, others are including CI as part of their more comprehensive developer Platform as a Service (PaaS) that manages the entire build-test-deploy process.
What is Continuous Integration?
Continuos Integration is the practice of integrating software very often, typically on any change to an artifact in the source repository. Anytime a team members commits a change to the repository, continuous integration ensures their change was safe and the software can be successfully built, tested and deployed.
CI is the software industry’s version of Toyota’s Stop the Line principle, where work is stopped immediately if a quality problem is found. Because code integration is often a manual, error prone effort, especially on large systems or those that build infrequently, the traditional thinking has been to do it less often in order to avoid the cost of integrating (keep costs low). CI runs counter to this and adopts opposite philosophy, “if it’s hard to do, do it more often”. This means you should integrate more often, not less often. Taken to the extreme this means on on every change to the repository.
Doing this without automation would be prohibitively expensive for most teams. But by leveraging automation to detect changes in repositories and subsequently execute a script that downloads, compiles, packages, tests and deploys the software – the cost to complete this cycle dramatically drops and you get a wealth of benefits.
Continuous Integration is the key building block for Continuous Delivery, the practice of keeping your application in a state where it is always possible to deploy into production. This in turn is the building block for Continuous Deployment, the practice of deploying every change into production. These capabilities provide the business agility start-ups like Facebook, Netlflix and Groupon rely on to deploy changes to their products daily to continually improve their customer’s experience.
The primary benefits of CI for organizations and teams include:
Improved Quality – Quality is primarily improved through reduced time to detect and resolve issues. When CI is in place teams find and fix issues quickly and generally “keep the build green” throughout development. When issues do occur, teams “stop the line” and address integration defects right away. Automation of most all key tasks to build and deploy helps reduce human error and improve repeatability.
Faster Release Cycles – Teams using CI can release code faster because the code always stays in a deployable state. Teams using Continuous Deployment take this concept to the extreme, deploying releases to production everyday by automating all tasks from source repository until running in production.
Improved Velocity – CI helps teams achieve a higher velocity (throughput) but automating the mundane development tasks so teams can focus more of their time on building new capabilities. When used in conjunction with automated regression testing, CI can also shorten the testing window at the end of each iteration.
Scalability – How do you get 20, 50 or 100+ agile developers all actively working on the same codebase? Without CI, this is very difficult because of the rapid pace of change to the codebase. With CI, your teams can scale as your business scales without sacrificing quality and speed.
Reduced Costs – This comes as a by-product of the benefits above. CI can lower the development and maintenance costs through automation and improved quality.
Improved Culture – Perhaps the biggest impact on agile teams is the improved culture of quality that’s instilled through CI. Quite simply, CI becomes the heartbeat of development and a wonderful metaphor for the overall health of the project or product. Teams using CI take pride in consistently producing high quality software that always passes the tests and is ready for deployment.
In addition to these, CI in the cloud offers a few more:
Faster Startup – No more hassles with provisioning a new build server, downloading software, installing, configuring and managing your own CI environments. Not to mention patches, upgrades and maintenance. All of these services can be provided by CI products designed to run in cloud either stand-alone or within broader development platform as a service tools.
Elasticity – Need more resources to build, test and deploy your applications faster? Why not leverage the cloud’s elastic benefits to provision these on-demand and greatly reduce the time to run large builds or regression test suites.
Cloud Computing Impact
Continuous Integration came from the XP community with Kent Beck and Martin Fowler as early advocates. Today it has grown into one of the most common development practices in agile development with a marketplace of tools for doing continuous integration with a variety of technology platforms. As development and deployment tools have moved toward the cloud, so have CI products.
While most CI products are still designed to download and run in locals environments, the cloud has created a new breed of products that provide CI services on hosted platforms that teams can began using quickly. Now you can avoid the hassles, costs and time of procuring a new environment, downloading, installing and configuring the CI software. Now it’s hosted in the cloud for you, just configure and go. Need more horsepower to speed up build times? Not a problem, some of these products can leverage the cloud’s elasticity to provision extra build resources on-demand and even parallel the build tasks.
The market for cloud-based CI solutions continues to grow as many established CI vendors now offer hosted versions of their products. Many of these started life as downloadable CI products but are now hosted in the cloud as a part of a larger integrated development Platform as a Service (dPaaS) solutions.
These solutions integrate all parts of software development and delivery including requirements and project management, source control, building, testing, release management, issue tracking and collaboration from a single cloud-based product. Larger dPaas solutions that include CI include Bamboo, CloudBees, CloudForge and Electric Commander. These solutions are focused on enterprise customers and their needs whereas focused CI solutions for the Ruby (tddium) and iOS (hostedci) markets are primarily focused on the developer community and their needs.
Ruby Focus and Support
Integrated Automated Testing
In addition to these hosted solutions above, downloadable CI products exist for private clouds or internal virtualized environments. Popular ones include Jenkins, Anthill Pro, CruiseControl, go, Visual Studio TFS 2010, TeamForge, BuildMaster, Mojo and Buildbot. For those teams not ready to move entirely to a hosted CI solution, many are still migrating internal build servers running CI to Amazon EC2 instances for improved management and reduced costs.
For those wanting to learn more about Continuous Integration, I recommend these resources:
Welcome to Agile Cloud Development, the newest series in the Virtualization Practice that brings analysis to the methods and tools at the intersection of agile development and cloud computing. We believe together these are the future of software delivery for global organizations of all sizes in all industries, especially those wanting to increase business agility, reduce costs and ultimately improve their customer’s experience.
In this series introduction we’ll cover what is Agile Cloud Development, why it’s important and how it’s unique.
What is Agile Cloud Development?
First, let’s cover the basics. The term agile is an umbrella term for a set of related methods with four common principles as defined in the agile manifesto. It’s been a formal movement since 2001 and since then has transformed the way software is developed and delivered. Faster time to market, higher quality, improved productivity and happier teams are some of the common benefits agile organizations enjoy. The Standish Group in 2011 stated, “The agile process is the universal remedy for software development project failure”.
For these and many other reasons we believe agile is the future for how most all software will be created and delivered to customers. The most popular agile methods include Scrum and Extreme Programming (XP) but there are others. With respect to this series, we’ll cover the aspects of agile development that integrate with cloud computing.
In addition to agile, within this series we’ll also include newer methods based on Lean principles in particular Kanban, which is enjoying success in not only software delivery but also IT operations. Kanban’s focus on continuous flow and limited work-in-progress is a natural fit for organizations doing continuous delivery – a capability that’s built on virtualization and cloud computing.
Also within this series we’ll explore DevOps and the growing community of methods and tools for bridging the gap between agile development teams and (typically not agile) IT operations teams. Many of these tools integrate with virtualized and cloud environments and help the IT operations teams be as agile as their development counterparts.
So the integration of all these concepts will form the basis for articles in Agile Cloud Development.
Why is it important?
Organizations that leverage agile and cloud have enormous competitive advantages over their competitors with respect to higher quality, faster market responsiveness and greater business agility. This ranges from large global enterprises to Lean Startups – a new movement that wouldn’t exist without the capabilities of agile and the cloud.
Since 2001, agile has gradually become the norm for most software product organizations due to its flexibility and rapid time-to-market. It is also gradually replacing waterfall projects in many larger corporations, a fact not lost on the Project Management Institute (PMI) who recently rolled out agile certification and are updating their classic PMBOK™ text to include agile practices. Even PMI is going agile!
Since 2006, cloud computing has also grown dramatically, becoming a $41 billion industry by 2011 with estimates of up to $241 billion by 2020. Whatever the eventual size, there’s broad consensus that cloud computing is going to be a large part of the growth in the tech industry.
For these factors and many more, we think Agile Cloud Development is an important topic for global organizations of all shapes and sizes today and into the future.
How is it unique?
The topics at the intersection of agile development and cloud computing are unique for a few reasons:
Cloud is an Agile Accelerator – Cloud computing, when combined with agile engineering and DevOps practices, enables faster release cycles of higher quality code. This shrinks the time it takes to get an idea from concept to running in production and directly…
Enables Business Agility – Organizations leveraging agile and the cloud can quickly test these new ideas in the marketplace, gather feedback and adjust course based on customer demand. They can do this in days and weeks, not months and years. Leveraging the cloud keeps initial investments in hardware low and allows organizations to invest more as the ROI improves. Organizations that do this very well can translate faster cycle time and higher quality into…
Reducing IT Costs – Through agile process improvements and cloud-based SaaS, PaaS and IaaS, bringing an idea from concept to production costs less. Cloud-based tools for management, development, testing and monitoring are bringing ever-greater capabilities to organizations for lower costs, improving the bottom line. While important, reducing IT costs are secondary to…
Improving Customer Experiences – Today this is being recognized as one of the most important drivers of revenue and long-term business success. The ability for organizations to engage their customers in meaningful experiences, not simply ship more features or become more efficient, is the future for organizations large and small. Examples include Amazon and Zappos, Starbucks and Apple. This is also on the radar of VMWare’s CEO who is seeing a shift to customer experience in the market.
Growing the Economy – The so-called new new economy is one of the driving forces behind global economic growth. Small start-ups are leveraging the cloud and agile development to not only compete with large corporations in traditional markets but create new markets too. From Google to Twitter, Netflix to Groupon, today’s startups want to grow their businesses quickly and do it with less up-front investment – a perfect recipe for agile development and cloud computing.
What will be covered in this series?
Analysis in the series will generally have one of these themes:
Emerging Trends and Topics
Top Vendors and Tools
Practical How-To Articles and Case Studies
In keeping with the virtualization practice standard, in the coming months we will create a whitepaper that describes the Agile Cloud Development market and vendors that serve the market.
We believe Agile Cloud Development is the future of software development for global organizations of all shapes and sizes in all industries. For those seeking improved business agility, lower costs and ultimately better customer experiences – the Agile Cloud Development series is designed for you.
I start my day, every day, with what I call a daily news briefing. I first read the daily newspapers, Financial Times, International Herald Tribune, and the Wall Street Journal. During my daily news briefing last Saturday (July 25th), I was captivated by an article that appeared in the Wall Street Journal Saturday Edition titled, “Silicon Valley Survivor“. This piece was on one of this industry’s CEOs that I admire and respect highly for his insight into the long-term direction of the industry; John Chambers of Cisco Systems.
In a recent discussion with a group of executives, I stated “these are very tough times, but the technology is there to be revolutionary and make great things happen for the business. I told them that they just need to be that revolutionary person to drive change.” I’ve had a couple of comments online and more offline directly to me around innovation. I wanted to share some of my thoughts on being innovative and some things that you can do to help push the “innovation envelope” and create a dynamic IT environment.