DevOps Case Study

October 30, 2015

A major top-20 regional bank turned to Levvel in an effort to drive architectural design and implementation of DevOps practices within a newly created Internal Cloud Team.

INTRODUCTION

A major top-20 regional bank turned to Levvel in an effort to drive architectural design and implementation of DevOps practices within a newly created Internal Cloud Team. This cross-functional team was responsible for developing new procedures around CI/CD, TDD, source control, infrastructure as code, rapid deployment processes and inclusion of new application/middleware/database operations. As with many large enterprises they were forced to deal with the complexity of a diverse application landscape, well-defined responsibility silos, and the learning curve of new technologies and processes specific to the DevOps movement.

PROBLEM SCENARIO

When new servers and networking infrastructure were required, application teams would submit a ticket or make a request to the operations team for the required components. Submission of this request kicked off the provisioning of the VMs followed by installation of the base operating systems. After the OS was in place a lengthy process requiring the active involvement of up to 6 teams was initiated to install all the packages sitting on top of the OS (middleware, database, monitoring, networking, security, messaging, etc.).

The end to end execution of this process required up to 6 weeks, a large number of man hours, contract resources, and consistent management to ensure the delivery process was progressing.

SOLUTION

Levvel worked with the client to build an efficient delivery process enabled by Chef, Jenkins, Git/Gerrit and Artifactory – all operating within a larger environment that allowed the client to retain other services that had already been invested in such as VMWare’s vRA, ServiceNow and PAM. In addition to designing and building the technical components of the system the Levvel team provided training and best practices guidance to enable the client resources. This knowledge transfer included regular group trainings followed by intense hands-on activity centered around pair-programming and instructor-led development workshops.

As a first step a small team was tasked with building a proof of concept for utilizing Chef as the primary configuration management utility. This was done by standing up a private HA Chef server using DRBD on premise and combining the installation of Chef client into the VM provisioning process. Using open source cookbooks and a basic run-list. The POC team was able to quickly deliver a working prototype of a base Linux VM (RHEL) and move on to the implementation of a full solution.

Realizing the full benefit of this proof of concept required a complete, customized Chef implementation as well as the introduction of tools and process to support the transition to zero-touch deployment. Levvel facilitated this process in the following ways:

  • Implementing custom Chef cookbooks for required packages including Centrify, NTP, Legato, and LVM.
  • Establishing an Artifactory server to manage the Chef artifacts utilized in the provisioning process.
  • Teaching development best practices including unit testing, TDD, version control, and automated testing.
  • Pair-programming with client resources to help them understand how to translate their existing processes into Chef recipes and empowering them to own the process going forward.

devops-case-study

IMPACT

The major benefits of this project are realized on two fronts. First and foremost, both the time and resources required to deploy servers were drastically reduced. IT velocity is a major force for success both in the ability for the client to rapidly bring solutions to the marketplace as well as providing fiscal savings to the organization

The provisioning of a server now takes 6-11 minutes on average and requires only the passive oversight of 3 departments and no external resources.

The second major area of improvement was the consistency, stability, and reproducibility of the deployment process. Prior to the use of Chef for configuration management the installation and configuration of packages required many manual steps that were executed by system administrators without stringent code tests, specs or reviews. Most of the system administrators had one-off scripts, stored in a shared drive, they used to automate pieces of the work and/or procedures they performed in a unique way.

Moving to a zero-touch deployment process removed variability and risk from the deployment process. Supplementing this practice with development best practices like code review and automated testing ensures consistently higher quality deployments.

CONCLUSION 

Levvel was able to work in partnership with client resources to establish a zero-touch deployment environment built on Chef. The execution of the project included both technical delivery and mentoring on the new technology being introduced.

Successful delivery of this project helped the client realize both operational improvement and expense reduction.