Rational Scrum

Recently I tried out a variant on methodology that I’ll dub Rational Scrum. I’ve been trying to put together a few thoughts about the overall process for months, and finally found some time for it.

Just as people have specializations, so do processes. Applying one process to all situations is just as wrong as calling your dentist when you need brain surgery.

Rational itself is an excellent methodology and scales very well. Starting with only a few Rational artifacts, it can almost feel like Extreme Programming. Yet, the Rational body of work provides a framework that supports a much more comprehensive methodology… something less than full-on Spiral development, but still robust enough to handle large-scale, distributed team development.

However, for all that I like Rational, it does have some holes… and these are plugged most wonderfully by Scrum. In fact, Rational and Scrum benefit each other so well I’ve started referring to the combination as “Rational Scrum.” As with most methodologies, it does not define every possible response to every possible situation, but the combination of these two techniques is very robust and complete.

The processes complete each other by addressing mutual weaknesses. Scrum steps in to fill a gap in organization and team management. Conversely, Rational brings a structured, risk-driven approach to Scrum that is lacking.

Most recently I was able to put the principles of Rational and Scrum into effect with a client. The initial engagement was, let’s say, “unfortunately typical.” The client had about 45 technical staff working in a disorganized fashion on unclear requirements. Most of the work was outsourced to a third party, further distancing the team from key business knowledge. Product releases had been very slow or nonexistent for the past year—what releases there were had been seriously compromised in quality. Development felt isolated from the business and quality assurance was a poorly funded afterthought. The production support group of about five people were out of the loop on software deliveries and constantly scrambling to stay on top of customer driven emergencies.

My first task was to get my mind around what was going on. I took a trip to the outsource vendor (amazingly, this was the first trip anyone from the company had taken to the vendor) to get to know the team. I spent several weeks understanding the issues and bringing the team up to speed on what the business was really all about. Because of time differences (about thirteen hours) this sort of cross-pollination had never happened. Ideally, I would have set up a program where the entire team would have rotated through on-site “shifts” of several weeks duration, but budget was limited. This would have created a far more integrated team and given each person on-the-ground experience with the business, its goals, its priorities. I also delivered some training on Rational methodology in a compressed one-week course to the entire development team.

One of the most glaring problems was a complete lack of information share between the business and the development teams. We immediately purchased and implemented task and defect tracking (with Atlassian Jira) and a document repository (with Atlassian Confluence). Effective immediately, all information had to be migrated to the new system. If it wasn’t in there, it didn’t exist. I went so far as to return, unread, any technical email or document I received unless it was in Jira or Confluence. Absolutely no defect, feature request or deliverable could be discussed until it was put into Jira, given a once over to make sure the issue was stated clearly and, in most cases, given a quick first-pass effort estimate from development.

We also started having daily Scrums. This proved to be the most difficult challenge—the time difference pretty well necessitated a tight meeting window. Once back in the U.S., I ended up scheduling nightly telephone calls with the development teams from 8:00 PM until about 11:00 PM every day. This went on for months, but was necessary to drive improvement. We typically kicked off an evening with the “management Scrum,” limiting it to no more than half an hour (I doubled the time limit because it was virtual, organization was difficult, and at least initially there were a lot of issues to work through). Once the management Scrum wrapped up, splinter Scrums started up with each of the individual development, quality assurance, testing and professional services teams. Even following Scrum guidelines to keep the meetings on-track, we initially ended up with plenty of overflow. This was followed with a management Scrum on U.S. timetables, the following morning. Initially, it was a grueling meeting schedule for me, but we kept individual team member involvement to a minimum—preferably one daily Scrum.

Another major change was implementing the Rational Unified Process. This took upwards of six months to effect as we introduced light processes initially, adding to the Rational methodology over time. A few of the most important aspects of Rational that we adopted early on were risk-driven development and truly iterative releases. While development had been using iterative terminology, the fact of the matter is that monolithic development practice was still being followed (for example, creating massive specifications and trying to build to the specifications rather than focusing on short-term release goals and continuous improvement). By focusing the team on short-term releases, iterative release cycles and a product backlog (all done in Jira and Confluence) we quickly raised the level of understanding, across the board, of what was coming and what it would take to deliver. This information started to flow throughout the organization and we started to see its effects in how product features were prioritized, how releases where scheduled, and in customer service and the product life cycle itself. Soon, the company went from half a dozen silos of locked away information to a single, relatively smoothly operating organization with across-the-board visibility.

During this evolutionary phase of these changes we discovered all kinds of interesting things were going on. We found separate development teams working on the same foundation technology. Optimizations that could be made with small shifts in the business goals. Streamlining and efficiency gains by changing our release cycle. Massive effort savings by disregarding a few relatively unimportant features. More streamlining that came from the information sharing. The product backlog started to become self-prioritizing as business owners and other stakeholders participated in setting the upcoming release goals. Distractions, fire-fighting, context-switching and misdirection disappeared and the development team started producing true forward progress for the first time in a couple of years.

In fact, the team went from essentially zero forward progress to a steady, major new software release every six months.

We also trimmed operational costs by about $1.2 million dollars per year. Of the 45 outsourced development staff originally on the project, we reduced the count to fifteen in less than a year. And they still kept producing a major new software release every six months. Despite a team size of about one third, we were delivering more and getting more done in less time.

We implemented improved quality assurance measures and actually increased the size of the software testing team from one person to three (all within the 15 person head-count). Software quality started to improve at a dramatic rate.

Through Rational Scrum we had realized a productivity increase of somewhere in the vicinity of 300-400% if we account for the team size reduction. We had achieved positive forward progress where it had been lacking for well over a year, if not two, and had done so after cutting the development team down to one third of its original size.

Toward the tail end of my involvement, after about 18 months on the project, the team size had been scaled back a bit more. The daily Scrum meetings had become smooth and unintrusive, going from two or three hours down to about 15-20 minutes per team. New technology teams focusing on completely new product lines were being hired and integrated into the process very effectively. Even the professional services group had adopted the process and, likewise, had reduced its staffing requirements from five to two and was operating more smoothly. Customer satisfaction was up and the fire-fighting had pretty well come to a stop.

The two processes—Rational Unified Process and Scrum—gave the company the focus and clarity it need to get back on track. Working together, these processes had delivered improvement to every aspect of the organization:

  1. All stakeholders in the project become a part of the development process. Everyone involved is genuinely interested in the project output and actively contributing torward progress.
  2. Visibility is raised across the board through information sharing. This improves efficiency, mindshare and contributes to progress as well. The group contribution is far more valuable than individual “information silos.”
  3. Truly iterative development delivers working, progressively improving software rapidly. Incremental change is easily accommodated by avoiding the “big bang” and monolithic development approach.
  4. Rational’s focus on risk-driven development keeps the team focused on the most difficult, potentially risky aspects of the project first. This contributes on a number of fronts: Major problems stemming from unknowns become identified and are dealt with early in the project. Major change stemming from these unknowns has less impact because it happens early. The riskiest, most complex portions of the project are developed first, and thus mature first and receive the most testing.
  5. Risk analysis and mitigation “bubbles to the top” through the daily Scrum meetings.
  6. Priorities come in-line with business and technical reality as the whole team works on sensible, informed schedules.
  7. Ownership and accountability become the mainstay as everyone develops a long-term interest in the project and pride of work product in their contribution. Commitments are taken seriously as a team.
  8. Visibility into the entire process means there are no surprises and no last minute bad news. The business can plan according to reality and influence the overall program in an informed manner.
  9. Progress is realized for effort and planning, not for putting “hours in the cubicle.” People are rewarded for good work and not for punching a clock.
  10. Quality becomes integral and continuous. The Rational process integrates quality assurance and structured testing from the very beginning of the project. It is never an afterthought.

Rational Scrum delivers a solution for small-scale as well as large-scale project development. It supports the essential aspects of project and program management that make quality software development a possibility. It adds the dimensions of reliability and repeatability to projects and focuses the team on continuous improvement and productive work.