Elegance Can Be the Enemy of Efficiency

I’ve been auditing some of the advanced Scala courses offered by École Polytechnique Fédérale de Lausanne, and as always am curious about my own solutions compared against what other people come up with.

One of the things I love about Scala is its elegance. Once you understand the language, you can achieve some really complex things quickly and easily — and, elegantly. So, a problem posed in the coursework is to write a function that, given a character in a game field, returns the first index of that character in the field.

After writing my own solution, I Googled a few other solutions. Here’s one:

def findChar(c: Char, levelVector: Vector[Vector[Char]]): Pos = {
   val ys = levelVector map (_ indexOf c)
   val x = ys indexWhere (_ >= 0)
   Pos(x, ys(x))
}

This is pretty elegant. In fact, I was first taken by how functional it looked — more functional than my own code. But there’s a problem. This elegant code is potentially very inefficient.

Let’s assume the game field is big. It consists of a Vector of Vectors, and the above code searches the entire game field. That is, even if the character matches the very first element in the field, all the rest of the elements will be scanned:

  1. ys maps over the entire Vector of Vector scanning for the index of c.
  2. x then scans the vector ys for a positive hit.
  3. Then the coordinate is returned from the points described by x and ys(x).

So the obvious problem here is our elegant, functional, good looking code could be terribly inefficient. Here’s my implementation:

def findChar(c: Char, levelVector: Vector[Vector[Char]]): Pos = { 
   @tailrec def iterate(x: Int): Pos = levelVector(x).indexOf(c) match { 
      case y if y > -1 => Pos(x, y) 
      case _ => iterate(x + 1) 
   } 
   iterate(0) 
}

Ok, it’s not as pretty, it’s longer, but — it’s still functional by design. More important, it will look for the character using a linear search, and stops when it finds the character. It’s efficient:

  1. We start in the first Vector, checking for the index of the character.
  2. As soon as we find it, the function returns. Otherwise, it recurses and moves to the next Vector.

It’s not as pretty but it preserves efficient design. Just because we can write pretty, elegant code doesn’t mean we always should.

Incidentally, you could also achieve nearly the same efficiency using something like this:

val row = levelVector.indexWhere(_.contains(c))
val col = levelVector(row).indexOf(c)

Personally, I prefer to avoid any potential inefficiency — but, over-optimizing can also be a problem. It’s probably best to find a happy balance. Don’t ignore efficiency, but don’t spend all your time optimizing situations that, quite likely, don’t really need the effort.

You’re Testing It Wrong…

I really like test driven development. It lets me be lazy. I don’t have to worry about my software quality, or that something I did broke some other thing. And with good dependency injection to make sure every component is working right, “it just works.” Now I code using TDD (writing my tests first, then coding to fulfill them), and I focus our QA efforts on making sure we have great test plans, and great coverage.

Closed System
A closed system wants to be tested.

So, when one of my project teams kept telling me they couldn’t write tests because the database wasn’t ready I got worried. Our team had been immersed in TDD for months — and every single engineer had nodded vigorously when I set expectations. The team leader recited the definition of “dependency injection,” just to drive home how ready they were to embrace it!

But when I asked to see the what was wrong, I knew we had a problem. The team’s tests were not injecting mock objects the right way. The idea behind dependency injection is to replace the smallest component possible in a closed system with another object, a “mock.” That mock can then monitor the system around it, inject different behaviors, and create desired results.

For example, let’s say we have a program that connects to a gizmo — your home thermostat. The thermostat itself is a separate component that lives outside your program. We can expect the thermostat to behave like a thermostat should… reporting the current temperature, and letting the home owner enter a desired room temperature. Pretty straight forward.

So the first step is to write a program that talks to the thermostat. We can wire up a real thermostat, but we’ve got a problem right off the bat. We want to know how our program behaves as the ambient temperature changes — 65 degrees, 32 degrees, or 100 degrees. But a real thermostat is only going to report the actual room temperature, and making the room frigid or boiling just isn’t going to be very comfortable or practical.

Not Mocking
Faking is not mocking.

This is where dependency injection comes in — wouldn’t it be great if we could inject a new gizmo, one that behaves according to our test plan?

It turns out that my team had been taking the wrong approach — but one that is pretty easy to make if you’re new to the idea of mocking and dependency injection. Unfortunately, it meant that they weren’t really testing the application. The were testing something else entirely.

Once we walked through the system, the mistake was clear. During application start up, it created a connection to a database. My team’s approach had been to add a “mocking” variable to the application. In effect, it created a test condition; if the application was in “mocking mode” it would only simulate database calls. If the application was not in “mocking mode” it sent real requests to a real database. Sounds great, right?

But it’s all wrong. Here’s the problem: The application was faking real world behavior. That is, throughout the program there were dozens of little tests, in effect asking, “if we are mocking, then don’t check the database for a user record, instead just return this fake user record.”

This meant that the real program — the actual application logic that would be deployed to the real world — was never tested. Instead, an alternate branch of logic was tested — a fake program, if you will. So two things happened:

  1. We weren’t testing the real program, we were testing something else altogether.
  2. The program itself became terribly complicated because of all the checks to find out “are we mocking?” and the subsequent code to do something else entirely.

And all of that is why my team said they couldn’t really test the system, because the database wasn’t up and running.

So what does real dependency injection look like? It’s simple: You want to change the actual gizmo, but change it in the most subtle way possible — and then you want to put that actual gizmo right back into your program.

Mocking
Real mocking doesn’t affect the original program flow.

Getting back to the thermostat example, an ideal solution would be to modify a real thermostat. You could crack it open, remove the temperature sensor, and add a little dial to it that lets you change the reported temperature. Then you plug the “mock thermostat” into your program, and you change the temperature manually! A potentially better approach would be to change the software that talks to your thermostat, and instrument it so that you can override the actual reported temperature. Your program would still think it’s talking to a real thermostat, and the connecting software could change the actual temperature before handing it off to your program.

In our case, the right solution could be injecting a simple mock component at the just the right point in our program.

For example, lets say our application uses an Authenticator object to log in users. The Authenticator checks the validity of a user in the database, and then returns a properly constructed User object. We can use dependency injection to substitute our own test data by overriding the single function we care about:

object fakeAuthenticator extends Authenticator {
    override def getUser(id: Int): Option[User] = {
        Some(User(id: -1, name: "Fake User"))
    }
}

On line 2, we replace the real Authenticator’s getUser function. The overridden method returns a hard-wired User object (in this case, one that clearly doesn’t represent a valid user account). By overriding the Authenticator in the test package only, the original program is not altered — all that’s left is to inject our altered Authenticator into the program.

The old fashioned way of doing injection is still reliable: Don’t tell, ask. Use a factory object to ask for the Authenticator. Given a factory in the application (let’s call it the AuthenticatorFactory) we can override what the factory actually returns in our test case only:

AuthenticatorFactory.setAuthenticatorInstance(fakeAuthenticator)

A slightly more modern approach is to use a dependency injection framework, but the underlying principle is exactly the same.

Likewise we can take the concept of mock objects further by using frameworks such as Mockito (a framework that works wonderfully with specs2). Mockito makes it easy to instrument real objects with test driven behavior. For example, Mockito will produce a mock object that acts just like a real object, but fulfills expectations (such as testing to make sure that a specific function is called a certain number of times).

Whatever tools and frameworks you use, test driven development has proven itself over the past decade. My own experience is the same: Every TDD project has produced more predictable results, has better velocity, and has been more reliable overall. It’s why I don’t do any coding without following TDD.

Organizing Overseas Teams

Hi, I’m Zacharias Beckman, president of Hyrax International. When it comes to coordinating international projects, one of biggest challenges we hear about is staying on top of the project.

As an international project manager, you have to know how to stay organized, and you need to know what your team is doing. When you have several different teams, all spread around the world, that’s not always easy. You also need to make sure that one of your teams isn’t being held up, waiting on another team.

This is what Tanya ran into, at one of our clients. She had been managing a U.S.-based team. Her company had just bought a smaller firm in India, intending to set up a “follow the sun” strategy. With teams in the U.S. and India, they could move faster because one team would hand off work, at the end of the day, to their overseas counterparts.

But there was a problem. After a few months efficiency was falling, not improving. Tanya found that the teams were poorly coordinated, and more often than not one would end up waiting on the other one. Tanya needed to change her strategy to accommodate a global team. She had to refocus, and figure out how to get these teams collaborating smoothly despite a separation of over 10 hours.

She made two major changes, both of which focused on improving coordination.

She took a critical look at their project management system, and decided that it wasn’t up to the job. It had worked great when everyone was in one office. But now it had to deliver a new level of coordination. She needed something that could better drive the process, improve visibility to her management team, and show dependencies between team members. It was absolutely critical that everyone know, at any time, who was waiting on them. They also needed better requirements management, and better collaboration tools. Her new system gave them the tools, but it couldn’t solve the communication issues on its own.

Tanya also changed the team schedule, setting up short, collective meetings every day. To avoid burdening one team, she set a rotating schedule: meetings where held at 9am in the US twice a week, and 7pm twice a week, with no meeting on Friday. Team members had to join at least two meetings each week, but it was up to them to pick which ones.

Tanya’s changes showed almost immediate results. The teams became more coordinated, and situations where one team was held up waiting for another pretty much vanished.

In a multinational organization, it’s important to remember that remote teams can feel like they are in a vacuum, lacking communication or cut off. To compensate, a good manager has to be extra vigilant and put in good processes, and good tools, and also make sure that no one team becomes the favorite. Tanya spread the meetings out to share the burden of after hours meetings. By doing so, she also sent the message that both teams are equally important.

Creating An International Culture Of Success

The International Business Dimension

Multinational teams present new challenges for the International manager. There are logistics problems: How do you coordinate teams that work in different time zones? What kind of collaboration can you create in a team that rarely sees one another?

As well as the logistic problems come cultural problems. For example, successfully creating a culture of innovation can be a challenge. Honeywell experienced this, according to a November, 2013, Time article, when Rameshbabu Songukrishnasamy began working as general manager of the company’s R&D centers in Shanghai and Beijing. He found his employees were not innovating. They weren’t tinkering or inventing on their own — not a positive sign in an R&D lab! “They were happy just doing what they were asked to do,” Rameshbabu says. The problem is, R&D is about doing something new.

A project manager for a large corporation in Brazil recently told me that the PMI Book of Knowledge is used infrequently at best inside Brazilian projects. He also warned against assuming that someone with a PMI certification has extensive experience, as is the case in the US. — Moore, Brandi, The Little BRIC Book.

Rameshbabu found that his Chinese workers had a fear of failure. They worried that the company would be upset if their work did not yield positive results, so they didn’t experiment. Another problem is that some Chinese engineers “tend to shy away from critical questioning,” a process that is fundamental in R&D. “The reason they are able to make so much innovation in Silicon Valley is that people question the status quo and find alternative ways,” says Rameshbabu. But he found that Chinese culture and education focused on rote learning, not critical thinking.

Creating A Culture Of Success

Creating successful International programs requires understanding and adapting to different business cultures. Applying Western management practices in Asia will fail, just as surely as transplanting Western employees into an Eastern environment. Imagine an independent, critical thinker from Silicon Valley landing in Foxconn, Shenzhen — where challenging the status quo is forbidden.

Team dynamics play a huge factor in management style, objectives, and capabilities. Building a culture of innovation is just one example of where these dynamics become complicated. Power distance will affect everything from goal setting to how problems are socialized. Communication style can quickly lead to misunderstandings. Differences on the fluidity of time can mean completely missing the mark with customer deadlines. And differences in identity and engagement style can lead to initial confusion, bad first impressions, or distrust.

This is why understanding business cultural practices is so important. Hyrax International LLC has a program that explores each of these five preferences. The program examines each of 27 different management disciplines, such as goal setting, risk management, change management, and assessing outcomes. The affect of business culture on each discipline is explored and explained, providing a road map to success on the International management scene. The company also offers many free resources to explain and explore International project management, and is also sponsoring Successful International Project Management, an in depth book that maps project management processes to cultural preferences.

We’ll be posting five more parts to this article (read Part 2, or see the entire series right here) in the coming couple of weeks. Each post will look at one of the five business cultural preferences, and briefly introduce how that preference impacts and affects the 27 management disciplines.

Hyrax International LLC’s Global Project Compass™ is the only visual map that clearly shows the connection between business culture and business process. This is what makes Cross Cultural Management™ so much more effective than traditional management.

The Compass maps 27 project management disciplines directly to business cultural preferences, and shows how these preferences affect business. The goal of the Global Project Compass, and Hyrax International’s associated management program, is to show how culture affects businesses worldwide — and to provide a clear map on how businesses can adapt successfully.

Why we can’t just burndown everything

Prashad is the project manager for a new product. After reviewing the project scope, Prashad’s team gives it an initial estimate of about three to four months to complete, which is good enough for management. The project kicks off, and using Scrum along with a little bit of project management, the project gets underway. The first month goes quickly, there is great progress, and everyone loves what they see. But in the second month, things start to slow down — so, Prashad adds more resources to the project to compensate. But by the third month, there are real problems and it looks like the project will be late. Prashad adds even more resources. Now the team is almost double the original size. He adds a lot more quality assurance and testing support for the team, too, because a lot of problems are cropping up. The team cranks away, and they finish the project in the fifth month, and claim “success” because they almost hit the deadline (it was, after all, just an “estimate“).

But what about all those additional resources? Is it really success if the project cost twice as much as originally estimated? The problem is, that cost was never formally stated. It was never tracked as a metric, so the fact that the project went horribly over budget is quietly swept under the rug — at least this time. There are plenty of cases where the budget isn’t there, or the resources aren’t there, or the product just can’t ship late.

So, if so many projects are missing their mark, what’s causing the problem?

The fallacy of so called Agile methods

According to studies conducted by KPMG, as much as 70% of projects fail to meet their goals. In this case, “goals” mean quality, schedule, and function (or a combination of those). Clearly, being able to accurately estimate a project’s delivery date is important. Without knowing when a project is done, you can’t predict cost, plan business operations, or dovetail releases with marketing, training, and customer service.

Agile methods like Extreme Programming and Scrum make promises that, in my opinion, they can’t keep. For well over a decade we’ve seen an increasing trend in project failures, and a large number of those failures are the result of “unending projects.” Projects that go on and on, continuing to deliver improvement but slowly creeping over budget and never reaching an end state. Part of the problem is the lack of formal estimation and metrics.

Both Scrum and Extreme Programming dodge the entire issue of project life cycle estimation. They focus on the near term, providing estimates for the next one or two sprints. This works great if you don’t have budget or time constraints, but in the real world that’s rarely the case.

Burndown just doesn’t work

Most Agile methods don’t explicitly define how you estimate progress, but the most common method is burndown — the measurement of completed effort against the planned sprint goals. This is fine for a quick-and-dirty project but as a strategy it utterly fails to pinpoint problems with budget and timeline. For anything larger than a few months effort, it doesn’t do the job.

And the problem is, few of today’s engineers seem to be getting the formal education and training needed to use proper estimation methods.

There are better solutions than burndown

Earned Value Management, or EVM, is nearing it’s 40th year of practice. Throughout this 40 year history, EVM has repeatedly evolved to meet new demands as technology and innovation made new leaps forward. It’s a proven method for measuring progress and it’s been proven time and again.

Of the dozen or so software engineers I spoke with about this article, none of them had even heard of EVM.

Earned Value Management, as well as a variation known as Performance Based Earned Value® (PBEV), are my favorite choices for estimating a project. The PBEV approach provides an incredibly robust method for measuring progress and staying on top of your project. It’s only marginally more difficult than other techniques of measurement. And most project managers are already capturing the metrics needed to make it work.

PBEV is an improvement over Earned Value Management Systems’ (EVMS) national standard. It supplements EVM with guidelines for the integration of project cost, schedule, and technical performance — thus created a comprehensive and highly accurate method for measuring true progress in a project. The key here is integration of effort with schedule and performance.

Had Prashad been using PBEV, he would have had a very different experience. The early estimates would have been more accurate, since PBEV estimates the entire project. It would have measured not only resources and cost, but also the actual performance — the rate at which work is being done. Had the project gone off track, Prashad would have known much more quickly, and would have had a more precise picture of how to correct the project’s course.

There will definitely be situations in which PBEV is not warranted, and a skilled project manager will need to know when to use other methods, and why. EVMS, for instance, addresses only the quantity of work completed, ignoring cost and schedule metrics. The implication here is that a quick burndown or EVMS-based evaluation of a project might look A-OK, when in fact the only reason it’s on track is because resources are being thrown at it — and costs are skyrocketing.

Burndown versus critical path thinking

Burndown is a very simple method for estimating work effort over a short term. It explicitly avoids timeline, cost and performance metrics. That’s fine for projects that are not constrained by time, budget, or resources. The problem is, burndown is used as a defacto standard too often, on the promise that using an Agile method will deliver a project faster or cheaper than its alternatives. I don’t disagree on this front. Agile methods generally are faster and cheaper — but the question is, faster and cheaper than what?

Most business need to be concerned with time, money, and staffing. Projects that have those constraints need to think about the critical path — the sum of all activities to reach the end of the project. Having a solid understanding of the fundamentals of critical path management is important to managing any project schedule. Depending on the nature of your project, you may be able to rely on casual effort estimates — or, you may be required to carefully identify, analyze, and attack critical paths to shorten your project lifecycle and stay within stringent guidelines. Without understanding the principles, and knowing where to turn for more information, neither is possible.

I’ll revisit the topic of PBEV in a future article and provide more depth into how it can be applied to a project. If you’re eager for more information, check out Performance-Based Earned Value®, Paul J. Solomon, Ralph R. Young, IEEE Computer Society, Wiley Interscience, 2007.

Solving cultural and logistic International project problems

Sometime in 2001, a New England firm that later became a client decided to outsource all of its software development to India. It seemed like an excellent idea at the time, as Indian intellectual property wage rates were roughly one-tenth of their U. S. counterparts. But the project went poorly: U.S.-based employees struggled to manage programmers located halfway around the world, and much of the work coming back from India didn’t meet the standards of quality expected by the U. S. firm. While the industry has improved, this is still a common problem today. How can a team located in a different culture, a different business environment, and surrounded by completely different ideas regarding acceptable customer service, adequately meet the quality demands of a foreign customer?

There’s a solution?

Before exploring a solution that solves the complex cultural and logistic issues of International projects, let’s take a quick look at the elements of a typical project. We can start by using a guide such as the Project Management Institute’s PMBOK®, or Project Management Body of Knowledge. The PMBOK quite effectively lays out much of the scope a project manager needs to be prepared to handle.

According to the PMBOK, every project has five phases in common:

  1. Initiating.
  2. Planning.
  3. Executing.
  4. Monitoring and Controlling.
  5. Closing.

During each of these phases, the project manager’s objective is to balance the competing constraints of scope, quality, schedule, budget, resources, and risk.

These constraints are, in and of themselves, often daunting. Add in the human element and often inevitable business politics, and projects can become difficult exercises in communication, motivation, and human psychology. In fact, the PMBOK spends a good bit of time discussing key areas where these factors play a major role. Stakeholder involvement, communication plans, getting an unbiased statement of work or progress, managing risks, and managing subcontracts — just to get the list started — are critical to achieving success on any project.

All of these factors combine to create a web of constraints that push and pull at the fabric of a project, often so much that the fabric doesn’t survive intact. For example, stakeholders may not have the project’s best interests at heart or may simply be uninterested in their day-to-day responsibilities. Budget constraints may require the project manager to make difficult decisions that affect the team, or the objectives of the project. Subcontracts are, by the very nature of business, first and foremost motivated by their own fiscal health and profit, not necessarily your best interests.

This is the landscape a project manager steps into on a daily basis. The larger an endeavor, the more significant these challenges become — even when the entire project is still in a single building.

International project problems… Solved

Now, expand the dynamics of the project to a global effort, involving International team members, foreign partners, and vendors that you seldom meet in a face to face setting. In fact, many global projects are “communications deprived,” due to geographic challenges. A global organization can often have project members spread across as many as 15 time zones. While advances in technology have led to tremendous strides in delivering reliable communication, teleconferencing or video conferencing a few times a week, in the early or late hours of day, doesn’t compare to frequent, in-person communication. Teams operate in a vacuum much of their day, making decisions they would otherwise not make on their own.

Think back to the New England based firm: All of these factors contributed to quality problems, as the team in India tried to deliver what they believed the client wanted, but failed. Lack of communication, cultural misunderstands, and business context that we take for granted was missing — and the project failed.

This is the vacuum into which Rational Scrum was created. Project management methodologies today don’t address the complex issues imposed by multicultural teams, working in widely distributed, International contexts. That’s what Rational Scrum is all about: Finding solutions for International project problems. Fuel for this blog comes from a few different sources, including my book Successfully Managing International Projects. I hope you’ll find the contents interesting!

For more information on Successfully Managing International Projects, sign up for our mailing list. You’ll get occasional, relevant updates and announcements as the publishing date grows near.

Tell me three times: The importance of quality assurance

Tell me three times

The earliest military applications of quality control came about from needing to send messages with reasonable confidence. The protocol was simple: Send every message three times. This triple-redundancy provided a checksum against failure, and could be used to correct broken messages. Any difference between the three copies would usually only show up in one, so the remaining two could be treated as accurate. The incorrect third could be discarded. In time, of course, advances in technology and process made it possible to introduce far more advanced — and secure — methods of communication. But the principle still lives on today in formal quality assurance: By introducing a redundant check on a program or process, we improve our chances of success.

As an industry, software folks have invested a huge amount of time toward figuring out what goes wrong with most projects. The root cause is complexity and our ability to accurately manage that complexity. Finding a method that enables reliable and repeatable results is a tough problem, especially given the variables involved in every project: Changing business environments, customer demands, technical capability and understanding, and team makeup are just a few of the factors that affect every project in a multitude of ways.

To combat the problems of complexity different best practices have become popular — some good, and some abysmal. The more successful techniques have a common theme: The idea that we can manage all this complexity by introducing multiple checkpoints.

Quality assurance and checkpoints

This is what quality assurance and structured software testing is all about — and yet, at least in the commercial industry — there’s always pressure to cut quality assurance or testing budgets. Take, for example, a recent project that ran about 18 months, involved well over 25 people, and launched to huge success (and zero defects in the product). The immediate aftermath of this successful effort? One might imagine kudos were in order. How many times do 18-month long, multi-million dollar projects get out the door without major problems? Instead, the project sponsors criticized the cost of development and, rather arbitrarily, said the team had “spent too much on quality assurance.” The reasoning behind this was simply that, since there were no bugs in the finished product, all that money spent on quality must have been wasted.

Management then demanded the quality assurance budget be cut dramatically — In fact, insisting they would not spend another dime on quality assurance. It was one of the most counterintuitive situations I’ve ever encountered.

The unfortunate consequence of this is an antagonistic relationship between project team and project sponsor. It brings into question how much visibility the sponsor should have when it comes to internal project budgets, and that’s a dangerous line to tread. At the same time, taking the position that QA will not be funded is crossing the line between budgetary management and meddling with technical process — and in this case, a process that had worked with stellar success.

The politics of management

Visibility should not be compromised. The project sponsor needs to know where costs are, and most project sponsors are not going to be happy with a single budgetary line item for “project development.” However, it is equally important that project teams and their organizations maintain a uniform front defending what works.

Stated another way, when you have something that works, treat it as a whole product that cannot be “sliced and diced.” Software development, at least those processes that work, cannot be subject to arbitrary and partial budget cuts. Cutting just the quality assurance department alters a working process. In this case, we had an unreasonable project sponsor that was not interested in understanding the complexity of building a product.

I held a firm line with our sponsor. We could not run a project with arbitrary cuts to parts of the program. Our compromise ended well enough. I was able to tell the sponsor that we would cut the budget across the board, not just in one department. At the same time we came to a mutual understanding: The sponsor was really just concerned with the big picture, the total number of dollars spent. As such, we agreed he would not see another line item labelled “quality assurance.” Future budget reports had a single line item for the total engineering cost, all inclusive of quality-related expenses.

The team, and the project manager, need to defend a system that does a good job. The sponsor needs to be informed that budget cuts cannot arbitrarily target specific program components: Instead, the right way to tackle this problem is for the sponsor to cut the overall budget and let the project organization decide how that cut will be implemented. Most likely, cuts will need to be applied equally across the project — thereby reducing overall output, but not messing with a process that works.

Unfortunately, everyone needs to recognize that this can lead to losing a client or canceling a project. The question is, would keeping the project alive be worth the long-term headaches, knowing that cost will be a constant challenge? You might gain near-term budget cuts, but the problem will come back ten-fold when poor quality and schedule slips lead to unhappy customers.

Once you have a working program and methodology, don’t compromise on what it takes to deliver a project right. It’s better to decide it’s too expensive and walk away, rather than put everyone through the mess of a poorly run project.