Overview

A printer manufacturer with many printers and of course device drivers, is suddenly experiencing a heavy volume of support issues.  The printer product has been in production for many years.  There is a bottle neck of recent issues that stem from the fact that customer support is not able to resolve issues without direct interaction with a single developer on the development team.  This has a double impact in that this KEY developer has fallen behind on delivery of new features and capabilities for the product line. 

Structure

It must be pointed out here that this is a specialized printing device and not a 'typical' printer one might expect attached to a PC computer.  The printer device driver is installed onto the 'user' machine and is expected to appear as an available printer to the user when selecting 'printer' to use. 

The Stated Problem

A large number of installations are experiencing problems ranging from printer simply does not print, printer is not available in printer list on computer, to the user computer system crashing when attempting to build the available printer list.

This issue is recurring and has been present for a period of  months.  Every time the issue is reported an exhaustive review of source code is performed to determine exactly why the device driver is not functioning correctly on THAT customer’s installation.  This of course is the classic fix the symptoms and never really address root cause.  The problem simply resurfaces.

The Engagement

Our initial contact with the case was with the Customer Support Director.  His goal was to determine if training would increase his teams’ ability to resolve client issue and reduce the backlog of issues dependant on the development team.  We suggested a training session wherein his team (and the development team) would be exposed to a 'methodology'.  This training would focus on problem resolution and how it can be efficiently managed across the group. We assembled a group of 12 people for the training. 

It was obvious within the first 10 minutes on site that the development team was opposed to any 'improvements' in their methods.  In fact the development director declared that TWO production affecting issues would take precedence over the training agenda and since everyone was assembled those issues would be addressed.

The development director was completely opposed to any suggestions as to how BEST to root because might be identified.  IN fact the training session turned into the key developer displaying source code on projection screen and detailing HOW it works to all present. 

During the first biological break of the morning, the Support Director was engaged in a conversation about how things are going astray of the intended goal. He asked for suggestions.  It was suggested that a TOOL be used to collect test data from the client experiencing the issue.  The client was on another continent.  He emailed the tool and its instructions to the client... it took over 4 hours for the test results to return.

During this time the development director was hard at work, deciphering code, picking brains with all as to what effect this action MIGHT have, and which effect this action might have related to the reported issue. The classic war room approach we all know and love... 

After receiving the client recording, I suggest that it would be useful to review the test data... the development director declared that his team was deep into the problem and could not afford a 'distraction'.  The support director was told the trainer would return the next day. 

Next morning, with a large group of bleary eyed technicians before him, the instructor began the second day (or was it the first day) of software problem resolution training.  He asked how long were you all here yesterday?  Their reply 11PM.  But we found it and it.... a raised hand from the instructor choked off the resolution to the issue.  Shortly thereafter the development director appeared and asked if there was really any value to HIS developers being tied up for such training.

He was asked; did yesterdays issue(s) get resolved? He said yes. And about what time.... he stated around 10:30 PM.  The math is simple, 9:30AM to 10:30PM is 13 hours times the number of people in this room.  Sir, I do not know the resolution to the issue, and if you will sit here for 10 minutes I suggest that using the test data collection from YESTERDAY morning,  we will show you how that resolution might have been reached in far less time by only ONE person.

The Root Cause

Of course he was compelled to sit, and produce a copy of the test data that had yet to be reviewed.  As it flashed onto the projection screen, three mouse clicks into the maze of data and there on the wall was clear data that this machine had TWO device drivers installed on their computer and the two drivers were different versions.  The OLDEST version was the one being activated by the client requesting features only available in the NEWEST device driver.   

Recall that the day before there were TWO production affecting issues that needed focus.  This fact caused the development manager to divide the war room into two teams... one for each issue.  The instructor turned to the lead on issue number TWO and asked.. Is this also the resolution to the second issue as well.  The reply was a silent nod.

The Results

It was determined within minutes of this revelation that the version upgrade code was not performing its job correctly.  So was the root cause the device driver being in two places, or was it the installation code for allowing the old version to remain in place. 

DEPENDS on what you wish to accomplish... in this case YES they required multiple drivers to be resident at the same time... but not to confuse the reference to which is to be used.  So we took another few moments to drill into exactly why the multiple device reference was being confused.

Summary 

Many issues stem from being able to reproduce the EXACT conditions in a lab that a remote computer is operating under.  It is important to understand that this requirement is NOT necessary IF one is able to obtain an X-RAY picture into that remote computer for a clear view of exactly WHY software is performing as it is for THEM.

Recall that it took 13 hours for this war room technique to determine cause of the issues. Using our process and after the initial seconds of analysis, the resulting conversations about the need for multiple device drivers and the research into why the multiple reference was not working correctly was maybe one hour.   That is a ratio of 12x13 to 1...... OR 99.9 Percent!!!! 

Also notice that we pointed out what APPEARED to be root cause and what was in fact determined the day before as a resolution.  HOWEVER, the real root because was not found until after discussion of WHY there are two, so the actual root cause was identified, fixed and the flood of calls eased back to normal level.  It’s this fact that is most glaring between a 'fiddle-till-its-fixed' approach and one based on sound proven methodologies aimed at uncovering root cause.

We admit this is not a typical difference, and was of course obtained by the actions of the development director....and the instructor proving a point. But we still contend that if only ONE person was involved in this issue, and if that person does not need to reproduce the problem, cost for issue resolution drops dramatically.

Such an impressive difference,  one which prompted the Director of Customer Support to announce; 'As of this moment, the support team will adopt the process we have just witnessed, for what we have just seen requires neither development experience nor interaction!'.