Stamp of Approval

Syndrome 1: You’ve developed a beautiful supply chain process model, metrics model, with depth and subtlety that allows you to probe with anything from automated software, to six-sigma statistics, is easy-to-explain to executives and has that rare power to illuminate business rather than merely describe. You team performs analysis, and begins redesigning processes to be more efficient and effective, and… when you work with process owners, they look at the starting process and ask the hated question “What’s this? I don’t recognize this…”.

Syndrome 2: You’ve developed a beautiful supply chain process model… the business recognizes it, and you have completed some redesign, and are approaching implementation. Software is in place, training has been rolled out, data converted, test plans are complete, and we’re about to do the go-live with the new process when someone asks the hated question, “but what about…?” What about this spreadsheet? What about the call center? What about our supplier?

Syndrome 3: You’ve developed a beautiful supply chain process model… the business signs off on it, and your reporting out to the audit committee, or SOX team, or the ISO9000 team, and you get, a day a week a month later, the hated email: noncompliance.

You’ve fallen into the trap of developing supply chain process models without validation. Validating supply chain process models is much more subtle than imagined. Many teams may take a model, and independently check with their sources, who agree that their piece is “OK”, but if they were in a room together (like in a design stage), they may disagree. Sometimes everyone agrees on a model, but you may have overlooked asking for keep inputs/outputs from an organization which people don’t realize sends inputs or receives inputs to processes, and you have incomplete models. You may have interviewed the wrong people! Managers love to describe processes as how they should be, rather than how they are. You should work with owners, and then double-check. Auditors know this, and check people who actually perform the process transactions, which is where you can get interesting non-compliance issues on processes. Realize, also, that no model is perfect. Validation is the art of balancing time spent (i.e. invested in) checking and double checking process definition with the risk of bad definition. Good teams get the balance right.

So what’s a typical validation? Gross disconnects. Do all the processes have inputs and outputs connected to processes? Are there missing processes which people don’t know why they’re missing? This is the classic area of “staple an order to your head” walk-through inspection for supply chain process models. Also, frameworks greatly enhance this type of validation by providing pre-defined inputs and outputs so you can check if something is obviously missing, and accounted for. I worked with one group who, much to their shock, found that the models they were using for analysis were grossly incomplete. The business simply didn’t realize a lot of standard inputs and outputs were present, and though in practice performed the work, hadn’t been able to identify it to process teams because they hadn’t considered the work a ‘process’. Using the SCOR framework dramatically improved the quality and validity of their supply chain process models.

What’s another typical validation? Large-team review. Isolated reviews tend to produce blinkers on the supply chain process model which are just as bad as gross disconnects sometimes. People accept definitions, though they may not own them, or are not aware that they don’t own the definition. It’s amusing to do walk-throughs with large teams, and see the illuminations on how work is actually performed arise. We have recorded sessions for modelers to review comments and specific works, and later, listening to talks over breaks during validation sessions, hear process owners talk about their shock and dismay that what they had privately understood was correct was, in fact, not.

What’s a very complex validation? Auditing a process by going into live process situations and notating every step, decision, input, output, to check conformance to the description provided by the process owner. Guess what! ISO9000 audits, internal SOX compliance audits and other types of audits do this. More than a walk-through, it’s more like a frame-by-frame analysis of a movie… expensive, but the most accurate validation of all.

Another type of validation is to check supply chain process model conformance to a given reference framework. No surprise process categories and types generated on the fly. No single-instance processes with complex sub-workflows hiding multiple instances of actual processes. No processes of type “A” misclassified as type “B” (receiving as manufacturing; planning as ordering). No new metrics, no if/then/else decisions secretly embedded. This validation is usually performed by central conformance teams on extended process teams in companies.

So what’s the balance we spoke of? We usually posed process audit/validation complexity/risk in terms of cost-of-change. For low-cost programs, where the process change was likely to be less than US$100K – US$300K, inspection and signoff audits were sufficient to remove most risk of badly modeled process, because there were usually so few processes modeled. For US$300K – US$800K programs, we usually wanted to make sure that there was not just a sign-off that processes were accepted by managers, but that they were complete. Usually enough processes here were modeled that we should make sure that all inputs/outputs were around and that walking through the process with transactions ‘worked’. For the US$800K – US$1M and above programs, we believed you need to sign-off processes, ensure that they worked, large-team review, and statistical samples of the actual processes to ensure they were as real as possible. Imagine what the cost of non-compliance for a US$30M SOX program, or where you’ve spent US$2M to get ISO9000 certification, and simple random samples show an unacceptable level of conformance. We also validate processes at this level that they are ‘standard to the company’, so we don’t get conflicting and expensive re-definition of processes in large complex systems.

Where is this done? Well, usually at supply chain process model capture time. Formally, I believe I would call this validation a type of analysis, but practice has shown that it’s best done as soon as possible at capture time (combined capture and analysis), rather than waiting until all analysis starts. Who does it? Usually the capture team. Another version is where we’ve worked with companies which have created process audit teams in central organizations to oversee all supply chain process models in all projects. It depends on how much your company invests in an ongoing basis in process, how much they want to avoid process program rework (which is what we’re talking about when you have invalid models: you have to ‘rework’ the model, the analysis, the change programs when the models are incorrect), and how much they must avoid process failure.

Takeaway today:

  1. A model isn’t enough – it must be validated prior to being used for subsequent analysis, design, and deployment.
  2. There are many types of validation, but in general validation is a ‘team experience’.
  3. The cost of validation should be balanced against the risk of failure of process change
  4. Validation can be performed by a variety of supply chain organizations; it depends on the company setup of supply chain management.