The reason as to why silent failures are the worst failures of automation is discussed.
I had it in mind previously that a broken automation would be reported to be broken.
Everything seems obvious, but I was once in the illusion quite a long time.
Melodramatic in my mind were failures. An error message of red color. A Slack alert. Upon a roll profile of acrimonious icons. Something loud, which I could hear it, and prevent any actual hurt before it were incurred.
It was the fantasy.
It was worse still than that.
The automations were the most dangerous, the ones that I have always constructed, which seemed to be ideal.
They were punctual.
Green checkmarks were seen on them.
They provided me with confidence.
And as they were methodically erasing records, circling the branches, cashing family records and leaving behind them little islands of havoc that would not be noticed until days later, when somebody in an organization would say, Hey, this number is out of the window.
At this point my paradigm changed in this respect.
I ceased inquiring, Did the workflow fail?
And began to ask, was the workflow that I believe it was?
The question presents itself differently.
It is not only a technical problem of failure. It is a question of trust. It makes you believe that the machine is carrying you on when in fact it is letting things fall in the cracks.
And that is why I believe that silent failures are most dangerous automation failures by all.
I Was Swindled by the Day Green Checkmarks
It was also among the first instances when this happened to me and it was in the process of routing a lead workflow.
Nothing fancy. The form was accepted, the data was purged, a check on the owner of the lead was done, an owner was assigned and a record was forced into a CRM. Pretty standard stuff. I was comfortable at this stage since I had already come up with a sufficient number of automations. Maybe too comfortable.
Then after seven days all appeared fine.
The workflow ran.
No error logs.
No obvious complaints.
Then sales would question themselves why a campaign was not so well off. The amount of money used on advertisements was actual. The traffic had been genuine. The qualified leads in the CRM were however abnormal.
I first put the blame on attribution.
Then I charged the excellence of form.
After that I literally followed the workflow step by step.
That was the go to the gut.
The requirement to treat a single source value differently as a result of an insignificant formatting difference was one of the conditions of a splintering step. Same campaign family. Slightly different label. They are not books that were dropping their mistakes. They just were getting into the wrong road and never reaching the CRM.
The automation did not have a discontinuity.
It had been repaired more than repaired.
Without a scrupulous moment it was mixed up.
The Failure That Does Not Scream
When a loud failure happens, it is inconvenient. An unspoken one costs a lot.
And this is what is so ugly in simple words in reference to silent failures:
- The workflow remains to be healthy.
- Particular parts of the process are still active.
- There is some part of the data that is through.
- Until downstream damage is evident, nobody pays attention.
- And days or weeks of bustle had overthrown the root cause when it was found.
The trend may be traced everywhere.
A sync job is a job which stops the updating process of just one field and continues to update the other.
A procedure where one group is not approved with a notification owing to change of role name.
A reporting pipeline that is running continuing to run every morning but taking in the cache of the previous day as input.
A time out enrichment of 8 percent implies enrichment of a silent pass only.
No alarms. No drama. And bad work in the pretext of trustworthiness.
It is there that it is dangerous.
The Elusion of Success Fugitive
This is one of the concepts that I had to learn again bitterly as I believed that a good run is a good outcome.
It does not.
An execution can be informed by a workflow platform. That is a help, but does not amount to telling you that you are right in business.
Such difference altered my construction style.
Previously I was mostly searching in case of technical breakdowns:
- Response of API?
- Has this node been performed?
- Was there a crash of the script?
- Has the workflow been finished?
then now am more disturbed with logical failures:
- Is it all the records that we expected we have dealt with?
- Were there any branches where it was getting nothing when it should have got something?
- Was the number of inputs of these and the number of outputs of these sufficient?
- Was it really necessary to make blank a field that was necessary?
- Did we have the correct version of the data in the correct system?
It is an opaque type of transition.
It is not.
It completely changes your position when the machine has made its way to the machine has done well.
The Branch Which Silently Eateth Records
I want to provide you with real example since it is where the lesson actually hit me.
My workflow was the processing of the support tickets. The received tickets were tagged and sorted into different paths according to urgency, degree of customers and product field. I possessed a switch node which was clean. Organized. Logical.
It was sincere, of which I was proud.
It is then that I then noticed that there was a slightly low rate of processing enterprise tickets compared to the normal process. Not a catastrophe. Sufficiently to be strange.
I excavated and discovered the cause of trouble. The switch logic had no coverage of a newly added product tag. These were not defective tickets. They were sliding in to a generic direction of lesser priority.
Everything still “worked.”
Nevertheless, the most precious tickets were getting average service.
This kind of bug is a violent one since it does not leave debris. It does nothing but create drift. And less apparent than is wreck drift.
Why this matters
The majority of automation failures are not binary since many of them are not binary. They are partial. Selective. Context dependent.
And half-failures are always first likely to hurt your best work-processes.
The VIP customer.
The edge case which is not normal.
The campaign name has been slightly modified and that is why this version is named so.
The single area with another date format.
The process of work continues and you unwind.
It is at this point that it gets dangerous.
Stale Data Is a Failure Not a Failure
This was offered later and I was surprised at this fact because I assumed that silent failure will imply that records are missed.
Sometimes it does.
It however, has wrong records.
I had created a reporting process, which had built the data of the orders, combined it with the product metadata and forwarded a summary to a dashboard and a daily mail. During a certain period it was fantastic. Clean output. Consistent delivery. Happy stakeholders.
One of them then asked a question, why were non-continued products under a category summary.
It was neither an issue of logic of reporting. The new input was the freshness input. The reason why one of the upstream data pulls was to a cached export was not being refreshed correctly anymore. It is the way the working process was prepared during the day with technically valid information which was structurally sufficient and totally obsolete.
Again, no erythromelalgia.
Just stale truth.
I still consider that expression annoying.
Selling is stale truth. It appears to be more in the reality that individuals begin to make decisions on it.
Why this matters
Even the workflow that is not broken is even more hazardous than the broken one.
An unsuccessful workflow generates friction. People notice.
Ad hoc confidence is caused by a workflow immobility. People act on it.
Bad judgements proliferate in such a way.
What I Changed When I got Burned
When these were more than enough, observability no longer seemed to me a pleasant optional quality of mature systems. I began to approach it the same way as the real cost of admission.
Nor can observability either sight. Clinically positive and to some extent, obsessively observable.
The following things have become the additions that I am making to virtually every valuable workflow:
- Verifications of the records in between significant phases.
- Notices on abnormal drops or spikes.
- Logging of branch distribution
- Rules of critical field validation.
- Checking of the data updatedness of the sources.
- Fallback paths which log loudly rather than silently fail.
- Short reports are dispatched when there is high impact execution of workflow.
This resembles overhead.
It is overhead.
And worth it all, worth it all.
Spending three hours a week to save time on working is not very efficient and, at the same time, every month, your pipeline is ruined. It is a trap.
Mini Case Study: The Misplaced Finance Approvals
The standard silent failure mode was observed in one of the operations that I helped to perform the review in the finance operations.
The invoices were sent through email, attachments were done, values were accessed, approval rules were applied and approvers notified. Everything looked polished.
Invoices were not escalated in the right manner as well and that was too far.
Why?
The parser returned the total sometimes as some text with an unexpected character. The reasoning of approval failed to go amiss. It simply could not do the comparison of it passing the invoice on the normal route and not the high value approval route.
And that is a dangerous sentence to write.
A high value invoice, which happens on the normal route due to a parsing anomaly is the kind of thing which may be left lying around until the audit season or until someone with a great deal of influence is much vexed.
It was not fixed by more complicated workflow. It was a less objectionable check.
We added:
- Prethreshold type checks.
- Parsing anomaly alerts.
- A run-invoice approval-count.
- It is that which is suspicious in a review bucket.
Very boring changes.
Very effective.
The old fashioned and the new fashioned
I would develop automations like a carpenter.
I make them now with doubtful unconsciousness.
Old way:
- Focus on the cheerful attitude to work.
- Errors are the primary threat to be considered.
- Assume, it is the production of the well-being successful operations.
- Check logs where something does not work.
New way:
- Pay attention to the fact that it is possible to have a defective workflow but not a crashed one.
- Take into account the silent drift as the main danger.
- Dissociate performance and performance accuracy.
- Faith was given without being verified.
The latter is noteworthy.
There should be credibility gained.
It is not expected to be the planned workflow too and there is the row of green executions.
The Sincere Cons will no one talk of
I ought to be just here.
The creation of silent failure detection is expensive.
It means:
- More setup time
- More alerts to tune
- Additional logging to look at.
- Other inferences on the look of normality.
- Increased maintenance caused by change of workflow.
It is at times of creating a spy on a spy system.
And of doing excessively of it is hazard. There is no observability of small automation being enterprise grade. You do not require the whole layer of anomaly detection when renaming files in a folder after 1 week of work.
But all on revenue, customer communication, finance, operations or reporting?
Different story.
And there failures in business are silent failures.
Should You Care?
Yes, mind, yes, mind, since you are one of these people:
- Minor safety net critical automation managers.
- Multi tool Ops group, multi step process.
- Founders which utilized dashboards and automated reporting.
- Lead alignment and associatedness of the marketing teams.
- It has physical effects of operations that have been lost in either the support or the finance department.
- He who continued to say, It worked, and so I guess it was all right.
Particularly the latter one.
Either you are decision making, money, customers or compliance and then you will be more concerned with silent failure than visible failures.
Labor breakdowns are experienced.
Silent failures pervert reality.
The Reckoning of Where This Is Going
I believe that automation will become smarter in the future not only in the field of workflow building.
It is more intelligent workflow scepticism.
Those systems will be coming up not just along with the report of the fact that a run has been made, but with whether the output is a suspicious one. Increased automatic baseline verification. Freshness awareness more. Increased disclosure of the levels of the branch. Greater instruments which not only inform us about what happened, but what did not happen without our knowledge.
It is the composition that I am the most interested in at the moment.
Not just automation.
Doubt at the price of robotization.
That would be a funny thing I say. The systems of the day will never be able to operate forever with confidence even the best systems in the coming years. They will be aware when one does something, but there is wrong about it.
An intelligence which is far more practical.
The Next Thing I Would Do to Your Workflows Tomorrow
The first things that I would audit in the case of the auditing of a workflow tomorrow involve the following:
- Find the maximum cost silent failure.
- Compare outputs and inputs of every significant stage.
- Check all default paths and backup code.
- Determine whether the source data can become outdated or not.
- Check field type and check field format.
- Use one alert in case of workflow done and the result appears to be abnormal.
- Establish a mini manual review queue of uncertain cases.
There is no need to take it all to pieces.
You only need to stop believing that the fact that there is no error is the manifestation of the rightness.
A change in attitude of this kind changes a lot.
That Part That Sticks With Me
It is not the technical teaching that made me change.
The sentimental one it was.
I found myself to be giving confidence on systems that I was not checking. I referred to workflows as reliable since they were silent. I was confusing calmness with stability.
Such a mistake is so human.
We do it anywhere.
In software. In teams. In processes.
When we are not aware of anything wrong we sleep.
Not all most hideous evils are proud. Such people merely go on with their ways of life in the background until the price is ultimately settled either way which cannot be overlooked.
It is through failures like this that I am so annoyed at the moment.
Since they are not merely disruptive to the workflow.
They destroy your illusions in yourself.
And then, there will be no need to question whether the automation is being implemented.
You start asking whether you can believe what you are seeing.
That is a much deeper question.
And an even better one, at any rate.
And now it is my time to wonder.
Have you ever had an automation that was functioning well until a time when you have realised that it was not functioning good the entire time?
Leave a Reply