System defects come in well-behaved and badly-behaved varieties.
The well-behaved kind stop clearly. If an error appears, someone notices and fixes it.
The badly-behaved kind fail silently.“Saved” is displayed, but nothing was saved. A list looks like all of it but stops partway. Business runs on unnoticed, and it surfaces six months later where the numbers do not add up.
This instalment is the story of the period spent killing those quiet failures.
It says “Saved,” and it is not saved.
This was the first pattern we found. The field names the screen sent and the field names the receiving side knew did not match.The receiving side silently discarded the fields it did not know, saved the rest, and returned “success.”
The screen says success. But what you corrected is not in there. It passes tests too, because the communication succeeded.
So we changed the whole design of the receiving side.If an unknown field arrives, do not discard it silently — return it as an error.That covers all 42 entry points.
It looks like a change that increases errors. It is the opposite.Typos and missed corrections now show up on the spot.Better to stop noisily than to fail quietly.
A list was being silently cut off partway.
The next thing we found was truncation of lists. When there are many records an internal limit applies, andit looks like everything is shown when in fact it stops partway — in six places.
On screen this is normal too. It says “20” and 20 are lined up, so nobody doubts it. Even if there really were 35, there is no clue anywhere to notice it.
On a screen whose counts are used for business decisions, this becomes an incident.Someone is deciding today’s workload from the number remaining. All six places were fixed to count the actual total.
We replaced apparent figures with aggregation over real data.
With the same thinking there was one more thing we fixed. Of the aggregate values shown on screen,the numbers hard-coded for the demo were replaced with counts from the actual data.
Nothing changes in appearance. Butnot changing was the problem. If you change the data and the number does not move, that number is a lie. And whoever saw that number during requirements definition takes it as “this metric is being captured.”
For the same reason, for the integration with the external service we also made it explicit on screen that it is not connected yet. Stating “not connected” from the start moves the conversation along faster than making it look connected and explaining later.
Every fix broke something, so we ran review after review.
This period was an endless round trip between fixing and verifying.
Fix something and something else breaks.That is unavoidable. So with each fix we ran an AI review, confirmed end-to-end behaviour with automated screen operation, and fixed the side effects found. We repeated that round trip many times over.
Traditionally this is a contest of manpower and time. A single review pass costs person-days, sothe judgement “we cannot go that far” always creeps in.The reason quiet failures remain in enormous numbers in systems out in the world is not that nobody can find them — it is that nobody had room to look.
Put AI at the centre and the unit cost of that round trip changes. You can run it any number of times, so“check it because it bothers me” becomes something you can always do. It is less that the quality bar rises and more the feeling ofno longer having to give up.
Quiet failures are an enemy of requirements definition too.
There was a reason we did this work before requirements definition.
Discuss with the customer over a prototype that fails silently, andthe content you agreed on becomes a lie itself.A requirement you proceeded on after confirming “it saves, doesn’t it” was in fact not saving. The later this kind of drift is caught, the more it costs to fix.
If you are going to discuss on the basis of something that works,that working thing being honest is the precondition..
Next time: cutting promises we could not keep out of the material. That one is about words and contracts rather than technology.
* The counts in this article are our own side’s work records. Nothing identifying the customer or the system is included. Approach and time required vary with the scale and structure of the project.
Series “Project Diary” (7 parts)Part 1: 82 fixes to the prototype on the day we won the projectPart 2: Cutting our own company introduction out of the first meeting’s agendaPart 3: Counting what we did not understand and putting it on the tablePart 4: Bringing the prototype closer to the real buildPart 5: Killing “it looks like it worked but nothing happened” (this article)Part 6: Cutting promises we could not keep out of the materialPart 7: Dropping the paper comparison and deciding by measurement