The final instalment. Here is the judgement that mattered most on this project.That we did not decide performance by a paper comparison.
Requirements documents usually say “main screens within three seconds.” And at the moment it is written, nobody knows whether it can be met. You find out after building it and putting production data in.
So we changed the order.Before entering requirements definition, we stood up an environment and measured.
We loaded 10,000 records and confirmed that no screen worked.
We stood up one verification server, made it reachable from outside, andloaded 10,000 project records.
The result was this.
Displaying the list took21.6 seconds. Keyword search took22.4 seconds. Even the detail screen, showing a single record, took11.7 seconds. Andall 11 screens failed to display even after waiting 90 seconds.
Chasing the cause, it was plain. Even when only one record was wanted, all records were being read. And because the record count shown in common across every screen called the same processing,every screen you opened hit those 22 seconds.
What matters here isthat we could fail like this before requirements definition. Had it surfaced after “within three seconds” was written into the design document, the design would have had to be redone.
We took 22 seconds down to under 3.
We applied the countermeasures in stages.
When taking one record, take only one. Put indexes on the fields used for searching. Reuse the results of identical queries. Andwe separated the database used for reading.
We created30 indexes. Eight of them are special ones for speeding up partial-match searching. We combed out every field that could be searched on so that nothing was missed.
The results came out like this. List 21.6s →1.9 seconds. Search 22.4s →2.6 seconds. Detail 11.7s →0.04 seconds。All 11 items came in under three seconds.Screens that would not display even in 90 seconds now open in a few.
These numbers change the requirements-definition discussion. Instead of answering “yes we can” to “within three seconds please,” you canproduce “measured at 10,000 records, here are the figures.”.
An incident happened, and it exposed a hole in the design.
Finally, a story about what did not go well.
A setup with a separate database for reading requires a mechanism to copy what was written over to the reading side. That copying processdeleted a large amount of data one day.The reading side, which should have held 10,000 records, was down to 82.
The writing side was unharmed, so the actual damage was recoverable — but chasing the cause, several things had stacked up. When the copying process is stopped partway and restarted,a state can be produced in which only delete operations are applied and add operations are skipped. On top of that, two people were carrying out recovery at the same time and breaking each other’s work.
The countermeasure was to put a mechanism that always skips changes already applied into the copying process. After putting that in, we forced the same conditions once more andconfirmed that not a single record was lost.
And there is something we learned.This hole was cheap because we could hit it before requirements definition.Had we hit it in production, the business would have stopped. The value of running the real thing at an early stage is not speed butgetting the failures over with while they are still cheap.
This much changed between sales and kickoff.
Let us line up, at the end, what has been written across seven instalments.
On the day we won the project we raised the prototype to a quality that could be touched. We cut our own introduction from the first meeting and made time to hear the other side’s conditions for success. We counted 37 things we did not understand and put them on the table. We brought how data is held into line with production and replaced apparent processing with the real thing. We combed out and killed the places that failed silently. We cut promises we could not keep out of the material. And we measured at 10,000 records, hit an incident, and fixed it.
All of this is finished before the kickoff.
With the conventional approach, most of it would have been done after requirements definition, in the implementation phase. And because it is built once and then corrected, it was called rework.
The effect of AI-driven development is not “the same things finish faster.”Because the real thing exists first, what you can do at the early stage changes. What was argued on paper can be argued by touching. Performance that was estimated by imagination can be answered by measurement. Failures you were going to hit in production can be hit before requirements definition.
It is not a story about becoming faster; it is a story about the order being swapped.And this swapping of order matters more to whether a project succeeds than efficiency in implementation does.
This series pauses here. The record of requirements definition itself we will put together in another form.
* The figures in this article are measurements in a verification environment (10,000 project records). Nothing identifying the customer or the system is included. Performance varies with environment, data volume and feature composition.
Series “Project Diary” (7 parts)Part 1: 82 fixes to the prototype on the day we won the projectPart 2: Cutting our own company introduction out of the first meeting’s agendaPart 3: Counting what we did not understand and putting it on the tablePart 4: Bringing the prototype closer to the real buildPart 5: Killing “it looks like it worked but nothing happened”Part 6: Cutting promises we could not keep out of the materialPart 7: Dropping the paper comparison and deciding by measurement (this article)