I spoke too soon last week about the imminent release of the release of a major upgrade to our software. Beta (test) sites late last week reported a several problems with the beta release which, at first, looked serious. While the problems all related to a single change, that they got out at all is causing us to tighten even further our in house testing.
Part of the challenge is the complexity of vertical market software. Beyond traditional business functions there are many industry specific functions which interact in ways with can be unexpected. A small change last week had unintended consequences. While we tested the change, we missed a couple of steps and delivered some frustration for our beta tech colleagues. It’s our fault and we fixed it quickly.
We have played with commercial testing software and found this wanting. There is nothing like real world testing because you cannot code to emulate user behaviour.
In the past we have used pre beta sites to play with the software before it goes to a larger beta community. We’ll do this again and track the impact.
It would be easy to not blog here about the beta blues of the last week. To do so would deny a problem common to every software company. Software bugs, no matter how stringent the testing, get through. Design, development and testing are human processes. The keys are how you respond and what you learn.