Just how much testing does an app really need?

Feb 19, 2020

Earlier this month [February 2020] a new software app that was time critical, mission critical, brand new and that had the eyes of the world on it was used. To put it mildly, it was less than a success. One of the key causes of the problem was a lack of software testing. But what form does software testing take and how much of it do you really need to do?

We are of course referring to the US Democratic Party in Iowa’s use of their new mobile app to record the result of the caucus for their presidential nominee. But app failure due to lack of testing isn’t restricted to the big boys, it could affect a project you’re working on too.

As software developers ourselves, it’s plainly not good for our industry to have failures like this. By way of balance, it’s true to say that there are software projects that don’t suffer as badly as this one did. One of the reasons that can make or break software development is often software testing.

Building software testing into your development lifecycle is very important. With some applications an iterative or agile design may be used to continually improve on the code base. While that gives you more room for maneuver than in, say, a waterfall project delivery schedule it still mean thorough testing should be undertaken at each step. After all, iterative development is only as good as the feedback that is provided for each development cycle.

While software testing sounds like a no-brainer there are reasons why many companies (and, I am sorry to say, some development teams) shy away from it. Primarily it’s because software testing isn’t actually that easy to do, at least if it’s done properly.

Let’s firstly consider who does the testing. Candidates might be the development team themselves, the end user, the customer (if they are different) or a professional testing house. In each case there are pros and cons to each selection.

The end user Pros: As close as we can get to a genuine user! Cons: May not be motivated, Difficult to engage / compensate, May not report bugs in a useful way, Can only be used a few times until they are no longer an “average” user. The customer / paying client Pros: Highly motivated. Cons: Know exact requirements. May not report bugs in a useful way. May lead to project scope-creep. Development team Pros: Very good at capturing required information for fixing the bug. Cons: Can be expensive. May not know exact requirements for edge cases. Do not act like end users. Professional testing team Pros: Very good at capturing required information for fixing the bug. May have access to testing pools. Cons: Can be expensive. May not know exact requirements for edge cases. May not act like end users.

One of the most difficult feedback types to trap and which often only comes out later is the edge case. These cases are bugs that can only be identified with a very narrow range of possibilities. For example let’s say someone manages to set a value to 101% which crashes a calculation, or they use the browser’s back button after a certain operation. None of these are planned actions yet users will do them at some point. If you’re planning the app for use in a critical environment with a non-trained user base (as the team over in Iowa did) then they’re highly likely to occur. Add to that the fact that even a very small chance can easily scale to a significant problem if your user base starts to get large.

Let us not forget that when a bug is identified a developer will need to step in and fix it. In order to do the fix, they really need to reproduce it (not least so they know when it’s been fixed) and therefore knowing the exact setup and steps that were taken would be ideal. That makes testing rather a chore.

But let’s not focus entirely on user interface and user experience (UI / UX) testing. There is nearly always a business logic layer. This is key to how the application adds value to the entered data and is behind reporting or output that businesses and users may rely on. From experience, we have learned that business flows of this kind can be extremely complex and so testing these in an end-to-end fashion can be extremely time consuming.

Automation can play its part of course. There are software applications that automatically report bugs when they are detected and good developers can write automated tests to ensure that application logic does not change with updates over time. The latter are useful for business logic in particular.

So when faced with the Iowa app the flow seemed simple - download the app, enter the security code for the area, tap in the results and hit send. So simple in fact that testing wasn’t factored into the app development or roll-out at all. Furthermore the opportunity for training was not taken up either, which given that they knew who the end users would be should have been quite possible to do.

So there’s plenty that can go wrong and having a development team that are professional and in tune with such potential pitfalls is a must for your project. With mission critical apps you often only get the chance to get things done correctly once so make sure the partnership you forge with your development team is a good one.

« Read more posts