The Shallow Testing Antipattern
Jun 27th, 2005 by phil
Stop me of you’ve heard this one. The developers build a product that they are happy with and send it to QA or to the stakeholders for verification. As far as the developers are concerned the product is “finished”. The QA group (or other stakeholders) come back with a handful of bugs that “must be fixed” in order to release. The developers roll their eyes and fix the bugs making a new release to QA. Wash. Rinse. Repeat. This cycle continues until the project is cancelled or the stakeholders get fed up and force it into production warts and all.
This is the Shallow Testing antipattern. What has happened here is that each round of testing does not go “deep” enough and many bugs remain undetected after each testing iteration. Sometimes the bugs that show up in subsequent iterations were either hidden by bugs that were fixed by the developers or were genuine regressions. However, often these bugs are just not caught by QA.
There are several root causes for this antipattern. While it is possible that QA just isn’t doing their job, the problem usually lies further upstream. The number one culprit is a lack of requirements. The immediate cause of the shallow testing is usually an insufficient test plan. Since the test plan writer must rely on the documented requirements when creating the test plan, a poor test plan may be an indicator of poorly documented requirements. The only solution to this is to document the requirements properly and then use that to build a proper test plan.
Many organizations may be in near crisis mode when they discover that they are falling into the Shallow Testing antipattern. Then, in a rush to release, they beef up the test plan without first documenting the requirements. In that situation the test plan becomes the de facto requirements. This is a bad position for QA to be in as they are now the owners of the requirements instead of a project champion or business analyst. It is the job of QA to verify that what the developers built is what the stakeholders asked for. Once the QA group becomes the owners of the requirements, that becomes impossible.
Proper defect tracking and communication between the developers and QA are also important for avoiding shallow testing. The test plan needs to be updated as issues are reported and fixed. This serves to guard against regressions and to clarify the intent of vague requirements. It is a good practice that no code be committed to the version control repository without an associated issue in the tracker. This way, every code commit corresponds to a documented issue. QA can then use the issues in the tracker to keep the test plan up to date.
Developers must also accept a certain amount of responsibility in this area. Unit testing can identify regressions and other issues before the code makes it to QA. Also, it is important that developers provide release notes covering all user-visible changes to the code and referencing the issue tracking system.
Shallow testing can seriously undermine a project that is close to completion. It is important to identify this antipattern early and to take steps to mitigate it. The best thing that a development team can do is to ensure that there are requirements, that someone has the responsibility for maintaining those requirements and that the test plan stays up to date. If those steps are taken, then this antipattern should not show up.
