Why do we need Constrained Path-based Testing (CPT)?

Why do we need Constrained Path-based Testing (CPT)?

Classical PT, without the possibility of defining constraints, might not be enough to handle several practical situations in system testing. To give a few examples:

Test cost management. Certain situations can be too costly for the given tests and have to be excluded from test scenarios. 

Feasibility of certain tests. When a test reaches certain parts of the system workflow, the internal state of the SUT might change, and it disables the continuation of the test in another certain part of the workflow. Such a situation in the industry is usually resolved “ad hoc when it happens” in the test design or in the test execution. However, such an approach will be very likely sub-optimal from a cost viewpoint. 

Outages of particular parts of the test environment. In more complex projects, parts of the processes might be temporarily unavailable due to hardware or software limitations. This fact has to be also reflected in the tests – it’s faster than letting 20 testers spend one or two hours discovering that a certain part of SUT in the test environment is not working and what they are experiencing, if not just a random set of bugs. How many MDs in total will we effectively lose by such an outage? Four? Eight? Quite enough to motivate us to find a systematic solution for this, isn’t it? Good test environment monitoring and management can help here and reduce these losses. When further powered up by good test case management, even better. 

Time-consuming asynchronous operation.

Imagine that action A represents the start of an asynchronous operation and action B represents its end. When we exercise action A in the test and wait for B to happen, the rest of the system can be tested in the meantime. In this case, such a test case that would call again the operation between actions A and B must not be present in the exercised set of the test scenarios. When it is, it may cause confusion or even a false bug report.

A need to use only a reduced subset of resources. There might be processes in a SUT that use some resources extensively, and when the availability of such resources is limited, they need not be executed. For example, an energy-intensive process might not be executed on a system with a low battery level.

Special testing cases. If you consider the previous situations as just “test management” or “resource management” issues, this is another class of problems that cannot be easily dismissed. Testing systems in various edge situations from a process or End-to-End viewpoint means focusing the test paths specifically on these situations and not testing the rest. Can this be entirely handled by test path prioritization or by test requirements? Few years we dealt with a problem of how to test the reliability of the IoT systems in case of a component outage, limited network connectivity, or signal jamming or spoofing. Systems of mission-critical nature, where one really likes to be sure that the system works reliably and safely when such edge conditions happen. Previous path-based testing concepts rendered highly ineffective from a cost viewpoint to handle such problems. Introducing constraints to the test path generation process turned out to be the right way.

 

What to do in such situations, having a classical path-based testing toolset only?

Few intuitive solutions are available:

  1.  “Twist the reality” when modeling a SUT process and model it in a way that test cases that are not possible will not be generated. Honestly, this is the wrong approach and will very probably lead to confusion later. In test automation, one of the golden rules is “Don’t change SUT because your tests would need it to overcome some obstacle. At least do it if you really don’t have other options”. We can rephrase it to “Don’t twist the SUT model to compensate a limit of an MBT method”. One of the biggest challenges in MBT (and probably one of its biggest drawbacks) is keeping the model up-to-date with the design documentation and with the SUT. In such a situation, only two out of ten test engineers remembering that “there are certain adjustments in the SUT model just done artificially to force the test paths to be executable in the SUT” is exactly what would just add frustration later.
  2. Exclude test paths (from the generated test set) that would violate the given condition. However, there are drawbacks: (1) it might take time, (2) you can make a mistake in this process, and (3) you reduce the test set perhaps more than is necessary (making your tests weaker and enabling bugs to remain present in the SUT).