Hi, welcome to this lesson where we will look at the pitfalls of testing strategies.

What content is covered in this lesson?

In the next few minutes we will talk about

• the Test Case object,
  • test cases,
  • testing actions,
  • test targets,
  • test results,
  • test chains,
  • test series,
  • test runs and
  • test templates,
• manual and automatic testing,
• the two basic automatic test methods,
• the influence of test data on the test strategy,
• positive and negative testing,
• the consideration of older inventory data during testing,
• how we deal with the continuously growing number of test cases,
• the organization of test templates,
• the impact of test strategy on sprint planning and team staffing,
• the configuration strategy,
• the result of a strategy on product quality,
• and we ultimately look at the financial impact of a sustained successful testing strategy versus typical missteps.

In advance, again the definition of technical terms in this lesson:

What is a GUI?

A GUI is a Graphical User Interface, which is the screen content of a software application that a software user is shown to use the software. The screen content (also called the GUI) has various elements for display and input such as lists, menus, buttons, input fields, etc.

What is an Exception?

Exceptions are situations in which the program does not continue to run as desired. These exceptions require solutions in the program code that take effect when a user or the code itself performs an inappropriate action. The exception handling not only prevents a program crash with a solution operation, but also gives the user a hint for the correct execution of an action by means of a message.

What is Dev?

“Dev” (as in Development) refers to development units that are organized into teams below the program level.

What is Ops?

“Ops” (like Operations) refers to units for operating an application. DevOps is a form of organization where Dev and Ops work together as one unit with a common organizational structure.

What is a Bug?

A bug is an error in the programming or configuration of an application that leads to a result that does not meet the requirement expectation.

What is a Production Instance?

A production instance is the hardware and software infrastructure required for the day-to-day operation of an application.

What is a Test Instance?

A test instance is the hardware and software infrastructure required to test an application.

But now to the content of this lesson:

What does the life cycle of a test case look like?

Let’s first start with something very basic: unlike almost all other requirement objects, which usually have a lifecycle e.g. from “Backlog” to “Done”, test cases have a completely different lifecycle, namely:

• “In Preparation”
• “Active”
• “Inactive”
• and “Archived”.

Of course, this status network is extended on a company-specific basis to reflect the company’s own organization. But basically it is the two status values “Active” and “Inactive” that distinguish test cases from almost all other requirement objects. Only the Non-Functional Requirement object, the Challenge object in the Value Stream, and the Solution object in the Solution Train have a similar lifecycle. A test case is not set to “Completed” after a test run, but is used again and again with the status “Active” until the test case is no longer applicable due to fundamental changes and is therefore “Archived”. A test case is adapted to new circumstances during its continuous reuse, but must never lose sight of its original purpose. In case of a detected bug, a test case is set to “Inactive” so that it does not interrupt the further progress of the test run until the bug is fixed. After the error is removed, it is reactivated. Even if a test case is archived because the basis is no longer applicable, it can be reactivated when the basis is applicable again. A test case is therefore never finished, but only used. So much for this basis.

What is a test action?

A test case contains several test actions, a test target and a test result. If the test result matches the test target, then the test was successful. Unsuccessful tests lead to root cause investigation and creation of a bug if necessary.

What is a test chain?

Multiple test cases can be strung together in a test chain that maps to a use case.

What is a test series?

Several test chains can be set up in parallel in a test series. Several test series can then be started in one test run. A test run is complete when the longest test chain has been completed. For a continuous test, the test run can be automatically restarted immediately after completion.

How does manual testing differ from automatic testing?

Now let’s talk about the difference between manual and automatic testing:

In manual testing, a human works through one test case at a time, executing the steps described in the test case in the application under test, and is expected to achieve the defined result at the end of a test case. This procedure and the associated effort must be repeated with each test run. Test runs are performed per sprint for unit tests, other test types per release depending on the test and deployment strategy. In extreme cases, continuous deployment, which is mainly used for standardized cloud solutions, involves continuous test runs that run in an endless loop. In such a case, I think everyone will quickly realize that manual testing quickly leads to a dead end. The dead end in reality means that a test case is performed only once, contrary to its meaning, and is not repeated. Today, manual testing is mainly used where a high testing effort is associated with a limited number of users, corrections can be implemented quickly internally and quality defects cannot damage the image of customers. Today, this primarily affects in-house applications – even in large companies with several thousand users. In addition, there are applications in end products. And that has been an adequate way so far. However, since end products contain more and more electronics and thus more software, the previous approach no longer works. And so it happens that companies are not only questioning the previous path due to their own quality requirements, but regulatory requirements are also forcing a reorientation.

And that brings us to automatic testing. In automated testing, a human sets up the test case once in a test management system and a test execution system. This setup requires a much higher level of expertise and also takes longer than setting up a manual test case in a test management system. The qualifications required for this are rare and therefore expensive – especially when experience that is absolutely essential for management tasks is a prerequisite. In addition, automated testing requires a test infrastructure that incurs significant one-time investments and ongoing costs. However, this disadvantage is quickly amortized because the execution of the tests does not generate any further manual effort and can thus be scaled flexibly and cost-efficiently.

What are the approaches to automated testing?

There are two different approaches to automated testing:

GUI-based test automations use a recorder that records work steps in the form of keystrokes and mouse actions in the application and then plays back this recording in the test run in the current development version. This automation approach is easy to set up and just as easy to use. However, this approach has a big catch: mouse actions are recorded with pixel reactions on the screen and played back at exactly this pixel. If the screen resolution now changes, then the recorded action is executed at a completely different location and the test case generates an error. Also, if the position of a GUI element changes due to a redesign or an addition, the test case runs to nothing because there is no longer an input element at the recorded pixel. In practice, this leads to the fact that existing elements are not touched because of the high effort in the adaptation of test cases and therefore the solution development is strongly limited and thus provides suboptimal solutions. But mouse actions are not the only problem here. Also tab actions from one input field to the next do not lead to the goal if an additional input field is inserted into the sequence. Also in this case, all test cases must be revised or the sequence of the new input field must be placed suboptimally at the end of the sequence. Als Fazit bleibt zu sagen: GUI-basierendes Testen ist günstig und einfach umzusetzen, aber ungenügend beim Ergebnis und der Pflege – vor allem bei langfristiger Betrachtung, der Berücksichtigung der Nutzerfreundlichkeit und dem Bedarf fortwährender Anpassung bestehender Test-Fälle bei Änderungen in der GUI.

And where there is a problem, there is often a professional solution: With interface-based solutions for test automation, there are no dependencies on a GUI, because the test case is managed via the interface between the test management system and the application under test without GUI interaction. Usually, the REST or SOA interface of an application is used for this purpose. The test actions of a test case are executed directly in the code of the application. But setting up and maintaining such test cases requires the high level of qualification already mentioned, as well as the existence of an interface with appropriate documentation.

How is software with status networks tested?

Now we could say to ourselves: hey great! Solution there. Let’s assume that we have also found the necessary human resources with sufficient qualifications and experience. And we also have a test management system with a connection to a good test automation tool. So we’re ready to go. Yes, basically. But unfortunately, not one application under test is like another. Each application under test is a unique test project that requires an individual test strategy – or at least the adaptation of an enterprise-wide test strategy.

In ERP, PLM, and ALM systems, objects typically move through a status network, e.g., from “New” to “In Preparation,” “In Process,” “In Review,” to “Released” and “Archived.” If we now test objects with the status “New”, then these objects are no longer available for the next test run with the origin status, but only with the result status. So we lose the test base with each test run, while more and more objects with an advanced status accumulate on the other side. In this example, we could still use the test object only to perform the archiving step. I would like to discuss an approach to this in a few minutes.

What is positive testing and negative testing?

Another challenge is that not only the ideal way to achieve a goal should be tested, but also all conceivable error actions of a user and the feedback required for them. What good is a positive test result to the user if every action that does not meet the developer’s ideal crashes the application or triggers an incorrect follow-up action? This cannot be the goal of testing! So testing includes a variety of negative tests in addition to positive tests. A good indication is an average of 5 negative test cases per positive test case. At least. The ratio varies greatly depending on the application. The more test actions there are in a test case and the more options there are in the execution of a test case, the more negative test cases are necessary to ensure error-free operation. This can be as many as 50 negative test cases or more per positive test case if all combinations of error possibilities are checked. However, this ideal is often opposed by the purse strings, so the definition of negative test cases is limited to the most common and foreseeable sources of error. Moreover, a good application developer already takes into account all the necessary exceptions and has already built in solutions to catch program crashes and related messages to the user. But: nobody is perfect and therefore negative test cases belong in every test strategy with the goal of a high-quality application.

If we now assume that in a company for a single software application only 1,000 requirements (means user stories) are implemented per year, each requirement has 3-5 acceptance criteria, these are backed up with positive test cases and 5 negative test cases are set up per positive test case, then we quickly arrive at an enormous number of test cases per year – for just one application. And a large company has several hundred of these. Please also keep in mind that we have not yet spoken about test cases for risks in this consideration! In the end, we are talking about 20-50 million running test cases in a company after 10 years! Well, some of them will fall out over time and not all positive and negative test cases need to be recreated for every requirement. We can often extend existing test cases. But ultimately, it remains a huge and exponentially increasing number of test cases that must be managed with a test strategy that works over the long term.

How is inventory data tested?

With this amount, however, the flagpole is far from being reached! So far, we assume that only new data (means inventory data of a single generation or version) is tested. However, the data structure evolves over the course of versions. The goal must therefore be that a user must also be able to work error-free with the data of the previous version and even older inventory data – regardless of whether it is migration data or unchanged legacy data. We have not even considered these test cases yet. But basically the calculation is simple: the already calculated number of test cases is multiplied by the number of supported previous versions. 5 supported previous versions result in 100-250 million test cases in the aforementioned calculation. Wowww! And these need to be organized and maintained in a structured manner in order to keep the number and the associated computing effort (i.e., the test runtime and the infrastructure capacity required for this) within limits.

The only question is how?

How can the effort for automatic testing be reduced?

Basically, the task is to classify test cases, test chains and test series in a clear structure and then provide a standardized template for each test case of a class. This gives the test manager an overview of where and in what form a new test case is found, added to and maintained. And yes: with a comprehensive test strategy as described above, it takes several times to insert a test case in the right places in the existing system. In order to maintain an overview and work cleanly, a very special qualification of technological competence and a clean and structured way of working is required. The requirement of an agile team for task rotation and knowledge dissemination within the team is difficult to implement with these very specialized qualifications – and yet test managers should remain part of the agile development team because only then can it be ensured that test cases are set up in line with customer requirements. However, if a test manager in the team has sole responsibility, then the dependency and pressure to perform is great. Especially in the case of illness or vacation, it would not be possible to set up or maintain test cases. And in the event of a complete loss, the knowledge would be lost as well. Does not go at all! Redundancy in the team is expensive and may not be fully utilized. So that’s also hard for the project budget to convey. What remains is the spin-off into a tester unit, which provides the necessary capacity centrally and still works as closely as possible with the development teams in order to be able to understand in which application area the respective test case is located. Another benefit this provides is the standardization of the test strategy across all areas, solutions, programs and teams. And that’s what I would suggest to most companies. A tester organization that sits in between Dev and Ops and thus works closely with both areas.

How are negative test results handled?

If we consider the enormous number of test cases, it is easy for all of us to imagine that the further development of an application will result in a lot of negative test results. The question is: how do we deal with it? Disregard? Then all the effort was for nothing and we could have saved the money. Investigate any negative test result and depending on the cause, adjust the test case or report a bug and temporarily disable the test case? This takes a lot of time, but yes: this must be the necessary consequence to ensure the quality of an application. And the budget required for this must be planned into a project. Nowadays, bug reporting can be automated by deactivating the test case and creating a bug object in the backlog of the deposited development team. With such comprehensive test strategies, however, the cause of negative test results is mainly due to the need to adapt the test cases. The development team would therefore be flooded with tasks that would then be passed back to the tester unit after all. We can then save this expensive and nonsensical effort and leave the processing in the tester unit, which then only creates bug reports for the responsible development unit in the applicable case.

How much effort is required for bug fixing?

Another challenge is the processing of bugs in the sprints of the development teams. Until today, it is common that bugs are categorized and prioritized and are always processed when there is still capacity available in a sprint after the assigned requirements have been processed. The result? We are all familiar with this: only the worst bugs are eliminated, all others are left to mold in the backlog and lead to consequential errors in the application, which often have negative effects on the application architecture and make an application so opaque that no one can see through it anymore. Things look better in companies that schedule a fixed capacity share per sprint for bug processing. But here, too, the majority of bugs are moldy in the backlog. The sheer amount of bugs that results from a comprehensive testing strategy can in no way be worked through within this framework. And if bugs are not worked through, then the goal of a comprehensive testing strategy will not be achieved. In other words, we can save ourselves all the effort here, too, and prefer to accept the consequence that we deliver poor quality and do not meet regulatory requirements and thus ultimately drop out of the market. Oh, I see: no, that’s not possible then either. So the only thing left to do is to consistently work off bugs. The solution that has proven most effective so far is a bug sprint between multiple dev sprints. The 3:1 ratio has worked quite well, but is also highly dependent on each company. This budget must also be included in a project and program. Nothing comes from nothing, even if management levels or purchasing departments would like to enforce it that way. Logic can no longer be banned during implementation at the latest.

Which test strategy fits the project?

Now let’s move on to one of the most important points for the test strategy – in my opinion, even the most important point, because this is where the greatest potential for success and failure lies. By failure, I mean primarily the negative cost impact when the strategy has led to a dead end. To be precise, I am talking about the configuration strategy of a test environment.

In the simplest case, imagine that you have a test case with 3 test actions, a test target and a test result. This test case is started with a test run. Quite simple. No problem.

Now, as discussed before, you have not only one test case, but several test cases. If you configure them one after the other in a test chain and then start them in a test run, the test run will run for an extremely long time – much too long with exponentially increasing test cases.

Here’s just one real-world example: an automotive manufacturer in Germany has an ambitious strategy to move from today’s 4 releases per application/year (which is not bad!) to monthly, then weekly releases to achieve the goal of continuous deployment. This means that application changes are continuously made available as functionality in the applications via process automations after all steps have been completed. For testing, this means that the test cases can never be processed in a long chain one after the other.

O.k. Then let’s just configure the test cases all in parallel in test rows. Each test case in its own test series. And we pack all the test series into one test run. This should be completed in no time at all. Unfortunately, this is only true in theory, because imagine: on the application infrastructure, all actions are executed simultaneously. There you need computing power without end, which goes far beyond the performance requirements of the productive instance – so much that it cannot be provided in reality. The test instance immediately crashes at the test run start and we unfortunately achieved nothing with it.

Since both extremes are, as expected, not purposeful, we recall the previously mentioned work with classification and templates. Our goal must be to keep the test run length short enough to meet the company’s individual requirements for release times, while at the same time not overburdening the test infrastructure and keeping the necessary costs within reasonable limits. Here, too, there is a budget requirement that is often significantly underestimated and thus underbudgeted. Who ever thinks that the test infrastructure might have a higher budget requirement than a production instance?

The goal of our efforts is to map complete use cases in a test chain. For short use cases, 2, 3 or more use cases can be strung together. Always with the goal of achieving the planned test run length. And because experience shows that use cases are constantly expanding, it is better to have a decent buffer, because otherwise the chain will have to be split up with additional effort. Unfortunately, this cannot be avoided either and will become a regular additional task. But we can’t afford too much buffer either, because otherwise the demands on the test infrastructure will become too great and the expensive infrastructure will not be utilized economically.

The ideal result is a closed block of test chains and test rows that makes the best possible use of the infrastructure and still includes extension buffers. In this context, flexibly expandable cloud offerings have proven their worth, if security aspects permit.

And now, as before, let’s start with the requirement extensions:

If we test data objects with a changing state network, then we must first create the necessary data objects for each test chain upstream of the test case. To avoid cluttering our test infrastructure, we should delete these data objects at the end of the test chain with administrator privileges – if the application allows deleting objects at all. If we don’t set this up appropriately, then quite quickly we have a mountain of data objects with an advanced status that overloads the test system and ultimately has to be cleaned up with manual database intervention. It is therefore better in any case to clean up at the end of a test chain.

And if we work with test series and a user can only perform actions once at a time due to the system, then we must first create the users of the test chain with the necessary roles and rights for each test series upstream of the test case. Within the test chains, the generated users log on and off during the test run in order to be able to map the entire use case. To avoid cluttering the test infrastructure, we need to delete the created users with administration rights at the end of the test chain.

So, for each test chain, we generate ourselves all the test data with state networks that are needed in the application under test. Other test data that do not have a status network can be used permanently as in any other application. Well, that looks good now, doesn’t it? Let’s just say: that was a good start!

After all, now there are the many negative test cases per positive test case. So we add the negative test cases to the existing test chains by adding the negative test cases before each positive test case. In the way: wrong doesn’t work (what means: expected test goal of negative test case is reached), wrong doesn’t work, ahhh: right works, so it can go on. This extends the previous test chain by a good factor of 5. This must be taken into account when setting up the positive test chain, or the test chain is set up from the start with all negative and positive test cases. But beware: the negative test cases are constantly being expanded by experience gained from user support, complaints, risk management and regulatory additions. Well, it looks really good now, doesn’t it? Let’s say: this is now a good intermediate status!

After all, up to now we have only been working with the latest inventory data. We now still need a defined number of previous versions, whose inventory data generated in them must be tested with the latest version. The process is quite simple and not that complex. The test chains and test series required for this do not have to be created manually again, but can simply be copied so that they access the respective assigned older inventory data. In some cases, it even makes sense to set up a separate test instance for each dataset – this always depends on the architecture of the application to be tested.

What are the advantages of a structured test strategy?

So much for the configuration strategy. The result takes into account the exponential growth of test cases and the performance limits of the test infrastructure. With such a comprehensive testing strategy, regulatory goals and quality objectives are achieved. The personnel, investment and operating costs are kept within limits despite the enormous number of test cases. The sheer volume of test cases is at least made more manageable by classification and a template system, even if – as discussed – very special skills are required for this. The dead-end situation as with manual testing is permanently avoided. And the test duration can be adjusted to the release interval by adjusting the test chains.

Is there no alternative to such a well thought-out testing strategy? I mean yes, because any evolved construct without these considerations leads to a dead end. Be it the non-achievement of the goals or the unaffordability of the requirements. We have already discussed possible undesirable developments in this regard in the course of this lesson.

Is the testing strategy described here the panacea for all companies? To this I say quite clearly: no! Every company is individual, just like its organization, products and applications. Basically, it takes a close look at the company’s organization, products, and application architecture, as well as a bold look into the company’s future, to come up with the right testing strategy. The considerations I have posed in the course of this lesson can serve as a guide, but they are by no means a blueprint.

Now let’s summarize what we have covered in this lesson:

• we have dealt with the test case object,
  • had a look on test cases,
  • testing actions,
  • test targets,
  • test results,
  • test chains,
  • test series,
  • test runs and
  • test templates,
• we have talked about the difference, the advantages and disadvantages of manual and automatic testing,
• learned the two basic automatic testing methods,
• considered the influence of test data on the test strategy,
• we have recognized the need to complement positive testing with negative testing,
• learned the need to consider older inventory data when testing,
• have looked at a solution for handling the continuously growing number of test cases,
• we recognized that test strategy can impact sprint planning and team staffing,
• have dealt with the configuration strategy a little more comprehensively,
• recognized the result of a strategy on product quality,
• and we looked at possible misdevelopments and their effects throughout the lesson.

In the lesson after next, we’ll delve a little deeper into testing and look at unit tests. But first there is a little surprise for you:

The following lesson includes a short quiz that allows you to check for yourself what content from this lesson has stuck with you. Have fun with it, see you again in the lesson after next!