Actions

Difference between revisions of "Risk Based Testing"

Line 7: Line 7:
 
*Risk-based testing can also involve using risk analysis to identify proactive opportunities to remove or prevent defects through non-testing activities and to help us select which test activities to perform.
 
*Risk-based testing can also involve using risk analysis to identify proactive opportunities to remove or prevent defects through non-testing activities and to help us select which test activities to perform.
 
The goal of risk-based testing cannot practically be – a risk-free project. What we can get from risk-based testing is to carry out the testing with best practices in risk management to achieve a project outcome that balances risks with quality, features, budget and schedule.<ref>Explaining Risk Based Testing [http://tryqa.com/what-is-risk-based-testing/ Try QA]</ref>
 
The goal of risk-based testing cannot practically be – a risk-free project. What we can get from risk-based testing is to carry out the testing with best practices in risk management to achieve a project outcome that balances risks with quality, features, budget and schedule.<ref>Explaining Risk Based Testing [http://tryqa.com/what-is-risk-based-testing/ Try QA]</ref>
 +
 +
 +
'''Risk Based Testing Process<ref>Risk Based Testing Process [https://testfort.com/blog/risk-based-testing-and-associated-challenges Testfort]</ref>'''<br />
 +
The Risk Based Testing Process is diagrammatically illustrated below:
 +
 +
[[File:RBT_Process.jpg|400px|Risk Based Testing (RBT) Process]]<br />
 +
 +
 +
Priority of RBT is to determine and identify risks so that perceived risks were classified and categorized. Thus, determining higher risks, that will be addressed in first place. Risk-Based Testing, despite being a foundation of the modern international testing standards, has been quite rarely applied successfully to its full capabilities due to certain challenges associated with such analysis.
 +
 +
For majority project test managers introduction to Risk-Based Testing initially happens while they are testers. Testers primarily use RBT while system testing phase and it is a valid approach. But at the level of test manager while establishing testing strategy RBT is so much more useful and productive. Risk testing analysis, calculating and defining what particular test phases to use, addressing greater levels of risk in first place. A lot of managers while writing test plan for a project, many times include only testing activities that they directly control.
 +
 +
Test managers must include in test plan all of the testing that will be done across entire life cycle by developers, testers, and stakeholders. It is the test manager’s primary responsibility to take control over testing cycle. Risk-Based Testing works best when interaction between test manager and developers is on a high level. When this happens it reduces chances of poor development that might lead to higher cost testing.
  
  
 
'''The Various Approaches to Risk Based Testing<ref>The Various Approaches to Risk Based Testing [https://www.researchgate.net/profile/Michael_Felderer/publication/259635976_Integrating_risk-based_testing_in_industrial_test_processes/links/00463531e0fc070d61000000/Integrating-risk-based-testing-in-industrial-test-processes.pdf Michael Felderer,  Rudolf Ramler]</ref>'''<br />
 
'''The Various Approaches to Risk Based Testing<ref>The Various Approaches to Risk Based Testing [https://www.researchgate.net/profile/Michael_Felderer/publication/259635976_Integrating_risk-based_testing_in_industrial_test_processes/links/00463531e0fc070d61000000/Integrating-risk-based-testing-in-industrial-test-processes.pdf Michael Felderer,  Rudolf Ramler]</ref>'''<br />
Bach (1999) presents a pragmatic approach to risk-based testing grounded on a heuristic software risk analysis. Bach distinguishes inside-out risk analysis starting with details about a situation and identifying associated risk, and outside-in risk analysis starting with a set of potential risks and matching them to the details of the situation.  
+
Bach (1999) presents a pragmatic approach to risk-based testing grounded on a heuristic software risk analysis. Bach distinguishes inside-out risk analysis starting with details about a situation and identifying associated risk, and outside-in risk analysis starting with a set of potential risks and matching them to the details of the situation.<br />
Amland (2000) defines a risk-based testing approach that is based on Karolak's risk management process (Karolak 1995) comprising the following steps and the corresponding risk management activities: planning (risk identification and risk strategy), identification of risk indicators (part of risk assessment), identification of the cost of a failure (part of risk assessment), identification of critical elements (part of risk assessment), test execution (risk mitigation), and estimation of completion (risk reporting and risk prediction). Amland was one of the first in introducing a systematic risk-based testing approach. As such, it is not aligned with the standard test process.
+
Amland (2000) defines a risk-based testing approach that is based on Karolak's [[Risk Management|risk management]] process (Karolak 1995) comprising the following steps and the corresponding risk management activities: planning (risk identification and risk strategy), identification of [[Key Risk Indicator (KRI)|risk indicators]] (part of [[Risk Assessment|risk assessment]]), identification of the cost of a failure (part of risk assessment), identification of critical elements (part of risk assessment), test execution ([[Risk Mitigation|risk mitigation]]), and estimation of completion (risk reporting and risk prediction). Amland was one of the first in introducing a systematic risk-based testing approach. As such, it is not aligned with the standard test process.<br />
Redmill (2004; 2005) provides a thorough discussion of risk-based testing (Redmill 2004) as well as a proposal for practical application suggesting a single factor risk assessment, either on probability or on impact, or a two-factor risk assessment, in which probability and impact are combined (Redmill 2005). The application of the resulting risk classification in the test process, e.g., for test planning, design, execution or reporting, is beyond the scope of the approach.
+
Redmill (2004; 2005) provides a thorough discussion of risk-based testing (Redmill 2004) as well as a proposal for practical application suggesting a single factor risk assessment, either on probability or on impact, or a two-factor risk assessment, in which probability and impact are combined (Redmill 2005). The application of the resulting risk classification in the test process, e.g., for test planning, design, execution or reporting, is beyond the scope of the approach.<br />
Felderer et al. (2012) propose a risk-based test process including the phases risk identification, test planning, risk assessment, test design as well as static and dynamic evaluation. The focus of their work is on the risk assessment model defined on the basis of an industrial project. In this model the risk coefficient is assigned to features and determined by impact, probability and time factors. Each factor is determined by
+
Felderer et al. (2012) propose a risk-based test process including the phases risk identification, test planning, risk assessment, test design as well as static and dynamic evaluation. The focus of their work is on the [[Risk Assessment Framework (RAF)|risk assessment model]] defined on the basis of an industrial project. In this model the risk coefficient is assigned to features and determined by impact, probability and time factors. Each factor is determined by criteria which are defined by metrics. Metrics are determined manually, semiautomated or automated.<br />
criteria which are defined by metrics. Metrics are determined manually, semiautomated or automated.  
 
 
The Practical Risk-Based Testing Approach (PRISMA) (van Veenendaal 2012) distinguishes business and technical risks determined by weighted criteria to calculate the overall risk of the risk items. Additionally, PRISMA defines a process consisting of concrete activities, i.e., initiating, planning, kick-off meeting, extended risk identification, individual preparation, processing individual scores, consensus meeting, and define differentiated risk-based testing approach. The activities are defined in a very concrete way with detailed instructions.  
 
The Practical Risk-Based Testing Approach (PRISMA) (van Veenendaal 2012) distinguishes business and technical risks determined by weighted criteria to calculate the overall risk of the risk items. Additionally, PRISMA defines a process consisting of concrete activities, i.e., initiating, planning, kick-off meeting, extended risk identification, individual preparation, processing individual scores, consensus meeting, and define differentiated risk-based testing approach. The activities are defined in a very concrete way with detailed instructions.  
  
In addition to generic risk-based testing approaches, also several model-driven approaches to risk-based testing have been introduced.
+
In addition to generic risk-based testing approaches, also several model-driven approaches to risk-based testing have been introduced.<br />
Chen et al. (2002) use activity diagrams to represent requirements and use a risk analysis to select test cases for regression testing purposes. The risk of a test case is determined by its cost, measured based on the cost of the requirements the test covers, and its severity probability, measured based on the severity-weighted number of defects uncovered by the test case. The approach is intended to support regression testing of requirements, and does not consider a broad spectrum of arbitrary risk items as in our approach. Chen et al. focus on the phases risk assessment
+
Chen et al. (2002) use activity diagrams to represent requirements and use a [[Risk Analysis|risk analysis]] to select test cases for regression testing purposes. The risk of a test case is determined by its cost, measured based on the cost of the requirements the test covers, and its severity probability, measured based on the severity-weighted number of defects uncovered by the test case. The approach is intended to support regression testing of requirements, and does not consider a broad spectrum of arbitrary risk items as in our approach. Chen et al. focus on the phases risk assessment and test execution but not consider the overall risk-based test process.<br />
and test execution but not consider the overall risk-based test process.
+
Stallbaum and Metzger (2007; 2008) introduce a model-driven risk-based system testing approach that is based on the Factor-Criteria-Metrics model (Cavano and McCall 1978). The focus of their approach is the annotation of risk assessment data in UML-based models and the automated generation of risk-based test suites, however, without a standard-aligned risk-based testing methodology and without an approach for its introduction in existing test processes.<br />
Stallbaum and Metzger (2007; 2008) introduce a model-driven risk-based system testing approach that is based on the Factor-Criteria-Metrics model (Cavano and McCall 1978). The focus of their approach is the annotation of risk assessment data in UML-based models and the automated generation of risk-based test suites, however, without a standard-aligned risk-based testing methodology and without an approach for its introduction in existing test processes.
 
 
Finally, the model-driven risk-based testing approach of Wendland et al. (2012) formalizes requirements as integrated behavior trees, augments the integrated behavior tree with risk information, identifies for each risk in the integrated behavior tree an appropriate test directive, and finally passes both the risk-augmented integrated behavior tree and the test directive definition into a test generator.  
 
Finally, the model-driven risk-based testing approach of Wendland et al. (2012) formalizes requirements as integrated behavior trees, augments the integrated behavior tree with risk information, identifies for each risk in the integrated behavior tree an appropriate test directive, and finally passes both the risk-augmented integrated behavior tree and the test directive definition into a test generator.  
  
  
 
=== See Also ===
 
=== See Also ===
 +
[[Governance]]<br />
 +
[[Corporate Governance]]<br />
 +
[[IT Governance]]<br />
 +
[[Risk Analysis]]<br />
 +
[[Risk Assessment Framework (RAF)]]<br />
 +
[[Risk Management]]<br />
 +
[[Risk Management Framework (RMF)]]<br />
 +
[[Information Technology Risk (IT Risk)]]<br />
 +
[[Enterprise Risk Management (ERM)]]<br />
 +
[[Risk IT Framework]]<br />
 +
[[Risk Assessment]]<br />
 +
[[Risk-Adjusted Return]]<br />
 +
[[Risk-Adjusted Return on Capital (RAROC)]]<br />
 +
[[Risk Matrix]]<br />
 +
[[Risk Maturity]]<br />
 +
[[Risk Maturity Model (RMM)]]<br />
 +
[[Risk Mitigation]]<br />
 +
[[Operational Risk]]<br />
 +
[[Operational Risk Management (ORM)]]<br />
 +
[[Architectural Risk]]
  
  

Revision as of 21:15, 20 March 2020

Risk Based Testing is an approach that takes a scientific approach when accounting for risk. It is mainly based on the factors of the business impact and the likelihood of failure, although there could be more.[1]

Risk based testing uses risk to prioritize and emphasize the appropriate tests during test execution. In simple terms – Risk is the probability of occurrence of an undesirable outcome. This outcome is also associated with an impact. Since there might not be sufficient time to test all functionality, Risk based testing involves testing the functionality which has the highest impact and probability of failure. Risk-based testing is the idea that we can organize our testing efforts in a way that reduces the residual level of product risk when the system is deployed.

  • Risk-based testing starts early in the project, identifying risks to system quality and using that knowledge of risk to guide testing planning, specification, preparation and execution.
  • Risk-based testing involves both mitigation – testing to provide opportunities to reduce the likelihood of defects, especially high-impact defects – and contingency – testing to identify work-arounds to make the defects that do get past us less painful.
  • Risk-based testing also involves measuring how well we are doing at finding and removing defects in critical areas.
  • Risk-based testing can also involve using risk analysis to identify proactive opportunities to remove or prevent defects through non-testing activities and to help us select which test activities to perform.

The goal of risk-based testing cannot practically be – a risk-free project. What we can get from risk-based testing is to carry out the testing with best practices in risk management to achieve a project outcome that balances risks with quality, features, budget and schedule.[2]


Risk Based Testing Process[3]
The Risk Based Testing Process is diagrammatically illustrated below:

Risk Based Testing (RBT) Process


Priority of RBT is to determine and identify risks so that perceived risks were classified and categorized. Thus, determining higher risks, that will be addressed in first place. Risk-Based Testing, despite being a foundation of the modern international testing standards, has been quite rarely applied successfully to its full capabilities due to certain challenges associated with such analysis.

For majority project test managers introduction to Risk-Based Testing initially happens while they are testers. Testers primarily use RBT while system testing phase and it is a valid approach. But at the level of test manager while establishing testing strategy RBT is so much more useful and productive. Risk testing analysis, calculating and defining what particular test phases to use, addressing greater levels of risk in first place. A lot of managers while writing test plan for a project, many times include only testing activities that they directly control.

Test managers must include in test plan all of the testing that will be done across entire life cycle by developers, testers, and stakeholders. It is the test manager’s primary responsibility to take control over testing cycle. Risk-Based Testing works best when interaction between test manager and developers is on a high level. When this happens it reduces chances of poor development that might lead to higher cost testing.


The Various Approaches to Risk Based Testing[4]
Bach (1999) presents a pragmatic approach to risk-based testing grounded on a heuristic software risk analysis. Bach distinguishes inside-out risk analysis starting with details about a situation and identifying associated risk, and outside-in risk analysis starting with a set of potential risks and matching them to the details of the situation.
Amland (2000) defines a risk-based testing approach that is based on Karolak's risk management process (Karolak 1995) comprising the following steps and the corresponding risk management activities: planning (risk identification and risk strategy), identification of risk indicators (part of risk assessment), identification of the cost of a failure (part of risk assessment), identification of critical elements (part of risk assessment), test execution (risk mitigation), and estimation of completion (risk reporting and risk prediction). Amland was one of the first in introducing a systematic risk-based testing approach. As such, it is not aligned with the standard test process.
Redmill (2004; 2005) provides a thorough discussion of risk-based testing (Redmill 2004) as well as a proposal for practical application suggesting a single factor risk assessment, either on probability or on impact, or a two-factor risk assessment, in which probability and impact are combined (Redmill 2005). The application of the resulting risk classification in the test process, e.g., for test planning, design, execution or reporting, is beyond the scope of the approach.
Felderer et al. (2012) propose a risk-based test process including the phases risk identification, test planning, risk assessment, test design as well as static and dynamic evaluation. The focus of their work is on the risk assessment model defined on the basis of an industrial project. In this model the risk coefficient is assigned to features and determined by impact, probability and time factors. Each factor is determined by criteria which are defined by metrics. Metrics are determined manually, semiautomated or automated.
The Practical Risk-Based Testing Approach (PRISMA) (van Veenendaal 2012) distinguishes business and technical risks determined by weighted criteria to calculate the overall risk of the risk items. Additionally, PRISMA defines a process consisting of concrete activities, i.e., initiating, planning, kick-off meeting, extended risk identification, individual preparation, processing individual scores, consensus meeting, and define differentiated risk-based testing approach. The activities are defined in a very concrete way with detailed instructions.

In addition to generic risk-based testing approaches, also several model-driven approaches to risk-based testing have been introduced.
Chen et al. (2002) use activity diagrams to represent requirements and use a risk analysis to select test cases for regression testing purposes. The risk of a test case is determined by its cost, measured based on the cost of the requirements the test covers, and its severity probability, measured based on the severity-weighted number of defects uncovered by the test case. The approach is intended to support regression testing of requirements, and does not consider a broad spectrum of arbitrary risk items as in our approach. Chen et al. focus on the phases risk assessment and test execution but not consider the overall risk-based test process.
Stallbaum and Metzger (2007; 2008) introduce a model-driven risk-based system testing approach that is based on the Factor-Criteria-Metrics model (Cavano and McCall 1978). The focus of their approach is the annotation of risk assessment data in UML-based models and the automated generation of risk-based test suites, however, without a standard-aligned risk-based testing methodology and without an approach for its introduction in existing test processes.
Finally, the model-driven risk-based testing approach of Wendland et al. (2012) formalizes requirements as integrated behavior trees, augments the integrated behavior tree with risk information, identifies for each risk in the integrated behavior tree an appropriate test directive, and finally passes both the risk-augmented integrated behavior tree and the test directive definition into a test generator.


See Also

Governance
Corporate Governance
IT Governance
Risk Analysis
Risk Assessment Framework (RAF)
Risk Management
Risk Management Framework (RMF)
Information Technology Risk (IT Risk)
Enterprise Risk Management (ERM)
Risk IT Framework
Risk Assessment
Risk-Adjusted Return
Risk-Adjusted Return on Capital (RAROC)
Risk Matrix
Risk Maturity
Risk Maturity Model (RMM)
Risk Mitigation
Operational Risk
Operational Risk Management (ORM)
Architectural Risk


References

  1. Defining Risk Based Testing Rajnish Mehta
  2. Explaining Risk Based Testing Try QA
  3. Risk Based Testing Process Testfort
  4. The Various Approaches to Risk Based Testing Michael Felderer, Rudolf Ramler


Further Reading