Test suite Effectiveness is one great way to describe which test suite produce the most important defect discoveries. You wish to return to that test suite from time to time to see that the number of defects discovered by the execution of those tests is reduced over time. A test suite is a collection of test cases in a consistent sequence that used by potential end user. The Test Suite can also be a collection of test cases performing a less specific task such as Sanity, Load, Regression, Progression, etc. Now that you have those test suites you wish to measure how effective they are in discovering defects and what is the severity level of those defects detected. Putting this data into a report will show interesting highlight on those test suites that generate the worse defects. YOU WANT TO HIT HARD ON THAT SPOT. The following figure show test suites by name and the defect discovery breakdown by severity.
Tip: You want to eliminate test suites that produce less than 10 defects which means this is too early to measure them.
Using ORCANOS filter manager you can create a filter that will return all the defects reported in ORCANOS system. Adding to the return result the Execution Set Name (Supported from 188.8.131.52) will include the name of the execution set that the defect was generated from.
Note: To limit the return results only to the execution set that generated more than 10 defects you will need to add to the special criteria of the defect filter the following statement.
execution_set_name in (select execution_set_name
where execution_set_name <> ”
group by execution_set_name