The article below, describes the tools Orcanos provides for managing your Tests, and your Tests Run Results.
The process described below is the conventional methodology for running the Testing process, and is built of the following basic actions:
- Writing down the Tests
- Adding the needed Ex.Sets and populate them with Tests
- Adding a new Parameter to the Ex.Set
- Running the Tests
- Controlling the Tests Results using the Dashboard
Lets have a further look.
Writing Down the Tests
Writing the Test Cases for the V&V process is an overall methodology, that we will not get into in this article. but the basic principles are important :
- Decide whether the written Test Case is a Validation or a Verification Test Case type (could be both)
- Write a Test Description and Prerequisites for the test in order to live no place for misunderstanding by the Tester.
- Make sure the Steps defined are Numeric measured (with an Actual numeric Result) if possible
- Make sure there arent to many steps defined for each Test Case. (Maximum around 10 Steps per a Test Case)
- Add your new Test Cases under a specific Test Folder on the tree view, so it will be easy to find
The new test written, can be managed using its Status (and more attributes such as Priority, Category, etc.).
The Test Case Status reflects the Test Case maturity level, Its readiness to be executed. Orcanos enables a Test Case managing workflow based on the Test Case Status. Its values can be: New, In Edit Process, Ready for Test. It enables the Writer himself to have a good control over his work, and enables the mid layer management to have a status of the ongoing writing work.
Adding the needed Ex.Sets and populating them with Tests
The Execution Set item type purpose is to encapsulate all running data of a defined Test Cases batch. It can be used for a certain chain management in the process (Unit Testings Ex.Set, Sanity Testings Ex.Set, Integration Testings Ex.Set , etc.) or for a certain testing domain (Load Testings, Efficiency Testings, etc.). The Execution Set shall hold a common concept Test Cases that should be run under the same topic.
Adding the Test Case to the new Execution Set should be done using the ‘Execution Set – List of Test Cases’ tab. An Orcanos user can duplicate the tree view using this tab, and choose the list of Test Cases to be inserted into the desired Ex.Set.
Notice the Test Run Status of each Test Case is currently equals to ‘No Run’ in this stage, since the new inserted Test Cases still werent being executed under the specific Ex.Set. The Ex.Set Status will get the same value on this phase: ‘No Run’, while the Test Case Status will still maintain its last manually updated Status – “Ready to Run” (Actually, the Test Case Status will remain the the same value till the end of the Testing process, unless being changed manually).
Adding a new Parameter to the Ex.Set
Before start executing the Tests, an Orcanos user can add a new Test Run Parameter to the Execution Set, on the execution Set “Run Parameters” tab. It is possible to add an existing parameter or a brand new one. This added Run Parameter value will be added to each Run of any of the Tests located under the specific Execution Set. It can be usefull when Testing a new web feature that needs to be validated using different types of browsers, for example. In this example, we will define a new Run Parameter that is called: “Browser”, and its possible selected values will be: Chrome, Firefox, Explorer, etc. After the Run Parameter was defined on the Execution Set level, selection of one of the parameters values will become mandatory for each Run.
Running the Tests
Managing your Ex.Sets is based on the Orcanos tools provided. The Ex.Set Status is a very good tool that enables the mid and high management layers to know the current status of all tests ran under the specific Ex.Set. For example :
assuming we have 10 Usability Test Cases, called UTC1 to UTC10, located under a Usabillity Ex.Set, called UES1. (its important to emphasize: a single Test Case can belong to different Ex.Sets on the same time).
Now, lets assume UTC1 to UTC4 were executed successfully, having all its run results equal to ‘Pass’. In this case, the Execution Set Status will be ‘In Testing’ since not all Test Cases were executed. Notice that the Execution Set Status value is automatically updated, with no action needed by the user. In this case, the Status log will be as follows:
|UTC1-UTC4||Test Case Status||Ready to Work|
|UTC5-UTC10||Run Status||No Run|
|UES1||Execution Set Status||In Work|
After failing TC5, the Status log will be as follows :
|UTC1-UTC5||Test Case Status||Ready to Work|
|UTC6-UTC10||Run Status||No Run|
|UES1||Execution Set Status||Fail|
Once a Test Case Run was Failed, the Execution Set Status will be updated to ‘Fail’.
If UTC1-UTC10 were successfully executed, All UTC’s and UES Statuses will be set to ‘Pass’.
Controlling the Tests Results using the Views and Dashboards
Controlling the process is significant important. Orcanos provides a Filtering tool and a Dashboard, for making all this information accessible and visual.
There are three main entities we would like to control, when regarding a Testing process:
A Test Case, An Execution Set, and a Test Run.
Each one of the three has a dedicated filtering tab (Under Work Items tab). Below, you can find few recommended tools for each one of the three:
- Test Run
- Test Run Results : Orcanos enables setting a view, to present all Run Results, including the last Run Results, with the Tester Name and Run Date.
- Testing Results Tranding on Dashboard : A chart placed on the Dashboard, shows the Pass vs Fail Tests Run along time.
- Execution Set:
- Tests Run – Last Run Result per Execution Set : Shows the distribution of the Run statuses under each Execution Set.
- Execution Set Status: for a multi execution Sets environment.
- Test Cases:
- Test Cases Traceability Coverage: Shows all uncovered Requirements by Tests or the other way around: All Test Cases with no Requirement related
- Traceability Matrix for an overall view
These tools and more – Email Notifications, on time Alerts, enables aמ excellent platform for managing the Testing process, from beginning to the end.