Connecting Practice

In “How Can You Develop A Test Specification Documents?” topic, it describes the specific activities to perform and achieve the development of test specification documents. In this piece, it describes the typical contents of test specification template for documenting test cases characteristically below.

Template Header
1. Project number - It should describe the project number accurately.

2. Test Specification name - It should describe the test specification name clearly.

Template Details
1. Test No. - It should describe the test number to ensure every test cases in a project has a unique number for traceability. Note: To ensure every test case in a project has a unique number for traceability, this number should start with the Use Case number (for example Use Case 3 Search, numbering will be 3.1, 3.1.1, 3.2, etc.) - 3.1, 3.2, etc will be section headings and not have tests associated with them, while 3.1.1…., 3.2.1 will contain the test cases. If a test does not have a Use Case associated with it, the test number must start with 0. There must be no blank rows in the “Test No.” column (Column A).

2. Tracking No. - It should describe the associated defect tracking number reference with the particular test case for traceability.

3. Found in Build - It should be the date of the build when the test case was run.

4. Use Case No. - It is important for traceability of every test case that this number matches the use case number.

5. Subject, Object, or Function Name - It should be the subject, object, or function Name where the test case was run.

6. Test Case Description - It should be clear, concise, and accurately describe the test case. It should have some mandatory fields like:

• Objective - General objective of the test case that clearly and accurately describe it.
• Pre-Test Condition - Optional, Pre-requisite for testing the feature
• Verification - Optional, Need to verify against the pre-condition or requirement
• Steps to Reproduce - It should be the clear step by step instructions on how to reproduce the test case. It should provide a verbatim transcript of the session and show what commands you typed and what output in response. It should be minimized and easy-to-follow steps.
• Expected Behavior - It should be the description of the correct or expected results where the defect is not present.
• Actual Result - It should be the clear and detailed description of what happens after the step by step instruction are followed. If you saw test failures then describe the errors, carefully and precisely, what they were. The developers need to know what has gone wrong. If available, include the error message number to get information or search for clues.

Important:

• All scenarios must be considered including negative testing.
• Boundary Analysis must be performed.
• It is not simply a case of sticking verify in front of each requirement - analysis must be carried out for completeness, effectiveness and efficiency.
• If something is not clear in the use case, now is the time to raise questions and ensure the answers form part of this document - it’s too late in the test execution phase.
• Consider information and/or things you will need to support testing, again now is the time to request for these information and/or things.

7. Test Data - Test Data to support every single test scenario must be included

8. Run - It is used to indicate which test are to be run for re-testing and regression testing. Setting the field to 1 indicates that the test case must be executed for re-testing.

9. Pass/Fail

• Pass - Test Passed
• Fail - Test failed if the actual results differ significantly from the test expectations.
• Can’t Test (CT) - A bug in another piece of functionality is restricting you from running this test
• Functionality Not available (FNA) - Functional no longer available or will be available in further phases
• Not Tested - Leave the pass/fail cell blank if you decide not to test the test case.

10. Comment - It should be any additional comments - keep them as consise as possible.

11. Tested By - It should be the tester’s name for traceability and accountability.

12. Tested Date - It should be the date when the test case was run.

13. Test Time (Hours)

• Estimated - Time estimated to execute test case
• Actual - Actual time spent executing test case (include time logging Defects, etc.)

14. Guidelines

a. The layout / format must not be changed from the original template.
b. There must be no blank rows in the “Test No.” column (Column A).
c. To insert further rows to allow you add more defects, highlight a row and click insert row (standard excel). Do not highlight row 5 when inserting rows!
d. If you need to add more test cycles, highlight the last eight columns and copy, then highlight the next eight columns and paste. Don’t forget to change the test cycle number. If you follow this instruction the formulas will also be copied and work!
e. Remember, you must keep the revision history sheet updated. If you update something - put a comment in the revision history. All test cycles must be tracked in the revision history. This number should match the SI revision number - this makes rolling back easier if needed.
f. To group test cases for convenience:

• Click on Data
• Then Settings
• Select “Summary Columns to right of detail” only
• Steps 2 and 3 will only need to be done once
• Select rows beneath your heading that you wish to group (heading should not be selected)
• Click Group and select Rows (Data/Group and Outline/Group)

Join QAST Practitioner Group, download the material in MS Excel version, and/or give comment(s) at:
http://daniloalsonado.com/qastforum/index.php?topic=56.0

In “How to Report Bugs Effectively?” topic, one can identify the twofold strategy below to provide defect information effectively to developers.

1. Show the defects to the developers personally; Or
2. Provide all the information that the developers need; give them careful and detailed instructions on how to make it fail in the same way; and/or find pattern, symptoms, and/or answers to their own defects that concern them to know the cause. More information is almost always better than less - Simon Tatham. The intention is to enable them to reproduce the defects.

Moreover, reporting defects are task of showing or documenting the defects so that the developers responsible for repairing the defects can reproduce the failures and fix the defects.

Further, one can document the defects using a template for reporting defects, tracking defect status, and generating test summary report. The typical contents of the template for documenting defects includes:

Template Header

1. Project number - It should describe the project number accurately.

2. Test Specification name - It should describe the project test specification name clearly.

Template Details

1. Defect No. - It should describe the defect number to ensure every defects in a project has a unique number for traceability.

2. Tracking No. - It should describe the associated defect tracking number reference with the particular defect for traceability.

3. Priority - It should correspond to how urgent it is that the defect be fixed. If the defect needs to be fixed immediately, it should have its priority set to P1 (P1 being the highest priority and P5 being the lowest). Most serious bugs should have their priority set to P2.

P1 - Very High
P2 - High
P3 - Medium
P4 - Low
P5 - Very Low

Moreover, priority should be based on the failed functionality importance and the risk of failure to the business. Where possible the priority should be assigned based on discussions between QA group, Project Manager, BA, and Software Development Lead.

4. Dev Assigned - It should be the assigned developer for fixing a bug.

5. Reported By - It should be the tester’s name for traceability and accountability.

6. Reported Date - It should be the date when the defect was recorded.

7. Found in Build - It should be the date of the build when the defect was found.

8. State - It should be New, Open, or Closed.

9. Status - It should be:

• Disregarded - Defects that were disregarded in the bug report.
• NTReview - Defects that were New in the bug report for review.
• CNNWE - Defects that were Closed-NeverFix as not worth the effort.
• CWAD - Defects that were closed-Withdrawn as designed.
• CWCR - Defects that were closed-Withdrawn as cannot reproduce.
• CWD - Defects that were closed-Withdrawn as duplicate.
• CWNAB - Defects that were closed-Withdrawn as not a bug.
• CWTP - Defects that were closed-Withdrawn as third party.
• CWUE - Defects that were closed-Withdrawn as user error.
• CDLI - Defects that were closed-Deferred as Low Impact.
• Withdrawn - Defects that were closed-Withdrawn.
• OTClosing - Opened Defects that were found for closing.
• Fixed - Defects that were closed as fixed.
• ToTest - Defects that were opened to test.
• ToFix - Defects that were opened to fix.
• ToDefer - Defects that were opened to defer. It will be addressed in a later release.
• OTNMI - Defects that need more information.
• OTCR - Defects that can’t reproduce, functionality no longer available, or will be available in further phases. It is returned when the person assigned cannot replicate the incident.
• OTHNI - Defects that have new information.
• OTHD - “Defects that have dependency or a bug in another piece of functionality is restricting you from running this test.
• OTFNMI - Opened ToFix Defects that need more information.
• OTFHNI - Opened ToFix Defects that have new information.
• NNMI - New Defects that need more information.
• NCR - New Defects that can’t reproduce.

10. Bug Severity - It should correspond to the seriousness of the defect. it should be set to Critical, Major, Minor, or Low severity type. Severity Types are scheme to assist the project team in identifying the severity of the defect and managing the defect log.

Moreover, the number and severity level of open defects may affect timely completion of a project, so it is important to a have a consistent approach to assign severity level to errors; and critical that everyone know how to identify the severity of defects, and how severity levels affect the delivery of the system.

The following are definition of severity levels for defects. Notice that in addition to identifying what the severity levels are, we also present some example of what types of defects qualify for each severity. This helps the project team subjectively in deciding the severity of an error.

Critical: Business process and system functionality are SEVERELY affected - software HALTS. In most cases, they are defects in which will not allow the tester to proceed with the tests - a system inoperable or showstopper issue. Critical defects must be corrected before the code is released to production. Examples are as follows:

• Endless loop - A sequence of instructions in a computer program that are constantly repeated. In most cases, you get an incomplete page displays, or are unable to control the status of the mouse.
• Fatal errors - A condition that halts processing due to program bugs, read errors, or other anomalies. In most cases, generally you cannot recover from it.
• Program Hangs - A condition that software stops responding.

Major: Business process and system functionality are seriously affected; system and/or data is exposed to potential loss or interruption. It is a severe error that causes a program not to perform properly or to produce unreliable results. Normally, the user cannot find an appropriate “workaround” for this type of error. Also it needs immediate attention. It must be corrected before the code is released to the production. Near-term system workaround is required. Examples are as follows:

• Computation errors - Incorrect/Inaccurate calculation - a great deal. Sign errors are surely the most common errors of all. Causes: Loss of invisible parentheses, belief that a minus sign means a negative number, or out of formula/carelessness.
• Incorrect functionality - Fails to satisfy the target specifications or requirements.
• Erroneously displayed/unexpected display of error messages
• Database/table errors

Minor: Business process and system functionality are moderately affected - software is not at risk. Critical system(s) and/or data are not at risk. It needs to be resolved as soon as possible but not a showstopper. Also it is an error in which a “workaround” solution can be found or available for the user. These defects do not significantly impact the execution or performance of the software and need not necessarily be corrected before the code is released to production. Examples are as follows:

• If you click the button, the code does not work, but if you choose the command from the menu, it works fine.
• Incorrect display - In most cases, it is a condition wherein the display contains inconsistencies with the specification or requirement documents.
• Incorrect location - In most cases, it is a condition wherein the display contains inconsistencies with the specification or requirement documents.
• Data was excluded from display - In most cases, it is a condition wherein the specified or required data is not displayed.
• Unexpected behavior of link - In most cases, link is broken and goes not to the correct/intended page.
• Unexpected display of alert message - In some cases, the system responds with an alert message to the user.
• Major grammatical errors in warning/error messages
• Incorrect value for essential fields in the logs
• Obvious cosmetic errors in screens
• Incorrect default values

Low: Business process and system functionality are marginally affected or unaffected. System(s) and/or data are not at risk. In most cases, they are defects in which minimal error, a cosmetic change, or an enhancement is needed. Low level defects do not need to be corrected before release to production. Examples are as follows:

• Incorrect format
• Undeleted line space
• No line space
• No space between character
• Undeleted character
• Undeleted text
• Wrong alignment
• Wrong caption
• Wrong spelling
• Minor grammatical errors
• Wrong text position on screen
• Wrong data position on screen
• Wrong window position on screen

11. DC No. - It should be the defect category Number for analysis.

12. Subject, Object, or Function Name - It should be the subject, object, or function Name where the defect was found.

13. Defect Description - It should be clear, concise, and accurately describe the defect with actual facts so that developer doesn’t have to come looking for tester to get them. and it should have some mandatory fields like:

• Title - General description of the defect that clearly and accurately describe the defect.
• Steps to Reproduce The Bug - It should be the clear step by step instructions on how to reproduce the bug. It should provide a verbatim transcript of the session and show what commands you typed and what output in response. From Development point of view, it is very important. It should be minimized and easy-to-follow steps.
• Actual Result - It should be the clear and detailed description of what happens after the step by step instruction are followed. If you saw test failures then describe the errors, carefully and precisely, what they were. The developers need to know what has gone wrong. If available, include the error message number to get information or search for clues.
• Expected Behavior - It should be the description of the correct or expected results where the bug is not present.

14. Remarks - It should be any additional comments - keep them as consise as possible.

15. Screen Dump - It should be the captured image that could help in motivating developers to fix bugs that are hard to recreate.

16. Test Time - It should be the estimated (Time estimated to re-test the bug) and actual (Actual time spent re-testing the bug) in hours.

Join QAST Practitioner Group, download the material in MS Excel version, and/or give comment(s) at:
http://daniloalsonado.com/qastforum/index.php?topic=58.0