Tips for Writing Test Cases & Bug report


a. Test Cases are prone to regular revision and update:
We live in a continuously changing world, software are also not immune to changes. Same holds good for requirements and this directly impacts the test cases. Whenever, requirements are altered, TCs need to be updated. Yet, it is not only the change in requirement that may cause revision and update to TCs.
During the execution of TCs, many ideas arise in the mind, many sub-conditions of a single TC cause update and even addition of TCs. Moreover, during regression testing several fixes and/or ripples demand revised or new TCs.
b. Test Cases are prone to distribution among the testers who will execute these:
Of course there is hardly the case that a single tester executes all the TCs. Normally there are several testers who test different modules of a single application. So the TCs are divided among them according to their owned areas of application under test. Some TCs related to integration of application, may be executed by multiple testers while some may be executed only by a single tester.
c. Test Cases are prone to clustering and batching:
It is normal and common that TCs belonging to a single test scenario usually demand their execution in some specific sequence or in the form of group. There may be some TCs pre-requisite of other TCs. Similarly, according to the business logic of AUT, a single TC may contribute in several test conditions and a single test condition may consist of multiple TCs.
d. Test Cases have tendency of inter-dependence:
This is also an interesting and important behavior of TCs that those may be interdependent on each other. In medium to large applications with complex business logic, this tendency is more visible.
The clearest area of any application where this behavior can definitely be observed is the interoperability between different modules of same or even different applications. Simply speaking, wherever the different modules or applications are interdependent, the same behavior is reflected in the TCs.
e. Test Cases are prone to distribution among developers (especially in TC driven development environment):
An important fact about TCs is that these are not only to be utilized by the testers. In normal case, when a bug is under fix by the developers, they are indirectly using the TC to fix the issue. Similarly, where the TCD development is followed, TCs are directly used by the developers to build their logic and cover all scenarios, addressed by TCs, in their code.
So, keeping the above 5 factors in mind, here are some tips to write test cases:
1. Keep it simple but not too simple; make it complex but not too complex:
This statement seems a paradox, but I promise it is not so. Keep all the steps of TCs atomic, precise with correct sequence and with correct mapping to expected results, this is what I mean to make it simple.
Now, making it complex in fact means to make it integrated with the Test Plan and other TCs. Refer to other TCs, relevant artifacts, GUIs etc. where and when required. But do this in balanced way, do not make tester to move to and fro in the pile of documents for completing single test scenario. On the other hand do not even let the tester wish you had documented these TCs in some compact manner. While writing TCs, always remember that you or someone else will have to revise and update these.

2. After documenting Test cases, review once as Tester:
Never think that the job is done once you have written the last TC of the test scenario. Go to the start and review all the TCs once, but not with the mind of TC writer or Testing Planner. Review all TCs with the mind of a tester. Think rationally and try to dry run your TCs. Evaluate that all the Steps you have mentioned are clearly understandable, and expected results are in harmony with those steps.
The test data specified in TCs is feasible not only for actual testers but is according to real time environment too. Ensure that there is no dependency conflict among TCs and also verify that all references to other TCs/artifacts/GUIs are accurate because, testers may be in great trouble otherwise.

3. Bound as well as ease the testers:
Do not leave test data on testers, give them range of inputs especially where calculations are to be performed or application’s behavior is dependent on inputs. You may divide the test item values among them, but never give them liberty to choose the test data items themselves. Because, intentionally or unintentionally, they may use same test data and some important test data may be ignored during the execution of TCs.
Keep the testers eased by organizing TCs according to the testing categories and related areas of application. Clearly instruct and mention which TCs are inter-dependent and/or batched. Similarly, explicitly indicate which TCs are independent and isolated so that tester may manage his overall activity at his or her own will.

4. Be a Contributor:
Never accept the FS or Design Document as it is. Your job is not just to go through the FS and identifying the Test Scenarios. Being a quality related resource, never hesitate to contribute. Suggest to developers too, especially in TC-driven development environment. Suggest the drop-down-lists, calendar controls, selection-list, group radio buttons, more meaningful messages, cautions, prompts, improvements related to usability etc.

5. Never Forget the End User
The most important stakeholder is the ‘End User’ who will actually use the AUT. So, never forget him at any stage of TCs writing. In fact, End User should not be ignored at any stage throughout the SDLC, yet my emphasis so far is just related to my topic. So, during the identification of test scenarios, never overlook those cases which will be mostly used by the user or are business critical even of less frequent use. Imagine yourself as End User, once go through all the TCs and judge the practical value of executing all your documented TCs.

Conclusion:
Test Case Writing is an activity which has solid impact on the whole testing phase. This fact makes the task of documenting TCs, very critical and subtle. So, it should be properly planned first and must be done in well-organized manner. The person who is documenting the TCs must keep in mind that, this activity is not for him or her only, but a whole team including other testers and developers, as well as the customer will be directly and indirectly affected by this work.
So, the due attention must be paid during this activity. “Test Case Document” must be understandable for all of its users, in an unambiguous way and should be easily maintainable. Moreover, TC document must address all important features and should cover all important logical flows of the AUT, with real time and practically acceptable inputs.


How to test software requirements specification (SRS)?


Do you know “Most of the bugs in software are due to incomplete or inaccurate functional requirements?”  The software code, doesn’t matter how well it’s written, can’t do anything if there are ambiguities in requirements.
It’s better to catch the requirement ambiguities and fix them in early development life cycle. Cost of fixing the bug after completion of development or product release is too high.  So it’s important to have requirement analysis and catch these incorrect requirements before design specifications and project implementation phases of SDLC.
How to measure functional software requirement specification (SRS) documents?
Well, we need to define some standard tests to measure the requirements. Once each requirement is passed through these tests you can evaluate and freeze the functional requirements.
Let’s take an example. You are working on a web based application. Requirement is as follows:
“Web application should be able to serve the user queries as early as possible”
How will you freeze the requirement in this case?
What will be your requirement satisfaction criteria? To get the answer, ask this question to stakeholders: How much response time is ok for you?
If they say, we will accept the response if it’s within 2 seconds, then this is your requirement measure. Freeze this requirement and carry the same procedure for next requirement.
We just learned how to measure the requirements and freeze those in design, implementation and testing phases.
Now let’s take other example. I was working on a web based project. Client (stakeholders) specified the project requirements for initial phase of the project development. My manager circulated all the requirements in the team for review. When we started discussion on these requirements, we were just shocked! Everyone was having his or her own conception about the requirements. We found lot of ambiguities in the ‘terms’ specified in requirement documents, which later on sent to client for review/clarification.
Client used many ambiguous terms, which were having many different meanings, making it difficult to analyze the exact meaning. The next version of the requirement doc from client was clear enough to freeze for design phase.
From this example we learned “Requirements should be clear and consistent”
Next criteria for testing the requirements specification is “Discover missing requirements”
Many times project designers don’t get clear idea about specific modules and they simply assume some requirements while design phase. Any requirement should not be based on assumptions. Requirements should be complete, covering each and every aspect of the system under development.
Specifications should state both type of requirements i.e. what system should do and what should not.
Generally I use my own method to uncover the unspecified requirements. When I read the software requirements specification document (SRS), I note down my own understanding of the requirements that are specified, plus other requirements SRS document should supposed to cover. This helps me to ask the questions about unspecified requirements making it clearer.
For checking the requirements completeness, divide requirements in three sections, ‘Must implement’ requirements, requirements those are not specified but are ‘assumed’ and third type is ‘imagination’ type of requirements. Check if all type of requirements are addressed before software design phase.
Check if the requirements are related to the project goal.
Some times stakeholders have their own expertise, which they expect to come in system under development. They don’t think if that requirement is relevant to project in hand. Make sure to identify such requirements. Try to avoid the irrelevant requirements in first phase of the project development cycle. If not possible ask the questions to stakeholders: why you want to implement this specific requirement? This will describe the particular requirement in detail making it easier for designing the system considering the future scope.
But how to decide the requirements are relevant or not?
Simple answer: Set the project goal and ask this question: If not implementing this requirement will cause any problem achieving our specified goal? If not, then this is irrelevant requirement. Ask the stakeholders if they really want to implement these types of requirements.
In short requirements specification (SRS) doc should address following:
Project functionality (What should be done and what should not)
Software, Hardware interfaces and user interface
System Correctness, Security and performance criteria
Implementation issues (risks) if any
Conclusion: 
I have covered all aspects of requirement measurement. To be specific about requirements, I will summarize requirement testing in one sentence:
“Requirements should be clear and specific with no uncertainty, requirements should be measurable in terms of specific values, requirements should be testable having some evaluation criteria for each requirement, and requirements should be complete, without any contradictions”
Testing should start at requirement phase to avoid further requirement related bugs. Communicate more and more with your stakeholder to clarify all the requirements before starting project design and implementation.

Sample bug report


Below sample bug/defect report will give you exact idea of how to report a bug in bug tracking tool.
Here is the example scenario that caused a bug:
Lets assume in your application under test you want to create a new user with user information, for that you need to logon into the application and navigate to USERS menu > New User, then enter all the details in the ‘User form’ like, First Name, Last Name, Age, Address, Phone etc. Once you enter all these information, you need to click on ‘SAVE’ button in order to save the user. Now you can see a success message saying, “New User has been created successfully”.
But when you entered into your application by logging in and navigated to USERS menu > New user, entered all the required information to create new user and clicked on SAVE button. BANG! The application crashed and you got one error page on screen. (Capture this error message window and save as a Microsoft paint file)
Now this is the bug scenario and you would like to report this as a BUG in your bug-tracking tool.
How will you report this bug effectively?
Here is the sample bug report for above mentioned example:
(Note that some ‘bug report’ fields might differ depending on your bug tracking system)
SAMPLE BUG REPORT:
Bug Name: Application crash on clicking the SAVE button while creating a new user.
Bug ID: (It will be automatically created by the BUG Tracking tool once you save this bug)
Area Path: USERS menu > New Users
Build Number: Version Number 5.0.1
Severity: HIGH (High/Medium/Low) or 1
Priority: HIGH (High/Medium/Low) or 1
Assigned to: Developer-X
Reported By: Your Name
Reported On: Date
Reason: Defect
Status: New/Open/Active (Depends on the Tool you are using)
Environment: Windows 2003/SQL Server 2005
Description:
Application crash on clicking the SAVE button while creating a new
user, hence unable to create a new user in the application.
Steps To Reproduce:
1) Logon into the application
2) Navigate to the Users Menu > New User
3) Filled all the user information fields
4) Clicked on ‘Save’ button
5) Seen an error page “ORA1090 Exception: Insert values Error…”
6) See the attached logs for more information (Attach more logs related to bug..IF any)
7) And also see the attached screenshot of the error page.
Expected result: On clicking SAVE button, should be prompted to a success message “New User has been created successfully”.
(Attach ‘application crash’ screen shot.. IF any)
Save the defect/bug in the BUG TRACKING TOOL.  You will get a bug id, which you can use for further bug reference.
Default ‘New bug’ mail will go to respective developer and the default module owner (Team leader or manager) for further action.


How to write a good bug report? Tips and Tricks


Why good Bug report?
If your bug report is effective, chances are higher that it will get fixed. So fixing a bug depends on how effectively you report it. Reporting a bug is nothing but a skill and I will tell you how to achieve this skill.
“The point of writing problem report(bug report) is to get bugs fixed” – By Cem Kaner. If tester is not reporting bug correctly, programmer will most likely reject this bug stating as irreproducible. This can hurt testers moral and some time ego also. (I suggest do not keep any type of ego. Ego’s like “I have reported bug correctly”, “I can reproduce it”, “Why he/she has rejected the bug?”, “It’s not my fault” etc etc..)
What are the qualities of a good software bug report?
Anyone can write a bug report. But not everyone can write a effective bug report. You should be able to distinguish between average bug report and a good bug report. How to distinguish a good or bad bug report? It’s simple, apply following characteristics and techniques to report a bug.
1) Having clearly specified bug number:
Always assign a unique number to each bug report. This will help to identify the bug record. If you are using any automated bug-reporting tool then this unique number will be generated automatically each time you report the bug. Note the number and brief description of each bug you reported.
2) Reproducible:
If your bug is not reproducible it will never get fixed. You should clearly mention the steps to reproduce the bug. Do not assume or skip any reproducing step. Step by step described bug problem is easy to reproduce and fix.
3) Be Specific:
Do not write a essay about the problem. Be Specific and to the point. Try to summarize the problem in minimum words yet in effective way. Do not combine multiple problems even they seem to be similar. Write different reports for each problem.
How to Report a Bug?
Use following simple Bug report template:
This is a simple bug report format. It may vary on the bug report tool you are using. If you are writing bug report manually then some fields need to specifically mention like Bug number which should be assigned manually.
Reporter: Your name and email address.
Product: In which product you found this bug.
Version: The product version if any.
Component: These are the major sub modules of the product.
Platform: Mention the hardware platform where you found this bug. The various platforms like ‘PC’, ‘MAC’, ‘HP’, ‘Sun’ etc.
Operating system: Mention all operating systems where you found the bug. Operating systems like Windows, Linux, Unix, SunOS, Mac OS. Mention the different OS versions also if applicable like Windows NT, Windows 2000, Windows XP etc.
Priority:
When bug should be fixed? Priority is generally set from P1 to P5. P1 as “fix the bug with highest priority” and P5 as ” Fix when time permits”.
Severity:
This describes the impact of the bug.
Types of Severity:
  • Blocker: No further testing work can be done.
  • Critical: Application crash, Loss of data.
  • Major: Major loss of function.
  • Minor: minor loss of function.
  • Trivial: Some UI enhancements.
  • Enhancement: Request for new feature or some enhancement in existing one.
Status:
When you are logging the bug in any bug tracking system then by default the bug status is ‘New’.
Later on bug goes through various stages like Fixed, Verified, Reopen, Won’t Fix etc.

Assign To: 
If you know which developer is responsible for that particular module in which bug occurred, then you can specify email address of that developer. Else keep it blank this will assign bug to module owner or Manger will assign bug to developer. Possibly add the manager email address in CC list.
URL:
The page url on which bug occurred.
Summary:
A brief summary of the bug mostly in 60 or below words. Make sure your summary is reflecting what the problem is and where it is.
Description:
A detailed description of bug. Use following fields for description field:
  • Reproduce steps: Clearly mention the steps to reproduce the bug.
  • Expected result: How application should behave on above mentioned steps.
  • Actual result: What is the actual result on running above steps i.e. the bug behavior.
These are the important steps in bug report. You can also add the “Report type” as one more field which will describe the bug type.
The report types are typically:
1) Coding error
2) Design error
3) New suggestion
4) Documentation issue
5) Hardware problem
Some Bonus tips to write a good bug report:
1) Report the problem immediately:If you found any bug while testing, do not wait to write detail bug report later. Instead write the bug report immediately. This will ensure a good and reproducible bug report. If you decide to write the bug report later on then chances are high to miss the important steps in your report.
2) Reproduce the bug three times before writing bug report:Your bug should be reproducible. Make sure your steps are robust enough to reproduce the bug without any ambiguity. If your bug is not reproducible every time you can still file a bug mentioning the periodic nature of the bug.
3) Test the same bug occurrence on other similar module:
Sometimes developer use same code for different similar modules. So chances are high that bug in one module can occur in other similar modules as well. Even you can try to find more severe version of the bug you found.
4) Write a good bug summary:
Bug summary will help developers to quickly analyze the bug nature. Poor quality report will unnecessarily increase the development and testing time. Communicate well through your bug report summary. Keep in mind bug summary is used as a reference to search the bug in bug inventory.
5) Read bug report before hitting Submit button:
Read all sentences, wording, steps used in bug report. See if any sentence is creating ambiguity that can lead to misinterpretation. Misleading words or sentences should be avoided in order to have a clear bug report.
6) Do not use Abusive language:
It’s nice that you did a good work and found a bug but do not use this credit for criticizing developer or to attack any individual.
Conclusion:
No doubt that your bug report should be a high quality document. Focus on writing good bug reports, spend some time on this task because this is main communication point between tester, developer and manager. Mangers should make aware to their team that writing a good bug report is primary responsibility of any tester. Your efforts towards writing good bug report will not only save company resources but also create a good relationship between you and developers.



Bug life cycle


What is Bug/Defect?
Simple Wikipedia definition of Bug is: “A computer bug is an error, flaw, mistake, failure, or fault in a computer program that prevents it from working correctly or produces an incorrect result. Bugs arise from mistakes and errors, made by people, in either a program’s source code or its design.”
Other definitions can be:
An unwanted and unintended property of a program or piece of hardware, especially one that causes it to malfunction.
or
A fault in a program, which causes the program to perform in an unintended or unanticipated manner.
Lastly the general definition of bug is: “failure to conform to specifications”.
If you want to detect and resolve the defect in early development stage, defect tracking and software development phases should start simultaneously.
We will discuss more on Writing effective bug report in another article. Let’s concentrate here on bug/defect life cycle.
Life cycle of Bug:
1) Log new defect
When tester logs any new bug the mandatory fields are:
Build version, Submit On, Product, Module, Severity, Synopsis and Description to Reproduce
In above list you can add some optional fields if you are using manual Bug submission template:
These Optional Fields are: Customer name, Browser, Operating system, File Attachments or screenshots.
The following fields remain either specified or blank:
If you have authority to add bug Status, Priority and ‘Assigned to’ fields them you can specify these fields. Otherwise Test manager will set status, Bug priority and assign the bug to respective module owner.
Look at the following Bug life cycle:
Bug life cycle
[Click on the image to view full size] Ref: Bugzilla bug life cycle
The figure is quite complicated but when you consider the significant steps in bug life cycle you will get quick idea of bug life.
On successful logging the bug is reviewed by Development or Test manager. Test manager can set the bug status as Open, can Assign the bug to developer or bug may be deferred until next release.
When bug gets assigned to developer and can start working on it. Developer can set bug status as won’t fix, Couldn’t reproduce, Need more information or ‘Fixed’.
If the bug status set by developer is either ‘Need more info’ or Fixed then QA responds with specific action. If bug is fixed then QA verifies the bug and can set the bug status as verified closed or Reopen.
Bug status description:
These are various stages of bug life cycle. The status caption may vary depending on the bug tracking system you are using.
1) New: When QA files new bug.
2) Deferred: If the bug is not related to current build or can not be fixed in this release or bug is not important to fix immediately then the project manager can set the bug status as deferred.
3) Assigned: ‘Assigned to’ field is set by project lead or manager and assigns bug to developer.
4) Resolved/Fixed: When developer makes necessary code changes and verifies the changes then he/she can make bug status as ‘Fixed’ and the bug is passed to testing team.
5) Could not reproduce: If developer is not able to reproduce the bug by the steps given in bug report by QA then developer can mark the bug as ‘CNR’. QA needs action to check if bug is reproduced and can assign to developer with detailed reproducing steps.
6) Need more information: If developer is not clear about the bug reproduce steps provided by QA to reproduce the bug, then he/she can mark it as “Need more information’. In this case QA needs to add detailed reproducing steps and assign bug back to dev for fix.
7) Reopen: If QA is not satisfy with the fix and if bug is still reproducible even after fix then QA can mark it as ‘Reopen’ so that developer can take appropriate action.
8 ) Closed: If bug is verified by the QA team and if the fix is ok and problem is solved then QA can mark bug as ‘Closed’.
9) Rejected/Invalid: Some times developer or team lead can mark the bug as Rejected or invalid if the system is working according to specifications and bug is just due to some misinterpretation.

No comments:

Post a Comment