Estimates are important to test leads and test managers for a number of reasons. As test lead or a test manager, I would want to know,
- How many test cases do we need to write?
- Can we use any of the existing test cases?
- How many existing test cases do we need to modify?
- How long will it take to execute the test cases?
- How many resources do I need to write and execute the test cases?
There are no easy answers to any of these questions and we can not and should not make up numbers out of thin air (although I have seen some do just that). The goal of this article is to discuss ways to create simple and easy to use metrics to do better estimates. Primarily, there are two ways to do estimates.
Estimating the estimates: This approach of doing estimation is merely a speculation or taking a guess. It is not to say that these are wild guesses. On the contrary, these are rather informed guesses and the information comes from previous projects. Previous projects are used as a baseline for doing estimates. One can argue that this makes it a valid estimate. There is some truth to this argument and besides estimates are not meant to be an absolute. But, if the purpose of doing estimates is to come as close as possible to an absolute, then there is a chance that this approach will fall short. If estimated numbers are far apart from the actual numbers, then we are probably not doing a good job on our estimates and as a result we will be writing either too few or too many test cases than we should. In order to take the guessing out of estimates, we do need to create documents (of some kind) which will help us to do a better estimate. One indication of a good estimate is that the estimated numbers reflect actual numbers (within an acceptable difference). A difference of up to 5 to 10 percent may be considered acceptable.
Documenting the estimates: The documenting process of doing estimates is to create some simple metrics which are easy to use. As a test lead or a test manager we need to have some kind of measuring unit which would tell us the depth and breadth of our test coverage in some detail. With that in mind I had created a metric which looks something like this:
Test Coverage & Estimates |
||||||
| User Stories | Description | Test Scenarios | New t/c | Existing t/c | Modified t/c | Total t/c |
| 3.1.3.1.1 | In Lists & Spreadsheet theuser selects one or two lists and selects quick graph option from menu. |
In L&S create and select, – One list and do a quick graph – Two lists and do a quick graph – More than two lists and do a quick graph – Two adjacent lists and an empty column and do a quick graph – One empty column and another list and do a quick graph |
5 | n/a | n/a | 5 |
| 3.1.3.1.1a | Add “Quick Graph” as a menu item in the Data menu | Quick Graph as a last item in Data menu |
n/a | n/a | 1 | 1 |
| 3.1.3.1.2 | Add “Program Editor” as a menu item in the Insert menu | Program Editor as a last item in Insert menu | n/a | n/a | 1 | 1 |
| 3.1.3.1.3 | “Line Number” edit field in “Go to Line” dialog only accepts numeric | – Rational numbers (ex: 11, 22, -11 – 4/11 etc.) – Irrational numbers (ex: square Root of 2) – Decimal – Alphanumeric starts with alpha – Alphanumeric starts with numeric – Alphabetic – Special characters – Spaces – Numeric – No entry (blank) – Numeric with spaces – Numeric, spaces and numeric |
1 |
1 1 1 1 1 1 1 1 1 1 |
n/a | 12 |
| Total t/c | 7 | 10 | 2 | 19 | ||
This metric serves two purposes. First, it describes all the possible test scenarios and thus puts the focus on test coverage and second, the document also serves as a guide to do an estimate on how many test cases we need to write for these user stories. It is always a good idea to have a quick review of this document to see if we are duplicating or missing test scenarios. It is also possible that we may want to combine two or more test scenarios into one test case thus changing the number of test cases we need to write. For example, in case of the last user story (3.1.3.1.3), we may want to have a Data Driven Test case (DDT) where all the test scenarios are listed in a spreadsheet thus making it one test case instead of eleven. This metric is fairly easy to create, easy to follow and easy to maintain.
In addition, we may also want to use the following metric to identify the regression test cases associated with these user stories. This metric is not driven by user stories or requirements as it is the case for the previous metric. Here the layout is completely different and the focus is on different controls (Buttons, edit fields, combo box, spin control etc.). Here is the example of such an estimate for the user story 3.1.3.1.3:
Checklist for Estimates |
||||
| Standard | Criteria | New t/c | Existing t/c | Modified t/c |
| Functional | Functional tests include all aspects of a dialog where there is underlying code that performs a certain task or executes a certain function. | |||
| Dialog Initialization | <Dialog Name> “Go to Line” dialog |
n/a | 1 | n/a |
| Close dialog (Esc, Click x) |
<Dialog Name> “Go to Line” dialog |
n/a | 2 | n/a |
| Default values | Default value for Line Number is always zero | n/a | 1 | n/a |
| Buttons | OK and Cancel buttons in “Go to Line” dialog work correctly | n/a | 2 | n/a |
| Spin Control | Line Number only allows a max of current lines in Program Editor Line Number only goes down to zero |
n/a | 5 1 |
n/a |
| Edit field | Line Number only accepts numeric | 2 | 10 | n/a |
| Hot key | Ctrl+G: Opens “Go to Line” dialog | n/a | 1 | n/a |
| GUI | GUI tests will test aspects of a dialog that are not covered under the functional tests. These include the placement of controls and labels in the dialog, properties of the controls, accelerator keys, etc. | |||
| Minimize/Maximize | <Dialog Name> “Go to Line” dialog |
n/a | 1 | n/a |
| Tab order | <Dialog Name> “Go to Line” dialog |
n/a | 1 | n/a |
| Tool Tip | <Dialog Name> “Go to Line” dialog |
n/a | 1 | n/a |
| Total new t/c | 2 | |||
| Total existing t/c | 26 | |||
| Total modified t/c | 0 | |||
| Grand Total: | 28 | |||
Here, the total number of estimated test cases is 28 and it includes the regression test cases. Since the user story (3.1.3.1.3: Line Number edit field only accepts numeric ) has to do with the changes made in “Go to Line” dialog, we may want to run some of the test cases (GUI and functional) associated with the dialog and this metric helps us find those test cases.
Conclusion: The important thing to remember is that these metrics are dynamic, meaning changes can be made as needed. For example, if new controls are introduced (i.e. scroll bars, check boxes etc), we can add them to the list. Once these metrics are in place, doing an estimate becomes a simple matter of plugging the numbers in the appropriate columns. With such ease of use of these metrics, there is no reason for speculation.
About the Author
Mohammad Alam I started as a mainframe programmer and gradually migrated to c/s environment in mid 90’s with the intention to become a software developer. In order to make a smooth transition from mainframe programming to software development, I decided to start as a Software QA Engineer and thought I would work my way to be a developer. Because of my programming experience, I was asked to be a part of test automation which gave me an opportunity to write code. As I learned more about the world of test automation and QA, I realized that this is what I want to do and I have been doing software QA ever since in various capacities, such as tester, test lead, automation engineer, automation lead and coordinator for offshore test teams. Occasionally, I do share my experience (some are more painful than others) with the QA community hoping that others may benefit from it. But ,on a serious note, I would love to travel around the world and taste the local delicacies and if I could get paid for doing this (like that dude on Food Network), it would be awesome!