Congratulations! You’ve just joined a company as the new QA Manager and you’re eager to dive right in and be successful. What will be your first steps besides familiarizing yourself with the applications under test? Our recommendation is to thoroughly review the processes and policies within the company as well as those internal to QA. Your success significantly depends on close collaboration with each of the touch points QA has. Therefore, your first task is to assess
these relationships and improve them as necessary. This article will arm you with a “checklist” that will help you with this endeavor.
In a typical organization, QA interacts with the following departments: Product Management (PM), Development including Build Management, Customer Support (CS), the Program Management Office (PMO) and sometimes Sales that provides deployment solutions to the prospects. Below are the questions that you should ask each respective department that will guide you to developing a viable test strategy and keep your department and your relationships running smoothly. We do not pretend to provide an all encompassing questionnaire. It is however, based on our view and practice. We’ve also added comments to the questions to describe our position.
Product Management
Does the company use documented requirements? Are they written in testable and measurable terms?
Comments: In many organizations the requirements specifications arewritten very generically. It is hard to design test cases based only on PM documentation without referring to the functional design spec. This obstacle defers test case development until the programmers produce the code and extends testing cycle. Helping the PM to produce unambiguous testable requirements willhelp QA to solidify their testing strategy.
Are the requirements documented as static requirements where each requirement isolates a thing the system must do?
Comments: This is the traditional way of documenting requirements as features that still takes place in many organizations. A feature is a set of logically related functional requirements that provides a capability to the user that satisfies a business objective (Wiegers, Karl E. 2003, Software Requirements. Microsoft Press). The problem with static requirements is that they do not describe how the requirements interact. This can lead to testing Essential Software Testing. CRC Press)
Are requirements documented as use cases (Rational Unified Process) or user stories (Agile methodology)?
Comments: These define dynamic requirements that help to connect individual features and improve test coverage
Does your organization use a requirements management system to track the requirements?
Comments: We observed that in many companies requirements are still documented using MS Word or Excel. Since there is no attachment capability, the requirements and supporting documents are spread among numerous files connected by hyperlinks. When the number of requirements grows, it is hard to search, extract, and group the requirements based on a given criteria. Only a fully functional requirements management system will give the ability to flexibly manipulate requirements.
Does PM produce non-functional requirements for the products performance and reliability?
Comments: In many companies we ran into a situation where the PMs had no real experience in how to specify performance criteria: transaction response times and throughput, etc. Very often that task was delegated to QA to determine the application boundaries. However, performance can only be discussed with respect to specific platforms, which can be different than an operational QA test environment.
How does the PM notify Development and QA about requirement changes?
Comments: If it is done manually through the emails, this can be error prone. Only using either a requirements management system or a content managing system (Belorusets, Vladimir. 2009, Managing Content, Software Test & Performance, Volume 6, Issue 9) will make this task automatic and reliable.
Development
What software development methodology does the company follow (Test-Driven
Development, Agile, Unified Process, etc)?
Comments: The answer to this question gives you immediate clues how the work is allocated between different departments and what the QA role is in the software development life cycle.
Does Development produce the relevant technical specifications that will help QA to understand how the requirements are implemented in the product?
Comments: These technical specifications, external if performing black box testing, and internal if performing white box testing, will provide QA an enormous insight as to what is being developed. Without these documents QA will not be able to provide detailed schedules, define their test strategies and methodologies or begin to understand the workload ahead. This will impact resource allocations, scheduling and budgeting.
What are the prerequisites to accept the build in QA? Is unit test formal and consistent within the development department? How is the build functionality validated before the build comes to QA?
Comments: It has been our experience that Development rarely formalizes unit testing, has not integrated the test function, and essentially delivers the product to QA for testing to have them tell the developers where the issues are. In order to maintain the overall schedule and in particular QA’s schedule, a working build is a must. Nowadays one of the best practices to validate the build is continuous integration. Under continuous integration once the build is created, all automated tests should run to confirm that it behaves as expected.
What are the relationships between developers and QA? Are testers considered as “friends” helping to improve quality of the code or “adversaries” humiliating developers by revealing their blunders?
Comments: One cannot overly stress the importance of a close knit working relationship with Development. The relationship should be based on mutual respect for each group responsibilities. After all they both have the same goal of delivering a quality product on time.
How often does Development release patches and hot-fixes?
Comments: Testing of unplanned patches will affect the project schedule. advance to allow for a proper resource allocation. All these deliveries should be a part of the PMO project plan.
QA
Are the test plan and test cases reviewed by PM and Development for completeness?
Comments: It is important that each group who are defining, developing implementation and testing the requirements is on the same page. For QA, this will ensure that the testing being planned will provide the correct amount of coverage in the significantly changed areas of the product while also ensuring that testing is not wasted on areas that have had no meaningful updates.
How does QA know about code changes in the upcoming build?
Comments: We highly recommend that each build be accompanied with the Build Notes briefly describing the changes introduced along with the unit and initial integration test results. This will help QA to focus on the most risky areas affected by the code change first.
How often does the build come to QA? Does QA have enough time to verify resolved defects and conduct the regression tests?
Comments: The scheduling of individual builds will be necessary to ensure that each build be given its proper scrutiny. Testing of a given build must be completed based on the changes annotated in the Build Notes and must allow enough time for new functional testing as well as regression. Each of these builds should be planned according to their content.
Does QA use any test case management system? How does QA share test cases if the teams are distributed in the distant geographical locations (off-shore teams)? How easy is it to find a test case among thousands of other test cases? How does QA select and assemble test cases into the test sets?
Comments: Unfortunately test cases, as requirements, very often are documented using MS Word or Excel. Since it is not web based, everyone should have his own copy, and version control must be imposed on merging the joint document.
Is the test case management system integrated with the defect tracking system and requirements management system?
Comments: If you link the failed test case with the related defect, then as soon as the defect status changes to fixed, you can see it in the test case management system. This way you immediately know that you need to rerun the test case. Integration with the requirements management system will help you to track proper requirements coverage.
How does QA track the progress in test case execution? How are quality reports with the test results generated?
Comments: The progress reports should be generated by the test case management/execution system and not by the QA Manager manually. We use reports to track new feature coverage progress and overall project progress by monitoring the percentage of execution of the test cases assigned to every tester.
Does QA do white box testing, for example for testing an API?
Comments: White box testing requires a different skill set from both the testers and the QA Manager. In some companies, the white box testers are called software engineers in test to emphasize the development component for the position.
Do they have Open Source or commercial test automation tools?
Comments: The main advantage of Open Source tools is that they are free. However, although many Open Source projects use forums where everyone can ask a question, there is no guarantee that your problem will be solved.
What is a test automation methodology? Is every script created from scratch? Where do you organize script data?
Comments: We support keyword-driven test automation methodology where test scripts are assembled by using the special keywords from the pre-defined vocabulary. The vocabulary can be expressed in the low level GUI actions or in the business terms. One of the approaches on how to organize test data can be found in the article: Belorusets, Vladimir. 2008, Construct a Data Framework for Seamless Testing, Software Test & Performance, Volume 5, Issue 9.
How does QA handle the reusable test environments and test data?
Comments: With advent of virtualization, QA saves a lot of time for recreating test environments by simply saving them as images. Test automation supports populating test environments with the known repeatable data that drive the product to a predefined initial stage.
What is the release criterion?
Comments: We noticed that in many companies the release criterion is based only on the defect matrix. The comprehensive release criterion should be also based on the product performance, operational reliability, configuration coverage, and successful upgrade testing.
Does the organization support beta testing?
Comments: Beta customers are the great source of product validation with real environments, scenarios, and data. We observed that many companies do not formalize this process with their beta customers and therefore do not gain any real feedback. We recommend that QA help Beta customers to develop the acceptance test plan. Acceptance test plan will identify the requirements and scenarios that are the most important to the customers for successful business. The feedback from beta customers may be later incorporated into the release either in the current version or versions to come. Beta testing is basically a dress rehearsal for the release.
Customer Support
How does the CS notify QA and Development about customer problems?
Comments: In some companies, the customer problems are recorded in a separate issue tracking system and then duplicated in the QA defect tracking system. In our view, it is inefficient to have two systems and synchronize test results between them. One of the solutions to let QA know that a defect came from the customers is to define a special field ‘Customer’ in the defect tracking system.
Does the CS publish any analysis data for the customer issues?
Comments: All information from the CS should be used by QA to identify missing test coverage.
Program Management
Office How does QA interact with the PMO regarding test planning and ongoing project changes?
Comments: QA sometimes underestimates its relationship with the PMO. The PMO stands above all departments participating in the product release and its goal is to coordinate their activities. QA testing schedules, resources and budgets will need to be factored into the PMO’s initial program schedule. In the event that the testing schedule should need to be revised (often compressed), QA will need to work with the PMO to discuss various options having to do with increasing resources or reducing functionality, reducing the test matrix or asking for specific wavers from the original PMO release plan.
Sales
Does Sales inform QA about the configurations of deployed customer environments?
Comments: This information will tremendously help QA to plan and prioritize the configuration testing.
When Sales presents the product to the prospective customers, do they use a “scripted scenario”?
Comments: In this case the Sales scenario must be thoroughly tested by QA to avoid the possible embarrassment in front of the customers. Finally, it is important to note that in a mature organization the responsibility for the overall quality of the release is not just the responsibility of the QA organization but of each department that has a hand in creating the product. We would like to remind you that quality is embedded in the code not by QA but by developers based on the business requirements. We hope that the questions raised in this article will help the new QA Manager to make a roadmap for quality improvements in their new company.
About the Author
Dennis Aaron Dennis Aaron is Principle Consultant working with Start-up companies to help them formulate implementation strategies and costs for their emerging QA and Support organizations. He has 20+ years of extensive experience in software development, end-to-end test methodologies and product delivery process improvement. Dennis has held a number of senior positions at NEC, Sun Microsystems, Oracle, EMC and GAP, including various startups. He was a speaker at NEC’s Zero Defects conference in Tokyo and teaches computer courses at a local community college. He earned his BSCS at Virginia Tech and can be reached at CachetConsulting@aol.com, (408)-888-8535