Automation Demands Special Skills; Field The Best Players Possible
People are at the core of any successful automated software testing program implementation, and hiring competent and qualified people with the matching skill set is critical to the success of AST.
Despite the vendor hype, automation requires more than simply running a capture/playback tool, and not just anyone can effectively use one. It’s also a bad idea to set your hiring goals around the lowest labor costs, or to fill most positions with interns or recent college graduates. For automated testing programs to be implemented successfully, effective hiring practices need to be in place to attract, select and retain the best-qualified people.
The Interviewer
Start by looking at the interviewer’s skills. Are nontechnical managers interviewing for technical positions? Nontechnical managers can evaluate a candidate’s ability to work well with others, their attention to detail and other administrative and organizational skills, but can hardly be expected to recognize a candidate’s ability to perform technical work. While nontechnical managers should be part of the interviewing process, a highly technical resource from the project should participate in interviews to evaluate the candidates’ knowledge of design, software development and other relevant technical areas.
Even if your automation process involves a vendor-provided testing tool (i.e. capture/playback), such tools require a degree of software development skill— some more than others. We recommend that proven software development techniques be used to produce efficient, maintainable, and reusable automated test scripts and software. Those software techniques include code modularity; loose coupling; iterative and incremental development (repeatable build and test); and development of libraries, to allow for reuse in various modules.
Granted there are times when a software tester wants simply to develop automated testing scripts, for example to set up a test environment or to populate a database with test data. The quickest way is to do just that. But for any larger AST effort, where automated testing is implemented throughout the testing program, other lifecycle considerations are required, such as project schedule, requirements, design and implementation of the AST software, AST configuration management of the product baseline, and quality assurance considerations throughout the AST implementation. In fact, one of the primary reasons many significant automation effort get in trouble is that they do not treat AST as a software development project.
The Requirements
A schedule for the AST project should be developed up front and tracked throughout the project. The test requirements to be automated need to be developed and verified as with any other software development project. AST software needs to be designed and developed, and the team is expected to use good techniques and practices as well.
The primary differences between the skill sets needed for AST and those needed for development of the software product are that the AST team also needs an in-depth knowledge of automated test tools, knowledge of the available automated software testing frameworks and approaches to implementing them, an understanding of the applicability of test techniques and how to implement an AST program.
The roles and responsibilities and associated tasks are organized into five basic areas, which our experience shows are key to implementing a successful AST program. Depending on your environment, areas might not always be separated; for example, the CM and/or QA role may be the responsibility of all involved.
- Program management
- Systems engineering
- Software development
- Configuration management
- Quality assurance
One person may have several skills, or one task may be covered by several people; mapping skills to people is not a one-to-one exercise. There are also varying degrees of experience. What is important is that the AST implementation team collectively has all has skills covered. But experience also shows that the least experienced person should never be put on the most complex task. This might appear to be an overly simplistic rule, but it might surprise you how often we’ve seen it ignored.
Other’ Skills
While specific technical skills are required to implement an automated testing program successfully, people skills and other intangibles are also required to enhance the chances for success. We call them “other” skills because few tests or interview techniques can predict whether a person who demonstrates those skills is genuine, whereas “technical” skills can generally be determined using tests.
Automated testing team members need to be adaptable. Experience has shown that automated testing needs to be performed in a variety of different technical environments—on different platforms, using different technologies, and with expertise in a variety of subject matter areas. Therefore the team members should be able to learn quickly, and are ideally already familiar with a broad array of new and mature technologies, processes, tools, and meth-odologies. It is advantageous if the automated test team possess the following “other” skills:
- Be quick learners who enjoy performing a variety of tasks, learning new things, and touching many different products
- Have an aptitude for conceptualizing complex activities
- Understand complex test requirements, be able to formulate test planning and design approaches to support requirements and be able to perform multiple tasks concurrently
- Be able to develop work-around solutions to problems encountered during test development and execution
- Be creative, with the ability to think of multiple ways to exercise a system or application to ensure that it works in all circumstances, and be able to identify all conditions under which the software or system could fail
- Be diplomatic and discrete, especially when having to report major defects, and be able to work closely and effectively with a variety of stakeholders
- Possess strong verbal, written, and grammatical skills, with the ability to communicate any issues and articulate ideas to the customer/stakeholders
- Be detail-oriented and able to identify difficult-to-find glitches, and have a strong interest in improving the quality of a software product, including being self-motivated and trying to find all defects possible
- Be process-oriented and skilled in the ability to understand inputs, logical sequences of steps, and expected outputs
Sometimes a project team is inherited. In this case, evaluate the current skill sets of the team and then map the skills back to the AST projects in the pipeline to ensure the skills needed are available—before the project starts. Then create a gap analysis of skills needed by the group. You can then create a training plan to acquire these skills or hire for them. It is important to note that a successful automation team consists of a mix of skills and levels of experience and expertise.
Preventing Road Blocks
We hear it so often: a good manager doesn’t solve problems, he prevents them. The main role of the AST program manager (PM) is to prevent any roadblocks, and if they do arise, to remove them in a quick fashion. As with the traditional program management role, he needs to lead the overall AST project, acting as the point of contact to the customer (whether internal or external), establishing schedule and cost budgets, tracking the actual schedule and cost against budgets, and resolving issues or assessing and mitigating program risks. His responsibility includes gathering input from all roles described here to the various schedules and processes being tracked to allow for better decision making. Key skills include good leadership and communication skills, planning, the ability to identify and track milestones, foresee and prevent issues, as well as a well rounded technical understanding of the project.
The AST PM needs to understand effective AST processes, including defect tracking processes. He verifies they are implemented and adhered to, and should understand tool training requirements to make the team effective in their use, whether requirements, defect tracking or automated testing tool use. For example, all roles listed here should be proficient in the use of any requirements management tools used on the project.
Table 1 lists some of the responsibilities of the program manager for the first four phases of this process, while the 5th phase “program review” requires the tracking of any defects that actually slipped into production, and related root cause analysis, to be sure this information is tracked for historical purposes, to avoid repeating the same mistake. Providing Subject Matter Expertise
AST systems engineering (SE) roles typically are associated with subject matter expertise (SME) regarding the project or system, development and management of requirements from the highest to the lowest priority, providing the technical lead with information for how the system works and why it was designed to work that way, development of the test plan and tests to confirm that the design meets the requirements, and providing the customer with a technical point of contact.
AST system engineers are also often referred to as the software testers. An understanding of the various software test data design techniques, such as boundary values, equivalence partitioning, pair wise or orthogonal arrays testing, is required. Strong communication skills, subject matter expertise, knowledge of how the product or system is used, the ability to develop and document automated test requirements, and the ability to design tests to validate requirements are all important skills for a systems engineer.
Table 2 lists some of the responsibilities of the AST SEs for the first four phases of this process, while the 5th phase “program review” requires for example, AST PM support and input to any tracking of defects that actually slipped into production, and related root cause analysis, to be sure this information is tracked for historical purposes, and again, to avoid repeating the same mistake.
Additionally, it is important to note that AST SEs support all AST PM activities already described in Table 1, and that no tasks can be done in isolation without input from all team members.
Implementing the AST Code
Software development responsibilities include the design and development of the ASTF and associated test software, as well as subject matter expertise regarding the processes, languages, and tools used during AST development. Critical skills include strong communication skills, the ability to understand requirements (and ask the right questions if the requirements are ambiguous) and then design software based on the requirements. Practition-ers must also possess software programming skills, and knowledge of the software development lifecycle processes.
The AST implementation needs at a minimum to be able to interface with the languages and operating systems used for the product to be tested, so it is very helpful for the software developers to be experienced with those same programming languages and operating systems. The software developers also need to be proficient in the scripting languages and development languages used on the project. Finally, they need to be experts in the use of the automation tools and automation frameworks or development to help train those that support the types of testing to be implemented.
Table 3 lists some of the responsibilities of the AST software developers for the first four phases of this process. The 5th phase “program review” requires, for example, AST PM support and input to any tracking of defects that actually slipped into production, and related root cause analysis, to be sure this information is tracked for historical purposes, and again, to avoid repeating the same mistake.
Other Roles
The configuration management (CM) role and quality assurance (QA) role are sometimes separate, but often are also the responsibilities of everyone on the team, i.e. that of program management, systems engineering, and software development.
CM includes management of all aspects of the product baseline, including version control, product release and problem reports. Necessary skills include strong communication, subject matter expertise in the technical skills to support version control and product release, including programming, and proficiency with CM tools, problem reporting tools, and the software development lifecycle process.
In addition to a familiarity with the CM tools used on the project, the configuration manager should be familiar with the scripting languages and development languages used on the project. Some expertise in the use of the available automated tools is also necessary for the configuration manager to be able to develop CM procedures and build scripts.
Quality assurance provides oversight that the planned processes are being followed and the ongoing evaluation of the product quality. QA engineers should be familiar with the software programming languages being used on the project. In addition, they should be highly proficient in QA tools and problem reporting tools. Familiarity with the automated test tools being used is also needed in order to support analysis for which they are responsible.
Communicating AST Roles
Once the roles and responsibilities and related tasks have been defined, it is important that they be communicated so that everybody involved understands who is responsible for what. AST program implementation is a team effort, and at times a person can have multiple roles and be responsible for a variety of tasks. It is important that team members work toward a common goal, and that it is clear that all are held accountable for the success of the AST program, without any single person being held responsible for its success or failure.
Task overview meetings and discussions of roles and responsibilities need to be held regularly. Training sessions should be scheduled to allow for cross-training to make people most effective. A process isn’t valuable unless it has been documented, communicated and provided with training. The same is true of a tool or automation framework. Tools can end up on a shelf if the tester is not trained on how to use or maintain it.
AST training efforts can include test tool or automated script training. For example, the AST implementation personnel and other stakeholders need to view test automation activities within the scope of a separate development effort complete with strategy and goal planning, test requirement definition, analysis, design and coding. This way they can learn to appreciate the fact that effective automated test development requires careful design and planning just like any other software application development effort.
The outcome of this training is that all stakeholders understand the AST phases and milestones, processes, roles, responsibilities and tasks associated with them. All stakeholders will know how to use the AST-recommended tool suite and will be familiar with the AST program and its implementation. For example, they would know how to run the ASTF and automated testing scripts, so anyone on the team could maintain the automated tool suite, if necessary.
The AST team needs to comprise diverse skill sets, including software development and knowledge of automated testing methodologies and tools. The AST team members cannot be expected to neatly and precisely fit into a specific discipline area; sometimes people have to wear many hats to get the job done, including each other’s hats. But it’s important that all the needed skills be present on the team.
Figure 1 (below) can be used to define a set of roles and responsibilities for each team member and serve as a top-level assessment of the skill sets of the AST team to assess and identify technical strengths and weaknesses. We encourage you to consider additional tasks that will need to be completed for your specific AST project. If there are tasks listed for which no one on the AST team currently has the requisite skills, either someone with those skills should be added to the team or additional training should be provided for someone on the team to develop the skills listed.
Only when roles and responsibilities have been communicated and tasks are understood can personnel’s performance be evaluated.
About the Author
Bernie Gauf