Continued from Part 1
5. Adopt a Tiered Approach to SAP Automation Architecture
An automation framework can be broken down into 3 distinct tiers. Provided below is a description of each tier that you should include in your SAP automation framework. Figure-1 provides an example of a tiered architecture.
Tier 1: The top tier of your SAP automation framework should include:
- SAP Solution Manager (SolMan) and Transport Management Tool (Conigma)
- Web-based Application Lifecycle Management (ALM) Tools (HP ALM, IBM RequisitePro, SmartBear QA Complete)
- Client-side testing applications like HP Sprinter (an add-on for HP ALM)
- Help Desk (incident reporting) System (Service Now)
- Applications Under Test (AUT), including: SAP ECC, SAP CRM, Legacy Systems, Workstations
Tier 2: The second tier of your SAP automation framework should consist of your automated testing tool. This could be HP QTP, IBM RFT, Janova, Panaya, Worksoft, or SAP TAO. As mentioned, pick what is best for your organization as a whole bearing in mind that SAP integrates with many other software applications (e.g., JDA, formerly Manugistics) which may be included in your overall SAP regression test suite. Many components exist within Tier 2, including:
- Reporting component — If you choose HP QTP as an example, the automated testing results (evidence) will be reported in real-time back to HP Quality Center or ALM. It is important for you to investigate your organization’s overall reporting strategy. Do you want real-time reporting, or are you satisfied with “point in time” snap shots of test execution and defect reporting? Asking yourself questions like that will determine which toolset you’ll ultimately end up purchasing.
- Primary (Driver) Testing Script components — Each of the end-to-end (E2E) test cases described previously in this white paper are driver test scripts. Each of these test cases consist of re-usable functions (e.g., create sales order with VA01 or view invoice with VF03).
- Error handling components — How many times does an unexpected yet irrelevant error appear? These components ensure that your automation framework is robust enough to log these types of errors, but continue running an end-to-end test script unattended. Other components include identifying errors which actually matter and should be logged as defects within the test management tool.
- Business Process Library of Reusable components — Unique SAP T-Codes will make up your re-usable library of components. These components can be used within the driver test scripts. As your SAP Automation engineers develop more automated test cases with different data sets (e.g., Switzerland vs. Chinese data) — they can easily interchange components in an automated driver test script.
Tier 3: The third tier is your relational database management and file system. All objects in ECC or CRM are stored in an object repository (SQL Database) that is referenced by the automated testing tool (e.g., HP QTP). Each object in the repository is re-usable for different testing scenarios. What’s more is that the third tier should contain portions of your golden test set (Note: test data may also exist in tier 1 in the form of parameters stored within manual test cases). The required test data (to be used with the automated test scripts) may live within the automated testing tool and/or a stand-alone XLS spreadsheet. The source of automated regression testing reporting will begin in Tier 3 and feed up into Tier 2.
Figure 1: Example SAP Automation Framework
Source: SAP.com (for Educational Purposes Only)
6. Strive for granularity on all levels when building your SAP Test Automation CoE
Granularity may very well be the cornerstone of developing a robust and maintainable SAP Test Automation CoE that can be called upon for the lifecycle of SAP. If you’re organization has implemented SAP, the odds of strategically changing direction to another ERP vendor is slim. Take into account each of these considerations when developing your SAP automated regression test scripts:
- Ensure that the common reusable library of components in HP Quick Test Professional (QTP) or another automated testing tool contains each unique SAP Transaction Code (T-Code) that your organization’s end users execute.
- Review your organizations critical end-to-end business scenarios. If not done already, granulate the E2E into high-level testing scenarios [MS Visio works great for this activity. Further granulate the high-level testing scenarios into test cases (conditions). Finally, identify the specific SAP T-Codes executed within each test case. Granulate, granulate, granulate!
- Leveraging your ALM or Test Management Tool (Quality Center, SmartBear QA Complete), a test case is created for each end-to-end business scenario. Designs steps (conditions) are added to the test set.
- A “driver script” is used for each end-to-end business scenario. The O2C (order to cash) business scenario for your organization would be driven by one script that calls reusable actions from the common reusable library of components. The unique parameters required to execute the necessary string of SAP t-codes for instance are passed from the component level to the driver script level and executed over SAP.
- As described previously in this white paper — do not underestimate the importance of error handling procedures throughout the common functions in your reusable library. In order to schedule unattended automated regression test execution, your automated suite needs to be designed with sustainability and autonomy in mind. What a waste of resources to automate the test scripts and have humans continually monitor the execution! These valuable resources can be using that time to design/automate more test scripts.
- As an SAP QA Director, you should study up on open architecture and ensure that the automated testing tool your organization chooses can seamlessly integrate (in real-time) with the ALM/Test management tool. If you choose on vendor for both your ALM toolset and automated tool, then you’re likely going to have no problem with seamlessly integrating the software applications. Just be careful and don’t take the vendors marketing collateral as fact. Ask to demo their software in a live environment and to pilot the integration (free of charge to you). Some vendors will ask for a “retainer fee” to pilot their product in your environment. Nonsense! Push back on them by showing them the growing list of QA vendors who would gladly provide a pilot program free of charge. Protecting your organizations assets and bottom line is your 1st priority. Don’t get soft with the vendors vying for your company’s hard earned dollars. Your shareholders will thank you.
Why is granularity so important? Because business processes change, enhancements are made, and software is upgraded. Granularity of test components gives your company the modularity and ease of maintenance required for long-term sustainability. We can’t stress enough the importance of seeing the forest through the trees when it comes to “planning ahead” for sustained operation of your SAP environment (after the expensive consultants off-boarded!). Additional considerations that are “best practice based one experience” include:
- Your SAP automation team should be small and effective — Navy SEAL style. Don’t allow you Solution Integrator or 3rd party vendor to sell you on the concept of growing a large team of non-value add testers. Keeping everyone in the world employed isn’t your responsibility as an SAP QA Director. Less is more when it comes to SAP test automation expertise. Foster, award, and nurture the talented team members. Cut lose the dead weight and very selective in whom you hire into the roles.
- Near sighted “Test Experts” will spend half their day creating and/or searching for data to use in SAP testing environments. This is a complete waste of valuable company resources. Identify your “Golden Test Set” (both master and transactional) early on and develop automated test scripts to create the data cross SAP environments after each client or system refresh. It’s common sense, but you wouldn’t believe how many organizations (and testers) blindly go about every day manually creating data to test against. It is necessary some of the time, but most of the time — you should rely on your golden test set across SAP environments. Most importantly, separate the test data from the scripts! Never hard code the master & transactional data within the test scripts themselves. Quality Center allows you to create parameters and distinct “configurations” for each test script. Take advantage of this functionality and it will save you in the long-run.
- Don’t over complicate the process of estimation and planning required to build your SAP automated regression test suite. If you were a PMP, you’d break everything down (granulate) and then begin the estimation process. The same thing applies when the PM asks you (Mr. Test Manager) how long it will take to completely build the SAP automated regression test suite. Break down the components in a project plan with an estimate of each. Tie each component to a test scenario, and multiple test scenarios back up to an end-to-end business scenario. Simple enough. Provide your estimate and call it a day.
- Your SAP Process Experts, Champions, and Power Users need to work closely with the SAP Test Automation CoE. You should anticipate language barriers if the team is outsourced. Effective collaboration tools like Microsoft Lync and Skype can help overcome any language barriers that may exist. A peer review should occur via Web-Ex. When a change occurs to the script — it is best practice to have it peer reviewed and approved.
- To keep things simple, attach any documentation (e.g., Visio test scenario flows) to the test case within the test management toolset. Don’t get into the habit of storing these files on your workstation or on SharePoint somewhere. Clump them together in one centralized location and you’ll thank yourself when a key player on the team up and quits one day.
- Just like programing — make sure your automation engineers follow proper coding (VBscript in QTP) and commenting guidelines. Document the policies & procedures within your test management tool.
- Review your “Golden Test Set” every 3 months in alignment with SAP environment refreshes to ensure validity. If your organization rolls on another country to SAP, it’s highly likely that your Golden Test Set will need to grow to accommodate the transactional/master data used in that region of the world. You may also need to include additional master data elements as a result of company mergers and/or acquisitions which change the scope & complexity of your business model.
- Dry run your E2E automated scenarios over your SAP QA environment a few times. Share the results with the client SAP Process Experts.
Last, but certainly not least for Step #6 is to turn on versioning control in your test management toolset! If you’re using HP ALM to manage requirements, test planning, design, execution, and defects — the version control feature will show you how a requirement has evolved over time. This is invaluable information to the SAP Test Automation CoE. Most test management tools come out-of-the-box with version control turned off. Be sure to turn it on.
7. After you’ve assembled the key components of your SAP automation architecture, develop a governance model with the outsourced vendor or internal SAP center of excellence team.
If your organization has made the strategic decision to outsource SAP automated regression testing, we suggest leveraging a platform such as VerticalCloud TestFarm (www.testfarm.com). TestFARM is a cloud-based platform that makes it easy for organizations to establish a managed services testing relationship with multiple vendors, one vendor, or freelance SQA experts. Finding specific software testing expertise (e.g., SAP ECC 6.0) is often difficult and very expensive. Global vendors often build offshore SQA/testing teams that aren’t properly trained or worse yet — paid hourly with a contractual agreement! Communication with global teams is also limited to the governance model established by clients and outsourcing vendors, often in the forms of e-mail communication and a static MS Word master test agreement.
TestFARM is a product by VerticalCloud that allows companies to quickly and easily establish a managed services testing relationship with SQA experts across the world. An organization can engage one vendor, multiple vendors, or tap into expertise only found through freelance SQA experts. The end-to-end process of engaging highly qualified test experts across “hard to find” skill sets is streamlined within TestFARM. Through TestFARM, companies can assemble their on-demand (virtual) team quickly and replace resources even quicker. A dedicated VerticalCloud project manager works with you to identify your specific needs and helps you “farm out” the services in the most efficient and risk-adverse manner. VerticalCloud also employs an internal team of highly qualified SQA testing experts with decades of real-world experience that you can leverage in a professional services manner.
The biggest challenge faced by SAP automation engineers is your organizations complex business scenarios, data requirements, and associated manual test cases. As requirements evolve and business process inevitably adapt to changing market conditions, the associated regression test cases need to adapt accordingly. There are several common sense methods to ensuring your outsourced SAP test automation engineers stay abreast of changes, including:
- In a global business environment, SAP Process Experts and Champions are often scattered throughout the world. As your on-demand automation team is assembled with TestFarm, your internal process experts and SAP power users should work collaboratively with the assigned SAP automation engineers.
- As the SAP Process Experts, Champions, and Power Users transfer knowledge to the SAP automated testing center of excellence, screen recording tools should be leveraged to capture every user action. HP Sprinter is an extremely effective tool for capturing a “storyboard” of an end users interaction with SAP. The assembled SAP testing CoE team can review the associated manual test script, review the screen shots, and ask questions of their client counterparts via VerticalCloud TestFarm.
- Leverage HP ALM’s robust requirements traceability features and integrate this functionality into your organization’s configuration management processes. As enhancements and defect fixes (corrections) are identified, the members of your on-demand SAP testing CoE will be notified immediately of the potentially impacted automated RGR test cases. If defects are slipping through to production in a certain area, alerts are created automatically for the experts covering that particular area.
8. Once the SAP automated testing CoE governance model is defined, focus your attention on how your organization will “assign out” tasks for regression testing cycle.
Organizations will often fall into the “trusted advisor trap” for smaller teams — particularly outsourced SAP testing teams. The team often starts out very small — 2 or 3 resources. The outsourcing vendor typically sets up shop in a lost cost country like India, China, or Taiwan as an example. Since the client often has a long-standing relationship with the vendor on other fronts, they assume that the same quality of services will be provided by the outsourced SAP automation engineers.
The client often thinks, “They work for the same parent company, we have a trusted advisor relationship with them on other fronts, certainly they will do an effective job with our SAP automated testing strategy and execution. They are reasonably priced too.”
Meanwhile, the resources on the team have grown from 3 people to almost 10 in the matter of 2 years — all full-time billable “warm bodies” that the outsourced vendor is more than happy to accommodate. As an SAP QA Director, you must always ask yourself this question:
“How can we cover more ground (functionally) with fewer resources, minimize manual testing efforts, and ultimately save the company bundles of cash that we would have otherwise wasted on supporting a local economy with warm billable bodies?”
By laser focusing your attention around the “managed services” model and the platform used to manage workload — you’re organization will not fall into the “trusted advisor trap”. The effectiveness of your SAP testing center of excellence (whether in-house of outsourced) is a direct reflection of the expectations set forth by the leaders in the outsourced organization.
Read Part 1
About the Author
Matthew Angerer- Global SAP Test Manager Mr. Angerer is a versatile, innovative, results-oriented Quality Assurance and Testing Manager with over a decade of experience in enterprise, mission-critical systems implementation, and support for large entities including Unilever HPC, Defense Logistics Agency, Nationwide Insurance, and Mettler-Toledo, Inc. He has solid expertise in SAP ECC and CRM software quality assurance, business process analysis, full lifecycle software development, large-scale software integration, interface design/testing, and Independent Verification and Validation (IV and V) methodologies. He successfully rolled out SAP Testing Center’s of Excellence (CoE) from the ground-up with 10 full-time resources supporting the global testing needs of organizations and reporting directly to him. Matt is an HP Certified Professional in Quality Center and QuickTest Professional, as well as an IBM Certified SOA Associate. He is available for contract SAP QA opportunities and can be reached at matt.angerer@verticalcloud.com.