Introduction
Anyone that has watched the movie A League of Their Own must remember the scene when Dottie quit the team right before the World Series. The conversation between her and Jimmy [Tom Hanks] has been one of the most unforgettable scenes for me. Dottie’s husband just returned from WW2 and she decided to quit the team and move back to the country. When Jimmy asked why she wants to leave baseball, she looked at him and said: “It just got too hard.” The look on the face of Tom Hank’s character could make your neck hair rise to attention. Stone faced, Jimmy leaned forward and said:
“It’s supposed to be hard. If it wasn’t hard, everyone would do it. The hard is what makes it great.”
Managing SAP projects is hard – nobody will question that statement. Ensuring that your automated regression testing suite is effective, balanced, and properly designed is even harder. The scene referenced above in A League of Their Own sheds light on why difficult situations shape character and ultimately drive professionals to search for better and more efficient ways to trap SAP defects. Sure, we could all throw up our hands and say “It just got too hard” and walk away from the sometimes messy field of SAP consulting, project management, and testing strategy for the matter. But, as Jimmy expressed to Dottie: “It’s supposed to be hard. If it wasn’t hard, everyone would do it. The hard is what makes its great.”
Overcoming SAP test strategy challenges with a balanced approach to people, process, and technology is sometimes a matter of embracing the chaos while riding the waves of good and bad. If you’re an IT professional of any kind, you can relate to Dottie when she wants to throw in the towel and walk away from baseball for good. On the flip side, some of us can also relate to the more experienced and battle scarred character that Tom Hanks played [Jimmie]. A Roadmap to Streamlining SAP Automated Regression Testing for Global Organizations is an attempt at making “the hard” achievable for organizations of all shapes and sizes that decide to implement SAP. Full of common sense advice that is often overlooked, we provide you with the necessary insights to make “the hard” great again.
Why Automate SAP Regression Testing?
The information age we live in today has turned into a boiler maker. Software is dynamic, project managers have lost their patience, and the CFO is focused on the bottom line. The project schedule is aggressive and regression testing cycle windows have shortened as a result. If one aspect of the project delivery schedule is delayed, often times the leadership team will look to cut corners. One of the first areas they look at is the testing phase of the project. Before Go-Live, a proper regression testing of existing functionality for all the live SAP productive units isn’t a luxury – it’s a necessity. But, guess what? The CFO is concerned about the millions of dollars on the line for each month that the project go-live is delayed (contracted SAP functional experts aren’t a cheap date). Before you know it, you are called into a meeting with questions like:
- How can “you” (test management) reduce the regression testing cycle even further due to delays in other phases of the project?
- How can “you” (test management) magically ensure sufficient testing coverage and effective QA for critical SAP business processes while slashing your test execution window in half. And, oh by the way, we (the business) can’t provide you with any more resources (people) to
execute test cases manually. - How can “you” (test management) mitigate the “ripple effect” associated with configuration changes across SAP modules?
All of these questions characterize the challenges that come with manual regression testing. Manual regression testing often requires end users of the business to “act as testers” during the regression testing cycle of major roll-ins. The advantage of using the business SAP Power Users during regression testing is that it truly provides them with assurance that their critical business functions will continue to work. The disadvantage on the other side of the coin is that manual testers provide less test coverage and offer inconsistent results – not to mention it’s time consuming!
If you’re still reading this white paper and nodding your head right now – then you’re in for a treat. SAP test automation can address many of the challenges outlined above. It’s not a silver bullet (cure all) by any means, but it can provide the following benefits:
- Instead of waiting until the last leg of an SAP roll-in to regression test, you are able to run scheduled SAP automation cycles – ultimately reducing the cost of defects [early detection] and helping to improve overall quality.
- Speed – enough said.
- Repeatability and consistency. Humans are humans. We make mistakes.
- Test cases can be scheduled to run independently (e.g., overnight) with very little human intervention.
- Automation scripts can be used to automatically setup master data, variants, and even transaction data across SAP testing environments.
- Organizations can develop “Navy SEAL Like” automated testing capabilities. The testing team size is small and highly effective. Regression testing cycles are no longer dependent on SAP Power Users from the business manually testing critical business functionality.
- With the right toolset such as HP Quick Test Professional (QTP), reporting and critical defect detection is automatically synchronized with leading test management tools like HP ALM or Quality Center. Project Managers have finger-tip transparency.
- Month-end financials, sales order replication from CRM to ECC, bank reconciliation, production planning, demand planning and other key mission-critical business processes are confirmed with automated regression testing.
- With SAP support pack stacks (SPS) and Enhancement Packages – an automated test suite ensures critical business processes continue to work without error.
The benefits outlined above make SAP automated testing a necessity. Yet organizations are still on the fence of whether to outsource SAP testing services to leading consultancies, build an internal SAP testing center of excellence (CoE), or develop a hybrid approach to outsourcing that involves both in-house expertise and offshore/outsourced SAP automation expertise.
Ask the CIO and he’ll recall conversations he had with other C-level executives about outsourcing. From the C-level executive perspective, outsourcing SAP testing (both automated and manual) is the perfect solution – especially for a global organization. Transfer risk and responsibility to a 3rd party consultancy and your problems are solved – your hands are wiped clean – or so you think. Even better, you save a bundle of money by hiring “SAP Automation Experts” in low cost countries like India or Malyasia! The CFO is happy and the shareholders on Wallstreet agree this is the right approach. The Solution Integrator (SI) is also tickled pink because they now hold the responsibility of bug fixing and regression testing. Can you say “revolving door”? This is the proverbial situation of allowing the “fox to guard the hen house”. The message is simple: just because offshoring (outsourcing) testing has been somewhat successful for other global organizations does not necessarily mean it will work for you.
An SAP QA Manager must be careful in choosing their battles – especially with upper management. We must learn to carefully present the facts of a situation and recommend the best solution for the organization. For some firms, 100% outsourcing of your SAP regression testing is the best strategy. The 3rd party firm finds the “right people”, employs the “right process”, and leverages the “right technology” to make it a fruitful relationship for many years to come. For other organizations, it makes sense to get your ducks aligned internally before engaging with a 3rd party vendor to outsource SAP automation testing.
In the words of Clint Eastwood, A Roadmap to Streamlining SAP Automated Regression Testing was written to provide you with: The Good, The Bad, and The Ugly. As we all learned during the financial collapse of 2008, sugar coating the facts will get us into a bad situation very quickly that requires years of hard work and sacrifice to dig ourselves out. Our goal is to present you with common sense steps and give you “real-world hints” that can only be provided by the people in the day-to-day operational activities of developing an SAP automated testing Center of Excellence (CoE). The first step we recommend is choosing the “right test tool” and the “right automation framework”.
1. The right testing tool and automation framework can make all the difference.
For global organizations that are implementing SAP ECC and CRM, choosing a true “application lifecycle management” [ALM] toolset can make all the difference. Many organizations are short sighted and decide that a Microsoft Access and Excel-driven approach to “everything test management” is the tried and true method of achieving success. This approach may work if you’re building out a web-based SaaS application that is designed to accomplish a few different functions. But, when you’re rolling out SAP ECC with every module known to mankind – you better hunker down and do the necessary analysis of the tools available in the marketplace. Choose a test management tool that integrates easily with your chosen automation toolset. Also make sure that your chosen automated toolset can automate testing in systems other than SAP. This is especially important for designing E2E testing of all business transactions (e.g., SAP CRM to SAP ECC).
Table-1 provides you with a format to evaluate leading test management tools. Toolsets to evaluate may include HP Quick Test Professional (QTP), IBM Rationale Functional Tester (RFT), Worksoft, Panaya, TestAnywhere, Janova, and SmartBear’s TestComplete. When making a selection, it is important to holistically view the entire suite of software applications throughout your organization. Some testing vendor’s focus specifically on SAP, but you may be well-suited to consider a vendor that offers strong requirements traceability for future change impact analysis. SAP is complex software. Identifying “what to regression test” is sometimes half the battle. Think “Quality Assurance” from the requirements definition process all the way through systems delivery.
| Decision Criteria | Vendor A | Vendor B | Vendor C | Comments |
|---|---|---|---|---|
| Cost | ||||
| Script Generation Language | ||||
| Object Identification | ||||
| Reporting Capabilities | ||||
| Change Impact Analysis | ||||
| Scripting | ||||
| Script Playback | ||||
| Script Execution | ||||
| Browser Support | ||||
| Descriptive Programming Approach | ||||
| Automation |
Table-1: Sample Criteria to Evaluate Test Management Software Vendors
Other considerations that should weigh heavily into your decision include the support apparatus offered by the vendor and the ease of implementation. Many vendors are offering their test management toolsets SaaS (software-as-a-service). Companies like Janova Software (www.janovasoftware.com) and Panaya offer cloud-based automated testing solutions. Other vendors like HP and IBM require local installation of their automated toolsets which integrrates with their web-based test management tools. Choose wisely, but choose based on your unique situation. Just because the vendor owns 60% of the market share (cough: HP), doesn’t mean you should blindly go in that direction. Some of the smaller players like Janova Software and Panaya really have something unique to offer.
2. Knowing which SAP automation framework to build is critical for maintainability.
Software is dynamic and ever-evolving. Nothing remains the same. As an SAP QA Director, you should skeptically validate all claims made by the implementation partner and internal project management team alike. Blindly accepting statements without challenging them on the spot will get you into a heap of trouble in the long-term. Things you might hear include: “There isn’t a need to stand up a virtual server for the automated software – our SAP automation engineers will install directly on their workstations.” Or even worse, “Managing requirements traceability in a centralized toolset is overkill. We can easily track requirements with Excel-based spreadsheets and use other sets of Excel workbooks for Gap tracking and Test Control.”
Obviate. In a few years, the implementation partner you originally chose to implement SAP is likely winding down. The KT (knowledge transfer) will occur to the internal team. Your QA & Test team is now responsible for the monthly SAP service packs. If your automation framework is not robust, or your framework is lacking requirements traceability – it’s a nightmare to manage. We’ll talk about managing change impact a little later in this whitepaper, but the critical success factor for choosing an automation framework is to think in terms of maintenance and long-term sustainability.
We will also describe key architecture considerations for your SAP automation framework later in this whitepaper, but you should pick a hybrid SAP automation framework that facilitates unattended test automation execution. A framework in which you can kick off a series of end-to-end test scenarios auto-magically (without real people monitoring). Some believe this is unrealistic, but with careful planning and design – it can be accomplished. The problem occurs when the automation framework was originally designed by an unknowledgeable resource that has to be “undone” and “redone” in another toolset and framework altogether. Don’t fall into this trap. Make the right decisions up front and you’ll save yourself months of frustration and wasted resource time.
3. Define SAP regression automation goals and communicate them often to business stakeholders.
Some people take others knowledge at face value based on their titles within an organization. Its common sense, but it helps to remind yourself at times that others in the organization are not always promoted into positions because of their knowledge. It’s a fact that certain people hold positions of decision-making authority due to whom they may have known prior to joining an organization, or being in the right place at the right time. Reminding yourself of this fact necessitates the importance of outlining the goals of automated regression testing to all stakeholders. Don’t assume that everyone understands the distinction between System testing and Regression testing.
you’ll save yourself months of frustration and wasted resource time.
The key is to identify what regression test cases would make good candidates for test automation. To do this effectively though, you’ll need to first define the goals for automated regression testing while also outlining what “regression relevant” truly means to the business. Since SAP releases are repetitive, it’s not feasible (nor is it necessary) to test all functionality including transactions, interfaces, configurations, etc.., during every regression testing cycle.
you’ll save yourself months of frustration and wasted resource time.
Many factors come into play when determining the goals of automated SAP regression testing. Largely it’s about ensuring that the critical path of the business’ operations are always “up and running” on SAP. If you’ve been around the block a few times, you’ve heard of the following E2E processes:
- Campaign to Cash
- Order to Cash
- Hire to Retire
- Engineer to Order
- Make to Stock
- Postpaid Service Contract to Billing
- Make to Order
- Lead to Prepaid Service Contract
- Purchase to Pay
Since project managers and the business are bottom-line focused (most of the time), they typically provide you with X number of days to get through your regression testing cycles. Often times, you have about 1 week (5 business days) to get through the automated regression testing cycle for monthly SAP service packs (sometimes even less). For a large roll-in or enhancement project, you can expect a few weeks – give or take. During this time period, your testing cycle should ensure that all key interfaces spanning systems and trading partners (not just SAP!) are still working as designed.
you’ll save yourself months of frustration and wasted resource time.
Remember: the focus of automated regression testing is to ensure the critical path of the business is not impacted by a new build (code) moving into production. As an SAP QA Director, you should outline this in the organization’s master test strategy. Excerpted below was my definition of SAP regression testing for a global manufacturer of precision instruments: you’ll save yourself months of frustration and wasted resource time.
Regression testing provides confidence that key subsets of existing functionality have not been adversely affected by new or amended functionality. It is not the purpose of regression testing to cover actual testing of new or modified functionality, as this will have been covered in earlier test stages (unit and functional testing).
you’ll save yourself months of frustration and wasted resource time.
Regression testing covers key areas of existing functionality, such as core business critical functions.
you’ll save yourself months of frustration and wasted resource time.
It is not feasible, nor desirable, for regression tests to cover every piece of functionality due to the effort involved in both setting up and maintaining the regression tests, and also the amount of time and effort involved in executing the tests. The degree of functional coverage in the overall regression test suite will depend on the number of agreed critical functions, with areas such as non-essential reporting, administrative views, and low volume queries/transactions being deemed out of scope. The functionality of a particular code release will determine the sub-set of the regression tests within the overall regression pack, which need to be run. However, the tests will cover as wide a range of functionality that is possible for a given amount of effort you’ll save yourself months of frustration and wasted resource time.
Problems in the following areas will be reduced by performing a regression test:
- Adverse impact on functionality in another area due to changes made in one area;
- Reduction in performance (a high-volume environment will be required for performance testing to be representative of production; however performance problems can still be identified in a medium volume environment if a particular test runs slower on the new code set.);
- Exception and error reporting (by identifying problems before they are put into production).
The main benefits of performing a regression test phase are as follows:
- Provides confidence that a new release will not adversely affect key functionality currently in the production environment;
- Reduces the risk of disruption to the production service immediately after a new code implementation, and associated business impact.
One more analogy to solidify our understanding of regression testing would be when a manufacturer enhances a motorcycle that is already in the marketplace. People all over the world are driving this motorcycle model today. The manufacturer decides to make a few enhancements that will improve handling and performance. Before releasing the new model into the marketplace, they first need to conduct a series of tests on the bike.
- Engine starts. Check
- Motorcycle accelerates. Check
- Motorcycle turns to the left and right – steering works. Check
- Bike slows down to a stop after acceleration – brakes work. Check
Confirming these functions continue to work after the enhancements (modifications) have been completed to the motorcycle is critical. If time and budget permit, the QA team can conduct additional testing (e.g., exploratory testing) to ensure other functions of the motorcycle continue to work as designed. Often times, project managers look at the regression testing phase as the “catch all” of the testing layers. When bugs slip into production, you’ll often hear: “Well, why wasn’t this caught during the regression testing that we do?” Save your fingers and outline it once within a master test strategy. People will ask “why wasn’t this trapped during regression testing” about bugs that surfaced in non-critical business functionality. When others do ask this question (as they inevitably will), you can use it as an opportunity to quickly reiterate the goals of regression testing. If you’re so inclined, you can take it a step further by discussing QA (prevention) techniques that would have trapped the bug much earlier in the systems delivery lifecycle. It is important that the organization as a whole understands the purpose and goal of the regression testing layer. It reduces (never prevents) finger pointing you’ll save yourself months of frustration and wasted resource time.
4. Determining what business scenarios are “critical” to the business takes collaboration.
On most SAP projects, the implementation partner and business conduct a series of “fit/gap” analysis meetings (otherwise known as JAD sessions) to determine what needs to be customized from the global SAP template that is out-of-the-box. In the late 90s and early 2000s, SAP experts categorized their knowledge based on module (e.g., SD or MM Expert). Today’s SAP professionals are more known for their cross-module expertise, rooted more deeply in process. With that said, you should always identify the end-to-end scenarios (outlined in Step #3) as critical – forming the basis of building your SAP automated regression testing suite.
Beyond the end-to-end scenarios, your SAP automation strategy should include the:
- Identification of functions that are used most often in SAP production. Vendors like Panaya offer SAP impact analysis tools to quickly identify what SAP transactions are most used by your business end users. Examples such as entering sales orders, invoicing, month end closing, creating material master records, and other high volume activities need to be tested and included as part of your SAP regression automation suite.
- Segmenting of functionality that includes a high degree of customization. If your SAP CRM Service template has been highly customized – it would make sense to increase automated regression testing coverage in this area.
- Interface communication testing of SAP with legacy systems and trading partners. SAP PI, EDI, and CRM Middleware are incubators for defect activity. As a build moves through an SAP landscape, manual configuration is needed (pre set-up work) in each environment. The human intervention needed to setup partner profiles as an example lends itself to errors. Automating the regression testing of sales orders created in SAP CRM to SAP ECC is a necessity.
After you’ve identified the business scenarios to include in your SAP automated regression test suite, it’s time to decide upon the automation architecture The example figure in Step #5 shows you a sample automation architecture and key considerations.
— End of Part 1, Part 2 —
About the Author
Matthew Angerer Mr. Angerer is a versatile, innovative, results-oriented QA Architect with over a decade of experience in enterprise, mission-critical systems implementation, and support for large entities including Unilever HPC, Defense Logistics Agency, Nationwide Insurance, Honda Motors, Tory Burch, SM Energy, Imperial Irrigation District (IID), Wisconsin Public Services Corporation, and Mettler-Toledo, Inc. He has solid expertise in SAP ECC and CRM software quality assurance, business process analysis, full lifecycle software development, large-scale software integration, interface design/testing, and Independent Verification and Validation (IV and V) methodologies. He successfully rolled out SAP Testing Center’s of Excellence (CoE) from the ground-up with 10 full-time resources supporting the global testing needs of organizations and reporting directly to him. Matt is an HP Certified Professional in Quality Center and QuickTest Professional, as well as an IBM Certified SOA Associate. He is an HP ALM Solutions Architect for ResultsPositive and can be reached at mangerer@resultspositive.com.