This article originally posted November, 2008
Party Time. Excellent. Take a Trip Backstage to Hang With 2008’s Test-Tool Heroes (We’re Not Worthy!)
E x-squeeze me! OK, pop quiz: In 2006, did you vote for Mercury Interactive as your starring supplier of testing tools? Chyeah! You all put your hands together for LoadRunner, QuickTest Professional and TestDirector for Quality Center.
I mean like last year, the players were the same, dude, but the awards went to Hewlett-Packard, which acquired Mercury in November 2006. Bonus! These tools have received our standing ovations for four years running. They are like truly amazing and excellent.
Just as last year’s tallies bestowed the awesome LoadRunner with our Grand Prize for most votes overall, so too does the versatile tool receive this year’s top honor, leading in the individual categories of Java/Test Performance, Data/Test Performance and Load/Test Performance. Schwing!
So here’s your backstage pass to party with the best testing tools to take the stage in 2008, along with the two closest warm-up bands. This year’s categories included defect management, automation, security, performance, free solutions and those from new players—seventeen categories in all. Maybe you’ll see your Testers Choice. Party on, Garth.
Data/Test Performance
LOADRUNNER took the most votes in the data/test performance category, which groups products in terms of their ability to apply test data to an application or system and evaluate how it handles the processing.
At the core of LoadRunner’s power is the Virtual User generator (VuGen), which creates test scripts for playback and test case simulation. Script parameters can be modified as needed to adapt to different cases, data parameters (such as for keyword-driven testing), correlation and error handling. A controller module runs the scripts and can simulate large numbers of users while the analysis module tracks and graphs behavior and performance results.
If you’re new to LoadRunner or need to pick up a few new techniques, you might refer to the March 2007 issue of Software Test & Performance for an excellent tutorial by consultant Francisco Sambade on using the tool to test multiple protocols.
Runners-up in this category were Compuware File-AID/CS and Red Gate’s SQL Data Generator. File-AID is an enterprise data management tool that permits testers to quickly build test data environments across a multitude of systems, including mainframes, MVS, DB2 and distributed systems (as the finalist CS edition).
Introduced just this past April, Red Gate’s SQL Data Generator has generated enough attention from testers to and unseat Intel’s VTune Performance Analyzer, last year’s runner-up in the category. The Red Gate tool’s claim to fame is the ability to populate “two million rows of intelligent data for 10 tables in less time that it takes to get a cup of coffee.” What’s more, it does so with a single mouse-click, according to the company.
Functional Test
One again, HP’s QUICKTEST PROFESSIONAL takes the top prize for functional testing. The 2006 Grand Prize winner, QuickTest Pro is a Windows-only record-and-playback tool for functional and regression GUI testing, and it includes a scripting language based on VBScript that can control and manipulate program objects and allow testers to script out test procedures.
Leading it to victory in 2006 were enhamcements to the tool’s team collaboration capabilities, a new object repository manager and the ability to share function libraries across tester workgroups. It also added keyword management and drag-and-drop test-step construction, XML output for test-result reporting, and a new and more accurate debugger that identifies errors while building or maintaining test cases. HP has released no major updates recently.
Runners-up on the functional test category were IBM Rational Functional Tester and Borland Silk Test. Neither of last year’s finalists—Parasoft SOAtest and Compuware Optimal Quality Management—came close this time.
Functional Tester is an automated function and regression test tool for GUI and data-driven testing. It allows scripting with its own ScriptAssure environment or can be driven by Visual Basic .NET, Java from within Eclipse, or custom controls built with Java or .NET through a proxy SDK.
SilkTest is also an automated function and regression testing tool, with cross-platform testing capabilities for applications built for client/server, Java, .NET and the Web. It also includes a scripting framework to help facilitate reuse of test assets.
Static/Dynamic Code Analysis
Testers handed the top honor in the the code analysis category to IBM’s RATIONAL SOFTWARE ANALYZER DEVELOPER EDITION, unseating three-time imcumbent PurifyPlus, a useful tool with long and storied roots, now also developed and marketed by IBM Rational. The automated runtime analysis tool spots memory leaks, profiles application performance and analyzes code coverage. Supported languages include C/C++, Java, the .NET languages, Visual Basic, and Visual C++. Versions are available for Linux, Unix and Windows.
Also unseated by this year’s runners-up—the Eclipse Test and Performance Tools Platform and HP’s DevInspect—were Compuware’s DevPartner Studio and Parasoft’s Jtest, last year’s finalists in the category.
Eclipse’s TPTP project for Linux and Windows got an infusion of new features last June as part of the Ganymede release train. Major new capabilities include increased test coverage through test creation, automation and expanded run time execution; the ability to execute multiple instances of the tool in parallel; simplified handling of test assets, including the ability to cut, copy, paste and rename. Also new are an improved thread profiler and new profiler API for Java 1.5 and above.
Inherited along with HP’s acquisition of SPI Dynamics near the end of 2007, DevInspect was widely praised this year by testers. The tool is intended primarily for automated security testing and remediation during development.
Test/QA Management
HP is once again on top with TESTDIRECTOR FOR QUALITY CENTER, which testers voted this year and last their favorite tool for test and QA management. TestDirector for Quality Center includes modules for requirements management, test planning, test lab and defect management.
TestDirector allows testers and managers to gather requirements, design and schedule manual and automated tests, and analyze the results, all through a browser. Also included are graphical reporting and integration with HP’s WinRunner and QuickTest Professional tools. Options include extensions for Oracle, SAP and SOA.
Finalists in this category were Borland’s SilkCentral Test Manager (as last year) and IBM’s Optim Test Data Management Solution, kicking VMware Lab Manager from its third-place perch last year.
Since being acquired by Borland along with Segue Software in February 2006, SilkCentral Test Manager has been integrated with VMware Lab Manager to streamline cross-platform testing, and with Eclipse to facilitate capturing of test results in a centralized console for impact analysis.
The browser-based environment for remote, simultaneous test execution and management of JUnit/NUnit and other third-party testing frameworks also includes a manual test client to ensure repeatable data collection and reporting. Reporting is provided by BIRT.
Optim Test Data Management Solution is IBM’s way to manage test data and improve application quality. The tool permits testers to apply specific standards of coverage, create error or boundary conditions and create an environment that closely simulated production.
The same tool can be used across enterprise apps, including PeopleSoft and Siebel; databases from IBM, Microsoft, Oracle, Sybase and others; and Linux, Unix, Windows and Z/OS.
Defect/Issue Management
Last year there were two winners in the defect and issue management category: HP TestDirector for Quality Center and the Mozilla Foundation’s Bugzilla were tied. This year, TESTDIRECTOR alone took the top spot and Bugzilla moved to third, edged out by Microsoft’s Visual Studio Team Edition for Software Testers.
The defect tracking module in TestDirector automatically checks its central defect database for similar defects each time a new defect is logged, helping to reduce duplicates. The tool tabulates defect statistics to aid management deployment decisions and identify trends.
A relative newcomer, Visual Studio Team Edition for Software Testers can perform functional and load testing on Web applications and Web sites, and manages and permits reuse of test cases, all without writing a single line of code.
Bugzilla began its life in 1998 and was originally written in Tcl by Terry Weissman. Thinking another language might get more traction within the community, Weissman decided to port it to Perl, which resulted in Bugzilla 2.0. In April 2000, the project was handed off to Tara Hernandez, who gained more developer participation, including that of its current custodian, Dave Miller. Bugzilla last year won the top spot in the free test/performance tools category, but this year slipped to runner-up.
Load Performance Test
This year’s winners in the load and performance testing category are a carbon copy of last year’s. LOADRUNNER again topped the list, with capabilities centered around its Virtual User Generator record-and-playback tool. VuGen generates an editable script of user actions that are routed through a protocol-dependant proxy during recording. So when scripts are generated using the Web/HTTP edition, testers can set LoadRunner to generate either URL– or HTML-based scripting, for example.
Runners-up in this category again were IBM Rational Performance Tester and Borland SilkPerformer. Performance Tester stresses Web applications and tests their scalability. Integrated with IBM’s Tivoli management environment, it enables large, multi-user tests while using minimal hardware resources. Distributed controller agents are available for Linux, Windows and z/OS. Test results are displayed in a Linux or Windows application with high-level and detailed tree views and a tree-based text editor.
SilkPerformer is an automated, sharable environment for executing load and stress performance tests across Java and .NET applications with server-side analysis. The tool can simulate thousands of users with no licensing restrictions for controllers or protocols. A visual script recorder enables script editing and manipulation. A plug-in extends test creation to Eclipse. There’s also a SOA edition.
SOA/Web Services Test
In the SOA and Web services testing category, last year’s winner was this year’s no-show. LoadRunner has fallen out of favor for testing these types of applications, replaced in 2008 by IBM’s RATIONAL TESTER FOR SOA QUALITY.
When your app relies on someone else’s Web service, you can’t leave availability to chance. For example, if you’re depending on a credit authorization service that’s accessed via the Web, how can you ensure that it won’t fail on Black Friday, traditionally America’s busiest shopping day? While you can’t control the credit agency’s service, you can use Tester for SOA Quality to simulate thousands of hits against its service and watch how your application behaves while simultaneously trying to conduct hundreds or thousands of transactions. The tool also works with services that present no GUI.
Runners-up in the category were HP’s QuickTest Professional and Parasoft SOA Quality Solution.
A perennial favorite of testers, QuickTest Pro is an automated functional and regression testing tool for GUI applications that records and plays back user actions on Web or native applications. Scripts are recorded in a proprietary scripting language based on Microsoft’s VBScript and can be modified, played back and reused.
Parasoft’s SOA Quality Solution incorporates the company’s multi-layer workflow approach to testing and quality assurance. Capabilities include policy enforcement with interoperability and consistency checking across SOA layers; multi-layer verification of the underlying implementation for auditing; service emulation with end-to-end testing and emulation of business logic or transactions; regression testing; penetration testing; and load and performance testing.
Security Test
Last year, SPI Dynamics took the top prize in security testing with WebInspect, its security scanning and assessment tool for Web applications. Now owned by HP, which took both runner-up slots in the category, WebInspect this year was displaced by IBM’s RATIONAL APPSCAN STANDARD EDITION.
Inherited along with its acquisition of Watchfire, AppScan is an automated security auditing tool for developers, QA staff and security auditors. It’s also available in multi-user and SaaS versions. IBM in October introduced AppScan Express Edition, a version of the utility aimed at small and mid-sized businesses with support for apps made with AJAX, Flash and other so-called Web 2.0 technologies.
HP’s Assessment Management Platform, a first-timer for the Testers Choice awards, is an automated, centrally controlled security scanning and testing platform for Web applications. It permits distributed teams to access and remediate security vulnerabilities through a dashboard-style interface.
WebInspect last year, prior to its acquisition by HP, got itself a shiny new profiler that scans Web apps and suggests configuration settings for the most effective testing. It also received a real-time traffic monitor that reports HTTP activity during a scan. A results window displays requests and responses sent by WebInspect during crawls and security audits. The tool was completely rewritten in January 2007, and SPI claimed performance improvements and compatibility with modern technologies and techniques. WebInspect 7.5 reportedly further improved auditing capabilities for spotting vulnerabilities in AJAX-based applications and strengthened its support for Windows Vista.
Test Automation
QUICKTEST PROFESSIONAL again comes out on top, this time in the test automation category. QuickTest Pro includes a pluggable architecture to allow customization. The tool includes plug-ins for ActiveX controls, Web applications and VB objects. Optional .NET-language objects and multimedia plug-ins are available. The tool also saves the AUT’s screens along with the recorded script, highlighting the portion of the screen being tested as an aid to regression testing. These are also useful for creating checkpoints, which can apply to images, tables and pages, and permit verification of expected application behavior during automated tests. Other features include virtual objects, output value for data verification, data- and transaction-driven testing, and exception handling.
Compuware TestPartner is an automated GUI function testing tool that uses Microsoft’s Visual Basic for Applications for scripting, debugging and error handling. Tests also can be created using “Visual Navigator,” a storyboard environment. By displaying screens of the AUT, testers select from lists of automation steps to assemble storyboards. They also can add test logic, variables and other particulars.
Another newcomer to the Testers Choice awards is HP Business Process Testing, a Web-based test design solution that the company says is intended to allow subject-matter experts to build and execute manual and automated tests without any programming knowledge. It automates testing and creation of documentation, and is also said to reduce test maintenance.
SCM/Build Management
Microsoft VISUAL SOURCESAFE was tops in the SCM/build management category for the second year running. Originally developed by One Tree Software as SourceSafe, this tool was first released in the early 1990s as version 3.1, which a Wikipedia author speculates was meant to match the current version of Windows at the time. Microsoft’s SCM at the time was called Delta and was said to be inferior. Microsoft acquired the 16-bit SourceSafe and released a 32-bit version as Visual SourceSafe 4.0 around 1995. It would be 10 years before the company released VSS 2005, the first client-server version; until then it was limited to local use.
Among VSS’ strongest attributes are its tight integration with Visual Studio, relative ease of use and price; it’s free with certain subscriptions to MSDN. It also offers numerous expansion possibilities thanks to a pluggable architecture. Its future in the enterprise seems uncertain, however, since many of its capabilities are implemented in Visual Studio Team Foundation Server and several good rival systems are available from the open-source community.
One such rival system is Subversion, which again this year is a runner up along with ElectricCommander from Electric Cloud.
The Subversion version control system, particularly popular among open-source developers, was launched in 2000 by CollabNet. Subversion commits are atomic and do not cause corruption if interrupted. All moves, adds, changes and deletions are versioned, including files, directories, trees, file metadata and symbolic links. It supports binary file storage with file-locking for unmerged files, and sends diffs in both directions.
Build automation tool maker Electric Cloud in late September began shipping ElectricCommander 3.0, which included the ability to “preflight” newly modified source code to determine if it will correctly build into an application. The new version also integrates with Eclipse and Visual Studio. ElectricCommander automates program component assembly without regard to development language or build utility. It supports numerous scripting languages, including Bash, perl, Python and Tcl. It also works with AccuRev, ClearCase, Perforce, Subversion and Synergy software configuration management systems.
Embedded/Mobile Test/Performance
Parasoft took the top prize in the embedded/mobile test and performance category with its C++ TEST, putting IBM’s Rational Test RealTime out of the top spot. C++ Test is a solution for code analysis and review, coverage analysis, regression testing and automation of unit and component testing. Parasoft offers versions for embedded and non-embedded apps for Linux, Solaris and Windows. C++ Test works with Eclipse and Visual Studio as well as in batch processes.
Runners-up in the category were TestShell from QualiSystems and Prevent from Coverity.
TestShell, the flagship of tool maker QualiSystems, is a line of products that includes tools for planning, building, executing and controlling tests and analyzing the results. The heart of the suite is TestShell Foundation, the engine and database that provide a pluggable infrastructure to grow with the needs of the testing organization.
Aptly named for its intended purpose is Prevent, Coverity’s code scanning tool for finding programming errors using static analysis before compilation. It works with C, C++ and Java source code, and integrates with an existing build environment and development process.
.NET Test/Performance
For testing performance of .NET applications, testers this year chose Microsoft VISUAL STUDIO TEAM SYSTEM EDITION SOFTWARE TESTER, moving it up from its second-place finish last year to trade places with HP LoadRunner.
Team System is an assemblage of server- and client-side components that trickled onto the market since 2005. The primary server-side components are Team Foundation Server and SQL Server. The server stores all source code and supports multiple simultaneous checkouts, branching, merging, conflict resolution and tracking of all changes. Security can be applied at any level.
Reporting is handled by SQL Server Reporting Services. Included reports can show code change over time, lists of bugs without test cases and regressions on previously passing tests. There’s also a build server with tracking. On the client side, Visual Studio enables code analysis, code coverage and test tools for build validation.
Support for .NET languages was added to LoadRunner beginning with version 8, which also added a diagnostics server, commander, mediator and probes for Java EE and .NET. Other improvements included a LoadRunner 8 that was more scalable, and easier to install, configure and use, according to the company.
LoadRunner permits hundreds of simulated users to be launched from one or several machines with the LoadRunner agent installed. These load generators, or “injectors,” are instructed on how and when to launch scripts using LoadRunner scenarios so that simulations can be ramped up.
The .NET version of Compuware DevPartner Studio also was a runner-up in this category. The company also offers versions for C/C++ and Java.
DevPartner analyzes quality and complexity of code and can detect memory leaks, timing issues and coverage. The software also can help with memory optimization, CPU, disk and network resource management, fault simulation, and error detection.
Java Test/Performance
LOADRUNNER is the testers choice in the Java test and performance category for the third year running. During load tests, the tool uses monitors to gauge the performance of the applications or components under test. Available monitors include those for virtual users, transaction rate, network latency, Web and database server responsiveness (these are server-specific) and for server resources.
Monitor data is saved for examination by the analysis tool, which processes and graphs the completed scenario results. By analyzing these graphs and combining them into reports, testers gain an understanding of the big performance picture and can make or suggest adjustments to tune performance.
JUnit and the Eclipse Test and Performance Tools Platform also received high marks.
Often heralded as the grandfather of unit testing frameworks is JUnit, one of the earliest and most successful of its kind. Celebrating its 10th birthday in 2007, JUnit has spawned ports to numerous languages including C# (NUnit), C++ (CPPUnit), Fortran (fUnit), Perl (Test::Class and Test::Unit), PHP (PHPUnit) and Python (PyUnit). There’s even a version for JavaScript (JSUnit). JUnit has been downloaded more than 2 million times in 10 years, according to one study, and is included as a plug-in to all major IDEs, including Eclipse.
JUnit 4.5 was released in August and simplifies JUnit extensions with new public extension points for inserting behavior into the standard JUnit 4 class runner. Advances in JUnit 4.4 included a new API with an “extensible and readable syntax,” according to the release notes. The API enables features like assumptions or the declaration of explicit dependencies when a tester has no control of forces that might cause a test to fail. Assumptions also give rise to theories, which can capture some aspect of behavior in “possibly infinite numbers of potential scenarios,” reads the notes.
When it was updated to version 4.5.1 in September, the Eclipse Test and Performance Tools Platform (TPTP) workbench and Agent Controller received Java 1.6 support, numerous bug fixes and increased test coverage through test creation, automation and expanded run-time execution. The update includes numerous enhancements over 4.5, its Ganymede release in June.
Integrated Test/Performance Suite
While awaiting Visual Studio Team System 2008, testers in 2008 told us they are quite fond of VISUAL STUDIO TEAM EDITION SOFTWARE TESTER, this year’s winner in the Integrated Test and Performance Suite category. Meanwhile, much is already known of Rosario, code name for Visual Studio Team System 2010.
Microsoft reportedly plans to build more application life-cycle management capabilities into the tool, will integrate it with its Office Project Server, and add full traceability across projects and resources as well as metrics and dashboards.
Testers last year chose HP Performance Center as their favorite integrated test/performance suite; this year it was runner-up. The tool combines all the capabilities of LoadRunner with test-asset and human-resource management features and reporting in a centralized repository accessible through a browser with role-based security. Load-test configurations and test scripts can be created, edited and scheduled for execution. Progress can be tracked, graphed and compared with data from past projects or releases. Performance Center also can reboot and deploy patches to remote nodes.
Also a runner-up was TestComplete from AutomatedQA. TestComplete is a record and playback tool that automates UI functional testing of Flash, Java, .NET, Web and Windows apps. It also can handle database and back-end HTTP load testing. Recorded tests can be modified and reused for regression testing.
Commerical Test/Performance (under $500/seat)
The prime choice for sub-$500 commercial testing solutions was SOFTWARE PLANNER, making 2008 a peak in the roller coaster ride that was the past three years for Pragmatic. The SaaS tool earned top marks this year, was a runner up in 2007 and took the top spot in 2006. Software Planner is a Web-based hierarchical project planner that uses task linking to prevent one task from beginning before another is completed.
Project managers can use the system to set baseline dates, costs and track deviations as the project proceeds. Team members are alerted to new tasks via e-mail; multi-level security limits access to authorized people. Billed as an application life-cycle management tool, it helps IT departments manage requirements, project deliverables, test cases, defects and trouble tickets. The tool integrates a calendar, dashboards, document collaboration and sharing, a knowledge base, and threaded discussions. Software Planner also is available as a self-hosted solution.
Runners-up were Mindreef SOAPscope Developer and Shunra Virtual Enterprise Desktop Standard.
Mindreef has been earning awards almost since coming onto the scene in 2002. SOAPscope 5.0, a tool for testing SOAP-based apps was awarded best solution from a new company in 2006. SOAPscope Developer allows teams to create, test, deliver, support Web services and SOA components, and automate XML-oriented tasks. The tool is included with SOAPscope Server 6.1, a server-based solution intended for use and collaboration by all members of the SOA and Web services team, including analysts and managers.
Shunra Virtual Enterprise creates a virtual network for use as a simulation environment for application testing. It permits testers to determine application performance and user experience under a variety of network conditions and “impairments,” such as high network load or traffic, excessive collisions, broadcast storms, and other types of real-world production scenarios. Test results are displayed graphically, and testers can drill into particular results for issue specifics and analysis.
Free Test/Performance
Of the free testing and performance tools available today, testers this year chose the TEST AND PERFORMANCE TOOLS PLATFORM project of Eclipse, bouncing Bugzilla, last year’s winner, into a runner-up. TPTP is a test and performance life-cycle tool, addressing a project’s needs from early testing to production application monitoring, and everything in between. Test editing and execution, monitoring, tracing and profiling, and log analysis capabilities all are core parts of this tool, which supports embedded systems, standalone applications, enterprise-grade apps as well as high-performance computing systems.
One of Eclipse’s earliest and most successful projects is still TPTP, and it has enjoyed tremendous support from the community. Now at version 4.5.1, its subprojects include those for building tools for monitoring and logging performance, and resource allocation of application servers; for building testing tools that edit, manage, deploy and execute tests; and for application trace and profiling.
Runners up were J
Unit and Bugzilla. It’s safe to say you’ve succeeded when your name is widely used as a verb. For instance: “Google that term and see what you get,” or “Be sure to JUnit your code before checking it in.” JUnit has become synonymous with unit testing and has been built into Eclipse for years now. Its success has even exceeded the expectations of co-creator Kent Beck.
Bugzilla, the open-source defect tracking system, celebrated version 3.0 last year, nine years after v2. Chief among the enhancements are huge performance gains, a Web services interface, and the ability to control and customize far more of the environment than in prior versions, according to project developers.
Best Solution From A New Player
VMLogix LABMANAGER 3.5 took this year’s top spot for Best Solution from a New Player, which we define as a company less than five years old. Released in May, version 3.5 added support for Citrix XenServer 4.1 to its existing support for virtualization systems from Microsoft and VMware. LabManager 3.5 also includes some major user interface enhancements and to virtual network management.
Last year’s winner in this category was Defender from Fortify Software. This year the company reached runner-up with another new product: Fortify 360, which takes aim at the software development life cycle with the ability to identify, prioritize and repair vulnerabilities during every phase of development.
Requirements management company Blueprint grabbed a runner-up slot with Requirements Center, which brings cloud-based collaboration to application users and business analysts. As the company puts it, the solution puts these people at the heart of the development process, “allowing them to collaboratively define, visualize and validate their requirements in business terms, all before a single line of code has been developed.”
About the Author
Edward J Correia