The Choices This Year Look A Lot Like Last Year’s Winners, With LoadRunner Again Leading the Pack
In 2006, you voted Mercury Interactive as your favorite supplier of testing tools, lavishing high praise on LoadRunner, QuickTest Professional and TestDirector for Quality Center. Today, the names of the most highly acclaimed tools are the same, but the awards go to Hewlett-Packard, which acquired Mercury in November, 2006. The tools have received our top honors for three years running.
Unlike last year’s tallies, which bestowed our Grand Prize on QuickTest Professional for the most votes overall, this year’s highest honor goes to LoadRunner, which topped five individual categories.
So here they are, your choices of the best testing tools the industry has to offer, along with the second- and third-place finalists. Categories include defect management, automation, security, performance, free solutions and those from new players; 17 categories in all. Perhaps you’ll see your Testers Choice.
Data Test/Performance
LoadRunner took the most votes in the data/test performance category, which groups products in terms of their ability to apply test data to an application or system and evaluate how it handles itself.
At the core of LoadRunner’s power is the Virtual User Generator (VUGen), which creates test scripts for playback and test case simulation. Script parameters can be modified as needed to adapt to different cases, data parameters such as for keyword-driven testing, correlation and error handling. A controller module runs the scripts and can simulate large numbers of users, while the analysis module tracks and graphs behavior and performance results.
If you’re new to LoadRunner or need to pick up a few new techniques, you might refer to the March 2007 issue of Software Test & Performance for an excellent tutorial by consultant Francisco Sambade on using the tool to test multiple protocols.
Finalists in this category, products with the next highest number of votes, were Compuware File-AID/CS and Intel VTune Performance Analyzer.
File-AID is an enterprise data-management tool that permits testers to quickly build test data environments across a multitude of systems including mainframes, MVS, DB2 and distributed systems (as the finalist CS edition).
If you’re part of a team building applications that target Intel-based systems, VTune Performance Analyzer should be a standard part of your testing toolbox. The compiler- and language-independent tool presents a graphical interface for visualizing performance characteristics and identifying bottlenecks. VTune for Linux supports 32- and 64-bit applications, offers compiler-generated tuning advice, can tune systems with large non-uniform memory architectures and sports an Eclipse-based GUI as well as a powerful command-line interface. The Windows version, which supports Vista, Intel quad-core processors and 32- and 64-bit apps, integrates with Visual Studio, includes Intel’s Thread Profiler and can tune Linux systems remotely.
Functional Test
QuickTest Professional, the automation tool for functional and regression GUI testing that took last year’s Grand Prize, this year occupies the top spot for functional testers. The Windows-only product includes a scripting language built atop VBScript that permits procedures to be specified and program objects and controls to be manipulated.
Leading it to victory in 2006, Mercury had enhanced the tool’s team collaboration capabilities with a new object repository manager and the ability to share function libraries across tester workgroups. It also added keyword management and drag-and-drop test-step construction, XML output for test-result reporting, and a new, more accurate debugger that identifies errors while building or maintaining test cases.
Functional test finalists were Parasoft SOAtest and Compuware Optimal Quality Management.
SOAtest, a regression tester for Web and SOA services, integrates with Microsoft Visual Studio Team System. With the October release of version 5.5, the company added support for Windows Communication Foundation and other protocols, allowing testers of .NET-based applications to exercise messaging in a multitude of open and proprietary protocols.
SOAtest 5.5 also automates the creation of intelligent stubs, letting testers emulate the behavior of a running system to test services in the context of their actual behavior rather than on a live system. The suite also integrates with Visual Studio Team System for Software Testers, and receives test results directly within Microsoft’s environment. It also runs on Linux and Solaris.
Part of a larger suite addressing the entire QA life cycle is Compuware Optimal Quality Management, which monitors and controls performance of an organization’s quality processes. The tool extends the company’s Test Management test creation tool to include automated workflow, metrics, dashboard reporting and document templates. Quality Management connects test cases and defects to requirements while tying in metrics, giving test managers something against which to set objectives, gauge results and make decisions.
Defect/Issue Management
It’s nice to see a free tool built by a volunteer workforce at the top of any list of favorites. Tied with HP TestDirector for Quality Center at the top of the defect/issue management list was Bugzilla, competing in a category with commercial products developed by companies with infinitely more money and resources than the open source community from whence it comes.
Originally written in Tcl by Terry Weissman, Bugzilla began its life in 1998 as a replacement for the defect tracker used by Netscape for the Communicator suite (it surely must have been quite loaded). Thinking another language might get more traction with the community, Weissman decided to port it to Perl, resulting in Bugzilla 2.0. As the Wikipedia story goes, Weissman in April 2000 handed the project off to Tara Hernandez, who succeeded in gaining more participation from the development community. She handed it off to its current custodian, Dave Miller, and the rest is history. Bugzilla won our top spot in the free test/performance tools category.
The defect tracking module in TestDirector for Quality Center automatically checks its central defect database for similar defects each time a new defect is logged, helping to reduce duplicates. The tool tabulates defect statistics to aid management deployment decisions and identify trends.
With two winners for this category, we selected a single finalist: Seapine TestTrack Pro. Among its greatest attributes, users repeatedly say, is its ease-of-use, thanks to its intuitive and customizable GUI. The interface is ready to use out of the box, but also can be patterned after your own processes, terminology, industry nomenclature or regulations, culture and methodology. All artifacts can be tracked, including defects, feature and change requests and tasks, in an automated workflow system. It supports multiple layers of security, links with popular SCM systems, and includes search and customizable reporting. All APIs are open.
Commercial Test Performance
This year’s prime choice for sub-$500 commercial testing solutions was TechExcel’s DevTest, a Windows and Web-accessible tool that addresses most aspects of the testing life cycle. The test coverage module lets testers create, manage, analyze and reuse coverage data in a central repository.
Wizards help testers schedule and assign tests with reusable queries that identify the tests that need to be run. Workflow rules can be defined, along with triggers that can automatically notify or assign tasks. Defects can be automatically linked to tests or other tasks.
Finalists were Pragmatic’s Software Planner Professional and SOAPscope from Mindreef.
Available as SaaS or a self-hosted solution, Software Planner is a hierarchical project planner that, through task linking, can prevent one task from beginning before another is completed. Project managers can use the Web-based tool to set baseline dates and costs, and can track deviations as the project proceeds. Team members are alerted to new tasks via e-mail; multi-level security limits access to authorized people.
When Mindreef came onto the scene in 2002 with a tool for testing SOAP, few had even heard the term. But in 2006, the company’s SOAPscope 5.0 was awarded best solution from a new company. While hardly new at version 5.0, SOAPscope is a groundbreaking tool that strips away the complexities of SOAP messages to help developers and testers quickly identify the root cause of Web services problems. There’s also a team edition.
Static/Dynamic Code Analysis
Handed the top honors in the code analysis category was three-time winner PurifyPlus, a useful tool with long and storied roots now developed and marketed by IBM Rational. The automated runtime analysis tool for Linux, Unix and Windows spots memory leaks, profiles application performance and analyzes code coverage. Supported languages include C/C++, Java, Visual C++, Visual Basic and the .NET languages. Versions are available for Linux, Unix and Windows.
As Wikipedia tells it, “Purify” was created by Reed Hastings of Pure Software, which eventually merged with Atria Software. The resulting company, Pure Atria Software, was later acquired by Rational Software, which in 2002 was famously gobbled up by IBM for (at the time the staggering sum of) $2.1 billion. Purify, a runtime and memory management and error detection tool for Windows, is still sold today. It works with Java, Visual C++ and .NET languages, and because it doesn’t require access to source code, also can be used with third-party libraries.
Compuware’s DevPartner Studio and Parasoft’s Jtest were finalists in this category.
DevPartner includes a memory checking tool (BoundsChecker) and an application error simulator (Fault Simulator) to help testers root out bugs and other application shortcomings, and test and tune performance. The Compuware tool comes with a performance analysis and reporting module, and works with Visual Studio and Java.
Jtest is a static analysis tool that also can automatically build Java test cases by applying what the company characterizes as a set of testing best practices. The code review and regression testing tool for Java EE, SOA, Web and other Java applications is intended to help identify functionality, maintainability, performance, reliability and security issues.
Embedded/Mobile Test/Performance
IBM’s Rational Test RealTime was the favorite test tool for testers of embedded and mobile applications, a versatile cross-platform solution designed for testing and runtime analysis of embedded-system components.
Test RealTime automates the creation and deployment of host- and target-based test harnesses, test stubs and drivers, enabling Ada, C/C++ and Java applications to be tested directly on the target, the best place for accurate results. Also included is automated C-code review using predefined rules, memory, thread and performance profiling for C/C++ and Java, code coverage analysis for Ada, C/C++ and Java, runtime tracing and system testing. An Eclipse plugin allows Test RealTime to work in Eclipse’s C/C++ Development Tools project for Windows without leaving that environment.
Finalists in the category were the Eclipse Device Software Development Platform (DSDP) and Wind River Workbench.
First proposed in March 2005 by embedded giant Wind River Systems, the DSDP broke new ground for the company and for Eclipse. Long a proponent of its proprietary systems and tools, Wind River seemed to complete its embrace of open source development with its proposal to lead the project to build an Eclipse framework for embedded-systems testing and development. The company first turned the corner on open source in December 2003, when it abandoned its BSD Unix strategy in favor of Linux.
Development of the DSDP project has been a long road. Encompassing six subprojects—device debugging, embedded RCP, mobile tools for Java, native application builder, target management and tools for mobile Linux—the project has also required modifications to core functionality of Eclipse itself. But the project is now well along; TM is at version 2.0, eRCP at 1.1, and DD at 0.9.
Wind River took the other finalist spot with Workbench, its open framework for embedded systems development that stands alone or works as a plugin to Eclipse. The environment targets its proprietary VxWorks real-time operating system, Linux or systems that combine the two. It runs on Linux, Solaris and Windows, supports dozens of target architectures and multiple target connections with debugging for processes, tasks and threads. Workbench is sold in multiple configurations and price points, including those for developing platforms or applications, and with on-chip debugging capabilities.
Security Test
For security testing, SPI Dynamics took the top prize for WebInspect, its security scanning and assessment tool for Web applications. Now part of Hewlett-Packard, SPI in August released WebInspect 7.5, sporting a new profiler that scans Web apps and suggests configuration settings for the most effective testing.
Also new is a traffic monitor that reports in real time HTTP activity during a scan. A results window displays requests and responses sent by WebInspect during crawls and security audits. The tool was completely rewritten in January, when the company claimed performance improvements and compatibility with modern technologies and techniques. WebInspect 7.5 reportedly further improves auditing capabilities for spotting vulnerabilities in AJAX-based applications and better supports Windows Vista.
AppScan from Watchfire (now owned by IBM) and Hailstorm Enterprise ARC from Cenzic were finalists in the security test category.
AppScan is an automated security auditing tool available in single-user editions for developers, QA staff and auditors. Watchfire also offers multi-user modules for reporting and Web-based distribution about the enterprise, as well as SaaS versions.
Cenzic Hailstorm Enterprise ARC, which is short for application risk controller, is designed to do just that: scan Web applications and assess the risk involved in deploying them or leaving them deployed. Scan results are automatically prioritized—both individually and overall—and displayed dashboard-style in a Web browser with “eyes-only” role-based security. Report data are stored in a MySQL or Oracle database. New applications are detected and scanned automatically. Messaging is built in, enabling construction of custom workflows.
Test/QA Management
HP is again in the spotlight with TestDirector for Quality Center, which testers voted their favorite tool for test and QA management. TestDirector for Quality Center includes modules for requirements management, test planning, test lab and defect management.
The tool provides testers and managers with a browser-based window to requirements gathering, designing and scheduling manual and automated tests and analyzing the results. TestDirector for Quality Center includes graphical reporting and integrates with WinRunner and QuickTest Professional. HP offers extensions for SAP, SOA and Oracle.
Finalists in this category were Borland SilkCentral Test Manager and VMware Lab Manager.
SilkCentral Test Manager, acquired by Borland along with Segue Software in February 2006, is a browser-based environment for remote, simultaneous test execution and management of JUnit/NUnit and other third-party testing frameworks. A manual test client helps ensure repeatable data collection and reporting. The tool integrates with VMware Lab Manager to ease testing in virtualized environments. Reporting is provided by BIRT.
Slightly more than a year after its introduction in November 2006, VMware Lab Manager is already near the top of testers’ wish lists for its ability to streamline and help manage the testing process. Lab Manager allows virtual configurations—complete with networking, storage and other resources—to be stored and shared across development and test teams.
Lab Manager automates the capture, setup, storage and sharing of multi-machine software configurations. Saved systems can be deployed and running within minutes rather than hours or days. Regression testing is simplified by means of a shared library of complex, legacy configurations than can include operating systems, applications, third-party libraries and other drivers.
Free Test/Performance
Of the free testing and performance tools available today, testers chose Bugzilla, the open source defect-tracking system that celebrated version 3.0 in May, nine years after v2. Chief among the enhancements are huge performance gains, a Web services interface, and the ability to control and customize far more of the environment than in prior versions, according to project developers.
Faster performance comes by way of Apache’s mod_perl, which is now supported with back-end code that’s been refactored into Perl modules that interact with the database to deliver “extremely enhanced page-loading performance,” read the release notes. Bugzilla also will still run as a CGI application if performance isn’t critical, if system memory is an issue or for servers running something other than Apache.
Bugzilla now permits custom fields, custom bug resolutions and defaults (though fixed, duplicate and moved remain untouchable), and the ability to assign permissions on a per-product basis. Also new, developers and testers can now officially file and modify bugs via e-mail (previously unsupported) and set up default cc: lists to force certain addresses to always be added to lists for specific components. Administrators are now notified of Bugzilla updates when they log in.
A new globalwatchers parameter allows lists of addresses to receive all bug notifications generated by the system. All outbound e-mails are now controlled through templates, allowing them to be easily customized and localized as part of a language pack. A new mailform parameter offers control over which addresses show up in the from field on outbound messages.
Interface improvements include unchangeable fields that appear as such, warnings when a duplicate bug is about to be accidentally submitted (such as by going back in a browser or refreshing a page), and a navigation and search box at the top of each page in addition to the one that was always at the bottom. Also new are customizable skins (using CSS) and saved searches, which allow group members to subscribe to searches saved by others in that group. There are now QuickSearch plugins for Firefox 2.0 and IE7.
JUnit and Eclipse’s Test and Performance Tools Platform (TPTP) project were finalists.
Just as the term Band-aid is synonymous with bandage, JUnit has become synonymous with unit testing. Its success has exceeded the expectations of co-creator Kent Beck. As Wikipedia puts it, “Did you run your JUnits before you checked in?”
One of Eclipse’s earliest and most successful projects is TPTP, which has enjoyed tremendous participation from the community. At version 4.4.0.3 as of September, its subprojects include those for building tools for monitoring and logging performance and resource allocation of application servers, for building testing tools that edit, manage, deploy and execute tests, and for application trace and profiling.
Test Automation
For test automation, QuickTest Professional again comes to the fore, taking its second top award this year. The UI testing automation framework for Windows and Web-based applications works by identifying objects in the UI layer and applying mouse clicks, keyboard input and other test activities on them. Actions are recorded and captured as COM objects and stored as VBScript commands, which can be edited.
QTP is built with a pluggable architecture to allow scripting recording to be more conducive for the application under test. The tool includes plugins for ActiveX controls, Web applications and VB objects. Optional plugins includes those for .NET-language objects and multimedia. QTP also saves the AUT’s screens along with the recorded script, highlighting the portion of the screen being tested as an aid to regression testing and for creating checkpoints. Checkpoints, which can apply to images, tables and pages, permit verification of expected application behavior during automated tests. Other features include virtual objects, output value for data verification, data- and transaction-driven testing, and exception handling.
Finalists in the category were Borland’s SilkCentral Test Manager and CollabNet Cubit.
SilkCentral Test Manager becomes most powerful when used in conjunction with some of Borland’s other tools. Integration with CaliberRM (or a third-party requirements management system), for instance, enables a testing organization to adopt requirements-based testing, tying unit tests with requirements. The tool also includes native support for VMWare Lab Manager as well as Borland’s Gauntlet continuous integration tool, SilkTest automated regression and functional tester, and SilkPerformer performance and load tester.
Test automation today is about more than scripting end-user actions, and CollabNet’s innovative Cubit is a tool that that no testing organization should be without. It works by hosting applications under development (and test) and making them available via the Internet. Developers remotely control and test apps using RDP (on Windows), VNC or TCP/shell for Linux, Unix and Windows apps.
Code is stored in a Subversion repository, can be checked out and worked on locally using a developer’s own tools, and checked back in as usual. What’s different is that since code is hosted, builds and test runs are performed centrally, guaranteeing that everyone is executing their changes on the same hardware and operating system, and eliminating the predicament of having builds work on one system and not another.
An acronym for Centralized and Unified Build, Integration and Test environment, Cubit can stand alone or extend CollabNet’s Enterprise Edition collaboration platform for code design and development. The system allows for the rapid setup and teardown of centralized test beds in a variety of hardware platforms and operating systems—physical or virtual—including combinations of x86, x86-64 and Sparc on Red Hat, Solaris and Windows.
These configuration profiles also are stored in Subversion and can be copied, edited, versioned and diffed just like source code. When software is ready for deployment, an image of the host test bed can be sent along with the application. This simplifies and speeds up deployment to new machines, which can get applications and operating system installed in one shot.
For example, an enterprise might develop its application on Windows and Solaris, and when finished, request 1,000 deployment copies of each, which would be delivered as target-ready ISO images readable by management tools such as Tivoli and OpenView. Operating system licensing requirements are left up to the customer. Of course, apps also can be deployed without the operating system image.
Although similar on some levels to virtual lab automation tools such as Akimbi Slingshot, Cubit ties not just to the AUT and operating system, but also to the configuration of the test bed system, so testers are never in the dark when debugging an application long after the development systems have been wiped clean. Testers always know exactly what was used to build and test every version of every application, a boon to regression testing.
SCM/Build Management
Microsoft Visual SourceSafe was tops in the SCM/build management category. Originally developed by One Tree Software as SourceSafe, this tool was first released in the early 1990s as version 3.1, which a Wikipedia author speculates was meant to match the current version of Windows at the time. Microsoft’s SCM at the time was called Delta, and was inferior. Microsoft acquired the 16-bit SourceSafe and released a 32-bit version as Visual SourceSafe 4.0 around 1995, according to the site. It would be 10 years before the company released VSS 2005, the first client-server version; until then it was limited to local usage.
Among VSS’s strongest attributes is its tight integration with Visual Studio, relative ease of use and price; it’s free with certain subscriptions to MSDN. It also offers numerous expansion possibilities thanks to a pluggable architecture. Its future in the enterprise seems uncertain, however, since many of its capabilities are implemented in Visual Studio Team Foundation Server and several good rival systems are available from the open source community.
One such rival system is Subversion, which along with IBM Rational ClearCase was a finalist in the SCM/build category.
The Subversion project was launched in 2000 by CollabNet, which at the time was looking to replace the part of its SourceCast application-development hosting system that included CVS for version control. Subversion commits are atomic and don’t cause corruption if interrupted. All moves, adds, changes and deletions are versioned, including files, directories, trees, file metadata and symbolic links. It supports binary file storage, file locking for unmerged files and sends diffs in both directions.
Industry stalwart ClearCase also received high marks. First released for Unix in 1992 by Atria Software, the Purify developer that would ultimately be acquired by IBM, ClearCase continues to be popular among Unix and Windows developers and testers. Among its major features is its ability to store multiple replicas at different sites, build auditing, interoperability between repositories on Unix and Windows servers, and integration with numerous IBM and third-party products.
.NET Test/Performance
For testing performance of .NET applications, testers again chose LoadRunner. Support for .NET languages was added beginning with version 8. This release also added a new .NET diagnostics architecture including diagnostics server, commander, mediator, and probes for Java EE and .NET. The company also claims that LoadRunner 8 is more scalable and easier to install, configure and use.
LoadRunner employs a controller to launch scripts created by VUGen. The controller permits many hundreds of simulated users to be launched from one or a small number of machines with the LoadRunner agent installed. These machines are called load generators or injectors, and are instructed how and when to launch scripts using LoadRunner scenarios.
Microsoft Visual Studio Team System and Parasoft Test were finalists in this category.
Released piecemeal since 2005, Team System is composed of server- and client-side components. The primary server-side components are Team Foundation Server and SQL Server. All source code is stored on the server, which supports multiple simultaneous checkouts, branching, merging, conflict resolution and tracking of all changes. Security can be applied at any level.
Reporting is provided by SQL Server Reporting Services; canned reports include code change over time, list of bugs without test cases and regressions on previously passing tests, according to Wikipedia. There’s also a build server with tracking. On the client side, Visual Studio enables code analysis, code coverage and test tools to be brought to bear for build validation.
Parasoft Test is an automated static analysis, unit-testing and policy enforcement tool for .NET. With the June release of version 4.0, the company added the ability to find runtime defects by tracing and simulating execution paths that would otherwise elude manual tests or inspections. Also new was a code review module that performs automation preparation, tracks peer code reviews, and permits teams to define and manage distribution lists and groupings for code review notifications and routings.
SOA/Web Services Test
Here’s another category in which LoadRunner excels. Let’s say your Web service depends on public services provided by several major airlines. Part of your job is to make sure it doesn’t fail on, say, Thanksgiving, one of the travel industry’s busiest days. While you can’t control Jet Blue’s flight itinerary service, you can simulate thousands of hits against that service and watch how your application behaves while simultaneously trying to gain access to flight schedules.
That’s a scenario typical of those addressed by the venerable LoadRunner. Because once an activity has been defined, LoadRunner’s Controller module ensures that it’s consistently repeatable. This enables testers to identify issues and verify that patched code doesn’t repeat those issues.
Finalists in the SOA/Web Services category were Empirix e-TEST Suite and Performance Tester for SOA Quality from IBM Rational.
The Empirix suite consists of browser-based tools for helping to determine the quality, scalability and availability of Web services and Web-based applications. The e-Load module provides load testing and measures performance and scalability; e-Tester is a point-and-click environment for automated regression and functional testing; and the e-Manager Enterprise module provides test process management, requirements definition, manual and automated test execution and maintenance and defect tracking.
Testers also found favor with IBM Rational’s Performance Tester Extension for SOA Quality. The add-on to the company’s Performance Tester tool extends performance and scalability testing to SOA applications. It employs workload modeling to automatically generate Web-service test clients and performance tests, permitting testers to validate the ability of a SOA system to remain responsive as the number of operating system users grows.
The tool also monitors real-time server response and throughput times, visualizes server resource data to help locate performance bottlenecks, and supports Java code insertion for advanced data analysis and parsing.
Integrated Test/Performance Suite
Testers chose HP Performance Center as their favorite integrated test/performance suite. The tool combines all the capabilities of LoadRunner with test-asset and human-resource management features and reporting in a centralized repository accessible through a Web browser with role-based security. Load-test configurations and test scripts can be created, edited and scheduled for execution, progress tracked, graphed and compared with data from past projects or releases. Performance Center also can reboot and deploy patches to remote nodes.
Also highly rated were e-TEST Suite from Empirix and Compuware’s Optimal suite.
The Empirix suite combines requirements, tests and issues modules for .NET, Java EE, XML, Web services applications as well as Web-based CRM, ERP, PeopleSoft, Siebel and SAP systems.
With Optimal, Compuware merges modules for code quality, test management and quality assurance.
Java Test/Performance
Once again, LoadRunner is the testers’ choice. During load tests, the tool uses monitors to gauge the performance of the applications or components under test. Available monitors include those for virtual users, transaction rate, network latency, Web and database server responsiveness (these are server-specific) and for server resources.
Monitor data is saved for examination by the analysis tool, which processes and graphs the completed scenario results. By analyzing these graphs and combining them into reports, testers gain an understanding of the big performance picture and can make or suggest adjustments to tune performance.
Credited with high marks for Java testing and performance tuning were JUnit and Parasoft Jtest.
Often heralded as the grandfather of unit testing frameworks is JUnit, one of the earliest and most successful of its kind. Celebrating its tenth birthday in October, JUnit has spawned ports to numerous languages including C# (NUnit), C++ (CPPUnit), Fortran (fUnit), Perl (Test::Class and Test::Unit), PHP (PHPUnit) and Python (PyUnit). There’s even a version for JavaScript (JSUnit). JUnit has been downloaded more than 2 million times in 10 years, according to one study, and is included as a plugin to all major IDEs, including Eclipse.
Recent advances in JUnit 4.4 include a new API with an “extensible and readable syntax,” according to the release notes. The API enables features like assumptions or the declaration of explicit dependencies when a tester has no control of forces that might cause a test to fail. Assumptions also give rise to theories, which can capture some aspect of behavior in “possibly infinite numbers of potential scenarios,” according to the notes.
Parasoft’s philosophy is to identify and prevent bugs before they end up in code. Jtest attempts to accomplish this with Bug Detective, the component described above that finds runtime defects by tracing and simulating execution paths that would otherwise elude manual tests or inspections. Jtest also automates code review and checks it against more than 700 Java coding best practices along with an organization’s own custom coding rules.
Best Solution From a New Player
Fortify Defender was voted the best product from a company or organization less than five years old. The company was founded in 2004 with a mission to prevent vulnerabilities before they’re deployed by focusing on static source-code analysis based on CERT rules and those included in flagship Fortify SCA.
Fortify describes Defender as an “application-level intrusion prevention” tool for deployed Web applications. Call it trivial, but how can Defender prevent vulnerabilities if it works on applications that are already in use? Still, testers found favor with the tool introduced in 2006 as an “internal firewall” for Web applications. Defender was enhanced to support .NET languages in March.
The tool goes to work when pointed at an application executable. It creates a “fortified” version of the app that monitors and protects all API interactions and takes a variety of actions—from logging to blocking—when malicious activities are detected.
Finalists in this category were dynaTrace Diagnostics and Veracode SecurityReview.
dynaTrace Software GmbH was founded in 2005 in Linz, Austria. Diagnostics monitors Java and .NET applications for performance and stability issues—under load or in production—and helps to identify the root cause of problems. Among the company’s founders is Bernd Greifeneder, former CTO of Segue Software.
Veracode was founded in 2006, but the core technology in its SecurityReview security analysis service has been in development since 2002. SecurityReview is an automated hosted service that analyzes compiled code and presents vulnerability data to a Web browser. Security flaws detected by the SaaS include embedded malicious code, back-door issues and the absence of encryption in critical areas.
Load/Performance Test
LoadRunner again topped the list, this time in the load/performance test tool category, where it would seem most comfortable.
LoadRunner’s capabilities center around the Virtual User Generator (VUGen), a record-and-playback tool that generates an underlying script that can be edited as needed. User actions are routed through a protocol-dependent proxy during recording, which affects the resulting script and how it can be edited. For example, when generating scripts using the Web/HTTP edition, testers can set LoadRunner to generate either URL– or HTML-based scripting.
Finalists in this category were IBM Rational Performance Tester and Borland SilkPerformer.
Performance Tester is a load and performance testing tool for proving out Web application scalability. Built for the enterprise and integrated with IBM’s Tivoli management environment, it enables large, multi-user tests while using minimal hardware resources. Distributed controller agents are available for Linux, z/OS and Windows. Test results are displayed in a Linux or Windows application with high-level and detailed tree views and a tree-based text editor.
SilkPerformer is an automated, sharable environment for executing load and stress performance tests across Java and .NET applications with server-side analysis. The tool can simulate thousands of users with no licensing restrictions for controllers or protocols. A visual script recorder enables script editing and manipulation. An Eclipse plugin extends test creation to the popular environment. There’s also a SOA edition.
SIDEBAR: GRAND PRIZE WINNER: HEWLETT–PACKARD LOADRUNNER
Clearly the tester’s favorite tool is LoadRunner. In addition to getting the most votes overall, this versatile product received top honors for data testing/performance, load/performance testing, SOA/Web services testing, and .NET/test and Java/test performance categories.
LoadRunner examines application behavior and performance of running systems as it applies load to those systems. The tool is capable of simulating real-world scenarios with hundreds or thousands of simultaneous nodes banging away at an application, Web service, Web server, database or other program in Windows or Unix.
About the Author
Edward J Correia