He tests software, writes automation (java, selenium, python, ruby, take your pick) and yes, used to coach Lacrosse. Of course I’m talking about Adam Goucher, the Canadian Test Consultant and Contractor.
Beyond test contracting, Adam is probably best known as a writer. A co-editor of Beautiful Testing (2009, O’Reilly), Adam is a prolific blogger; he also tweets as @AdamGoucher.
If I had to pin his skills down, I’d put Adam in the school of context-driven testing (hey, that’s what he says on his about page on his website) but also with a focus on automation and enabling developers to create very complex test setups, rigs, harnesses, etc.
In fact, in my words (not his) I would say Adam is the guy people call when they want to do something tricky with test automation and get stuck.
Adam Goucher will now answer your questions.
Question
Lately our company has been purchasing applications that are client-based which I get asked to performance test. We use IBM’s Rational Performance Tester, which we use primarily for webbased applications and web services. I’ve also got access to Visual Studio Team System Test Edition, but the training material states you can only use one virtual user to drive GUI tests. Is driving traffic through the GUI’s something that is done routinely using a different set of tooling or is this a stumbling block for everyone? The idea of purchasing yet another expensive product won’t go over well so I’m curious what others are doing. – Michael Beckman
Goucher
There are currently at least two approaches to doing performance testing of applications through a browser that I use. Keeping in mind I am aligned with Selenium, they both involve it.
Approach one – death by a billion browsers: Using BrowserMob, I can unleash an army of browsers (controlled by Selenium) to assault the site. Since it is SaaS, I don’t need to setup or maintain the load generation machines and I only pay for what I need. It allows me to setup different load profiles to execute concurrently which lets me simulate the real (or expected) traffic patterns.
Approach two – you are the weakest link, good-bye: Using some tool, apply pressure on a key resource (network, cpu, memory) to simulate a constrained situation and then hit the server with a handful of requests. You can then see what goes POP by moving the constraint around from the load balancer, to the app tier, to the database, the caching server, etc.
Or combine both.
Notice how neither involve the build-out of your own massive infrastructure the way we might have had to do it ten years ago. Capital is too expensive and difficult to acquire today to waste it acquiring machines that just sit idle during the two months between performance runs (though the huge Sun machine we bought for it was pretty cool). There is still a monetary component, yes, but I would guess it is less than what you would spend on a tool for use in-house once you factor in annual maintenance contracts, hardware to run on, training, etc.
Full disclosure, I have relationship with BrowserMob as part of my consulting business.
Question
Selenium seems to be having an identity crisis. The Selenium IDE is the public face of the project, yet very few active users use it (or admit to using it). What’s the best way to get beyond the IDE, and help make the transition to using the real Selenium under the hood? Additionally, how can we help make that change occur with our testing friends? – Michael Larsen, San Franciso, CA
Goucher
Ya. We kinda got called out on that by Bret Pettichord at the Selenium Conference. And I don’t think we yet have a good answer to that. Part of it is education around what sort of things Se-IDE is good at (rapid bug reproduction scripts or driving the application to a known state to reduce startup time for exploratory testing are principle ones) and what the RC and WebDriver APIs allow for (data driving, integration with backend databases and web services, random input generators, sophisticated oracles, etc.).
But even that doesn’t really cover it all. There is an additional education burden to learn how to program (well enough to automate) and to make effective use of the various xunit frameworks that exist. And not everyone wants to learn how to do this. A motivated learner can learn enough Python to be effective with Selenium and Page Objects in 3 days of hard work.
The trick there is motivated.
And not everyone is.
The key to a person’s motivation is entirely unique to them, but you have to always be wary of not inflicting help upon them with automation. Some of the best testers I know would not list automation as their primary tool. And some of the worst I have seen do.
Circling back to the identity crisis, I suspect things are about to get a lot worse than better in the near term now that Selenium has commercial interests and their own marketing agendas. Not to mention what amounted to a declaration of war against HP at the conference. The ramifications of that could be interesting.
Question
Follow-Up Question – How can we help our testing friends make the switch from IDE to a more robust Selenium? – Michael Larsen, San Franciso, CA
Goucher
What we need is more turn key automation frameworks – along the same idea that things like QTP are turn key – that provide a structure around the automation, integrates the runner and are basically ready to go out of the box. Eventually, as the automation becomes more sophisticated then they will become hyper customized (see the ‘Lightsaber Heuristic’ below), but I have found that getting the initial bit of inertia is often the biggest challenge people face starting out with automation.
It is my hope that I’ll be able to make an announcement on something to this end in the next month or so.
Question
Hello Adam! At CAST 2009 you gave a talk, The Most Important Stakeholder Is You ( http://adam. goucher.ca/papers/ TheMostImportantStakeholder- You. pdf ). I’m curious, in what manner has the world changed since then? How has the testing community changed with respect to your statements? – Jeroen Rosink, The Netherlands
Goucher
The ideas in that paper are still just as relevant today; only now, instead of “just” Twitter we have “Twitter the juggernaut.” A lot of people, including myself, have reduced their blogging but instead discuss, spread and consume ideas and do a lot of our marketing through Twitter. One thing I didn’t mention in the paper is the value in having your name as your brand. Very few can pull off having something clever as their online persona and have it map back to them the way Elizabeth Hendrickson can with “test obsessed” for example – but I’m sure she has put in the requisite effort into making that happen. Unless you are going to make that effort, then just stick with being you.
But always remember, that like any social media, your past, current or even future clients (employers) could also be watching you.
Oh, and fair warning, I have been told I get chatty on Twitter when at airports.
Question
Adam, I have never worked with Selenium, although it looks interesting and I’d like to learn it. At STPCon, there were a group of testers sitting at a table next to mine who were talking about testing with Selenium. A heated point of discussion was whether Selenium would work for their environment or not. Without being any more rude than eavesdropping, which I was already doing, they peaked my interest with that. One fellow was adamant that there was no way that Selenium would help them. As I said, I am curious and have never worked with Selenium. Can you describe for me an environment or application that would be perfect for Selenium and one where Selenium would absolutely be of no value? – Pete Walen, Grand Rapids, MI
Goucher
Selenium does web content. That’s pretty much it. So if you have content in a browser then Selenium could be a perfect tool to interact with it.
There are exceptions. For example, if you are using Flash or Flex on your site then you will need to do extra work to allow Selenium to peer inside the black box that is the stage. But there are known solutions for that. Same for Silverlight or HTML 5’s Canvas tag (which was wonderfully demonstrated at the Selenium Conference earlier this month – certainly a video to watch once they are released). Java Applets are another black box, though I have heard of one team making a bridge so they can use Selenium with it as well.
Certain web frameworks are also a pain to deal with due to random id generation, buttons that are not buttons but styled javascript, etc., but there are techniques for working around that as well. (And they likely are a pain even in commercial tools as well.)
And with the introduction of the WebDriver API into the Selenium mix, other problems that used to plague Selenium like upload file dialogs are also less of a problem.
Not to mention its cross-platform, multi-language support that other tools cannot (or do not currently) provide.
Question
How would you best visually demonstrate the value of testing to a business person or someone who doesn’t understand testing? – Lanette “Dairy is Better than Non-dairy” Creamer, Seattle, WA
Goucher
Visually, I’m not sure. But as someone who has been hired a number of times to start testing groups from scratch, I take two approaches to conversations like this. (Both of which I got from Michael Bolton who in turn got them from Jerry Weinberg I believe.)
First, I need to understand why we are having this conversation in the first place. Why is it that now, after they have been doing things a certain way for a certain length of time, do they think they need to have testing. Understanding what event(s) led up to the conversation is important. With that information you now know the existence of a stakeholder, who they are and their motivations. Everything (including testing) is, after all, a people problem.
Second, I try to understand, or at least get them to explain what would happen without testing. Perhaps testing in their circumstance is exactly the wrong thing to do and is testing-for-the-sake-of-testing rather than an activity that has beneficial results. Perhaps spending a week on testing pushes their release date beyond an important industry event. Immediately after it being a people problem, it is a money problem.
As a consultant, I find a bigger challenge in a lot of organizations is When To Stop Testing rather than whether or not to do testing at all.
The Grand Rapids Airport Parking Lot Calculator is a fine example of this. Yes, there are massive, and fun, bugs in it – but it does work when used in the context it was designed for. More testing wouldn’t add any further value to either the airport or people wanting to calculate parking charges, but would result in more bugs being found.
Question
What’s the all-time weirdest question you’ve ever been asked about Selenium? – Catherine Powell, Bahston, MA
Goucher
I’m not sure that I’ve had an all-time weirdest question; or at least nothing that pops out. But at least once a month I see someone trying to automate either GMail or Hotmail or similar which always makes me shake my head in disbelief. Unless you are Google (or Microsoft respectively) you should not be trying to automate those things. There is always a better solution to what you are trying to do. Usually it involves reaching into the database to get the authentication token that was sent to the user’s GMail account and faking out the browser into thinking the user clicked the message – but sometimes it is just easier not to automate that particular task.
On the cool side of things instead, Chris McMahon won an iPad for crawling Craigslist and I have heard of people checking into their flights via Selenium as well. I also have got into the habit of very carefully reading the rules of online contests for entries via automated means and entry limits by email and/or household.
Question
Imagine that you have a newbie tester who dreams to become the automation tester one day, what would be your advice for him and what steps should he take ? – Aleksander Lipski, Poznan, Poland
Goucher
I’m pretty sure David Burns would have issue with someone else wanting that title, but
Step one: learn to test
Step two: learn to program
I think it actually should be in that order as well. One thing I am learning is that being a tester-that-automates is very different than a developer-that-automates. And there are too many of the latter and not enough of us as the former.
Automation is all about looking for the oracle and figuring out what switches and levers need to be pulled in order to meet your mission. It is not just making things flash across the screen at super-human speed. But it is easy to not know that if you don’t come from a testing background or have not become someone who identifies more as a tester than a developer.
As for what language to learn, since that is often the follow-up to this, I would say either Python or Ruby.
Question (Follow Up)
What is your opinion regarding Page Object Pattern. Should we in general create automation scripts with this pattern or do you have another approach for dry and bulletproof automation ? – Aleksander Lipski, Poznan, Poland
Goucher
To quote from a slide of mine: Page Objects FTW!
To me, the big benefit for the (Selenium) community so far has been the popularization of the Page Object Pattern which, for those not indoctrinated yet, is basically to represent pages under test as either a single class or group of classes. This brings the full power of OO to your automation framework. Done well, it leads to scripts that are extremely readable and completely removed from their implementation allowing for quick updates when locators change or even replacement of the underlying driver technology should the mood strike – without the script changing.
Yes, it is possible to write bulletproof and dry automation without it, but the separation between interface and implementation is a lot harder.
When I teach Selenium, I just start with Page Objects these days skipping the traditional way of writing scripts entirely.
Question
You’ve written recently that the next leap in the testing field will come from another discipline. Apart from something that might help any technical worker in general, such as advancements in learning or introspective techniques, do you think that a technological or purely testing-related methodology will manifest itself? If so, will it be related to automation (i.e. the next Selenium) or will it be a new school of testing, or of software development? And, of course, Why? – Alex Kell, Atlanta, GA
Goucher
Don’t suppose I can just say ‘No’, can I?
I still think that we’re pretty tapped out from looking inward. The best books, podcasts and conversation I have had on testing all had nothing to do with testing, at least directly. The ideas popularized in Malcolm Gladwell’s Blink for instance go nicely with (my ideas on) exploratory testing for example. Or Sian Beilock’s ‘Choke’ on how to deal with pressure (like, oh, say a pending product release). Or closer to the technology world, things like Continuous Delivery – which came out of more the IT Ops field than it did testing. And Agile before it came from the development side of the building.
I would be much happier if more testing conferences and magazines would have non-testers as speakers or authors. Their contribution is what will grow the field and the practice more than just the usual suspects.
Question
As a mostly commercial tools shop, we look at the open source tools frequently so see what all the hype is about. We have a very stable and solid framework using BPT, or Business Process Testing – (approaching 2 years and 400 tests with a very manageable maintenance load). We can never see how the open source tools can even begin to cover all the bases that our BPT framework does on complex applications. All of our searches for published success stories detail extremely simple systems. How do you implement these tools on more complex systems? Have you seen them succeed? – Jason, Pennsylvania, USA
Goucher
First, if you have a very stable and solid framework already then you would be silly for changing it just for the sake of change. Open source or otherwise.
Secondly, and this is where I become open source anarchist for a bit, what you end up with when you go the open source route is a solution that actually meets your needs rather than needs that meet your solution. For simple applications, the notion of cost can also be a factor, but in large organizations and complex systems even free solutions have a cost.
HP seems to have monopolized all search results for BPT so I’m guessing this is what you are talking about. From a cursory glance at their product page, it looks like it is in the same space as Cucumber or Robot Framework – only wrapped in a pretty GUI rather than being text. (Though depending on how the assets are stored, the GUI could actually be a hindrance.) But I’ve not used HP-BPT before, or even heard of it until now, so my interpretation could be wrong.
In any event, any tool that promises non-programming automation is peddling snake-oil including the open source ones. Automation is programming. Sorry. But that is just the way it is.
Finally, building your own automation environment is something I call the ‘Lightsaber Heuristic’. At one point, it was Star Wars canon that a Jedi needed to go on the Epic Quest to find the components to build their lightsaber. It after all was their weapon and tool that would one day save their life which meant they needed to understand it completely and ‘be at one’ with it. Same rules apply to automation – though hopefully it is not in a role where it can have your life in the balance. COTS tools are like getting your lightsaber from a vending machine in the mall; you don’t know how they are built, you don’t have access to spare parts, and even if you did have access to them, taking the back cover off voids the warranty.
Question
What can Selenium do for the iOS automation? How far can we take iOS (next up android) testing? – Jane Fraser, Bay Area, California
Goucher
Selenium 2, which has both the Remote Control and WebDriver APIs can do automation of IOS applications to some extent via WebDriver. It cannot drive all the native controls, which makes sense since Selenium is a browser automation library after all, but it can drive ones that use UIWebView (a WebKit based browser) both in the emulator and on an attached device. I suspect the information on the Selenium wiki might be a bit out of date, but I know it has been discussed on the webdriver mailing list a number of times.
Question
How did you manage to get all the chapters for Beautiful Testing? It must have been like herding cats! What is the secret to getting multiple contributors to step up to the plate on a book such as that? – Lisa Crispin, Denver, Colorado
Goucher
Lots, and lots of email. Unbelievable amounts of email actually. I did the total once and I think it was close to 2000 individual messages from the time we pitched the idea to O’Reilly to the time it was published.
In terms of getting contributors, it actually wasn’t as hard as you might think and I suspect it was due largely by its association with Nothing But Nets. Contributors, and editors, agreed to donate all revenues to Nothing But Nets to purchase mosquito nets for Malaria infested regions. Every 45 seconds a child dies of Malaria, so donating time to write an article was almost a humanitarian effort. There were a few people that were approached that I’m disappointed didn’t participate, but after a while you understand both how busy people can be, and the politics of the testing world. My next book will be just me though. 🙂
Question
Mercury Interactive is a former company that used to sell test tools. Is there any truth in the story that the Selenium name was chosen because Selenium is an antidote to Mercury poisoning? – Shmuel Gershon, Isreal
Goucher
Yes, though it was a bit of an indirect route to get there. Jason Huggins, creator of Selenium, told the full story at his Selenium Conference keynote. If I recall correctly, his original name was to be CheckEngine to continue with Thoughtworks’ car theme (they had CruiseControl already) but a conversation here and there led him down the road (another car metaphor!) to Selenium. His keynote was recorded so at some point in the future you’ll be able to get the full story.
Question
Which misconception about test automation is/are the most popular? Are there you have to correct or explain over and over again? – Shmuel Gershon, Isreal
Goucher
The biggest one around Selenium in general is that somehow Selenium is going to magically determine that your site is wonderful. Selenium is simply a library for controlling a browser, nothing more. The ability to data drive scripts, do clever modeling of workflows, assertions, etc. all come from the Runner (JUnit, RSpec, PHPUnit, Test::WWW::Selenium, etc.) not Selenium itself. Once that is explained, it both expands the world of possibilities that opens up since people realize they get everything they had in the language before – as well as the ability to control browser, and also limits their expectations as they realize that Selenium (and automation) is not the Silver Bullet they might have thought it was.
In automation in general, one conversation I tend to have a lot is the whole metrics/ROI question which to me lies in the land of rainbows, lollipops and unicorns. The real value in automation, to me, is the ability to do more focused, high-value manual exploratory testing sooner in the process than if you didn’t have the automation in place. How do you measure that?
Similarly, how do you measure the more complete model of the application you are testing? Writing sophisticated scripts means delving into the inner workings of the system to understand the data flows, to discover your true oracles, etc. One cannot help but learn more about the system. In my experience, the true value of automation (using the overly simplistic and often misleading metric of bugs found) is actually during the creation phase while you are digging around in this manner. Unless you have sloppy development practices with lots of regressions, then your automation should never find another bug once it is put into the main script pool.
About the Author
Adam Goucher Adam Goucher has been testing professionally for over 12 years at a range of organizations from national banks to startups. A large part of that time has been spent augmenting his exploratory testing with automation. After discovering Selenium during a capital budget freeze in 2006, it quickly became his tool of choice when automating web interfaces. As an active member of both the core development team and the larger Selenium community he has recently started to revitalize the IDE project in an effort to kickstart business adoption of it. Automation is not the only tool in his toolbox. As an experienced tester who believes in the principles of both the Context School of Testing and the Agile Manifesto he brings an open and modern view of testing to projects. His first book, Beautiful Testing was published in October 2009.