How to Safely Build Automation Using OSS

Building automated testing is always a delicate balancing act, with the best course of action heavily reliant on the type of testing, scale of the business and a host of other factors to boot. However, the choice between OSS and proprietary tools to safely build automated testing is a whole question in itself – we take a look at the key pros and cons to help you make the right choice…

The first and perhaps the biggest question is how easy is it to adopt the tools into your environment? Does the language, style, framework fit with your team needs and is it fully compatible with the company process requirements? There’s little point adopting a .Net architecture for a very JavaScript focused team, for example, but there are plenty of additional details here to catch out the unwary.

For example, if a business is testing web properties that use the Document Object Model (DOM) and the Shadow DOM, then the otherwise excellent choice of Selenium might cause issues, as Selenium struggles to ‘see’ the shadow DOM, which could cause considerable confusion when testing. Through limitations and ‘gotchas’ like this you might initially pick a tool that seems to fit but discover that it lacks a key capability once you have invested time and effort with your initial implementation. The exact functionality of the toolset is a bit of a hidden minefield. While OSS offers considerable flexibility, proprietary toolsets can also feature very strong integrations, so from a casual glance, it can be hard to pick a winner on functionality alone.

Cost of Free

Price is often seen as a consideration at this stage, with the ‘free’ price tag on OSS looking like a no-brainer, but there are hidden costs to ‘free’. Top of this list of hidden costs is creating bespoke integrations in-house, often seen as an alternative to paying a third party. Of course, this is a complete misnomer – not only will it take in-house resources time to build the project in question (thus using up expensive effort), but once built, tested and deployed the code will need to be maintained. The latter can easily snowball very quickly, taking a relatively simple workaround and creating a complex monster that becomes unstable and hoovers up even more resources.

Potential Plugin Pitfalls

OSS in particular often requires the usage of plugins to deliver particular functionality, Jenkins being a particularly good example here. These often seem like a great idea, but this is potentially an area of risk for a variety of reasons. Plugins come with their own sets of vulnerabilities, which in turn either mean they become outdated, opening an organization up to well-known exploits and the corresponding risk, or require the organization to patch them individually, potentially generating a mass of out-of-band patches to deploy randomly.

Plugin updates can also break other functionality, so the updates need to be tested before deployment. This prospect can easily create the situation that different teams within an organization are running different versions of various plugins, based on the functionality they need, an obviously undesirable situation. In contrast, while a proprietary toolset will also potentially suffer from the same problems, there should, in theory, be a more regulated patching routine that fixes multiple bugs and does not impact functionality.

Building in Vulnerabilities

Another aspect of this is that not all OSS plugin developers are equal – some may have been highly active in vulnerability testing and vetting their sources, libraries, and own dependencies. Some will not have been so diligent, potentially embedding common vulnerabilities into the organization. The fact is that security applies to your tools as well as your products, so it is certainly worth testing proprietary and OSS toolsets for vulnerabilities as part of the selection process.

OSS Licensing

Additionally, it is worth bearing in mind the vagaries of OSS licencing – there are 39 different types of OSS licence with various permutations and differences – some of which will specify ‘no commercial use’. While the chances of being invoiced over commercial use of a plugin are very small indeed, the fact is that there is a risk here and a potential cost if discovered as well as the ethical considerations.

This is not to say that OSS tools are not the right choice for testing – far from it – but the key is to take a long view on the testing architecture, rather than dashing in and just getting started with the obvious choices for today’s workload. By choosing the tools that will standardise across teams, and offer the most effective and efficient processes for the future, a testing function can achieve considerable savings just down the road. Think long, think security, and the right answer will be much clearer – whether OSS or proprietary!