Recently we had a chance to work with several software testing teams to help them get started with their test automation endeavors. The teams consist of mostly manual testers, and some who had experience with test automation before, but none of them were familiar with programming like developers. The applications are mostly web based, with companion mobile apps. We made a short list of several testing tools, including free and non-free ones, and then came up with the two best ones – Robot Framework andKatalon Studio. They are quite good choices in general, so we think it would be interesting to make a side by side comparison and see how they will perform in the future.
Technology
Both are built on top of Selenium — the automation framework that is currently backed by top web browser developers (Google, Microsoft, Apple, Mozilla). It is safe to say Selenium is the best way to automate your web browser for now. Selenium can also be used to automate mobile apps via Appium. Granted that Google and Apple both come up with their own way to do automation, using Selenium and Appium means that we can reuse source code and knowledge in multiple platforms — learn once, write everywhere.
So for technology, it’s a tie. Robot Framework – Katalon Studio : 1 – 1
Price
Both are free with no obligation. Running outdated “enterprise” software like UFT is just cost-prohibitive for us.
Robot Framework comes with two separate components: the test runner and the IDE. It is a little complicated to install these components, but once things are done, you are good to go. First, you need to install Python on your computer. Then run some command lines. If you use Windows, you might encounter issues because this OS is not Python’s first-class citizen, but they rarely happen, and solutions can be googled easily. There is actually a third “unofficial” component, which we will mention later. For Katalon Studio, from our opinion, this is one of the best installation experience. If you have installed Eclipse, then you would know it. For Windows, you will have to download a ZIP file and extract. For macOS, there is a DMG file ready for you. Pretty neat.
Robot Framework – Katalon Studio : 0.5 – 1
Test Design
Recording (codeless test automation)
Honestly, as a programmer, we am not a big fan of record and playback. It feels easier and quicker for us to just type the code. However, for those who have little experience with coding, the ability to record their actions and generate test cases is a huge productivity win. Unfortunately, Robot Framework does not provide a recorder — and this is where the third component shows up. The Selenium project includes an infamous Firefox add-on called Selenium IDE, which is essentially a test record & playback tool. Selenium IDE can have add-ons (it is add-on on add-on!), and someone had kindly written an add-on that helps Selenium IDE generate Robot Framework code. Some more installation works to be done, but it works. There are different types of Selenium Webdriver Exeptions that we face while working with Selenium WebDriver.
The downside of this is that Selenium IDE development has been inactive for a while, and it is based on Firefox’s old add-on framework, which they have decided to unplug at the end of 2017. Yes, Selenium IDE will stop working at that time for newer Firefox versions. Katalon Studio comes with its own add-ons for Chrome, Firefox, and even Internet Explorer. Installation is as easy as going to corresponding web browser add-on stores and click install. Katalon Studio extends its lead here.
Robot Framework – Katalon Studio : 0.5 – 1
Coding (Scripting)
Robot Framework has its own domain specific language (DSL). Custom keywords can be written in Python and Java, but when it comes to test cases content, you have to speak that DSL. Why it mimics human languages for simple test cases, complex ones are quite robotic. I personally find it less descriptive than common programming languages:
*** Settings *** Library String *** Test Cases ***
For-Loop-In-Range
: FOR ${INDEX} IN RANGE 1 3 Log ${INDEX} ${RANDOM_STRING}= Generate Random String ${INDEX} Log ${RANDOM_STRING} For-Loop-Elements @{ITEMS} Create List Star Trek Star Wars Perry Rhodan :FOR ${ELEMENT} IN @{ITEMS} Log ${ELEMENT} ${ELEMENT} Replace String ${ELEMENT} ${SPACE} ${EMPTY} Log ${ELEMENT} For-Loop-Exiting @{ITEMS} Create List Good Element 1 Break On Me Good Element 2 :FOR ${ELEMENT} IN @{ITEMS} Log ${ELEMENT} Run Keyword If ‘${ELEMENT}’ == ‘Break On Me’ Exit For Loop
Log Do more actions here …
Repeat-Action
Repeat Keyword 2 Log Repeating this …
But the Robot Framework IDE with syntax highlighting is good, so I guess I can get used to it quickly anyway.
Katalon Studio, on the other hand, supports Groovy and of course Java (Groovy is based on Java). Why it is not as popular as mainstream programming languages such as Java, C#, or JavaScript – it bears similar productive structures as in Ruby, Python, or JavaScript. Another good thing is that there are tons of free Java libraries to do just everything, so we did not have to rewrite “everything”.
But the coolest thing in Katalon Studio is that we can switch between Manual mode (with Recorder and point-and-click GUI) and Script mode (with Groovy editor) at any time. We took advantage of this to educate manual testers to write code. They at first design their test cases by point-and-click actions and elements, then switch to Script mode to see the equivalent Groovy code. Very nice feature.
Robot Framework – Katalon Studio : 0.5 – 1
Test Data
Both are good. Data for test cases can be fed from Excel files, databases etc. We got what we need, so nothing to complain by now.
Robot Framework – Katalon Studio : 1 – 1
Keyword extensibility
You can write custom keywords in both Robot Framework and Katalon Studio. They required some repeatedly ceremonial syntax, but testers can write their own keywords without the help from programmers. The documentation and tutorials are good enough, so we will leave this as a tie.
Robot Framework – Katalon Studio : 1 – 1
Execution, Reporting, and Integration
We expect modern test automation tools to be able to execute test cases via command line and in CI like Jenkins. We also need to have reports in-app to make it easy to investigate failures. The reports should also be prepared in archivable format (HTML, CSV, PDF) so that they can be looked up later.
With regards to using Comand Line Infterface (CLI), The TestQuality Command Line Interface allows you to upload your automated test results from Selenium to TestQuality. Automated test results must be output in JUnit XMLformat, which most test automation tools will provide. Test result attachments and related defects are also supported through test name tags or console outputs.
It seems that Katalon Studio is experimenting with a more native JIRA integration, which sounds good to me since we also use JIRA, but let me see how well it will work out. One thing missing in both is an image to cut down the time for CI deployment and configuration, but maybe it is just my laziness. Another tie for us.
Robot Framework – Katalon Studio : 1 – 1
Community and support
Both products are in active development and are well supported by their owners as well as the community. The questions are answered in very short time. Moreover, since Robot Framework and Katalon Studio are Selenium-based, we can pretty much resort to vanilla Selenium and WebDriver for complex scenarios.
It seems that Katalon Studio is built by a company doing software testing services, which means they know what features are practical and necessary. Robot Framework is a popular open source project on GitHub so I expect the same thing. The last tie here.
Robot Framework – Katalon Studio : 1 – 1
Conclusion
Robot Framework: 7.5/9 – Katalon Studio: 9/9
Both tools are great choices for test automation teams, with Katalon Studio has an upper hand on user experience and therefore is better than Robot Framework for tester-based test automation team. Competition is always good for consumers, and I look forward to seeing more advancements from the players in the future.