Part 1. Mobile Test Automation Framework
By: Slav Kurochkin
In this series of topics I want try to find out what is the best practices and right way of developing test automation framework. I want to understand how to provide solution to the company and help to improve quality.
Purpose and problem identifying.
Before starting designing, developing and implementing framework, we need to clarify why we want test automation framework, how it is going to help us, what problems we can solve with framework, where to start, what we want to see in framework, what functionality we want and what is possible failures and escape plan.
First of all let me start by saying I'm QA Engineer who is passion about his job, I like to study and learn new things, and most complicated topic for me always was test automation framework. I have seen lots of different frameworks, data driven, keyword driven, hybrids, frameworks based on Page Objects design pattern, some people call it POM. I was part of the team developing hybrid framework, I'm developing frameworks myself, but all of those frameworks looks blur to me I never been satisfied with all I have seen and participated in.
Lets start from analysis why I was not satisfied and what went wrong:
1. Keyword driven framework - best explanation keyword driven framework not the one your company want, you can find in blog post "How to write keyword driven framework" written by Alan Richardson, if you ask my opinion most of them is to simple and depend on spreadsheet, there is obvious limitations with functionality's, for example still can't see how you would implement file downloader classes explained on lazerycode blog post. And last one in my opinion spreadsheet approach is quite difficult maintain in case different engineers working on test automation and submitting code to GIT.
2. Data driven framework - so obvious reason you have to provide data, and if your application dynamic or there is not that much data that you can provide, it is become quite difficult to provide all required variables.
3. Hybrid framework - most of the time you will see mix of keyword and data driven frameworks, which in some certain case solution, but then I think it is become messy and it is not easy to maintain them, so the person who developing that framework defiantly looking for some job security. Yeah company can hire people with developers background, but it will cost a bit more expansive and as far as I have seen developers have different mentality then QA folks, most of them time they suffer quality to deliver product in time, and developers thinking more from unit testing prospective which is quite different from regression functional testing.
4. Page Objects design pattern - nova days, well I can't say that much bad about it, the only reason some people may avoid I think is because it quite difficult to find right pattern and level of abstraction your tests should be.
Lets start by identifying problems and possible solutions. I will as an example mobile application and most of the time it will be social networks, since it is field where I'm working right now.
I remember reading article how folks from Instagram started company and what they where trying to achieve, the pattern they where using is "What is the five common problem people facing and how we can solve it".
I think right approach before even starting designing framework would be to answer that questions, so lets try to do it, and we have as given write test automation framework for Social Networking App:
- Manual testing is slow and it is rely on the human, so if human is tired, sick or bored there is big possibility that we going to miss bugs.
- Machine does not have humanity. We want to develop framework which will provide system analyses and compare expected with actual. For example compare response time with previous runs and give as a note if there is significant slow down. We also want machine to learn about the system and if amount of bugs for specific module increasing we want machine give as a note.
- We want generate reports which will give as clear vision what is going on failure. We want to know what happened, why it happened, when it happened, we want screenshot of the fail stage and if possible generate solution. Example: let say image failed to upload: we would system not just saying that it failed, but give as hint, provide as multiple reasons why it might happen. And if system can manage to exactly problem and cause system give as solution.
- We want our testing code to be nice and clean, so any engineer may have a look and make contribution to framework. It is quite important to have clean and clear testing code. We would like that our system run unit tests against framework, for that reason we need to mock objects and be fluent in unit tests.
- System need to be integrated with Continues Integration Tool and run independent from specific person, it should run automatically in any case (SDET on vacation or sick). We want to be able perform easy upgrade and use project management tool such as Maven.