How is AI impacting Testing Automation?
Artificial Intelligence (AI) is a sought-after technology today as it performs complicated tasks faster than a human being and significantly reduces human errors. AI is penetrating all types of businesses and sectors like e-commerce, healthcare, and application development. The adoption of AI in the Quality Assurance (QA) sphere is increasing too. Using various AI techniques, we can automatically test changes in the UI and analyze reports to streamline testing and save time. Thus, using AI helps in faster and more accurate testing of the software developed and speeds up its deployment.
In automation testing, AI is widely used to analyze the UI (web pages, native), recognize elements as manual testers do, and finally, check whether they are present at the appropriate places or not. In the near future, AI will enable testers using the testing tools, created with the help of Computer Vision(CV) and Deep Learning(DL), to perform better quality testing.
Advantages of AI in Testing Automation
Here are a few advantages of using AI in testing –
• Improved Accuracy –Automated testing powered by AI will help in performing repeated tasks accurately. Manual testers will no longer need to perform monotonous tasks. The number of human errors will also decrease.
• Overcoming the limitation of people – With AI, we can simulate hundreds or thousands of virtual sets of users with different personas that can interact with a network, software, or web-based applications.
• Increase in overall test coverage – With automated testing, one can increase the overall depth and scope of tests resulting in an improvement of software quality.
• Faster time to market – With the change in source code every time, manual testing becomes time-consuming. But with automation using AI, we can do it at zero additional cost and at a much faster pace.
All types of testing can be enabled to produce a better outcome using AI. However, in this blog, we focus on only one type of testing using AI –UI Testing.
Zensar’s Solution for UI Testing
We, at Zensar, have built an AI-based accelerator called ScreenAssure to automate the UI testing of web and mobile-based applications with the least human intervention. The focus of the accelerator is to:
(a) detect the position of the required buttons/icons loaded on-screen.
(b) to extract the information being rendered on the screen after each operation on the UI and verify its correctness.
We use customized CNN (Convolutional Neural Networks) based object detection for detecting the assets and their positions. We also use our in-built capabilities in OCR for extracting characters from the user interface and perform the required analysis using our NLP accelerators. Customized OCR based text extraction techniques are used to understand the data being presented on screen and verify its correctness. The accelerator comes with plug and play modules (adapters) to configure the parameters and define methods for both asset detection and information verification.
Online Games Use Case
Using this accelerator, we addressed a potential problem that we came across in online games. The problem was to assess whether all the assets have been loaded properly on the UI and also to check whether the UI is displaying proper information about the credit and debit of points from the user account.
We solved the problems mentioned above by creating a tool named GameAssure using our accelerator. The process flow can be briefly described in the steps below:
• Locate the position of the start/spin button so that we can use them to start the game. We also locate the credit history mentioned on the screen and keep track of it. Now, we trigger those actions using the activator module.
• Next, we locate the location of different elements like credit/win/bet in a gambling game and extract the value to verify if the values or text are correct after each round of the game.
• The same solution is deployed for all devices independently, with very little effort required for each configuration.
GameAssure can be used on any gaming website with little customization. It comes with a customizable architecture and is easily integrable using restful APIs.
Some automation issues cannot be resolved using a conventional automation testing toolkit. This is where computer vision can play an important role. The deep learning-based methods used in computer vision enable higher accuracy and low error rates. To create ScreenAssure and GameAssure we have used the TensorFlow, OpenCV and Tesseract OCR Tools, which are open-source softwares.
We collected a dataset of images on different devices with varied resolutions/sizes and labeled them using a labeling tool (one-time process). We used it for training the object detection algorithm (deep-learning method) using CNN, to enable it to identify the location and type of the element. We used OCR based information extraction to validate the operations (debit/credit). We have performed the testing and analysis of four games and were able to achieve 91% accuracy. We also found that there were no false positives. So, the 9% error we observed was from false negatives. These were manually verified by the team.
AI is already altering how testing is done. In the future, AI-enabled tools will augment every aspect of testing (functional and non-functional testing). Scaling test automation using AI will enable a faster turnaround time for software development and deployment. Hours of manual mechanical effort will be reduced to a few minutes. The testing will evolve into a completely cognitive and creative activity as manual work is taken up by AI. In this direction, we need to be upfront and start taking steps towards creating AI-powered tools and utilize them for the creation of better softwares. Zensar’s UI testing accelerator – ScreenAssure and Game UI testing tool – GameAssure, are a couple of such steps in this direction.