Learning The Automatic Testing System of Nuomi

This paper introduces the automatic test scheme based on Baidu glutinous rice O2O system mobile technology framework, which is a breakthrough of the traditional mobile field test program, which consists of two parts: component automation testing and terminal monitoring and alarming.

background

Mobile App has evolved from the early Native architecture to the Hybrid framework, to the current component architecture. The development of the technology is constantly innovating, and the framework for testing automation is endless, including Appium (iOS and Android), Robotium, Calabash and EarlGrey IOS open source automation framework) and so on.

These tools have been instrumental in testing many products, but the development of testing techniques has lagged behind the changes in App development technology. The test automation framework always feels inadequate to meet the testing needs of existing products, such as components or React Native such architecture system products, the previously described those who test open source framework is somewhat inadequate.

To this end, after the road in the glutinous rice components change (component concept, please refer to the “movement of the structure of glutinous rice moving components of the road” is not described here), in the test automation technology has a new qualitative change above.

Component Automation Test Scheme

Early automation tools and the drawbacks of the program

The mobile App component team began to choose to use automated tools or frameworks to address the heavy task of manual testing, the lack of offline regression, and low test coverage. But gradually found that automated tools to some extent and can not be a good way to reduce the cost of manual testing, but distress in the ease of automation tools, increased learning costs, and run the success rate is not satisfactory. Some departments will automatically run the results of test cases as the last line before the line of defense, which we can think of the final data is true, whether the tester can really benefit from these so-called test tools.

From the current more popular test tools to do an analysis:

Based on the above tools, many companies will establish a series of cloud testing technology framework, but also with a lot of automation to meet the special test items, including the provision of test real machine. But for the component-based architecture of the app, and the development of these technologies are being applied more widely, based on the above program can not meet the needs of the product function iteration.

Components of the identification of components - components loaded into the iOS and Android platform, the code above is the same, if the appium can meet the current test requirements, but the recognition component page control properties is the only flaw, and at the same time identified on both ends The attribute elements that can be used are completely different, increasing maintenance costs for testers and increasing instability for running automated use cases.

Long-distance access to the stability of the component entrance - specific components of the page often need to log on, select the necessary steps to reach the components needed to test the page, there is the test focus, but the lengthy test steps to increase the automation of the Stability, halfway may be interrupted. If you can skip some unnecessary steps directly into the entrance, so you can avoid such problems.

Cross-platform stability issues - cross-platform stability has been a lot of companies worried about a problem, how to ensure seamless interaction between the client and the server, the service api is a long time to call and run can not request the failure, Are the shortcomings of existing tools.
Learning and maintenance costs - automation platform or tools If the environment configuration, the use of norms, such as the more complex requirements of the script, then the entry just for the test engineers have some learning costs.
Automatic Framework Solution for Glutinous Rice Components

In response to these questions, Baidu glutinous rice NA team to provide the following solutions:

1) The realization of the elements of recognition Google Chrome components through the structure of the page crawling, through the element attributes, or write support for xpath, jQuery selector expression. Elements of the positioning results for both iOS and Android at the same time.

2) For the test requirements of the page, and page elements, if the page path is longer, more steps before the entrance to the page will increase the instability of the case, glutinous rice through the Schema to achieve direct access jump straight Theme of the test program, shorten the distance, improve stability, for other products through the Schema Jump page can also refer to such programs.

For example, the following test is concerned about the test function within the cinema, to achieve the entrance of the intermediate links will often lead to instability of the case, through the existing program can be avoided.

3) The mobile phone installs the App to pull the cases of the server, disconnect the physical connection, so for the special instability or for special models of the needs, such as the latest iPhone series or the latest Android phone, the team can Run, fill the platform under the special circumstances of the resource gap.

4) a single case, to achieve a single business test requirements, the test suite is to achieve mobile phone test on the daily operation of the program, a number of single-scene test requirements into a unified complex test scenarios, component team, self-trigger daily run time segment.

5) In order to meet the fast iteration needs of the business, through the platform configuration, no special language skills, business test engineers only need to configure specific business scenarios to achieve script stable dialy operation, zero cost of learning and training investment.

To achieve the above features, through the client installation automation App, server configuration script and set the scene, you can use the automation to the extreme, the following is the Android mobile phone wireless automation framework:

Mobile terminal to install the automation App, HTTP request part of the back-end platform for the ability to support, including configuration information, script information, case operation report statistics on the platform to create, update and summary, to be prepared, the mobile terminal input or daily operation A designated test set ID number can be run in real time, resulting in the case of functional regression, including automation, as well as other special test performance indicators and other information.

UIAutomator framework based on the realization of the relevant packaging interface, the case of the various actions to achieve analysis, call the package to run the api page ui operation, the failure of the case steps to achieve the screenshot upload to determine the current failure of the specific circumstances. Because the support wireless connection, continues the job to trigger, therefore the parallel running ability also may satisfy.

Component-side monitoring of active inspection

Components from the App to the next release to the rendering, the whole process of each link will have all kinds of unexpected problems. On-line ANR, Crash, response time, etc. can be implanted through the App embedding program, or access to third-party sdk, the program can be used to monitor the real-time monitoring, Back-end platform to provide alarm information, real-time monitoring; but the complete rendering of the component page monitoring has been unable to achieve full coverage, such as back-end services back to the front normally, but the phone in a specific case the positioning function analysis problems led to the page has been The api initialization interface on the specific App end does not have the ready, the module does not carry on the judgment directly to call the internal method to cause the page to appear unusual and so on is the general offline log, or the burial point, the interface alarm platform can not discover. To this end the business side to ask the following questions:

How to load the page in the front-end components on the ui do abnormal monitoring;
How to locate abnormal after the need to provide the necessary auxiliary information to facilitate investigation;
How to deploy monitoring points, operating mechanism, how to achieve alarm strategy;
Component page load exception monitoring scheme

Glutinous rice team on the ui component page load anomalies do monitor, mainly through real machine real-time running components page list, polling cycle, page scanning, in addition to the components of the page element of the missing monitoring, also added js bomb abnormal block monitoring , Request the wrong monitoring, the specific program is as follows:

1) monitoring engine - the whole core is injected Luban.js file, the file can be components of the h5 page scan analysis (as shown in the figure provides three capabilities). Js how to be injected into the measured app? We injected into the file through the platform js, monitor the app monitor js files are updated to determine the load, the use of open the measured app switch to achieve js dynamic injection, so that each component or h5 Pages can refer to the file. In order to achieve the function of monitoring engine.

2) Monitoring page - The main function of the monitoring page is to pull the monitoring point, monitoring strategy, configuration information and so on for the monitoring engine to analyze, such as when the component 1 page is opened, the monitoring engine automatically triggers, Page of the information, the page scan, the success or failure of the logo will be back to the monitoring page, listening to the page uploaded to the platform.

We can look at what the program can do:

When listening to the page to get to our manually add the page when the monitoring items, the monitoring engine will do the following things

Dom elements of the existence of the judge, the request failed to judge, js abnormal shell frame to judge

The following is the actual online alarm monitoring caught the problem:

1) Failure of monitoring of page elements (Dom element presence judgment):

2) for a component of the page js error box (js abnormal shell frame to determine):

3) for a component of the page interface request specific link error (request failure to determine):

UI level The above three types of decision logic has basically covered the entire component page load anomaly monitoring.

Of course, the app framework itself will also exist log monitoring real-time upload to the back-end platform, the platform to do such as component package download successful monitoring, component update success rate monitoring, component end-to-end response time monitoring, which are online users to monitor Log for analysis, for our QA team, the first time to reflect the online user experience, you need such an active inspection of the way, do a great job in a large number of warnings before the potential risks identified in advance to eliminate.

Component Location Scheme

Online problem locating and locating has always been a very painful problem for the testers, especially when the user is feeding back an occasional problem, by looking at the phone number, log, time of occurrence, and user feedback, to find out what the user is doing Problem, this approach is more common, it is difficult to locate specific issues related to the user phone status type, network, system, app version, component version, whether or not to fight against cheating and other possibilities. If you can get the user’s request and back-end data, as well as the performance of the page at that time, plus some of the information mentioned before the basic 80% can locate some problems for online active inspection is also true.

We will luban.js file interface request, and json return the results of records, while the side automatically on the page screenshots, alarm time, run the system and other information to the various components side investigation: The following screenshot you can see the request_params, http_reponses is The request that the alarm obtains at that time and the result that the response should return to the request.

Alarm also provides accurate alarm time, alarm monitoring elements, and screenshots, through which you can locate the cause of the problem, such as the above take-out of a shop should have a menu of “hot” menu, back-end data gaps, Possible reasons for the back-end redis problem, the need for errorno: 114013 back-end log troubleshooting, this information largely to the duty officer to provide a smaller range of strong information based on the rapid investigation of online problems of great help, damage.

Monitoring point deployment and alarm strategy

Components page layout is generally divided into banner, category, searchfilter, list pages, etc., for these features, we will dom element monitoring involves this information, as shown below:

For the alarm can be set to component id continuous alarm log appears after a certain number, through text messages and e-mail interface to remind people to solve as soon as possible.

At present, the entire NAQA team of the sticky rice through the actual real machine for active inspection, divided iOS and Android side, mainly the core components, for other components to special scenes or special models under the monitoring, you can configure the monitoring task, the entire platform Can provide independent services, can also provide monitoring real machine services, this approach can greatly meet the monitoring needs of various components to protect the entire glutinous rice App real user experience.

reference articles: