Analyze Autotest Failures

After launching autotests, you can analyze the test run and identify the test failure causes to build up the analytics data for autotests. You can do the following:

  • Mark successful autotests that have no defects.
  • Specify a cause of failure for failed autotests by marking them with error categories: product, infrastructure, and autotests.
  • Create a regular expression from a stack trace in the failed autotest. You can associate the regular expression with an error category to automatically specify the cause of failure for failed autotests containing this expression.

Mark Successful Autotests

  1. Open the project.

  2. Select Autotests in the main toolbar. By default, the Test runs tab opens.

  3. Open a test run.

  4. Filter out successful autotests. To do that, click the button in the Result column or click the Passed tile in the Tests by results pane.

  5. Select all successful autotests using a checkbox next to the ID column.

  6. Click Specify the reason.

  7. Select No defect in the cause of failure window.

  8. Click Save.

After marking successful autotests, focus on the Analysis of error categories pane. The successful autotests are now included into the No defects category. You can use this information for further analytics activity.

Identify Failure Reasons for Autotests

  1. Open the project.
  2. Select Autotests in the main toolbar. By default, the Test runs tab opens.
  3. Open a test run.
  4. Filter out failed autotests. To do that, click the button in the Result column and select Failed or click the Failed tile in the Tests by results pane.
  5. Open the test result card of a failed autotest. Focus on the Stack Trace snippet that displays the code lines that caused the failure.
  6. When you identify what caused the failure, specify the error category in the Cause of failure pane. There are three options available:
    • Product
    • Infrastructure
    • Autotest (you can specify a test defect (TestDefect) or an unexpected element (NoSuchElementExpectation) as a cause of failure)
  7. Click Save.

Save Error Categories

The system can automatically specify the cause of failure using regular expressions. A regular expression is a stack trace that causes test failures. You can save this stack trace in the system and associate it with a certain error category. If the system detects a regular expression in a failed autotest, it automatically specifies the cause of failure for this test.

  1. Open the project.
  2. Select Autotests in the main toolbar. By default, the Test runs tab opens.
  3. Open a test run.
  4. Open a failed autotest, for which you want to save a regular expression.
  5. Copy the code lines from Stack trace. These code lines are used as a regular expression.
  6. Close the test result card.
  7. In the Autotests section, switch to the Error categories tab.
  8. Click Create.
  9. Enter the error category name.
  10. Paste the copied code lines into the Regular expressions field.
  11. Click Save.