Work with Flaky Autotests

You can mark an autotest as flaky in the autotest card. This can help you analyze testing activities and understand the percentage of flaky autotests. You can mark autotests as flaky using API or using GUI.

Identify Flaky Autotests

TestGear enables you to identify flaky autotests using the stability rate. The system tracks the launch history of autotests and builds the stability statistics based on it. The stability rate is calculated using the following formula:

$$ (1-M/N)*100 $$

Where:

  • N is the last 100 transitions of autotest status or all transitions if their amount is fewer than 100.
  • M is the amount of transitions of autotest status when it changed.

To identify a flaky autotest:

  1. Open the project.
  2. Go to the Autotests section.
  3. Switch to the Autotests tab.
  4. Open the autotest card.
  5. View the Stability scale in the card details. If the stability rate is not 100%, you can mark your autotest as flaky.

After identifying a flaky autotest, you can analyze its results and create an error category to detect it automatically during the future runs.

Mark Autotests as Flaky Using API

When creating a new autotest using the POST /api/v2/autoTests method, you can mark the autotest as flaky. To do that, set the isFlaky parameter to true.

In this case, the autotest cards created in TestGear are marked as flaky.

Mark Autotests as Flaky from UI

  1. Open the project.
  2. Go to the Autotests section.
  3. Switch to the Autotests tab.
  4. Open an autotest card.
  5. Click the button to make the autotest as flaky.

Mark Multiple Autotests

  1. Open the project.
  2. Go to the Autotests section.
  3. Switch to the Autotests tab.
  4. Select the autotests you want to mark as flaky using checkboxes.
  5. Click Make flaky in the mass actions pane above the autotest list.

You can delete a flaky mark in the same way. To do that, select the autotests and click Make unflaky in the mass actions pane above the autotest list.