Flaky tests · Gitlab 中文文档

# Flaky tests > 原文：[https://docs.gitlab.com/ee/development/testing_guide/flaky_tests.html](https://docs.gitlab.com/ee/development/testing_guide/flaky_tests.html) * [What’s a flaky test?](#whats-a-flaky-test) * [Quarantined tests](#quarantined-tests) * [Quarantine tests on the CI](#quarantine-tests-on-the-ci) * [Automatic retries and flaky tests detection](#automatic-retries-and-flaky-tests-detection) * [Problems we had in the past at GitLab](#problems-we-had-in-the-past-at-gitlab) * [Time-sensitive flaky tests](#time-sensitive-flaky-tests) * [Array order expectation](#array-order-expectation) * [Feature tests](#feature-tests) * [Capybara viewport size related issues](#capybara-viewport-size-related-issues) * [Capybara JS driver related issues](#capybara-js-driver-related-issues) * [PhantomJS / WebKit related issues](#phantomjs--webkit-related-issues) * [Capybara expectation times out](#capybara-expectation-times-out) * [Resources](#resources) # Flaky tests[](#flaky-tests "Permalink") ## What’s a flaky test?[](#whats-a-flaky-test "Permalink") 这个测试有时会失败，但是如果您重试了多次，它最终会通过. ## Quarantined tests[](#quarantined-tests "Permalink") 当`master`的测试频繁失败时，应创建[〜" master：broken"问题](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) . 如果无法及时修复测试，则会影响所有开发人员的工作效率，因此应通过分配`:quarantine`元数据将其放入隔离`:quarantine` . 这意味着除非使用`--tag quarantine`运行，否则它将被跳过： ``` bin/rspec --tag quarantine ``` **在对隔离进行测试之前，应确保存在〜" master：broken"问题，这样它才不会永远留在隔离区.** Once a test is in quarantine, there are 3 choices: * 测试应该固定（即摆脱其脆弱性）吗？ * 是否应将测试移至较低级别的测试？ * 是否应该完全删除该测试（例如，因为已经有一个较低级别的测试，或者它正在复制另一个相同级别的测试，或者它测试的太多等等）？ ### Quarantine tests on the CI[](#quarantine-tests-on-the-ci "Permalink") 隔离测试在允许失败的专用作业中在 CI 上运行： * `rspec-pg-quarantine` （CE 和 EE） * `rspec-pg-quarantine-ee` （仅适用于 EE） ## Automatic retries and flaky tests detection[](#automatic-retries-and-flaky-tests-detection "Permalink") 在 CI 上，我们使用[RSpec :: Retry](https://github.com/NoRedInk/rspec-retry)自动重试失败的示例几次（有关准确的重试次数，请参阅[`spec/spec_helper.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/spec_helper.rb) ）. 我们还使用自制的`RspecFlaky::Listener`侦听`RspecFlaky::Listener` ，该`RspecFlaky::Listener`将`RspecFlaky::Listener`示例记录在`master`上的 JSON 报告文件中（ `retrieve-tests-metadata`和`update-tests-metadata`作业）. 这最初是在以下[网址中](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13021)实现的： [https](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13021) : [//gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13021](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13021) . 如果要在本地启用重试，则可以使用`RETRIES`环境变量. 例如`RETRIES=1 bin/rspec ...`将重试失败的示例一次. ## Problems we had in the past at GitLab[](#problems-we-had-in-the-past-at-gitlab "Permalink") * [`rspec-retry` is biting us when some API specs fail](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29242): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9825](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9825) * [Sporadic RSpec failures due to `PG::UniqueViolation`](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28307#note_24958837): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9846](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9846) * Follow-up: [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10688](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10688) * [Capybara.reset_session! should be called before requests are blocked](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/33779): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12224](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12224) * FFaker 生成测试尚未准备就绪的时髦数据（测试应该是可预测的，所以很糟糕！）： * [使`spec/mailers/notify_spec.rb`更加健壮](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/20121) ： [https](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10015) : [`spec/mailers/notify_spec.rb`](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/20121) * [`spec/requests/api/commits_spec.rb`暂时失败](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/27988#note_25342521) ： [https](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9944) : [`spec/requests/api/commits_spec.rb`](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/27988#note_25342521) * [用序列替换 FFaker 工厂数据](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29643) ： [https](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10184) : [//gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10184](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29643) * [spec / finders / issues_finder_spec.rb 中的暂时失败](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30211#note_26707685) ： [https](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10404) : [//gitlab.com/gitlab-org/gitlab-foss/-//merge_requests/10404](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30211#note_26707685) ### Time-sensitive flaky tests[](#time-sensitive-flaky-tests "Permalink") * [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10046](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10046) * [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10306](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10306) ### Array order expectation[](#array-order-expectation "Permalink") * [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10148](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10148) ### Feature tests[](#feature-tests "Permalink") * [Be sure to create all the data the test need before starting exercise](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/32622#note_31128195): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12059](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12059) * [Bis](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34609#note_34048715): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12604](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12604) * [Bis](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34698#note_34276286): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12664](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12664) * [Assert against the underlying database state instead of against a page’s content](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/31437): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10934](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10934) * 在 JS 测试中，当元素在 Capybara 发送点击的确切时间移动时，移动元素会导致 Capybara 误点击 * [由于窗口大小和滚动位置，下拉菜单向上或向下渲染](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17660) * [延迟加载的图像可能导致水豚误点击](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18713) * [Triggering JS events before the event handlers are set up](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18742) * [Wait for the image to be lazy-loaded when asserting on a Markdown image’s `src` attribute](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25408) #### Capybara viewport size related issues[](#capybara-viewport-size-related-issues "Permalink") * [Transient failure of spec/features/issues/filtered_search/filter_issues_spec.rb](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29241#note_26743936): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10411](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10411) #### Capybara JS driver related issues[](#capybara-js-driver-related-issues "Permalink") * [Don’t wait for AJAX when no AJAX request is fired](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30461): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10454](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10454) * [Bis](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34647): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12626](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12626) #### PhantomJS / WebKit related issues[](#phantomjs--webkit-related-issues "Permalink") * 记忆在屋顶！（TL; DR：加载图像但阻止图像请求！）： [https](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12003) : [//gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12003](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12003) #### Capybara expectation times out[](#capybara-expectation-times-out "Permalink") * [Test imports a project (via Sidekiq) that is growing over time, leading to timeouts when the import takes longer than 60 seconds](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22599) ## Resources[](#resources "Permalink") * [Flaky Tests: Are You Sure You Want to Rerun Them?](https://semaphoreci.com/blog/2017/04/20/flaky-tests.html) * [How to Deal With and Eliminate Flaky Tests](https://semaphoreci.com/community/tutorials/how-to-deal-with-and-eliminate-flaky-tests) * [Tips on Treating Flakiness in your Rails Test Suite](https://semaphoreci.com/blog/2017/08/03/tips-on-treating-flakiness-in-your-test-suite.html) * [‘Flaky’ tests: a short story](https://www.ombulabs.com/blog/rspec/continuous-integration/how-to-track-down-a-flaky-test.html) * [Using Insights to Discover Flaky, Slow, and Failed Tests](https://s0circleci0com.icopy.site/blog/using-insights-to-discover-flaky-slow-and-failed-tests/) * * * [Return to Testing documentation](index.html)