Compatibility/System Addon/Automated Testing

Running the automated tests for site interventions

The easiest way to currently run the tests is to use a local build of Firefox on a desktop computer (artifact builds are recommended to save a lot of otherwise-wasted time).

Almost all of the automated tests run on their own in bulk. A desktop build of Firefox will test the desktop site interventions, while an Android build will start an emulator and test the Android interventions.

It is recommended to re-direct the output to a logfile.

<syntaxhighlight lang="bash">

1. run desktop/laptop tests

./mach bootstrap # choose desktop artifact ./mach build ./mach test-interventions --headless | tee desktop.log

1. run Android-specific tests (on an emulator)

./mach bootstrap # choose android artifact ./mach build ./mach test-interventions --headless | tee android.log </syntaxhighlight>

However, there are some tests which require a specific platform, VPN, or other conditions like logins, or they will be skipped (which should be mentioned in the resulting log):

Tests requiring Linux

<syntaxhighlight lang="bash">

1. samsung_com.py

./mach test-interventions --headless --bug=1799968 </syntaxhighlight>

Tests requiring Windows or MacOS

<syntaxhighlight lang="bash">

1. saxoinvestor.fr

./mach test-interventions --headless --bug=1719859 </syntaxhighlight>

Tests requiring Linux or Windows

<syntaxhighlight lang="bash">

1. livescience.com

./mach test-interventions --headless --bug=1650317 </syntaxhighlight>

Tests requiring Linux or MacOS

<syntaxhighlight lang="bash">

1. bankofamerica.py.com

./mach test-interventions --headless --bug=1472075 </syntaxhighlight>

Tests requiring a VPN connection to the USA

<syntaxhighlight lang="bash">

1. directv.com.co

./mach test-interventions --headless --bug=1610344

1. stream.directv.com

./mach test-interventions --headless --bug=1577519 </syntaxhighlight>

Tests requiring VPN connections elsewhere

(to be determined/confirmed)

Tests requiring a login

<syntaxhighlight lang="bash">

1. missingmail.usps.com
2. note this test can take several minutes to run, as the site is very slow

./mach test-interventions --headless --bug=1644830 --config=logins.json

1. Steam Community
2. note that Steam requires two-factor authentication, so you will be asked
3. to provide a code twice by the test as it runs.

./mach test-interventions --headless --bug=1570108 --do2fa --config=logins.json </syntaxhighlight>

Example logins.json file

The logins.json looks like this, only with actual passwords:

<syntaxhighlight lang="json"> {

 "1570108": {"username":"steam_username", "password":"steam_password"},
 "1644830": {"username":"usps_username", "password":"usps_password"}

} </syntaxhighlight>

Note that the file must be valid JSON, so of course care should be taken with characters like quotes and brackets and commas. For example, if a password itself contains a double-quote, it must be escaped with a backslash:

<syntaxhighlight lang="js"> "password": "password with a \" double quote" </syntaxhighlight>

Interpreting the results

The logfile will capture two summaries of which tests FAILED and were SKIPPED, once with the intervention active (to see if it is still working) and once with it disabled (to see if it is still even necessary, or if the site issues has been fixed).

For instance, here is one such summary:

=========================== short test summary info ============================
FAILED testing/webcompat/interventions/tests/test_1799994_vivobarefoot_com.py::test_enabled[with_interventions]
FAILED testing/webcompat/interventions/tests/test_1830752_afisha_ru.py::test_enabled[with_interventions]
FAILED testing/webcompat/interventions/tests/test_1836182_www_flatsatshadowglen_com.py::test_enabled[with_interventions]
SKIPPED [1] testing/webcompat/interventions/tests/test_1448747_discountcoffee_co_uk.py:12: Bug #1448747 skipped on platform (mac)
SKIPPED [1] testing/webcompat/interventions/tests/test_1448747_dylantalkstone_com.py:12: Bug #1448747 skipped on platform (mac)

The SKIPPED tests should be run on the appropriate platform, with logins, or checked manually if necessary.

FAILED tests require need to be checked to confirm whether the intervention is still working or necessary. Sometimes a test will fail because a site has changed, but it may fail simply due to a networking or other hiccup. As such it may be worth running each failing bug number's tests a second or third time, using the --bug= parameter as shown above (including the config file if the test requires a login).