Releases/Firefox 25/Test Plan

From New wiki
Jump to navigation Jump to search

« Firefox 24 | Firefox 25 | Firefox 26 »

Firefox 25 Test Plan

Summary

The following is the test plan for Firefox 25 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 25. After the release this document will be used as an archive of what was done to validate this release.

Status Currently in Aurora -- merging to Beta on September 19, 2013
Lead Anthony Hughes (irc: ashughes)
Builds aurora.mozilla.org
If you have some free time, please pick a task below and get in touch with one of the leads.

Schedule

June 24, 2013 Firefox 25 lands on Nightly
August 5, 2013 Firefox 25 moves up to Aurora
September 19, 2013 Firefox 25 moves up to Beta
October 22, 2013 Firefox 25 Release Candidate available for testing
October 29, 2013 Firefox 25 is released

Aurora

Branch Migration

Pre Sign-off Tasks

  • [DONE] Firefox branch migrated (Release Management)
  • [DONE] Mozmill Tests branch migrated (Automation Development)
  • [DONE] Updates throttled on aurora channel (Release Engineering)
  • [DONE] Updates verified throttled on aurora channel (QA)

Sign-off Tasks

  • [DONE] Mozmill Daily functional testruns pass with no unexpected failures/regressions (report)
  • [DONE] Mozmill Daily l10n testruns pass with no unexpected failures/regressions (report)
  • [DONE] Mozmill Daily endurance testruns pass with no unexpected failures/regressions (report)
  • [DONE] Mozmill Daily remote testruns pass with no unexpected failures/regressions (report)
  • [DONE] Mozmill Daily auroratest update testruns pass with no unexpected failures/regressions (report)

Post Sign-off Tasks

  • [DONE] Aurora sign-off email sent (QA)
  • [DONE] Updates unthrottled on aurora channel (Release Engineering)
  • [DONE] Updates verified unthrottled on aurora channel (QA)
  • [MISSED] Verify Stub Installers stage the new Aurora build (bug 910442)

Pre-Beta Sign-off

Checklist

  • [DONE] Mozmill-CI automation testruns checked for potential blocking regressions
  • [DONE] Features signed-off for Beta readiness, verified disabled if not
  • [ON TRACK] WebRTC spotchecked for potential blocking regressions, parity with Firefox 22
  • [ON TRACK] Flagged bug fixes verified

Automation

The following tracks the results of daily automation run against the Aurora builds as indicated by the Mozmill Addons, Endurance, Functional, L10n, Remote, and Update reports.

Top Failures Platform Instances Bug Status
/testSecurity/testGreenLarry.js All 130 bug 907101 FIXED (2013-08-21)
/testPreferences/testRestoreHomepageToDefault.js All 41 bug 912363 DISABLED (2013-09-05)
/testPopups/testPopupsAllowed.js All 36 bug 790218 FIXED (2013-09-09)
/testSecurity/testSafeBrowsingNotificationBar.js All 32 bug 905033 DISABLED (2013-08-12)
/testFindInPage/testFindInPage.js All 24 bug 909227 BLOCKED by l10n bug 913396 (2013-09-05)
/testSecurity/testDVCertificate.js All 24 bug 890487 FIXED (2013-08-06)
/testSecurity/testMixedContentPage.js All 24 bug 901951 FIXED (2013-08-06)
/testPrivateBrowsing/testAboutCache.js Windows 23 bug 888924 ASSIGNED (2013-09-05)
/testPopups/testPopupsBlocked.js Windows 13 n/a WORKSFORME (2013-09-04)
Resident Memory Usage Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Net
Windows XP 105 MB 106 MB 106 MB 106 MB 106 MB 104 MB -0.952%
Windows Vista 97 MB 97 MB 97 MB 97 MB 97 MB 97 MB no change
Windows 7 32-bit 100 MB 100 MB 100 MB 100 MB 100 MB 99 MB -1.000%
Windows 7 64-bit 102 MB 102 MB 102 MB 102 MB 102 MB 101 MB -0.980%
Windows 8 32-bit 130 MB 124 MB 122 MB 123 MB 122 MB 102 MB -21.538%
Windows 8 64-bit 132 MB 126 MB 123 MB 124 MB 124 MB 104 MB -21.212%
Mac OSX 10.6 138 MB 134 MB 134 MB 134 MB 134 MB 132 MB -4.348%
Mac OSX 10.7 203 MB 200 MB 200 MB 201 MB 199 MB 197 MB -2.956%
Mac OSX 10.8 195 MB 189 MB 190 MB 190 MB 191 MB 192 MB -1.538%
Ubuntu LTS 32-bit 103 MB 100 MB 100 MB 100 MB 98 MB 98 MB -4.854%
Ubuntu LTS 64-bit 135 MB 131 MB 131 MB 131 MB 131 MB 129 MB -4.444%
Ubuntu Desktop 32-bit 102 MB 100 MB 100 MB 100 MB 101 MB 99 MB -2.941%
Ubuntu Desktop 64-bit 135 MB 131 MB 131 MB 131 MB 132 MB 130 MB -3.704%

Features

The following features are tracked and tested as indicated by the release notes, relnote+, Firefox, and Platform feature bugs.

WebRTC

Using the Moztrap test as a guideline, verify that Firefox 25.0a2 is no worse than Firefox 22.0 when making AppRTC calls of >5 minutes in length. When testing, here are the guidelines to follow:

  • Have only one browser open on each machine at any given time
  • Have only one call running at any given time on a particular machine
  • Make sure the caller and callee are always on *different* machines.
  • Please don't test any 3+-way calls for this sanity check -- We just want to see the results for 1:1 (basic) calling
  • If you find regressions, report a bug and CC Randell Jesup and Maire Reavy; they can help track down if the regressions are real or not
    • Tip: be sure to provide extremely detailed steps to reproduce and witnessed results, as well as detailed information about your test environment; more information is better than not enough information.
Caller Callee Result
Firefox 25.0a2 on Windows 7 Firefox 25.0a2 on Windows 7 PASS
Firefox 22.0 on Windows 7 Firefox 25.0a2 on Windows XP FAIL (1*)
Chrome Stable on Windows 7 Firefox 25.0a2 on Windows 8 PASS
Firefox 25.0a2 on Windows 7 Firefox 22.0 on Mac OSX 10.8 PASS (2*)
Firefox 25.0a2 on Windows 7 Chrome Stable on Ubuntu 12.04 PASS
Firefox 22.0 on Windows XP Firefox 25.0a2 on Windows 7 FAIL (4*)
Firefox 25.0a2 on Windows XP Firefox 22.0 on Windows XP FAIL (10*)
Firefox 25.0a2 on Windows XP Chrome Stable on Windows 8 FAIL (5*)
Chrome Stable on Windows XP Firefox 25.0a2 on Mac OSX 10.8 PASS
Firefox 22.0 on Windows XP Firefox 25.0a2 on Ubuntu 13.04 FAIL (10*)
Chrome Stable on Windows 8 Firefox 25.0a2 on Windows 7 FAIL (8*)
Firefox 25.0a2 on Windows 8 Chrome Stable on Windows XP FAIL (7*)
Firefox 22.0 on Windows 8 Firefox 25.0a2 on Windows 8 PASS
Firefox 25.0a2 on Windows 8 Firefox 22.0 on Windows 8 PASS
Firefox 22.0 on Windows 8 Firefox 25.0a2 on Mac OSX 10.8 FAIL (6*)
Chrome Stable on Windows 8 Firefox 25.0a2 on Ubuntu 13.04 PASS
Firefox 25.0a2 on Mac OSX 10.8 Firefox 22.0 on Windows 7 FAIL (9*)
Chrome Stable on Mac OSX 10.8 Firefox 25.0a2 on Windows XP FAIL (9*)
Firefox 22.0 on Mac OSX 10.8 Firefox 25.0a2 on Windows 8 FAIL (11*)
Firefox 25.0a2 on Mac OSX 10.8 Chrome Stable on Mac OSX 10.8 PASS
Firefox 25.0a2 on Mac OSX 10.8 Firefox 22.0 on Ubuntu 13.04 FAIL (9*)
Firefox 25.0a2 on Ubuntu 12.04 Chrome Stable on Windows 7 PASS
Firefox 22.0 on Ubuntu 12.04 Firefox 25.0a2 on Windows XP PASS
Chrome Stable on Ubuntu 12.04 Firefox 25.0a2 on Windows 8 PASS
Firefox 25.0a2 on Ubuntu 12.04 Firefox 22.0 on Mac OSX 10.8 PASS
Firefox 25.0a2 on Ubuntu 12.04 Chrome Stable on Ubuntu 13.04 PASS

Detailed Results

  • (1*) The caller hears himself, as well. The callee hears delayed sound (1-2sec) after a few minutes
    • not reproducible if caller uses latest Aurora 25.0a2.
  • (2*) The caller got a crash after ~12 min, while opening a page in a new tab: https://crash-stats.mozilla.com/report/index/97cf87a4-284f-4cd0-9a17-ed6cf2130906
    • one time occurrence, not related to WebRTC
  • (4*) Both caller and callee hear delayed sound (1-2secs) - reproducible also if callee uses Firefox 23.0.1
  • (5*) The caller hears delayed sound
    • reproducible also with FF23.0.1 and 24b9; moreover, with 23.0.1 and 24b.9, the delay appears on both sides
  • (6*) The callee hear himself, as well
    • reproducible also with 23.0.1 and Firefox 24 beta 10
  • (7*) Callee's conversation page crashes when caller attends
    • reproducible with FF23.0.1 or 24b9 (used by caller)
  • (8*) when caller connects, both attendances remain stuck in “Connecting…” mode
    • working with 23.0.1 and 24b9
  • (9*) when caller connects, both attendances remain stuck in “Connecting…” mode
    • working with 23.0.1 and 24b9
    • the problem is the TURN server which is giving a bogus address
  • (10*) sometimes, the image and sound is choppy and there is a delayed sound: ~5 seconds at caller, ~10seconds at callee; callee hears himself, as well
    • reproducible with 23.0.1
  • (11*) attendants see themselves instead of seeing the other person in the call and there is no sound
    • reproducible with 23.0.1

Bug Fix Verifications

The following bugs have been fixed and flagged for verification. At a minimum, any critical/blocker bug should be verified before migration to Beta. <bugzilla> { 

"f1":"op_sys",
"o1":"nowordssubstr",
"v1":"Android Gonk",
"f2":"keywords",
"o2":"nowordssubstr",
"v2":"assertion intermittent",
"f3":"component",
"o3":"nowordssubstr",
"v3":"Build%20Config",
"f4":"product",
"o4":"nowordssubstr",
"v4":"QA mozilla SDK Boot2Gecko Testing",
"f5":"status_whiteboard",
"o5":"nowordssubstr",
"v5":"qa asan Leo",
"keywords":"verifyme",
"cf_status_firefox25":"fixed"

} </bugzilla>

Potential Blockers

The following bugs have been nominated to block this milestone from uplifting to Beta.

  • no blockers currently

Beta 1

Build Info

Check List

Criteria Description Status
Automation No unexpected test failures from Mozmill functional testruns SIGNED-OFF (report)
Regression Testing No unexpected failures from manually smoketesting past regressions and new features SIGNED-OFF (report)
Bug Fix Verifications No unexpected failures from manually verifying landed bug fixes SIGNED-OFF (report)
Stability Verification No unexpected spikes in top crash report for this milestone SIGNED-OFF (report)
Updates on betatest channel No unexpected failures from Mozmill update testruns on the betatest channel SIGNED-OFF (report)
Updates on releasetest channel No unexpected failures from Mozmill update testruns on the releasetest channel SIGNED-OFF (report)
Updates on betatest channel No unexpected failures from Mozmill update testruns on the beta channel SIGNED-OFF (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Test Coverage
Platform Coverage
  • Windows XP
  • Windows 7 64-bit
  • Windows 8.1 Preview 64-bit
  • Mac OSX 10.8
  • Ubuntu 13.04 64-bit
Instructions
  • Run these tests
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
  • When you get to the WebRTC smoketest refer to the Special Coverage for WebRTC section below for coverage details
  • If you encounter a bug, report it and add it to the Issues list below using the bug number template
Special Coverage for WebRTC
  • Caller Firefox/Platform -> Callee Firefox/Platform
  • Firefox 25.0b1 on Windows XP -> Firefox 25.0b1 on Windows 7
  • Firefox 25.0b1 on Windows 7 -> Firefox 25.0b1 on Windows 8
  • Firefox 25.0b1 on Windows 8 -> Firefox 25.0b1 on Mac OSX 10.8
  • Firefox 25.0b1 on Mac OSX 10.8 -> Firefox 25.0b1 on Ubuntu 13.04
  • Firefox 25.0b1 on Ubuntu 13.04 -> Firefox 25.0b1 on Windows XP

Issues

List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.

Bug Status
bug 917818 - Findbar buttons remain in pressed state on Linux NEW - does not block 25b1
bug 912975 - Windows and tabs are not always restored after Reseting Firefox NEW - does not block 25b1
bug 860249 - [Mozmill] Exception "Sorry, cannot connect to jsbridge extension, port 24242" NEW - does not block 25b1
bug 791195 - Session is not restored after Firefox crash NEW - does not block 25b1

Beta 2

Build Info

Check List

Criteria Description Status
Automation No unexpected test failures from Mozmill functional testruns SIGNED-OFF (report)
Regression Testing No unexpected failures from manually smoketesting past regressions and new features SIGNED-OFF (report)
Bug Fix Verifications No unexpected failures from manually verifying landed bug fixes SIGNED-OFF (report)
Stability Verification No unexpected spikes in top crash report for this milestone SIGNED-OFF (report)
Updates on betatest channel No unexpected failures from Mozmill update testruns on the betatest channel SIGNED-OFF (report)
Updates on releasetest channel No unexpected failures from Mozmill update testruns on the releasetest channel SIGNED-OFF (report)
Updates on beta channel No unexpected failures from Mozmill update testruns on the beta channel SIGNED-OFF (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Test Coverage
  • Breakpad
  • MD5 Hash De-support
  • Plug-in Compatibility
  • Exploratory testing around Flash 11.9 Beta : etherpad
Platform Coverage
  • Windows Vista
  • Windows 7 32-bit
  • Windows 8 64-bit
  • Mac OSX 10.7
  • Ubuntu 12.04 64-bit
Instructions
  • Run these tests
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
  • If you encounter a bug, report it and add it to the Issues list below using the bug number template

Beta 3

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Regression Testing (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

  • Focus: Breakpad, Web Audio, WebGL
  • Platforms: Windows XP, Windows 7 64-bit, Windows 8.1 32-bit, Mac OSX 10.8, Ubuntu 13.04 64-bit
  • Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below

Beta 4

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Regression Testing (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

  • Focus: Breakpad, Web Audio, WebGL
  • Platforms: Windows XP, Windows Vista, Windows 8 64-bit, Mac OSX 10.7, Ubuntu 12.04 64-bit
  • Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below

Beta 6

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Regression Testing (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

  • Focus: Breakpad, Web Audio, WebGL, TLS/SSL
  • Platforms: Windows XP, Windows Vista, Windows 8 64-bit, Mac OSX 10.7, Ubuntu 12.04 64-bit
  • Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below
  • Exploratory testing: TLS/SSL etherpad

Beta 7

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Regression Testing (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

  • Focus: Breakpad, Bookmarks Library, Mixed Content Blocker, Add-on Compatibility
  • Platforms: Windows 7 64-bit, Windows 8 32-bit, Windows 8.1 64-bit, Mac OSX 10.9, Ubuntu 12.04 32-bit
  • Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below

Beta 8

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Regression Testing (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

  • Focus: Breakpad, Software update for root-owned files, Downloads (multiple files - small and large ones, cancel, retry, open folder, open downloaded executables), PDF Viewer (open multiple pdf files and interact with them), Session Restore after restarting from regular and safe modes, Windows 8.1 exploratory testing, Mac OSX 10.9 exploratory testing
  • Platforms: Windows XP, Windows 8.1 64-bit, Mac OSX 10.9, Ubuntu 13.10 32-bit
  • Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below

Beta 9

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Regression Testing (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

  • Focus: Breakpad, Plug-in compatibility, Session Restore with multiple windows and tabs (exploratory testing with different options).
  • Platforms: Windows XP, Windows 7 32-bit, Mac OSX 10.6, Ubuntu 12.04 64-bit
  • Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below

Beta 10

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Regression Testing (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

  • Focus: Breakpad, Web Compatibility, PDF Viewer (open multiple pdf files and interact with them), Exit Firefox (close Firefox all possible ways when open with differently loaded profiles - many tabs, plugin content etc)
  • Platforms: Windows XP, Windows 8.1 64-bit, Mac OSX 10.7, Ubuntu 13.10 64-bit
  • Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below

Beta 11

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Regression Testing (see below)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

OCSP Disabled
  • Use this Moztrap test as a guide
  • Verify OCSP stapling is disabled and look for rendering, layout, and functional issues
  • Check as many SSL enabled websites as possible, listing any issues found in the section below
  • Note: the list below is a bare minimum, please add to it as time allows
Sites Tested Windows XP Windows 7 32-bit Windows 8 64-bit Mac OSX 10.8 Ubuntu 13.04 32-bit
Amazon [DONE] [DONE] [DONE] [DONE] [DONE]
Baidu [DONE] [DONE] [DONE] [DONE] [DONE]
Bit.ly [DONE] [DONE] [DONE] [DONE] [DONE]
Bugzilla [DONE] [DONE] [DONE] [DONE] [DONE]
Digg [DONE] [DONE] [DONE] [DONE] [DONE]
EBay [DONE] [DONE] [DONE] [DONE] [DONE]
Facebook [DONE] [DONE] [DONE] [DONE] [DONE]
Flickr [DONE] [DONE] [DONE] [DONE] [DONE]
Google [DONE] [DONE] [DONE] [DONE] [DONE]
LinkedIn [DONE] [DONE] [DONE] [DONE] [DONE]
Outlook [DONE] [DONE] [DONE] [DONE] [DONE]
Pinterest [DONE] [DONE] [DONE] [DONE] [DONE]
QQ.com [DONE] [DONE] [DONE] [DONE] [DONE]
Reddit [DONE] [DONE] [DONE] [DONE] [DONE]
Slashdot [DONE] [DONE] [DONE] [DONE] [DONE]
SoundCloud [DONE] [DONE] [DONE] [DONE] [DONE]
Tumblr [DONE] [DONE] [DONE] [DONE] [DONE]
Twitter [DONE] [DONE] [DONE] [DONE] [DONE]
Vimeo [DONE] [DONE] [DONE] [DONE] [DONE]
Wikipedia [DONE] [DONE] [DONE] [DONE] [DONE]
Wordpress [DONE] [DONE] [DONE] [DONE] [DONE]
Yahoo [DONE] [DONE] [DONE] [DONE] [DONE]
Yandex [DONE] [DONE] [DONE] [DONE] [DONE]
Youtube [DONE] [DONE] [DONE] [DONE] [DONE]


Issues Found:

  • No issues found

Fix Verifications

  • [DONE] bug 929068 Tumblr logo does not show correctly at first load. (style sheet would not load at first load)
  • [DONE] bug 929524 Layout is broken on twitter.com at initial loading

Beta 12

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Fix Verifications

  • [DONE] bug 927901 Test has failed: Permission denied to access property 'encrypt'
  • [DONE] bug 930827 Hang when visiting icu-project.org

Release Candidate

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Regression Testing (see below)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on release channel (report)
  • [DONE] Verify update throttling level on release
  • [DONE] Verify what's new page for 25.0-pl via bug 927975

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Past Regressions
  • Focus: Breakpad, NVidia 3D-Vision, Partner repack update, and Windows share compatibility smoketests
  • Platforms: Windows XP, Windows 7 64-bit, Windows 8.1 32-bit, Mac OSX 10.9, Ubuntu 13.04 64-bit
  • Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below
OCSP Disabled
  • Use this Moztrap test as a guide
  • Verify OCSP stapling is disabled and look for rendering, layout, and functional issues
  • Check as many SSL enabled websites as possible, listing any issues found in the section below
  • Note: the list below is a bare minimum, please add to it as time allows
Sites Tested Windows XP Windows 7 64-bit Windows 8.1 32-bit Mac OSX 10.9 Ubuntu 13.04 64-bit
Facebook [DONE] [DONE] [DONE] [DONE] [DONE]
Google [DONE] [DONE] [DONE] [DONE] [DONE]
SoundCloud [DONE] [DONE] [DONE] [DONE] [DONE]
Tumblr [DONE] [DONE] [DONE] [DONE] [DONE]
Twitter [DONE] [DONE] [DONE] [DONE] [DONE]
Yahoo [DONE] [DONE] [DONE] [DONE] [DONE]
Youtube [DONE] [DONE] [DONE] [DONE] [DONE]
BT24 [DONE] [DONE] [DONE] [DONE] [DONE]
Google Mail [DONE] [DONE] [DONE] [DONE] [DONE]
DuckDuckGo [DONE] [DONE] [DONE] [DONE] [DONE]
MSN [DONE] [DONE] [DONE] [DONE] [DONE]
PayPal [DONE] [DONE] [DONE] [DONE] [DONE]
Reuters [DONE] [DONE] [DONE] [DONE] [DONE]
BBC News [DONE] [DONE] [DONE] [DONE] [DONE]
Moztrap [DONE] [DONE] [DONE] [DONE] [DONE]
Marketplace [DONE] [DONE] [DONE] [DONE] [DONE]
Mozilla Wiki [DONE] [DONE] [DONE] [DONE] [DONE]
Booking.com [DONE] [DONE] [DONE] [DONE] [DONE]
London Stock Exchange [DONE] [DONE] [DONE] [DONE] [DONE]
Moneybookers [DONE] [DONE] [DONE] [DONE] [DONE]
Oracle Academy [DONE] [DONE] [DONE] [DONE] [DONE]
Sports Direct [DONE] [DONE] [DONE] [DONE] [DONE]
Emag [DONE] [DONE] [DONE] [DONE] [DONE]
Imdb [DONE] [DONE] [DONE] [DONE] [DONE]
Aol [DONE] [DONE] [DONE] [DONE] [DONE]


Issues Found:

  • No issues found

Fix Verifications

  • [DONE] bug 929068 Tumblr logo does not show correctly at first load. (style sheet would not load at first load)
  • [DONE] bug 929524 Layout is broken on twitter.com at initial loading
  • [DONE] bug 927901 Test has failed: Permission denied to access property 'encrypt'
  • [DONE] bug 930827 Hang when visiting icu-project.org
  • bug 927975 Enable Firefox /whatsnew for /pl (polish) locale on Firefox 25 -- once updates are released

25.0.1

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Regression Testing (see below)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on release channel (report)
    • This was done via manual spot checks due to problems with the Mozmill infrastructure.
  • [DONE] Verify what's new page for 25.0.1-pl via bug 927975

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Past Regressions
  • Focus: Breakpad
  • Platforms: Windows XP, Windows 7 64-bit, Windows 8.1 32-bit, Mac OSX 10.9, Ubuntu 13.04 64-bit
  • Instructions: Force a crash with the crashme extension and verify that the report is successfully submitted
SSL Topsite Testing
  • Navigate to a website which uses SSL and do some exploratory testing on that site
  • Be sure to log in to the website if it has that functionality
  • Report any issues with rendering and general usage, in particular if you get any SSL errors
  • Suggestions for sites to test include email, banking, social networking, shopping, video streaming, and forum websites
  • Try to test at least 25 different websites
  • Add PASS if there were no problems or FAIL if you encountered an issue
  • Be sure to note any issues in the Issues Found section below
Sites Tested Windows XP Windows 7 64-bit Windows 8.1 32-bit Mac OSX 10.9 Ubuntu 13.04 64-bit
1. https://www.facebook.com/ PASS PASS PASS PASS PASS
2. http://www.youtube.com/ PASS PASS PASS PASS PASS
3. https://login.yahoo.com/ PASS PASS PASS PASS PASS
4. https://www.linkedin.com/ PASS PASS PASS PASS PASS
5. https://login.live.com/ PASS PASS PASS PASS PASS
6. https://twitter.com/ PASS PASS PASS PASS PASS
7. http://www.amazon.com/ PASS PASS PASS PASS PASS
8. https://accounts.google.com/ PASS PASS PASS PASS PASS
9. http://wordpress.com/ PASS PASS PASS PASS PASS
10. http://www.ebay.com/ PASS PASS PASS PASS PASS
11. https://www.tumblr.com/ PASS PASS PASS PASS PASS
12. https://www.pinterest.com/ PASS PASS PASS PASS PASS
13. https://www.paypal.com/ PASS PASS PASS PASS PASS
14. https://instagram.com/accounts/login/ PASS PASS PASS PASS PASS
15. https://login.alibaba.com/ PASS PASS PASS PASS PASS
16. https://www.adcash.com/en/index.php PASS PASS PASS PASS PASS
17. https://vimeo.com/ PASS PASS PASS PASS PASS
18. http://www.aliexpress.com/ PASS PASS PASS PASS PASS
19. https://ib.btrl.ro/BT24/ PASS PASS PASS PASS PASS
20. https://www.pcgarage.ro PASS PASS PASS PASS PASS
21. https://www.emag.ro/ PASS PASS PASS PASS PASS
22. https://en.wikipedia.org/wiki/Main_Page PASS PASS PASS PASS PASS
23. https://soundcloud.com/ PASS PASS PASS PASS PASS
24. http://www.booking.com/ PASS PASS PASS PASS PASS
25. http://www.bbc.com PASS PASS PASS PASS PASS
26. http://eu.battle.net/en/ PASS PASS PASS PASS PASS
27. https://mail.google.com/ PASS PASS PASS PASS PASS
28. https://plus.google.com/ PASS PASS PASS PASS PASS
29. https://quality.mozilla.org PASS PASS PASS PASS PASS
30. http://www.foxnews.com PASS PASS PASS PASS PASS
add more if time permits


Issues Found:

  • No issues found

Fix Verifications

  • [DONE] bug 933733 Page doesn't load/render if the mouse is not moving -- verified by reporter
  • [DONE] bug 935959 Update Mozilla to NSS 3.15.3 (new alternative NSS branch) to pick up a few fixes

Bugs Reported

  • [ON TRACK] bug 938075 Size of some Firefox 25.0.1 win32 l10n candidate builds is 45MB-72MB

Bugs

Potential Blockers

[ON TRACK]

  • no tracked bugs

[DONE]

  • bug 921457 [WebAudio] No sound is played when connecting a Delay Node
  • bug 904620 [WebGL] Hangs in SurfaceStream_TripleBuffer_Async::WaitForCompositor
  • bug 929068 Tumblr logo does not show correctly at first load. (style sheet would not load at first load)
  • bug 929617 Validity period for stapled OCSP responses is too short
  • bug 929524 Layout is broken on twitter.com at initial loading
  • bug 927975 Enable Firefox /whatsnew for /pl (polish) locale on #Firefox25

[DROPPED]

  • bug 923319 [WebAudio] Performance is extremely poor
  • bug 923106 [WebAudio] Sound is not changed in the Oscillator audio source node
  • bug 923058 [WebAudio] Jerky sound on Live Input node

Issues Found in Testing

Add any new issues found or old issues reopened to the list below. These issues will be triaged and escalated by the release leads.

<bugzilla> { 

"id":"927421,925680,923058,913407,917818,912975,791195,788423,919938,920044,860249,920151,914997,862687,880854,870705,868905,904620,921457,921442,921449,921462,921469,921735,923061,856900,923106,78414, 913854, 925735, 928379, 928335, 929442",

"include_fields":"id,summary,status" } </bugzilla>

Top Crashes

Conduct daily and weekly stability checks and list any tracked bugs below. Any crash in the top-20 should have the topcrash keyword added and be nominated to track. Use bughunter to help with URL correlations.

[ON TRACK]

  • no tracked bugs

[DONE]

  • bug 929009 crash in `anonymous namespace::downmix_to_stereo<float>(float*, long, float*, int)
  • bug 925459 crashes related to bitguard.dll
  • bug 921323 crash in OpenGL@0x32af (MakeCurrent from OMTC)
  • bug 920725 crash in nsHtml5StreamParser::WriteStreamBytes
  • bug 898156 crash in mozilla::net::FTPChannelParent::OnStartRequest
  • bug 881636 crash in mozilla::a11y::DocAccessible::UpdateTree
  • bug 873733 crash in npjp2 @ mozilla::plugins::parent::_retainobject with Java SE7
  • bug 800347 crash in mozilla::ipc::RPCChannel::OnMaybeDequeueOne

[AT RISK]

  • no bugs at risk

[DROPPED]

  • bug 928020 crash in mozalloc_abort(char const* const) | xul.dll@... | js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value*, JS::MutableHandle<JS::Value>)
  • bug 927081 crash in js::gc::IsAboutToBeFinalized<js::Shape>
  • bug 925407 crash in npswf32_11_9_900_117.dll@0x345b8d
  • bug 812683 [Win7] startup crash in _d2d_clear_surface @ CContext::RestorePipelineStateImpl (d3d11.dll, KB2670838, Asus Gamer OSD)
  • bug 767343 crash in nsSupportsStringImpl::SetData with abort

QA Wanted

[ON TRACK]

[DONE]

  • bug 929063 Twitter layout broken/missing items
  • bug 918612 [Mac] Firefox 24 doesn't start when home directory is on AFP network share
  • bug 913407 Rendering issue in Addon manager
  • bug 913043 [adbe 3627468] Flash 11.9 crash in F1987069282
  • bug 922875 Stencil test fails even if there is no stencil buffer in the current FBO

[DROPPED]

  • bug 919076 Saved files permissions set incorrectly on Linux and Mac
  • bug 917011 Video on second screen gets displayed on first screen in fullscreen mode
  • bug 886670 Crash in mozilla::gl::SharedSurface_GLTexture::Fence @ libGPUSupportMercury on citadel demo guided tour
  • bug 698296 crash in js::SweepBackgroundThings @ js::gc::Arena::finalize

Verifications

How to Verify a Fix
  • Pick a bug from this list
  • Download a build that is known to reproduce the bug and test to make sure you can see the bug
  • Download a build that is said to be fixed and test to make sure you cannot see the bug
  • If the bug reproduces in the fixed build:
    • set the STATUS field to REOPENED and provide a comment detailing your testing
  • If the bug does not reproduce in the fixed build:
    • set the status flag to verified for the version you tested
    • set the STATUS field to VERIFIED FIXED if the version tested is the same as the TARGET MILESTONE
    • any bugs marked verified will automatically appear in the Verified Fixes section below
Verified Fixes

<bugzilla> { 

"include_fields":"id, summary",
"f1":"op_sys",
"o1":"nowordssubstr",
"v1":"Android Gonk",
"f2":"keywords",
"o2":"nowordssubstr",
"v2":"assertion intermittent verifyme",
"f3":"component",
"o3":"nowordssubstr",
"v3":"Build%20Config",
"f4":"product",
"o4":"nowordssubstr",
"v4":"QA mozilla SDK Boot2Gecko Testing",
"f5":"status_whiteboard",
"o5":"nowordssubstr",
"v5":"qa asan Leo",
"cf_status_firefox25":"verified"

} </bugzilla>

Unconfirmed

Pick a bug from this list and try to reproduce the issue using the information provided. If you reproduce the issue, provide a description of the circumstances, set the status to NEW, and move the bug into a component which matches the issue (ie. a bookmarks issue would go in the Bookmarks component). If you can't reproduce the issue ask the reporter for more information by setting the need-info? flag to their email address.

Confirmed Bugs

<bugzilla> { 

"bug_status":"NEW",
"f1":"op_sys",
"o1":"nowordssubstr",
"v1":"Android Gonk",
"f2":"keywords",
"o2":"nowordssubstr",
"v2":"assertion intermittent verifyme",
"f3":"component",
"o3":"nowordssubstr",
"v3":"Build%20Config Tests",
"f4":"product",
"o4":"nowordssubstr",
"v4":"QA mozilla SDK Boot2Gecko Testing Thunderbird",
"f5":"status_whiteboard",
"o5":"nowordssubstr",
"v5":"qa asan Leo",
"f6":"version",
"o6":"anywordssubstr",
"v6":"25"

} </bugzilla>

Other Bugs

The following other queries should be triaged periodically, mostly by the QA Driver, and nominated for testing.

  • regressionwindow-wanted: nominate for qawanted if testing is needed or remove regressionwindow-wanted if the request is no longer valid
  • [ fixed]: nominate for verifyme if QA testing is needed, [easy verify] of community testing is needed, or [qa-] if no testing is needed

New Platform Readiness

Windows 8.1

Status COMPLETE (due 16 October 2013)
Owner Anthony Hughes
Manual Testing COMPLETE (see known bugs listed below)
Automated Testing COMPLETE (see known bugs listed below)
Bugs

[ON TRACK]

  • bug 922314 Add MFT Support for WebM
  • bug 924045 Metro icon shown after fresh install even if nightly is not the default browser

[DONE]

  • bug 921080 Clicking on item in download panel not working on Windows 8.1 RTM
  • bug 924038 Snapped view in Windows 8.1 64-bit
  • bug 924051 Display rendering issue if 2 monitors are used
  • bug 923241 Crash @ RtlAcquireSRWLockExclusive in d3d11.dll@0x17214

Mac OSX 10.9

Status COMPLETE (due TBD October 2013)
Owner Marcia Knous
Manual Testing COMPLETE (see known bugs listed below)
Automated Testing COMPLETE (integrated into Mozmill-CI workflow)
Bugs

[ON TRACK]

  • bug 917042 File picker often closes when trying to select a file the first time -- Needs regression range
  • bug 888605 Crash in [NSPopUpButtonObserver menuChangedItem] -- Needs regression range
  • bug 883859 Dialogs do not open on the screen where Firefox is active -- Possible Tech Evangelism

[DONE]

  • bug 918943 Duplicate symbol errors linking WebRTC when using the 10.9 SDK with --disable-optimize
  • bug 917526 Various duplicate symbols error building tree with the 10.9 SDK
  • bug 901348 Duplicate symbol errors building --with-intl-api
  • bug 886096 Crash in libdispatch

Post-mortem

  • Agenda & Notes
  • Follow-up:
    • OCSP - not tested well enough due to lack of live sites to test against
      • QA needs to be able to test features even without live use cases
      • Should Mozilla try to find a "launch partner" for these types of features in the future?
    • WebCrypto - undocumented uplift to Firefox 25
      • QA needs to be "involved" in uplift approvals, attending the RelMan meetings is one way we can achieve this
    • App Manager - flew under the radar completely
      • QA needs to escalate these "failures" as soon as they happen
      • Talk to dkl or glob about identifying "features" based on Bugzilla heuristics (patch size, number of files, component, etc)
    • l10n (24esr) - RelMan's mistake
      • QA is incapable of identifying these issues via manual testing and automation
      • RelEng should harden their system to mitigate exposure to human error
      • QA should try to involve l10n more in community events, reach out to localizers more
    • Mozmill-CI Instability
      • Cannot continue to force European staff to troubleshoot when they are on PTO, sick, or should be sleeping
      • A-Team needs someone in a North American timezone to maintain and troubleshoot issues during Beta/Release pushes
Retrieved from "https://mozilla.mediawiki.site/w/index.php?title=Releases/Firefox_25/Test_Plan&oldid=762349"