Releases/Firefox 36/Test Plan

From New wiki
Jump to navigation Jump to search

Summary

This page is to track testing of Firefox 36 from mozilla-central (Nightly) through to mozilla-release.

Bug queries and graphs for this release and Firefox iterations: Firefox QA Dashboard

How you can help

Would you like to help test Firefox 36? Join us! Here's some things you can do:

Schedule

Please consult the Rapid Release Calendar for more information.

Milestone Date Checks
Nightly 2014-10-13 Conduct daily stability and bug triage, ensure features are owned
Aurora Migration 2014-11-28 Conduct testing to ensure Firefox 36 Aurora builds are okay, sign off for updates on Friday
Aurora Updates enabled 2014-10-17 Continue daily stability and bug triage, ensure features are tested
Beta Uplift 2015-01-12 Review testing and sign off every beta release
Release 2015-02-24 Review tests and sign off release

Meetings

Meeting Purpose When Vidyo Room IRC Backchannel
Channel Status Raise quality concerns with the Release Management team 10:00 Pacific on Tuesday & Thursday ReleaseCoordination #planning
Firefox QA Meet the Desktop QA team and hear what they're working on 8:30am Pacific on Wednesdays DesktopQA #qa
QA Team Meet the entire QA team and hear what they're working on! 1:30am Pacific on Wednesdays QA #qa

Checks

Priority Check Description Frequency Last Checked
1 Stability Review the stability reports to identify new/rising crash signatures Daily 2014-xx-xx
2 Pushlog Review the pushlog to identify fixed bugs needing verification Daily 2014-xx-XX (not keeping up tho)
2.5 Iteration Triage the iteration bugs, add QA contacts Twice weekly 2014-xx-XX
3 QA Needed Bugs Review and assign qawanted bugs for investigation At least twice a week 2014-xx-XX
4 Unconfirmed Bugs Triage and test unconfirmed bug reports At least once a week 2014-xx-XX
5 Fixed Bugs Flag fixed bugs for verification At least once a week 2014-xx-XX
6 Verification Needed Bugs Test verifyme bugs to confirm fixed bugs are fixed At least once a week 2014-xx-XX
7 Reported Bugs Document and escalated reported bugs as necessary Every Monday 2014-xx-XX
8 Features Ensure features are owned and on track for release Once a week 2014-xx-XX
9 Automation Ensure automation bugs are owned and on track for resolution Once a week 2014-09-XX

Features

The following tracks features targeted for Firefox 36. For more information consult this wiki.

  • Ensure all features which do not require testing are flagged qe-verify-
  • Ensure all features which do require testing are flagged qe-verify+ and have an assigned QA Contact.
  • Ensure all features are either signed-off or disabled before release

XXX

  • Feature owner: Some One
  • Status: Unknown

<bugzilla> { 

"id":"",
"include_fields":"id,summary,status,resolution"

} </bugzilla>

Iterations

Firefox development now works on two-week iterations. This section tracks the bugs selected for development in these iterations. Please consult the Iteration Dev Walkthrough and the Firefox/IterativeDevelopment page for more information on this process.

  • Ensure all bugs have qe-verify status indicated.
  • Ensure all qe-verify+ bugs have a QA Contact (The Feature Owner page can help here)
  • Ensure all bugs important to be verified are VERIFIED before the end of the next iteration
  • See the Firefox QA dashboard (link at the start of this page) for bug queries and status.

Stability

Reports

Bug Triage

Pushlog

The following tracks triage of the Mozilla pushlogs.

  • Review each bug in the pushlog and determine if the bug should be
    • A) To be tested by the QA community --> add the qe-verify+ flag to the bug and set the status-firefox36 flag to fixed
    • B) To be tested by a new volunteer --> add the [good first verify] whiteboard tag
    • C) To be tested by the bug reporter --> set the needinfo flag to ask the reporter to confirm the bug is fixed
    • D) Not to be tested at all --> add the qe-verify-flag
  • If you are unsure, set the needinfo flag to the Assignee or QA Contact and ask if the bug needs testing
  • Be sure to set the status-firefox36:fixed flag
  • Ignore any bugs for non-Desktop platforms, with in-testsuite coverage, or which address test failures

QA Needed Bugs

The following tracks bugs which have requests for urgent QA assistance.

  • Choose a bug to test from the list below
  • Read it and make sure you understand what information is needed (this may be steps or URLs to reproduce, a regression range, or something else altogether)
  • Use the latest build of this Firefox version and test until you've discovered the information necessary
  • Report this information to the bug and remove the qawanted keyword
  • Document the bug in the Completed list

<bugzilla> { 

"include_fields":"id,summary,status,qa_contact,component",
"keywords":"qawanted",
"product":"Firefox Core Toolkit",
"f1":"op_sys",
"o1":"nowordssubstr",
"v1":"Android Gonk BSD Solaris",
"f2":"cf_tracking_firefox35",
"o2":"nowordssubstr",
"v2":"---"

} </bugzilla>

Unconfirmed Bugs

The following tracks triage and testing of incoming, unconfirmed bug reports.

  • Choose a bug to test from the list below
  • Use the latest Nightly build to confirm the bug reproduces
  • Flag with reporter with needinfo if you cannot reproduce and need more information
  • Set the status to NEW and ensure the bug is in the appropriate component if you can reproduce the bug (provide as much detail about your testing as possible)
  • If it's a Firefox bug you may want to add the firefox-backlog? flag to it.

All Unconfirmed Bugs

<bugzilla> { 

 "include_fields":"id,summary,component,op_sys",
 "version":"36 branch",
 "status":"UNCONFIRMED"

} </bugzilla>

Unchanged Since One Week

<bugzilla> { 

 "include_fields":"id,summary,component,op_sys",
 "version":"36 branch",
 "status":"UNCONFIRMED",
 "chfieldto":"-1w"

} </bugzilla>

Unchanged Since One Month

<bugzilla> { 

 "include_fields":"id,summary,component,op_sys",
 "version":"36 branch",
 "status":"UNCONFIRMED",
 "chfieldto":"-1m"

} </bugzilla>

Orphaned Fixes

Recent fixes with no target milestone set

  • See "Ongoing work" section of the Firefox QA dashboard for a Bugzilla query.

Recent fixes with status-firefox36 missing.

https://bugzilla.mozilla.org/buglist.cgi?j_top=OR&f1=cf_status_firefox36&o3=equals&v3=---&o1=equals&resolution=FIXED&o2=equals&chfieldto=Now&query_format=advanced&f3=cf_status_firefox34&chfieldfrom=-2w&f2=cf_status_firefox35&v1=---&v2=---&product=Core&product=Firefox&product=Toolkit&target_milestone=Firefox%2036

Verification Needed Bugs

  • Review the bugs below in the Pending Triage section and select a bug you want to test
  • Make sure you understand the bug before you begin testing it
  • Test on a previously known broken build to make sure you can reproduce the bug
  • Test on the latest build from the branches which are fixed to confirm the bug no longer reproduces
  • Mark the bug VERIFIED FIXED if the bug appears to be fixed and set the appropriate status-firefox flag to verified
  • Reopen the bug if it still reproduces
  • See the Firefox QA dashboard for bug queries and numbers.

Fixed Bugs

The following tracks triage and testing of bugs fixed for this Firefox version.

  • Select a bug and review it to determine if it should be
    • A) To be tested by the QA community --> add the qe-verify+ flag to the bug and set the status-firefox35 flag to fixed
    • B) To be tested by a new volunteer --> add the [good first verify] whiteboard tag
    • C) To be tested by the bug reporter --> set the needinfo flag to ask the reporter to confirm the bug is fixed
    • D) Not to be tested at all --> add the qe-verify- flag
  • If you are unsure, set the needinfo flag to the Assignee or QA Contact and ask if the bug needs testing. If it looks like it may be or should be in the testsuite, add the testsuite? flag.
  • Be sure to set the status-firefox35:fixed flag
  • See the Firefox QA dashboard for bug queries and numbers.

Reported Bugs

The following tracks bugs reported through day-to-day testing.

  • Once per week, Softvision will send an email identifying their found issues
  • Make sure the issues are documented below
  • Review the NEW issues to make sure they are actionable
  • Actionable bugs can be flagged "firefox-backlog?" according to the backlog triage criteria
  • Any regressions should be nominated to track and be escalated to Release Engineering
  • Review the FIXED issues to make sure they are verified

(To be updated ...) <bugzilla> { 

"include_fields":"id,summary,status,resolution",
"id":""

} </bugzilla>

Milestones

Aurora Migration

Post-merge Work (as soon as updates are disabled):

  • [DONE] MM-CI - configure the mozilla-aurora_update testrun on MM-CI to use the auroratest channel (before first Aurora builds are generated)

Pre-sign-off Checks:

  • [DONE] Automation - document all Mozmill test failures (on functional tests) and memory usage regressions (on endurance tests - note that those are only run on nightly, so you look for regressions that happened there). Bugs for those are usually filed by the automation team, but check that they are on top of that.
  • [DONE] Features - all scoped features are signed off as ready for Aurora by owners. (If you haven't heard anything to the contrary, that is probably true, but make sure we don't know of blocking issues.)
  • [DONE] Bugs - document and review any reported bugs and issues, nominate them for tracking if serious (i.e. make sure all bugs needed to be tracked have at least a request for that up).
  • [DONE] Stability - review crash stats before sign-off to raise any red flags.
  • [DONE] L10n - contact L10n drivers (usually :Pike) to see if there are any known localization issues.

Post-sign-off Checks (as soon as updates are enabled again):

  • [DONE] MM-CI - configure the mozilla-aurora_update testrun on MM-CI to use the aurora channel
  • [DONE] Updates - previous versions of Aurora update to the new version (check a few old builds manually once next round of new builds is available)

35-specific steps around Dev Edition landing:

  • [DONE] 11/07 - MM-CI - configure the mozilla-aurora_update testrun on MM-CI to use the auroratest channel (before first Aurora builds are generated)
  • [DONE] 11/10 - MM-CI - configure the mozilla-aurora_update testrun on MM-CI to use the aurora channel


Automation

Testrun Reports
Report Test Failures
Add-ons
Functional
Remote
L10n
Updates
Endurance
Memory Usage Report
Platform Min Memory Usage Max Memory Usage Avg Memory Usage Net Compared to Firefox 32
Windows 32-bit
Mac OS X
Linux 32-bit
Linux 64-bit

Memory regression being tracked/investigated in: <bug number>

Features

Feature Owner Aurora Sign-off Status Potential Blockers
name someone
name

Updates

Platform Source Build Target Build Result
Windows
Mac OS X
Linux

Reported Bugs

<bugzilla> { 

"id":"",
"include_fields":"id,summary,resolution"

} </bugzilla>

Betas

Sign-off emails

  • For Beta 1 and RC, sign off on beta-cdntest, RelMan needs to give the "go" for pushing out
  • For Betas in between, directly send "go" for pushing when beta-cdntest is OK.

Known Mozmill Failures

<bugzilla> { 

"id":"",
"include_fields":"id,summary,resolution"

} </bugzilla>

Beta 1

Build Information

Sign-off Criteria

  • Mozmill functional automation ran with/without unexpected failures
  • Mozmill update automation ran with/without unexpected failures
  • X beta blocking regression, stability, or support issues discovered

Testing

Bugs Reported

<bugzilla> { 

"id":"858032, 942130, 947079, 1089682, 1090518,1096357, 1099151, 1120967, 1120980",
"include_fields":"id,summary,resolution"

} </bugzilla>

Beta 2

Build Information

Sign-off Criteria

  • Mozmill functional automation ran with/without unexpected failures
  • Mozmill update automation ran with/without unexpected failures
  • X beta blocking regression, stability, or support issues discovered

Testing

Bugs Reported

<bugzilla> { 

"id":"942130, 962560, 1010976, 1099151, 1110751, 1117708, 1117716, 1122458, 1122491, 1053783, 856034, 1124171",
"include_fields":"id,summary,resolution"

} </bugzilla>

Beta 3

Build Information

Sign-off Criteria

  • Mozmill functional automation ran with/without unexpected failures
  • Mozmill update automation ran with/without unexpected failures
  • X beta blocking regression, stability, or support issues discovered

Testing

Bugs Reported

<bugzilla> { 

"id":"495040, 797830, 858032, 962560, 1010976, 1052569, 1099151, 1124171, 1124666, 1125822, 1125823, 1125842",
"include_fields":"id,summary,resolution"

} </bugzilla>

Beta 4

Build Information

Sign-off Criteria

  • Mozmill functional automation ran with/without unexpected failures
  • Mozmill update automation ran with/without unexpected failures
  • X beta blocking regression, stability, or support issues discovered

Testing

Bugs Reported

<bugzilla> { 

"id":"1000108, 1008178, 1102642, 1126317",
"include_fields":"id,summary,resolution"

} </bugzilla>

Beta 5

Build Information

Sign-off Criteria

  • Mozmill functional automation ran with/without unexpected failures
  • Mozmill update automation ran with/without unexpected failures
  • X beta blocking regression, stability, or support issues discovered

Testing

Bugs Reported

<bugzilla> { 

"id":"962560, 985986, 1000097, 1045528, 1051837, 1052569, 1096632, 1099151, 1124666, 1125842, 1127858, 1127879, 1127884",
"include_fields":"id,summary,resolution"

} </bugzilla>

Beta 6

Build Information

Sign-off Criteria

  • Mozmill functional automation ran with/without unexpected failures
  • Mozmill update automation ran with/without unexpected failures
  • X beta blocking regression, stability, or support issues discovered

Testing

Bugs Reported

<bugzilla> { 

"id":"777680, 1127884, 1128971",
"include_fields":"id,summary,resolution"

} </bugzilla>

Beta 7

Build Information

Sign-off Criteria

  • Mozmill functional automation ran with/without unexpected failures
  • Mozmill update automation ran with/without unexpected failures
  • X beta blocking regression, stability, or support issues discovered

Testing

After decision to ship only to Mac and Linux, and leave Windows with b6:

Bugs Reported

<bugzilla> { 

"id":"942130, 947079, 1043280, 1052569, 1066036, 1096357, 1130394, 1130414",
"include_fields":"id,summary,resolution"

} </bugzilla>

Beta 8

Build Information

Sign-off Criteria

  • Mozmill functional automation ran with/without unexpected failures
  • Mozmill update automation ran with/without unexpected failures
  • X beta blocking regression, stability, or support issues discovered

Testing

Bugs Reported

<bugzilla> { 

"id":"972658, 1131594, 1131604",
"include_fields":"id,summary,resolution"

} </bugzilla>

Beta 9

Build Information

Sign-off Criteria

  • Mozmill functional automation ran with/without unexpected failures
  • Mozmill update automation ran with/without unexpected failures
  • X beta blocking regression, stability, or support issues discovered

Testing

Bugs Reported

<bugzilla> { 

"id":"827042, 836732, 843218, 905902, 1018882, 1018973, 1124666, 1132420, 1132423, 1132429, 1132433, 1132860, 1132910",
"include_fields":"id,summary,resolution"

} </bugzilla>

Beta 10

Build Information

Sign-off Criteria

  • Mozmill functional automation ran with/without unexpected failures
  • Mozmill update automation ran with/without unexpected failures
  • X beta blocking regression, stability, or support issues discovered

Testing

  • Manual
    • [DONE] Exploratory testing covering: Web Compatibility, Video & Audio playback, Youtube Flash Compatibility
    • [DONE] Investigation of bug 1134052 on multiple Windows Vista x86 machines
    • [DONE] Verification of bug 1119535 on Windows XP (since Youtube serves HTML5 again for Windows)
    • [DONE] Bug verification

Bugs Reported

<bugzilla> { 

"id":"827042, 856034, 947079, 962560, 1010976, 1096357, 1124666, 1132433, 1134210",
"include_fields":"id,summary,resolution"

} </bugzilla>

Release Candidate

Build Information

Sign-off Criteria

  • Mozmill functional automation ran with/without unexpected failures
  • Mozmill update automation ran with/without unexpected failures
  • X beta blocking regression, stability, or support issues discovered

Testing

  • Automated (configs for ondemand)
    • This is the 36.0 release build. We should test updates on beta-cdntest for a preliminary sign-off (Release Management must give the "go" for pushing to beta), then on beta after the push to beta, and then on the day before release (Monday), after a full sign-off and push to that channel, run tests on release-cdntest.
    • [DONE] Automation Functional tests (report)
    • [DONE] Updates on beta-cdntest channel (report)
    • [DONE] Updates on beta channel (report)

Build 2 (done because of the AMD crash being >1% in build1 on beta):

  • Manual
    • [DONE] Sanity testing covering: Basic Web Compatibility, investigation of AMD crash
  • Automated (configs for ondemand)
    • This is the 36.0 release build. We should test updates on beta-cdntest for a preliminary sign-off (Release Management must give the "go" for pushing to beta), then on beta after the push to beta, and then on the day before release (Monday), after a full sign-off and push to that channel, run tests on release-cdntest.
    • [DONE] Updates on beta-cdntest channel (report)
    • [DONE] Updates on beta channel (report)
    • [DONE] Updates on release-cdntest channel (report)
    • [DONE] Updates on release channel (report)

Bugs Reported

<bugzilla> { 

"id":"827042, 942130, 947079, 1052569, 1096357, 1105939, 1124666, 1132433, 1135052, 1135055",
"include_fields":"id,summary,resolution"

} </bugzilla>

Release 36.0

Build Information

  • See Release Candidate Build 2

Testing

  • Update Testing - see Release Candidate Build 2
  • Throttle Testing (see Walkthrough for details)
    • Used commands:
./throttle.sh https://aus3.mozilla.org/update/3/Firefox/35.0.1/20150122214805/WINNT_x86-msvc/en-US/release/Windows_NT%206.3/default/default/update.xml
./throttle.sh https://aus3.mozilla.org/update/3/Firefox/35.0.1/20150122214805/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/Darwin%2013.4.0/default/default/update.xml
./throttle.sh https://aus3.mozilla.org/update/3/Firefox/35.0.1/20150122214805/Linux_x86-gcc3/en-US/release/Linux%203.13.0-45-generic%20%28GTK%202.24.23%29/default/default/update.xml
./throttle.sh https://aus3.mozilla.org/update/3/Firefox/35.0.1/20150122214805/Linux_x86_64-gcc3/en-US/release/Linux%203.16.0-30-generic%20%28GTK%202.24.25%29/default/default/update.xml
    • Results
      • After release push (expected to be @ 25%), 2015-02-24: Win32: 25% - Mac: 24% - Linux32: 21% - Linux64: 23%
      • After disabling updates (expected to be @ 0%), 2015-02-27: Win32: 0% - Mac: 0% - Linux32: 0% - Linux64: 0%
      • After fully enabling updates (expected to be @ 100%), 2015-02-xx: Win32: % - Mac: % - Linux32: % - Linux64: %

36.0.1

Build Information

Sign-off Criteria

  • Mozmill functional automation ran with/without unexpected failures
  • Mozmill update automation ran with/without unexpected failures
  • X beta blocking regression, stability, or support issues discovered

Testing

  • Automated (configs for ondemand, build2)
    • [DONE] Automation Functional tests (report)
    • [DONE] Updates on release-localtest channel (report)
    • [DONE] Updates on release-cdntest channel (report)
    • [DONE] Updates on release channel (report)

Bugs Reported

<bugzilla> { 

"id":"947079,1052569,1096357,1113637,1113647,1113743,1137602,1137603",
"include_fields":"id,summary,resolution"

} </bugzilla>

36.0.3

Build Information

Sign-off Criteria

  • Mozmill functional automation ran with/without unexpected failures
  • Mozmill update automation ran with/without unexpected failures
  • X beta blocking regression, stability, or support issues discovered

Testing

  • Automated (configs for ondemand, build1)
    • [DONE] Automation Functional tests (report)
    • [DONE] Updates on release-localtest channel (report)
    • [DONE] Updates on release-cdntest channel (report)
    • [DONE] Updates on release channel (report)

Bugs Reported

<bugzilla> { 

"id":"",
"include_fields":"id,summary,resolution"

} </bugzilla>

36.0.4

Build Information

Sign-off Criteria

  • Mozmill functional automation ran with/without unexpected failures
  • Mozmill update automation ran with/without unexpected failures
  • No blocking regression, stability, or support issues discovered

Testing

  • Manual
    • [DONE] Verify security fixes
  • Automated (configs for ondemand, build1)
    • [DONE] Automation Functional tests (report)
    • [DONE] Updates on release-localtest channel (report)
    • [DONE] Updates on release-cdntest channel (report)
    • [DONE] Updates on release channel (report)

Bugs Reported

<bugzilla> { 

"id":"",
"include_fields":"id,summary,resolution"

} </bugzilla>

Retrieved from "https://mozilla.mediawiki.site/w/index.php?title=Releases/Firefox_36/Test_Plan&oldid=1063513"