Releases/Firefox 31/Test Plan

From New wiki
Jump to navigation Jump to search

Schedule

Please consult the Rapid Release Calendar for more information.

  • 2014-03-17: Firefox 31 arrives on Nightly
  • 2014-04-28: Firefox 31 merges to Aurora
  • 2014-05-02: Firefox 31 Aurora updates enabled
  • 2014-06-09: Firefox 31 merges to Beta
  • 2014-06-12: Firefox 31 Beta 1 released

Meetings

Meeting Purpose When Where
Crashkill Raise stability concerns with the Crashkill team 10:00 PST on Monday Vidyo: Stability, IRC: #crashkill
Channel Status Raise quality concerns with the Release Management team 10:00 PST on Tuesday & Thursday Vidyo: ReleaseCoordination, IRC: #planning

Checklist

Priority Check Description Frequency Last Checked
1 Stability Review the stability reports to identify new/rising crash signatures Daily 2014-06-18
2 Pushlog Review the pushlog to identify fixed bugs needing verification Daily 2014-06-19
3 QA Needed Bugs Review and assign qawanted bugs for investigation At least twice a week 2014-06-03
4 Unconfirmed Bugs Triage and test unconfirmed bug reports At least once a week 2014-06-09
5 Verification Needed Bugs Test verifyme bugs to confirm fixed bugs are fixed At least once a week 2014-05-05
6 Reported Bugs Document and escalated reported bugs as necessary Every Monday 2014-06-03
7 Features Ensure features are owned and on track for release Once a week 2014-05-21
8 Automation Ensure automation bugs are owned and on track for resolution Once a week 2014-06-03

Features

Please consult the feature testing walkthrough for more information.

Generational Garbage Collection (lizzard)

<bugzilla> { 

"blocked":"619558",
"include_fields":"id,summary,status,resolution"

} </bugzilla>

HTTPS for Yahoo Searches (Camelia Badau)

<bugzilla> { 

"f1":"blocked",
"o1":"equals",
"v1":"994248",
"include_fields":"id,summary,status,resolution"

} </bugzilla>

In-content Preferences (Camelia Badau)

<bugzilla> { 

"f1":"blocked",
"o1":"equals",
"v1":"738796",
"include_fields":"id,summary,status,resolution"

} </bugzilla>

Mozilla PKIX (Matt Wobensmith)

<bugzilla> { 

"f1":"short_desc",
"o1":"anywordssubstr",
"v1":"mozilla::pkix",
"include_fields":"id,summary,status,resolution"

} </bugzilla>

New Tab Page Search (Petruta Rasa)

<bugzilla> { 

"f1":"blocked",
"o1":"equals",
"v1":"962490",
"include_fields":"id,summary,status,resolution"

} </bugzilla>

New Tab Page Tiles (Cornel Ionce)

<bugzilla> { 

"component":"New Tab Page",
"whiteboard":"[tiles]",
"include_fields":"id,summary,status,resolution"

} </bugzilla>

Plugin Whitelisting (Bogdan Maris)

<bugzilla> { 

"f1":"component",
"o1":"equals",
"v1":"Plugin Click-to-Activate Whitelist",
"include_fields":"id,summary,status,resolution"

} </bugzilla>

WebVTT (Alexandra Lucinet)

<bugzilla> { 

"f1":"blocked",
"o1":"equals",
"v1":"629350",
"include_fields":"id,summary,status,resolution"

} </bugzilla>

Iteration Development

Ensure these bugs have QA status as per Iterative Development guidelines.

<bugzilla> { 

"include_fields":"id,summary,status,qa_contact,whiteboard",
"f1":"status_whiteboard",
"o1":"anywordssubstr",
"v1":"s=it-31c-30a-29b s=it-32c-31a-20b s=it-33c-32a-31b"

} </bugzilla>

Stability

This section tracks bugs representing the most urgent stability issues. Consult this wiki for more information.

<bugzilla> { 

 "include_fields":"id,summary,component,op_sys,cf_status_firefox31",
 "f1":"op_sys",
 "o1":"nowordssubstr",
 "v1":"Android Gonk",
 "f2":"keywords",
 "o2":"anywordssubstr",
 "v2":"topcrash",
 "f3":"status_whiteboard",
 "o3":"nowordssubstr",
 "v3":"qa-",
 "f4":"cf_status_firefox31",
 "o4":"nowordssubstr",
 "v4":"--- unaffected verified wontfix"

} </bugzilla>

Bugs

QA Wanted Bugs

<bugzilla> { 

"include_fields":"id,summary,status,qa_contact,component",
"f1":"op_sys",
"o1":"nowordssubstr",
"v1":"Android Gonk BSD Solaris",
"f2":"cf_tracking_firefox31",
"o2":"nowordssubstr",
"v2":"---",
"f3":"keywords",
"o3":"anywordssubstr",
"v3":"qawanted",
"f4":"product",
"o4":"anywordssubstr",
"v4":"Firefox Core Toolkit"

} </bugzilla>

Unconfirmed Bugs

<bugzilla> { 

 "include_fields":"id,summary,component,op_sys",
 "version":"31 branch",
 "status":"UNCONFIRMED"

} </bugzilla>

Fixed Bugs

Review the pushlog and place each Desktop Firefox bug in one of the following categories:

  • A) Tested by the QA Team --> add the verifyme keyword to the bug and set the status-firefox31 flag to fixed
  • B) Tested by a new volunteer --> add the [good first verify] whiteboard tag
  • C) Tested by the bug reporter --> set the needinfo flag to ask the reporter to confirm the bug is fixed
  • D) Not tested at all --> add the [qa-] whiteboard tag

Tips:

  • If you are unsure, set the needinfo flag to the Assignee or QA Contact and ask if the bug needs testing
  • Be sure to set the status-firefox31:fixed flag
  • If the bug includes tests from a developer you can set in-testsuite+
  • Ignore any bugs for non-Desktop platforms, with in-testsuite coverage, or which address test failures
Untriaged Bugs

Reported Bugs

Ensure these bugs remain on track to resolve and are verified when fixed. <bugzilla> { 

"include_fields":"id,summary,status,resolution",
"id":"1003172,1003142,1003169,1005042,1005029,993359,993361,993402,993959,993974,994037,994063,995178,995193,1005038,1005394,1003155,993369,993373,993383,993339,993344"

} </bugzilla>

Bug Fix Verifications

  • Review the bugs below to identify a bug you want to test
  • Make sure you understand the bug before you begin testing it
  • Test on a previously known broken build to make sure you can reproduce the bug
  • Test on the latest build from the branches which are fixed to confirm the bug no longer reproduces
  • Mark the bug VERIFIED FIXED if the bug appears to be fixed and set the appropriate status-firefox flag to verified
  • Reopen the bug if it still reproduces

<bugzilla> { 

"include_fields":"id,summary,component,severity,op_sys",
"cf_status_firefox31":"fixed",
"keywords":"verifyme",
"f1":"op_sys",
"o1":"anywordssubstr",
"v1":"All,Windows,Mac,Linux",
"f2":"short_desc",
"o2":"nowordssubstr",
"v2":"Sora"

} </bugzilla>

Milestones

Aurora Migration

Pre-sign-off Checks:

  • [DONE] Automation - document all Mozmill test failures and memory usage regressions
  • [DONE] Features - all scoped features are signed off as ready for Aurora by owners
  • [DONE] Bugs - document and review any reported bugs and issues, nominate them for tracking if serious
  • [DONE] Stability - review crash stats before sign-off to raise any red flags
  • [DONE] L10n - contact :Pike to see if there are any known localization issues

Post-sign-off Checks:

  • [DONE] Updates - previous versions of Aurora update to the new version (check after updates live)
  • [DONE] MM-CI - configure the mozilla-aurora_update testrun on MM-CI to use the aurora channel

Automation

Testrun Reports
Report Test Failures
Add-ons 0 failing tests
Functional bug 999357: /testSessionStore/testUndoTab.js::testUndoTabFromContextMenu
bug 866599: /testPreferences/testPasswordSavedAndDeleted.js::testSaveAndDeletePassword
bug 904610: /restartTests/testAddons_changeTheme/test3.js::testChangedThemeToDefault
Remote bug 1003083 /testSecurity/testSecurityNotification.js::testSecNotification
bug /restartTests/testAddons_InstallAddonWithoutEULA/test2.js::testInstallAddonWithoutEULA
bug 992187 /restartTests/testAddons_InstallAddonWithoutEULA/test1.js::testInstallAddonWithEULA
bug 952034 /restartTests/testAddons_installFromFTP/test2.js::testAddonInstalled
L10n bug 1004827 /testAccessKeys/test1.js::testPrefWindowAccessKeys
Updates 0 failing tests
Endurance 0 failing tests
Memory Usage Report
Platform Min Memory Usage Max Memory Usage Avg Memory Usage Net Compared to Firefox 30
Windows 32-bit 78 MB 99 MB 95 MB +14 MB [17%]
Mac OS X 133 MB 140 MB 135 MB +17 MB [14%]
Linux 32-bit 90 MB 91 MB 90 MB +11 MB [14%]
Linux 64-bit 124 MB 125 MB 125 MB +15 MB [14%]

Memory regression being tracked/investigated in bug 1004608

Features

Feature Owner Aurora Sign-off Status Potential Blockers
Newtab page Tile Grid Cornel Ionce Ready for Aurora N/A
HTTPS for Yahoo Searches Camelia Badau Ready for Aurora N/A
In-content preferences Camelia Badau Ready for Aurora N/A
Search field in Newtab page Petruta Rasa Ready for Aurora N/A
WebVTT Alexandra Lucinet Ready for Aurora N/A
Plugin whitelist Bogdan Maris ON TRACK N/A

Updates

Platform Source Build Target Build Result
Windows Firefox 30.0a2 en-US 2014-04-06 Firefox 31.0a2 en-US 2014-05-02 PASS
Firefox 30.0a2 eu 2014-04-28 Firefox 31.0a2 eu 2014-05-02 PASS
Mac OS X Firefox 30.0a2 en-US 2014-04-10 Firefox 31.0a2 en-US 2014-05-02 PASS
Firefox 30.0a2 as 2014-04-25 Firefox 31.0a2 as 2014-05-02 PASS
Linux Firefox 30.0a2 en-US 2014-04-01 Firefox 31.0a2 en-US 2014-05-02 PASS
Firefox 31.0a2 zh-TW 2014-04-29 Firefox 31.0a2 zh-TW 2014-05-02 PASS with issues [1]

1. some locales are not updating passed Firefox 31.0a2 2014-05-01

Reported Bugs

<bugzilla> { 

"id":"1003155,1005394",
"include_fields":"id,summary,resolution"

} </bugzilla>

Beta 1

Read the Beta Coordination Guide for more info

Build1 (20140610163407)
Automated Testing

<bugzilla> { 

"id":"1024141"

} </bugzilla>

Manual Testing
  • [DONE] Smoketest new features
  • [DONE] Verify no HSB builds were generated
  • [DONE] Verify notice is shown to beta users when we turn telemetry on by default (bug 981689)

<bugzilla> { 

"id":"1023887,1000097,1005042,1007993,1008929"

} </bugzilla>

Beta 2

Read the Beta Coordination Guide for more info

Build1 (20140616143923)
Automated Testing
Manual Testing
Bugs

<bugzilla> { 

"id":"1026582,1026738",
"include_fields":"id,summary,status,resolution"

} </bugzilla>

Beta 3 [SKIPPED DUE TO BUG 1027983]

Read the Beta Coordination Guide for more info

Build1 (20140619131915)
Automated Testing
Manual Testing
Bugs

<bugzilla> { 

"id":"1028183,1028230,1028237,1027983,1028035",
"include_fields":"id,summary,status,resolution"

} </bugzilla>

Beta 4

Read the Beta Coordination Guide for more info

Build1 (20140623175014)
Automated Testing
Manual Testing
Bugs

<bugzilla> { 

"id":"1029480,1028230,1028237,1027983,1028035,1029480",
"include_fields":"id,summary,status,resolution"

} </bugzilla>

Beta 5

Read the Beta Coordination Guide for more info

Build1 (20140626181429)
Automated Testing
Manual Testing
Bugs

<bugzilla> { 

"id":"1027103",
"include_fields":"id,summary,status,resolution"

} </bugzilla>

Beta 6

Read the Beta Coordination Guide for more info

Build2 (20140630185627)
Automated Testing
Manual Testing
Bugs

<bugzilla> { 

"id":"1032770",
"include_fields":"id,summary,status,resolution"

} </bugzilla>

Beta 7

Read the Beta Coordination Guide for more info

Build1 (20140703154127)
Automated Testing
Manual Testing
Bugs

<bugzilla> { 

"id":"1034592",
"include_fields":"id,summary,status,resolution"

} </bugzilla>

Beta 8

Read the Beta Coordination Guide for more info

Build1 (20140707160635)
Automated Testing
Manual Testing
Bugs

<bugzilla> { 

"id":"1035776,1035803,1035835",
"include_fields":"id,summary,status,resolution"

} </bugzilla>

Beta 9

Read the Beta Coordination Guide for more info

Build1 (20140710141843)
Automated Testing
Manual Testing
  • Exploratory testing for stability issues - PASS
Bugs

<bugzilla> { 

"id":"",
"include_fields":"id,summary,status,resolution"

} </bugzilla>

Release Candidate 1

Read the Beta Coordination Guide for more info

Build1 (20140714151536)
Automated Testing

Note: keep an eye on addon hotfix bugs to verify them asap: https://bugzilla.mozilla.org/buglist.cgi?f1=blocked&list_id=10737327&o1=equals&resolution=---&resolution=FIXED&query_format=advanced&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&v1=928173

Sign-off for Beta

Verifications
  • bug 1038282 Remove sw during beta-release migration for Firefox 31, keep hsb - PASS

Sign-off for Release

Verifications
  • bug 1028972 crash in xul.dll@0x8e13cd | xul.dll@0x1ab44c | xul.dll@0x126047 | xul.dll@0x19effe | xul.dll@0x1c8403 | _MD_CURRENT_THREAD | PR_Unlock | xul.dll@0x11d2d4 | xul.dll@0x1c844d | xul.dll@0x34a70d
  • bug 988549 startup crash in _VEC_memzero | _VEC_memzero - N/A (latest change is NOT in Firefox 31 RC build 1)

Fixes landed on RC build (still need verification)

- https://bugzilla.mozilla.org/show_bug.cgi?id=812695 - [Read comment #414] [D2D] Text Rendering Issues on Windows 7 with Platform Update KB2670838 (MSIE 10 Prerequisite) or on Windows 8.1 ==> this is a rather old rendering issue affecting some Radeon cards. Going through the bug I could not find any Radeon model that we have available. There is only one machine with an AMD Radeon HD 3000, which may have been affected, but we haven't tried it yet.

- https://bugzilla.mozilla.org/show_bug.cgi?id=1034327 - Memory leak with TURN relay ==> according to Nils: "This was caught in the WebRTC QA lab and can be verified with that as well. Additionally manual verification has been done and can be done as needed."


Release Candidate 2

Read the Beta Coordination Guide for more info

Build1 (20140716183446)
Automated Testing
Manual Testing

Noteable Issues

  • bug 1015341, bug 1035534, bug 1035537 and bug 1035529 all related to Movie Toolbar (bandoo) and combine to be our #7 topcrash
    • Bandoo has been contacted and we'll be tracking this in Release (They are working on a fix)
  • bug 984033 is an OOM that is now #4, correlated with Crossrider framework add-ons
    • 0 crashes in Nightly since the patch landed in mozilla-central on July 17
    • too late for Release but it should be uplifted to 32 and might be good to consider as a ridealong if we do a .point release
  • bug 1012924 (experiments shutddown crash) is #2/#3 (depending on specific report) after OOM|small, competing with processMarkStackTop
    • this should be gone when we release (just like Fx30) but raises concerns about running experiments through the entire Beta cycle (perhaps disable it in the last Beta until stability settles down)
  • New tab page issues (etherpad) -- do not block as they are also reproducible in Fx 30 (ie. not regressions from backing out New Tab Page Tiles)
Retrieved from "https://mozilla.mediawiki.site/w/index.php?title=Releases/Firefox_31/Test_Plan&oldid=999265"