QA/Loop/Firefox 34/Aurora

Aurora

Firefox 34 Beta Readiness

Component	Tracking Bug	QA Contact	Health	Details
Client
Desktop	bug 971986	Anthony Hughes	[DONE]	No blockers for Beta
Mobile	bug 988276	Tony Chung	[DONE]	No blockers for Beta
Platform
Automation Support	pivotal tracker	Syd Polk	[DONE]	No blockers for Beta
MSISDN Support		Martijn Wargers	[DONE]	No blockers for Beta
Privileged API Support	bug 988285	Tony Chung	[DONE]	No blockers for Beta
Tokbox Support	---	Nils Ohlmeier	[DONE]	No blockers for Beta
WebRTC/H264 Support	bug 1004761	Nils Ohlmeier	[DONE]	No blockers for Beta
Services
Firefox Accounts		Richard Pappalardo	[DONE]	No blockers for Beta
Server APIs		Peter DeHaan	[DONE]	No blockers for Beta
Server Scalability	Roll-out Plan	James Bonacci	[DONE]	No blockers for Beta

Checklist

• [daily] Review dogfooding bugs to escalate new critical concerns and flag fixed bugs for verification
• [daily] Review uplift bugs to flag fixed bugs for verification
• [daily] Review results of Moztrap testruns for potential bugs
• Create a smoketest run for community testing
• Create a regression testrun for release sign-off
• Create a report of individual component readiness for Beta

Pre-uplift Sign-offs

The following bugs have patches landed on the Fig branch for testing prior to uplift to Aurora.

• bugs with [fig:verifyme] in the whiteboard will be verified before uplift
• bugs with [fig:wontverify] will not be verified before uplift by may still be verified after uplift
• bugs with [fig:verified] in the whiteboard have been tested and verified

There should be 0 bugs with [fig:verifyme] in the whiteboard before the uplift. <bugzilla> {

"id":"1073047,1073027,1072279,1071835,1070065,1070045,1069965,1069962,1069178,1068580,1068178,1067937,1067845,1067519,1066816,1066609,1066509,1066506,1066502,1066219,1065777,1065608,1065591,1065275,1065155,1065153,1065144,1065052,1062835,1062126,1061756,1060809,1059754,1059186,1055567,1055239,1055139,1053774,1049565,1048938,1047667,1047181,1047146,1045690,1044411,1042060,1035655,1035846,1035348,1032700,1017257,1008990,1002416,1000766,1000237,1000112,974873",

"f1":"flagtypes.name", "o1":"equals", "v1":"qe-verify+", "include_fields":"id,summary,qa_contact,whiteboard,status" } </bugzilla>

Uplifts

Pending Triage

<bugzilla> {

"include_fields":"id,summary,status",
"product":"Loop",
"f1":"status_whiteboard",
"o1":"substring",
"v1":"loop-uplift",
"f2":"flagtypes.name",
"o2":"notequals",
"v2":"qe-verify+",
"f3":"flagtypes.name",
"o3":"notequals",
"v3":"qe-verify-"

} </bugzilla>

Flagged for Verification

<bugzilla> {

"include_fields":"id,summary,status,cf_status_firefox35,cf_status_firefox34",
"product":"Loop",
"f1":"status_whiteboard",
"o1":"substring",
"v1":"[loop-uplift]",
"f2":"flagtypes.name",
"o2":"equals",
"v2":"qe-verify+"

} </bugzilla>

Dogfooding

Once the final uplifts occur on Aurora we'll start having Mozillians use Firefox Hello for their 1:1 meetings. Any bugs reported will be listed below. <bugzilla> {

"include_fields":"id,summary,status,last_change_time",
"product":"Loop",
"chfield":"[Bug creation]",
"chfieldto":"Now",
"chfieldfrom":"2014-10-06"

} </bugzilla>

Smoketesting

We need a Moztrap testrun for community to run, including the following tests:

• Verify you can view ToS/Privacy/Support docs
• Make a call in guest mode via copy
• Make a call in guest mode via email
• Make a voice only call
• Pop-out the conversation window during a call
• Mute/unmute audio/video during the call
• Unshare/reshare your devices during a call
• End a call and submit feedback
• Register a Firefox Account and sign in
• Manage your contacts
• Call someone in your contacts

Criteria:

• Call host should be using Firefox 34 on any of our Desktop platforms
• Caller should be using Firefox or Chrome on any of our Desktop platforms
• Tester should report the desktop platforms, browsers, and network environments of both host and caller when reporting a test failure

Regression Testing

Guest Mode Tests

• Conduct a 1:1 call via Copy
• Conduct a 1:1 call via E-mail with Thunderbird as the default email handler
• Conduct a 1:1 call via E-mail with Outlook as the default email handler
• Conduct a 1:1 call via E-mail with Mail.app as the default email handler
• Conduct a 1:1 call via E-mail with Gmail as the default email handler
• Conduct a 1:1 call via E-mail with Yahoo Mail as the default email handler
• Verify entering do-not-disturb mode changes the button appearance and prevents receiving a call
• Verify entering available mode changes the button appearance and resumes the ability to receive a call
• Verify the conversation window can be popped out and in without affecting the call
• Verify the conversation window can be minimized and maximized without affecting the call
• Verify you can view the Terms of Service via the link in the panel on first run
• Verify you can view the Privacy Policy via the link in the panel on first run
• Verify you can view the Support documentation via a link
• Verify you can sign in to a Firefox Account via the link and/or the gear menu, username should appear in the panel and the icon should turn blue momentarily
• Verify you can view your account information via the gear icon
• Verify you can sign out of your account via the gear icon
• Verify navigating away from the call page should be treated as a disconnect
• Verify you can submit various types of feedback when the call ends
• Verify you see an error notification when the network is disconnected
• Verify you can generate a call URL and have a call when the network is reconnected
• Move the button into the customize panel and make sure the panel renders properly and is functional
• Verify notifications of various failure states
• Try conducting a call in private browsing mode

Standalone Page Tests

• Loading a call URL using an unsupported browser (for example Internet Explorer and Safari) should show an "Incompatible Browser" page with a link to download Firefox
• Loading a call URL using a supported browser should load the call page and check the following:
  • The URL displayed on page should match the URL you loaded
  • You should be able to initiate a video call
  • You should be able to initiate a voice-only call
  • You should be able to mute and unmute the video during the call
  • You should be able to mute and unmute the audio during the call
  • You should be able to submit feedback about the call once the call is terminated
  • If the build is localized, so too should the page
• Trying to start a call for an invalid call URL like https://call.mozilla.com/#call/invalid results in an error page
• Verify notifications of various failure states
• Try initiating a call in private browsing mode
• Verify you can view the Privacy/Support/ToS documentation via a link

Platform Coverage

Call Hosts

• Firefox 34 en-US on Windows XP on ethernet
• Firefox 34 de on Windows Vista on wifi
• Firefox 34 es-ES on Windows 7 on ethernet
• Firefox 34 zh-TW on Windows 8 on wifi
• Firefox 34 ru on Windows 10 on ethernet
• Firefox 34 en-US on Mac OS X 10.9 on wifi
• Firefox 34 ja-JP on Mac OS X 10.10 on ethernet
• Firefox 34 en-US on Ubuntu 14.04 on wifi
• Firefox 34 he on Ubuntu 12.04 on ethernet

Callers

Firefox

• Firefox 32 en-US on Windows XP on ethernet
• Firefox 33 de on Windows 8 on wifi
• Firefox 34 ja-JP Mac OS X 10.9 on ethernet
• Firefox 35 zh-TW Ubuntu 14.04 on wifi
• Firefox 32 en-US on Android 4.4 on 3g
• Firefox 35 en-US on Android 4.3 on wifi
• B2G 2.0 en-US on wifi
• B2G 2.0 en-US on 3g

Chrome

• Windows XP on ethernet
• Windows 7 on wifi
• Mac OS X 10.7 on ethernet
• Ubuntu 14.04 on wifi
• Android 4.4 on wifi
• Android 4.4 on 3g

Internet Explorer

• Windows XP on WiFi
• Windows Vista on Ethernet
• Windows 8.1 on WiFi
• Windows 10 Preview on Ethernet

Safari

• Mac OS X 10.9 on WiFi
• Mac OS X 10.10 on Ethernet
• iOS 8 on 3g

Opera

• Windows 7 on ethernet
• Windows 8.1 on wifi
• Mac OS X 10.8 on ethernet
• Ubuntu 14.04 on wifi
• Android 4.4 on 3g

Android Browser

• Android 4.4 on wifi
• Android 4.4 on 3g

Exploratory Testing

The following should be tested on all platforms (Windows, Mac, and Linux):

• Try crashing the browser during a call and see if you can restart the call using the same URL
• Try moving the call tab to another browser window
• Try moving the call window to another screen
• Try changing the screen resolution during a call
• Try switching between speakers and headphones during a call
• Try switching between an integrated camera to a USB camera during a call
• Try testing in networks with different firewall applications installed
• Test primary use cases in e10s mode
• Test primary use cases in win64 builds

End-to-End Automation

• [ON TRACK] Automated test is running locally (2014-09-29: ewong)

Bug Triage

• Need to verify all high-risk fixes are flagged for verification
• Need to verify all flagged fixes are verified
• Need to verify that there are no blocking unresolved bugs remaining

Health

• Desktop
• Mobile
• Platform
• Automation
• Server
• Services