B2G/QA/2.1

From New wiki
< B2G | QA
Jump to navigation Jump to search

Team Breakdown

Here is a table indicating QA contacts by major areas for Firefox OS 2.1.

Area QA Contacts
Management Tony Chung, Brian Huang
Release Tony Chung
Productivity Edward Chen, Tony Chung
Media No-Jun Park, Alison Shiue
System Front End Marcia Knous, Eric Chang
Communications Eric Chang, Johan Lorenzo
Performance Geo Mealer
Networking/Connectivity Alison Shiue, Eric Chang
System Platform Gerry Chang
Multimedia Platform Edward Chen
Devices Alison Shiue
Graphics No-Jun Park
Metrics Shing Lyu, Gerry Chang
Stability Paul Yang, Walter Chan

Feature Complete

Release Signoff Status

October 13, 2014

Health: Yellow

Feature Complete Criteria
  • All feature landing exceptions must be resolved
    • No. Mozilla features landed and verified, but partner (Tako) features are not tested. awaiting partner test apps for SHB, THA, and themes
  • Bugs with keyword late-l10n should be zero
    • Yes
  • Chipset vendor requirements or bugs needed to meet their FC date must be resolved
    • CAF FC for 2.1 met
  • Completed one functional test run
    • Yes. 2 testruns completed. completing 3rd this week
  • QA defines 2 areas of "functional areas" for exploratory testing and lead will send a "signoff" note when he feels its qualitatively ready -
  • Smoketest Greenness - Latest length of time that we haven't had smoketest regressions needs to be greater than five days
    • No, have at least 1 blocker a day
      • 10/13 - 62/63 (Bluetooth)
      • 10/10 - 38/39 (MMS)
      • 10/9 - 37/38 (MMS)
      • 10/8 - 37/38 (MMS)
      • 10/7 - 37/37 = pass
  • MTBF Test Run lasts for 30+ hours:
    • No. last week 10 hours
  • Perfomance team sign-off to ensure we have met the FC ready criteria
  • No. Performance acceptance - a few apps are over 1000ms, and have shown regressions from 2.0 to 2.1
    • bug 1079585 - [Meta] Meta bug for 2.1 release acceptance performance issues. 4 bugs untriaged, 1 minus’ed with no reason.
Additional QA notes
  • Still open blockers being tracked across features. See notes from Oct 8th
  • Lots of "user" papercut and usability bugs that are still being seen through dogfooding and exploratory. Notes include:
    • Graphic artifacts still appearing. long black lines in the UI
    • Instance where Rocketbar doesnt accept feedback
    • Dialer on idle, leaves a white screen effect
    • with a lot of apps open, it seems like performance lags on response. Reproducible on 512mb
    • apps aren't being killed on 2.1 when too many apps are being opened
    • dialer has call log issues, wont paint correctly
    • when trying to launch a link into browser, it just launches a blank page
    • sharing a bookmark via SMS, is very slow to load (~2-3 seconds)
    • when device is sleeping, waking up will have a blank page with rocketbar at top. need to pull battery. SHB is both on and off
  • Branch and bug verifications since Oct 9th
Remaining QA items
  • Complete 3rd functional testrun
  • Complete 2nd l10n run
  • 2.1 final bug bash
  • Work with Device QA on testing against shipping device. also untested features from Tako's feature list

September 18, 2014

Health: Yellow

  • [DONE] One functional test run complete
  • [ON TRACK] Two functional areas receive signoff via exploratory testing
  • [AT RISK] Smoketests are green for five or more days
    • One smoketest blocker open - bug 1069465
  • [AT RISK] MTBF test run is at least 30 hours
    • Last test run lasted 20 minutes due to bug 1064800

Functional Signoff Status

  • [AT RISK] Systems Front End
    • Various parts of the System Browser (bug 945259) have not passed basic feature verification (bug 1067573, bug 1069064)
  • [ON TRACK] Communications
  • [AT RISK] Graphics
    • Bad regressions (bug 1063046 & bug 1064918) being seen from Multi-Layer APZ feature (bug 967844)
    • Accelerated inverted color and greyscale rendering (bug 1016539) still not passing basic feature verification (bug 1066664)
  • [ON TRACK] Productivity
  • [ON TRACK] Multimedia Platform
  • [ON TRACK] Media
  • [ON TRACK] Networking/Connectivity
  • [ON TRACK] System Platform
  • [ON TRACK] Devices

Feature Landing

Release Signoff Status

Last Updated: September 5th, 2014

Health: Green

Functional Signoff Status

  • [DONE] Systems Front End
    • Note - conditional signoff, as we're signing off with what we have on FL, even though MozTrap test coverage & verifications are still in process
  • [DONE] Communications
    • Note - conditional signoff, as we're signing off with what we have on FL, even though verifications are still in process
  • [DONE] Graphics
    • Note - conditional signoff, as we're signing off with what we have on FL, even though verifications are still in process
  • [DONE] Productivity
    • Note - conditional signoff, as we're signing off with what we have on FL, even though verifications are still in process
  • [DONE] Multimedia Platform
  • [DONE] Media
  • [DONE] Networking/Connectivity
    • Note - conditional signoff, as we're signing off with what we have on FL, even though MozTrap test coverage & verifications are still in process
  • [DONE] System Platform
    • Note - conditional signoff, as we're signing off with what we have on FL, even though MozTrap test coverage & verifications are still in process
  • [DONE] Devices

Smoketest Status

Nominations

<bugzilla> 

   {
       "keywords": "smoketest",
       "cf_blocking_b2g": "2.1?",
       "resolution": "---",
       "include_fields": "id, assigned_to, status, resolution, component, summary"
   }

</bugzilla>

Blockers

<bugzilla> 

   {
       "keywords": "smoketest",
       "cf_blocking_b2g": "2.1+",
       "resolution": "---",
       "include_fields": "id, assigned_to, status, resolution, component, summary"
   }

</bugzilla>

Gaia UI Test Regressions Status

Nominations

<bugzilla> 

   {
       "keywords": "qablocker",
       "cf_blocking_b2g": "2.1?",
       "resolution": "---",
       "include_fields": "id, assigned_to, status, resolution, component, summary"
   }

</bugzilla>

Blockers

<bugzilla> 

   {
       "keywords": "qablocker",
       "cf_blocking_b2g": "2.1+",
       "resolution": "---",
       "include_fields": "id, assigned_to, status, resolution, component, summary"
   }

</bugzilla>

Test Coverage Status

Test Coverage Needed

<bugzilla> 

   {
       "whiteboard": "[2.1-feature-qa+]",
       "flagtypes.name": "in-moztrap?",
       "include_fields": "id, qa_contact, status, resolution, component, assigned_to, summary"
   }

</bugzilla>

Test Coverage Completed

<bugzilla> 

   {
       "whiteboard": "[2.1-feature-qa+]",
       "flagtypes.name": "in-moztrap+",
       "include_fields": "id, qa_contact, status, resolution, component, assigned_to, summary"
   }

</bugzilla>

Test Coverage Declined

<bugzilla> 

   {
       "whiteboard": "[2.1-feature-qa+]",
       "flagtypes.name": "in-moztrap-",
       "include_fields": "id, qa_contact, status, resolution, component, assigned_to, summary"
   }

</bugzilla>

Verification Status

Overall Verification Summary

<bugzilla type="count" display="bar"> 

   {
       "whiteboard": "[2.1-feature-qa+]",
       "x_axis_field": "status"
   }

</bugzilla>

Verification Needed

<bugzilla> 

   {
       "whiteboard": "[2.1-feature-qa+]",
       "status": "RESOLVED",
       "include_fields": "id, qa_contact, status, resolution, component, assigned_to, summary"
   }

</bugzilla>

Verification Complete

<bugzilla> 

   {
       "whiteboard": "[2.1-feature-qa+]",
       "status": "VERIFIED",
       "include_fields": "id, qa_contact, status, resolution, component, assigned_to, summary"
   }

</bugzilla>

Definitions

Health Notations

  • [ON TRACK] QA thinks the health is trending towards green, but signoff isn't complete yet
  • [AT RISK] QA thinks the health is trending towards yellow, but signoff isn't complete yet
  • [DONE] QA has signed off to indicate the health is green
  • [MISSED] QA has chosen to not sign off to indicate the health is red

FL Functional Team Criteria

The functional team FL health was determined against the following criteria:

  • MozTrap test coverage done?
  • All user stories verified to spec?
  • Smoketests green around your area?

The lowest health score out of each of the criteria above is used as the team's overall health status.

FC Functional Team Criteria

The functional team FC health is determined against the following criteria:

  • Have all features gone through a functional test run?
  • Did any risky feature areas get exploratory testing?
  • Were smoketests around your area green for 5+ days?
  • Did each outstanding FC signoff blocker get fixed?

The lowest health score out of each of the criteria above is used as the team's overall health status.

Retrieved from "https://mozilla.mediawiki.site/w/index.php?title=B2G/QA/2.1&oldid=1042374"