B2G/QA/2.2/2015-04-27Status

From New wiki
< B2G | QA | 2.2
Jump to navigation Jump to search

V2.2 Highlight & Risk

  • Highlight
    • V2.2 FC moved to 2015/04/29.
    • NEXUS-5-L 5.1 PVT build MTBF 38.5 hrs & counting.
    • FLAME-KK 4.4 PVT build MTBF 34.42 hrs.
    • NEXUS-5-L 5.1 Regression test done.
    • 48 bugs with blocking=2.2+ http://mzl.la/1CDEaqi
    • 12 bugs with blocking=2.2? http://mzl.la/1CDEKV4

V2.2 Milestone

  • 2015/06/08 V2.2 CC
  • 2015/04/29 V2.2 Updated FC
  • 2015/04/06 V2.2 FC
  • 2015/02/23 V2.2 FL

V2.2 Testing Scope

<bugzilla> 

   {
       "cf_blocking_b2g":"2.2+",
       "cf_blocking_b2g_type":"contains",
       "include_fields": "id, assigned_to, summary, status, target_milestone"
   }

</bugzilla>

V2.2 Testing by Type

Smoke Testing

Flame-KK Green

  • No bug found blocking manual smoketests but we have couple automation bugs need to investigate.
  • New Bugs Breaking the Automation smoketests
    • [Investigating] test_clock_create_new_alarm.py - Bug 1138278- [Clock] After running test_clock_create_new_alarm.py, it should set the alarm time to 1 min more than the current time
    • [Investigating] [Not reproduced manually] test_ftu_skip_tour.py - Bug 1138278 - test_ftu_skip_tour should reflect new Send Data initial state and separate SIM- and network-available conditions
    • [Investigating] test_marketplace_packaged_app.py - Bug 1151172 - [marketplace-tests-gaia] "test_marketplace_search_and_install_app" is failing on b2g v2.2
    • [Not reproduced] test_sms_with_attachments.py - Bug 1151696 - [v2.2] Investigate test_sms_with_attachments.py failure . (Cannot reproduce locally)
    • [Not reproduced manually] test_music_artist_mp3.py - Bug 1149886 - using to replace at Music app results in unable to locate <span.list-song-index> error


Nexus-5-L 5.1 Red

  • Testing on PVT build from 2015/04/14
  • Cases blocking manual test
    • OTA not working
  • Manual - are available on Google Drive
  • Known issues - no FM radio support, no DSDS support and no UMS support.

Regression Testing

2nd Run (2015/04/14)

  • Result
    • 1119(total), 1090(passed), 8(failed), 21(invalid)
    • 0.007% fail rate
  • bugs
    • Total 5
    • http://mzl.la/1cYZZau (Bugs related to software home button, privacy or ui polish will be skipped for now)
  • Result
    • ​342(total), 334(passed), 5(failed), 3(invalid)
    • ​0.01% fail rate

1st Run (2015/03/03)

Functional Testing

Red

  • General
    • Details:
    • 2nd regression, NEXUS-5-L 1119(total), 8(failed) bugs http://mzl.la/1cYZZau
    • FLAME-KK 342(total), 5(failed) bugs http://mzl.la/1GifEOz
    • NEXUS-5-L MTBF Average: ​29.18 hrs
    • Flame-L, RD has just received complete code, but building fail.
  • System Platform (Gerry)
    • Health Status:
    • Details: 7 blockers
  • OTA/FOTA (Gerry)
    • Health Status:
    • Details:
      • 2.0/2.1 can’t upgrade to 2.2 via FOTA - Bug 1155035
      • 2.0 can upgrade to 2.2 via OTA with some issues
        • Bluetooth issue - Bug 1132388 & Bug 1134821
        • Bug 1153787 & Bug 1153778
      • 2.1 can’t upgrade to 2.2 via OTA - Bug 1151006
  • Ringtones (Alison)
    • Health Status: Green
    • Details: No blockers
  • MTP/UMS (Alison)
    • Health Status: Green
    • Details: No blockers
  • System Frontend bugs (Hermes)
    • Health Status: RED
    • Details: 7 blockers, 2 nom
  • L-porting V5.1 (Hermes)
    • Health Status: Green
    • Details: no blockers
  • CAF bugs (Eric)
    • Health Status: Red
    • Details: 3 new bluetooth bugs opened this week, RDs are working on them, http://mzl.la/1HWrTQa 28 in total
  • MUSIC (Eric)
    • Health Status: Yellow
    • Details: lots of regression recently http://mzl.la/1HWsSzS
      • dominic kuo/jim porter (cc Hema)
  • 256M RAM memory optimization for WFVGA, 5MP. (Teri)
    • Health Status: Green
    • Details:

Exploratory Test

Flame-KK & NEXUS-5-L

  • Regression bugs

Stability

NEXUS-5-L 5.1

  • Environment
    • Display DIM
    • Rest time with dummy case (30 sec)
    • Running tests ** 14 case http://bit.ly/1aWkJyY
    • NEXUS *4, ZTE *2

- Result

  • NEXUS-5-L 5.1
    • Current run - NEXUS-5-L 5.1 *4: 38.5 hrs and counting = [55+93(running)+93(running) +67(running)] / (4 devices + 4 crashes) = 38.5
    • Current run - ZTE-L 5.1 * 2: Just started
    • Previous run - NEXUS-5-L 5.1 *4: MTBF 34.42 hrs = (41.3 + 53.2 + 43.8 + 6.1)/4
    • NEXUS-5-L 5.1 Rest time: 10%
    • [script issue] OSError: [Errno 24] Too many open files.
    • [adb issue] https://bugzil.la/1154629


FLAME-KK 4.4

  • Environment
    • Display DIM
    • Rest time with dummy case (30 sec)
    • FLAME *9

- Result

  • FLAME-KK 4.4
    • LAME-KK 4.4 * 4: MTBF 25.81 hrs = 50.77/4.96/25.81 - max/min/avg

V2.2 Test Case Creation

On Track

Test Case Creation WIP

Test Case Creation DONE

  • [2.2-feature-qa+] & in-moztrap+ (19 bugs) --> WORK DONE
  • feature-b2g=2.2+ & in-moztrap+ (78 bugs) --> WORK DONE

V2.2 Testing Verification Status

Overall Verification Summary

Need Verification <bugzilla> 

   {
       "cf_qa_whiteboard": "[2.2-feature-qa+]",
       "status": "RESOLVED",
       "include_fields": "id, qa_contact, status, resolution, component, assigned_to, summary"
   }

</bugzilla>

Verification Done <bugzilla> 

   {
       "cf_qa_whiteboard": "[2.2-feature-qa+]",
       "status": "VERIFIED",
       "include_fields": "id, qa_contact, status, resolution, component, assigned_to, summary"
   }

</bugzilla>

Retrieved from "https://mozilla.mediawiki.site/w/index.php?title=B2G/QA/2.2/2015-04-27Status&oldid=1070447"