Automated Test Coverage

Project Description

This project included those activities designed to convert, fix, automate, and ensure complete coverage for testing. It also included process improvement work so that test failures result in code back-out. Parts of this project had dependencies on the Marionette Improvements, Kitkat Emulator Improvements and Raptor Activation on Treeholder projects. Work on the project ended with the sunset of Firefox OS. The rest of the wiki remains for archival purposes.

Requirements:

  * Convert integration tests from GIP to GIJ
  * Fix intermittently failing tests
  * Fix tests that work on mulet but not on device
  * Create a Test Matrix to define which platform the tests are going to run on (device, mulet, emulator)
  * Make sure every single commit in gecko/gaia triggers automated tests
  * Automate MozTrap scenarios using GIJ when feasible
  * Actively maintain working Mulet tests with reporting on Treeherder 
  * Ensure code coverage

Automated Test Coverage

Milestone	Due Date	Status	Notes
Convert integration tests from GIP to GIJ	12/31/15	AT RISK	All but 14 have been converted. Engineers out on PTO.
Intermittently failing tests fixed	01/31/15	CHALLENGED	Engineers are disabling the tests that fail intermittently and will fix them before re-enabling. Need more engineering help with this.
Tests that work on mulet work on device	01/31/15	CHALLENGED	It's being reported that some of the flagged tests actually work on device. Need to reassess all. Challenged due to resources not being assigned.
Test Matrix created that defines platform(s) the tests are going to run on	12/31/15	AT RISK	Draft of matrix completed but there are still some tests that need to be evaluated for appropriate testing environment. Currently out for review with eng. leadership. Won't get this done until January.
Automation of tests according to Test Matrix	05/31/16	ON TARGET	Draft of matrix completed but there are still some tests that need to be evaluated for appropriate testing environment. Won't get this done until January.
Every single commit in gecko/gaia triggers automated tests	05/31/16	ON TARGET	There's still work that needs to be identified and scheduled.
MozTrap scenarios automated using GIJ when feasible	05/31/16	ON TARGET
Working Mulet tests actively maintained with reporting on Treeherder	05/31/16	ON TARGET
Code Coverage Tool developed	12/31/15	AT RISK	At risk for making targeted date. Originally targeted for end of Q4, but assistance at Mozlando didn't happen, so this will be delivered in Q1. New due date TBD.

Status Key

Color	Status	Key
	On Target	The project or deliverable is expected to meet its due date.
	Challenged	The project or deliverable is facing an issue that might cause it to miss its due date, but a “get well” plan has been developed to get it back on track.
	At Risk or Late	The project or deliverable is blocked or facing an issue that might cause it to miss its due date, and there’s no “get well” plan to get it back on track, or it is already late.
	Done	The project or deliverable has been completed.
	On Hold	The project or deliverable has been placed on hold.

---

Program Timeline

---

Project Scope

Bugs for GIP to GIJ conversions <bugzilla>

   {
       "whiteboard":"[gip to gij]",
       "feature-b2g":"2.6+",
       "status":["NEW","REOPENED","UNCONFIRMED","ASSIGNED","RESOLVED","VERIFIED","CLOSED"],
       "include_fields": "id, summary, status, target_milestone, resolution, assigned_to, depends_on, blocks, whiteboard, feature-b2g"
   }

</bugzilla>

Bugs for Tests that Run on Mulet but Fail on Device <bugzilla>

   {
       "whiteboard":"[FOD]",
       "feature-b2g":"2.6+",
       "status":["NEW","REOPENED","UNCONFIRMED","ASSIGNED","RESOLVED","VERIFIED","CLOSED"],
       "include_fields": "id, summary, status, target_milestone, resolution, assigned_to, depends_on, blocks, whiteboard, feature-b2g"
   }

</bugzilla>

Bugs for Tests that Fail Intermittently <bugzilla>

   {
       "whiteboard":"[intermittents]",
       "feature-b2g":"2.6+",
       "status":["NEW","REOPENED","UNCONFIRMED","ASSIGNED","RESOLVED","VERIFIED","CLOSED"],
       "include_fields": "id, summary, status, target_milestone, resolution, assigned_to, depends_on, blocks, whiteboard, feature-b2g"
   }

</bugzilla>

Bugs for Code Coverage Tool <bugzilla>

   {
       "whiteboard":"[CC] [CI]",
       "feature-b2g":"2.6+",
       "status":["NEW","REOPENED","UNCONFIRMED","ASSIGNED","RESOLVED","VERIFIED","CLOSED"],
       "include_fields": "id, summary, status, target_milestone, resolution, assigned_to, depends_on, blocks, whiteboard, feature-b2g"
   }

</bugzilla>

---

Dependency Tracking

---

Detailed Program Plan

Action Item	Engineering Owner	QA Owner	UX Owner	Bugzilla ID	Planned Done	Actual Done

---

Program Stakeholders

Role	Name	IRC
EPM
EM
PM
TL
UX
QA

• EPM = Engineering Program Manager
• EM = Engineering Manager
• PM = Product Manager
• TL = Tech Lead
• UX = User Experience
• QA = Quality Assurance

---

---