Quality Module

The Quality module provides a comprehensive overview of quality across the entire Software Development Lifecycle.

The module consolidates data from multiple sources, enabling insights into trends and correlations within:

• Code quality metrics, including vulnerabilities, bug detection, and code smells, from platforms like SonarQube or the Faros Events System. 
• Testing and Deployment processes within the CI/CD pipeline.
• Bugs and Issue Tracking from Task Management Systems, for instance, Jira.
• Incident reporting, covering outages or service, application, or infrastructure degradations, from tools like PagerDuty.

 

Accessing the Quality Module

---

The Quality Module is accessible under the Software Quality folder within Faros Reports. It is a premium add-on in the Faros platform. Under Menu > Dashboards, select View All and Faros Reports.  

 

Quality Module Dashboards

---

The Quality Module is composed of two dashboards, an Organization-level QA summary and a Team-level QA summary. Read on for details about each.

Organization QA Summary Dashboard

The Organization QA Summary dashboard is designed for senior QA leaders with cross-organizational responsibilities and provides comprehensive insights into the quality metrics of all teams within the organization.

The dashboard is structured into three primary sections: Impact, Trends by Sub-Organization Over Time, and Relationships. 

All metrics can be segmented by team, environment, and a specific time period. 

• The desired time period is set by the Date Filter at the top of the page, and defaults to the previous six months. 
• The desired team can be set using the Teams filter, and defaults to all teams. It is also possible to click on any of the team traces to drill down to a specific team
• The environment is set using the Test Environment filter.

The Impact section summarizes how QA positively influences the business, while also pointing out areas where QA can be improved:

The subsequent section emphasizes five distinct metrics, with each metric displayed across three columns. These columns respectively showcase the monthly trend, a line chart illustrating values based on the time range specified by the time filter, and a categorical bar chart delineating values across different teams. 

The five metrics are as follows:

The Relationships section at the bottom lets you visualize how different quality measures correlate with deployment success across your team’s application. Each application is represented by a colored bubble:

 

Team QA Summary Dashboard 

The Team QA Summary dashboard provides detailed, granular data specific to an individual team. It’s a great place for Engineering Managers to assess various quality metrics for applications and repositories owned by their teams.

It is composed of three sections focusing on Code Quality, Testing, and Incidents. Similar to the Organization Summary View, all metrics can be analyzed by team, environment and for a desired time window, with optional filters by Repository and Application.

The Code Quality section provides additional metrics at the PR level (i.e., coverage is computed on the incremental changes in each PR instead of on the entire repository) and repository level:

The Testing section shows additional information on test execution, pass rate, and counts over time and by repository:

The Incidents section breaks incidents down by severity and over time. It is a useful  summary of the effectiveness of QA in terms of averting problems and incidents in production, at the team level:

 

Setting up the Quality Module

---

The Quality Module is centered around 5 key metrics:

• Code Coverage
• Test Pass Rate
• Open Bugs per Developer
• Incidents
• Change Failure Rate

To compute each of these metrics, several data sources should be connected to the Faros platform, as detailed in the following sections.

Note: For most of the sources below, you will find handy validation dashboards in the Getting Started folder under Faros Reports. These dashboards are provided to all Faros customers and let you inspect the data coming from different sources such as Incident Management Systems or Task Management Systems.

Platform Prerequisites

The following should be set up on Faros in order to get the full functionality of the Quality module:

• Version control, VCS: The data will be used to link any QA issues to specific code changes.

• Organization and Team Members: This will allow Faros to know the teams in your organization, the reporting structure, and the size of each team.

• Team Attribution: To help determine which team each of the metrics is associated with

  • Repositories and Repository Ownership: For code coverage and testing
  • Board Ownership: For bugs
  • Applications and Application Ownership: For incidents and change failure rate

Code Coverage

Schema location: qa_CodeQuality Coverage

Sources: SonarCloud or SonarQube

Validation Dashboard: QA Code Quality Source Metrics

There are two ways of reporting Code Coverage and both types are used in the Quality Module: 

1. It can be calculated at the Repository-level to give the overall code coverage state, and this is typically done after a PR is merged during the project build. 
2. It can be calculated at the Pull Request-level showing the coverage just for the incremental changes, and this is computed during the Pull Request review. 

Each metric value should be associated with a particular commit in VCS and “Repository Ownership” will determine which team a commit belongs to.

Test Pass Rate

Schema location: qa_TestExecution Test Status (Success/Failure)

Sources: Xray or Events or Test Results Reporter

Validation Dashboard: QA Testing Source Metrics

A testing suite is typically run as part of the build process before deployment. The results of these tests should be reported to the Faros platform and include at a minimum: 

• a timestamp
• the resulting test status (Success or Failure)
• the particular commit in VCS corresponding to the code being tested. 

“Repository Ownership” will determine which team a commit belongs to.

Bugs per Developer

Schema location: tms_Task

Sources: Task Management, TMS

Validation Dashboard: TMS Source Metrics

Tasks of type “Bug” should be reported to the Faros platform. The “Board Ownership” association will determine what team the bugs belong to and the Org/Team setup will determine the number of developers on a team.

Incidents

Schema location: ims_Incident

Sources: Incident Management, IMS

Validation Dashboard: IMS Source Metrics

Incident counts are derived from incidents reported to the Faros platform. The “Application Ownership” association will determine what team the incidents belong to.

Change Failure Rate

Schema location: cicd_Deployment

Sources: Events

Validation Dashboard: Deployment Table

Change Failure Rate is calculated as the number of incidents per deployment. Setup for Incidents is already described above and Deployments should also be reported to Faros. This metric is only computed on high-severity incidents (Sev1 or Sev2) and for deployments to the Production environment.