Data Freshness

Intro

Data makes its way to the Faros reports and dashboards in two separate steps:

  1. Data Ingestion: Data from various source systems is ingested, standardized, and stored in the Faros graph either via external tools such as Jira, GitHub, etc or via real-time events. This data can be accessed via an extensive and flexible GraphQL API. At this stage, data is not yet able to be translated into Faros AI reports and dashboards.
  2. Workspace Sync: Periodically the normalized raw data is then translated (joined, aggregated, etc) and pushed to the Faros Reports tables for easy analytics. In order for data to appear in Faros AI modules and dashboards, a workspace sync will need to run.

Data Ingestion

As outlined above, a data sync occurs after Faros AI connects to your external tools (i.e. Jira, GitHub, etc) or via real-time events. In this phase, data is ingested, standardized, and stored in the Faros graph, but is not yet able to be translated into Faros AI reports and dashboards.

After you’ve connected to your sources (Jira, GitHub, etc) Faros AI will run your first data sync. After that, data syncs will run roughly every 6 hours (or you can customize your data sync to run at a different cadence by navigating to that source under Admin Settings > Data Control > Sources and updating the Schedule sync every (hours) field under Advanced Settings.

Customize Sync Frequency (under Advanced Settings)

Customize Sync Frequency (under Advanced Settings)

You can see data sync history and logs as well as manually kick off a data sync by navigating to Admin Settings > Data Control > Sources

View History for a Source

View History for a Source

View Logs

View Logs

Sync Latest Data

Sync Latest Data

Inspecting Data

Once your data sync runs, you can inspect your data by navigating to Dashboards > Query Ingested Data.
Expand your desired artifact type, for example tms_task, and select the fields you’d like to see (for example, name, UID, and refreshedAt). This will show you the data that Faros AI has ingested thus far. This helps you confirm that Faros AI has pulled the tasks, pull requests, or other artifact types that you’d like to see in your reports.

Query Ingested Data

Query Ingested Data

You can also navigate to Dashboards > View Ingested Data Schema to see how Faros AI standardizes the data for use in modules and dashboards. Zoom in to the chart on the right of View Ingested Data Schema to see the exact field mappings that Faros AI will utilize to build reports. You can learn more about the Faros AI schema here.

View Ingested Data Schema

View Ingested Data Schema

Workspace Sync

In order for data to appear in Faros AI modules and dashboards, a workspace sync will need to run. The workspace sync takes raw data and translates it into a form such that it can be used for Faros AI reports.

Faros AI modules and dashboards will not populate until the first workspace sync runs. If needed, you can manually kick off a workspace sync by navigating to Admin Settings > Data Control > Workspaces.

Kick off a workspace sync

Kick off a workspace sync

Once Faros AI is running, you can assume that data in Faros AI modules and dashboards will only contain data through the most recent workspace sync. The workspace sync runs on a set schedule, typically every 6 hours.

On the Workspaces page, you can see a reports table snapshot, which gives you information on the freshness of the data. The timestamp will show when the last successful sync began, which tells you the lower bound of data freshness (in other words, that data in your report will be from that time frame, if not more up-to-date.)

Query Reports Table

You can navigate to Dashboards > Query Reports Table to see all the reports tables and the fields they contain.

Query Reports Table

Query Reports Table

After selecting an artifact type

After selecting an artifact type

A note on team-specific reports

If you’d like Faros AI reports broken down by teams, you will need to assign team ownership. See this page for more details.

Once teams have been assigned, you will need to run a workspace sync in order for your reports to update accordingly. To do this, go to Admin Settings > Data Control > Workspaces, and kick off a workspace sync.

Kick off a workspace sync

Kick off a workspace sync