Data Freshness
Intro
Data makes its way to the Faros reports and dashboards in two separate steps:
- Data Ingestion: Data from various source systems is ingested, standardized, and stored in the Faros graph either via external tools such as Jira, GitHub, etc or via real-time events. This data can be accessed via an extensive and flexible GraphQL API. At this stage, data is not yet able to be translated into Faros AI reports and dashboards.
- Workspace Sync: Periodically the normalized raw data is then translated (joined, aggregated, etc) and pushed to the Faros Reports tables for easy analytics. In order for data to appear in Faros AI modules and dashboards, a workspace sync will need to run.
Data Ingestion
As outlined above, a data sync occurs after Faros AI connects to your external tools (i.e. Jira, GitHub, etc) or via real-time events. In this phase, data is ingested, standardized, and stored in the Faros graph, but is not yet able to be translated into Faros AI reports and dashboards.
After you’ve connected to your sources (Jira, GitHub, etc) Faros AI will run your first data sync. After that, data syncs will run roughly every 6 hours (or you can customize your data sync to run at a different cadence by navigating to that source under Admin Settings > Data Control > Sources and updating the Schedule sync every (hours) field under Advanced Settings.
You can see data sync history and logs as well as manually kick off a data sync by navigating to Admin Settings > Data Control > Sources
Inspecting Data
Once your data sync runs, you can inspect your data by navigating to Dashboards > Query Ingested Data.
Expand your desired artifact type, for example tms_task
, and select the fields you’d like to see (for example, name
, UID
, and refreshedAt
). This will show you the data that Faros AI has ingested thus far. This helps you confirm that Faros AI has pulled the tasks, pull requests, or other artifact types that you’d like to see in your reports.
You can also navigate to Dashboards > View Ingested Data Schema to see how Faros AI standardizes the data for use in modules and dashboards. Zoom in to the chart on the right of View Ingested Data Schema to see the exact field mappings that Faros AI will utilize to build reports. You can learn more about the Faros AI schema here.
Workspace Sync
In order for data to appear in Faros AI modules and dashboards, a workspace sync will need to run. The workspace sync takes raw data and translates it into a form such that it can be used for Faros AI reports.
Faros AI modules and dashboards will not populate until the first workspace sync runs. If needed, you can manually kick off a workspace sync by navigating to Admin Settings > Data Control > Workspaces.
Once Faros AI is running, you can assume that data in Faros AI modules and dashboards will only contain data through the most recent workspace sync. The workspace sync runs on a set schedule, typically every 6 hours.
On the Workspaces page, you can see a reports table snapshot, which gives you information on the freshness of the data. The timestamp will show when the last successful sync began, which tells you the lower bound of data freshness (in other words, that data in your report will be from that time frame, if not more up-to-date.)
Query Reports Table
You can navigate to Dashboards > Query Reports Table to see all the reports tables and the fields they contain.
A note on team-specific reports
If you’d like Faros AI reports broken down by teams, you will need to assign team ownership. See this page for more details.
Once teams have been assigned, you will need to run a workspace sync in order for your reports to update accordingly. To do this, go to Admin Settings > Data Control > Workspaces, and kick off a workspace sync.
Updated 24 days ago