Similar to Faros Managed Sources, Airbyte connections can be run in either a full sync mode or an incremental sync. Full syncs do a complete refresh of data each time the Airbyte connection is run, while incremental syncs only updates the new and changed data. Depending on selected Airbyte Deployment Option you chose on of the following will apply:
To run an Airbyte incrementally there needs to be a notion of state to determine what should be updated. For users using the cli this is in the form of a state file. An Airbyte connection will do an incremental sync if it can find a matching state file.
- Append the argument
—state myConnectionNameState.jsonto your cli invocation
- Keep this file name the same for each run of that Airbyte connection
- Make sure each Airbyte connection has its own state json file
- Always run your Airbyte connection in the folder that houses your state file
- If you wish to do a full sync, delete the state file
Similar to manually running the cli command, an Airbyte connection will do an incremental sync if it can find a matching state file.
- Follow naming conventions and notes from section above
- In your scheduled job, after the Airbyte connection has run, make sure you persist the file somewhere permanent
- In your schedule job, before the Airbyte connection has run, make sure you pull the file back into the execution folder and that the name matches the cli state argument
When setting up the streams for your connection, set the sync mode. Choose between
Full refresh and
incremental. The Destination can be set to Append for both.
It’s possible some streams may not support incremental syncs. This just means those particular models will always be fully synced. Leave those as they are and set all streams that support it to use incremental mode.
If you ever want to force full sync of your data, in your connection select the
refresh your data option. This will do a complete re-sync of all data, then continue updating data as determined by the sync modes of your source.
Updated 14 days ago