Guides

Common Airbyte Connection Examples

Suggest Edits

Running the following commands will import data from the given external source into your Faros application. To understand more about how these commands are created and to connect to other sources check out Importing Data in Hybrid Deployment Mode.

To get configuration options for the source, you can run:

docker run -it farosai/IMAGE:latest spec-pretty

To run a wizard to help create that configuration, you can run:

docker run -it farosai/IMAGE:latest airbyte-local-cli-wizard

Images are either airbyte-SOURCENAME-source or airbyte-faros-feeds-source, in which case you usually need to append a --feed argument with the desired feed name.
To configure the upload to Faros, you can use similar commands on the airbyte-faros-destination image.

šŸ“˜

Prerequisites

Make sure you have the required bash, docker, jq, and tee installed.

The examples below pull and execute the Airbyte Local CLI bash script directly from its repo. If you prefer to check out the repo and execute from a local copy instead, here's the repo. The airbyte-local.sh script is all you need.

Azure Repos

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src 'farosai/airbyte-azure-repos-source:latest' \
  --src.access_token $AZURE_TOKEN \
  --src.organization "my-azure-org" \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "myazurerepossrc"

Bitbucket

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "bitbucket-feed" \
  --src.feed_cfg.workspaces '["my-bitbucket-workspace-id-1", ...]' \
  --src.feed_cfg.token $BITBUCKET_TOKEN \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mybitbucketsrc"

Buildkite

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \  
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "buildkite-feed" \
  --src.feed_cfg.organization "my-buildkite-org" \
  --src.feed_cfg.token $BUILDKITE_TOKEN \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mybuildkitesrc"

Changeset

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "changeset-feed" \
  --src.feed_cfg.github_cfg.auth.kind "token" \
  --src.feed_cfg.github_cfg.auth.token $GITHUB_TOKEN \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mychangesetsrc"

CircleCI

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-circleci-source:latest \
  --src.token $CIRCLECI_API_TOKEN \
  --src.project_names '["vcs-slug/org-name/repo-name-1", ...]' \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mycirclecisrc"

GitHub

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "github-feed" \
  --src.feed_cfg.auth_cfg.auth "token" \
  --src.feed_cfg.auth_cfg.personal_access_token $GITHUB_TOKEN \
  --src.feed_cfg.org_repo_list '["my-github-org-1/*", "my-github-org-2/repo-1", ...]' \
  --src.feed_cfg.repos_query_mode.query_mode "FarosGraph" \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mygithubsrc"

GitLab

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "gitlab-feed" \
  --src.feed_cfg.token $GITLAB_TOKEN \
  --src.feed_cfg.repos_query_mode.query_mode "FarosGraph" \
  --src.feed_cfg.repos_query_mode.group_names '["groupid1", "groupid2"]' \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mygitlabsrc"

Jenkins

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-jenkins-source:latest \
  --src.server_url $JENKINS_SERVER_URL \ 
  --src.user $JENKINS_USER \
  --src.token $JENKINS_TOKEN \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "myjenkinssrc"

Jira

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "jira-feed" \
  --src.feed_cfg.domain $JIRA_SERVER_URL \
  --src.feed_cfg.personal_access_token $JIRA_TOKEN \
  --src.feed_cfg.projects_query_mode.query_mode "FarosGraph" \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "myjirasrc"

Additional Resources

Create a GitHub App to Authenticate Faros to Multiple Orgs

You can create a GitHub App and install it on your multiple GitHub organizations to easily enable read-only access from the Faros connector to your desired organizations and repositories.

  1. Navigate to either your profile's settings page or your organization's settings page, navigate to Developer Settings, and create a new GitHub App.
  • At the bottom of the creation page, select "Any account" to allow your app to be installed any user or organization.
  • The Webhook option is not required.
  • The app requires the following read-only permissions:
    • Repository permissions: Administration, Contents, Issues, Metadata, Projects, Pull Requests
    • Organization permissions: Members
    • For the full list of permissions review the connections doc.
  • Optional permissions for specific functionality:
    • To fetch SAML user information (to connect GitHub users to their Workday identities): read-only Organization permissions: Administration
    • To fetch copilot usage information: read and write Organization permissions: GitHub Copilot Business
  1. On your app's settings page, create a private key and a client secret. These values, along with the App Id and Client ID, are required by the Faros connector.
  2. Install the GitHub app on your desired organizations. You can either grant access to all repositories in an org, or only grant access to specific repositories per org.
  3. Run the Faros GitHub connector:
bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "github-feed" \
  --src.feed_cfg.auth_cfg.auth "app" \
  --src.feed_cfg.auth_cfg.app_id $GITHUB_APP_ID \
  --src.feed_cfg.auth_cfg.private_key $GITHUB_APP_PRIVATE_KEY \
  --src.feed_cfg.auth_cfg.app_cfg.auth "client" \
  --src.feed_cfg.auth_cfg.app_cfg.client_secret $GITHUB_APP_CLIENT_SECRET \
  --src.feed_cfg.auth_cfg.app_cfg.client_id $GITHUB_APP_CLIENT_ID \
  --src.feed_cfg.org_repo_list '["my-github-org-1/*", "my-github-org-2/repo-1", ...]' \
  --src.feed_cfg.repos_query_mode.query_mode "FarosGraph"
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mygithubsrc"

Create a GitLab PAT to Authenticate Faros

You can create a GitLab PAT to easily enable read-only access from the Faros connector to your desired projects.

  1. In the top-right corner of your GitLab account, select your avatar.
  2. Select Edit profile.
  3. On the left sidebar, select Access Tokens.
  4. Enter a name and set the expiry date to "no expiration" for the token.
    • The PAT only requires the following read-only permission:read_api
  5. Select Create personal access token.
  6. Run the Faros GitLab connector:
bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "gitlab-feed" \
  --src.feed_cfg.token $GITLAB_TOKEN \
  --src.feed_cfg.repos_query_mode.group_name "mygroup" \
  --src.feed_cfg.repos_query_mode.repositories '["repo1", "repo2"]' \
  --src.feed_cfg.repos_query_mode.query_mode "GitLabGroup" \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mygitlabsrc"

Create a Jira PAT to Authenticate Faros

You can create a Personal Access Token in Jira to enable read-only access from the Faros connector.

  1. In Jira, select your profile picture at the top right of the screen, then choose Profile.
  2. Once you access your profile, select Personal Access Tokens in the left-hand menu.
  3. Select Create token.
    • Give your new token a name.
    • The integration user needs application access to Jira, the 'Browse Users' global permission, and the 'Browse Project' and 'View Development Tools' permissions for each project
    • Set the expiry date to "No Expiration"
  4. Click Create.
  5. Run the Faros Jira connector:
bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "jira-feed" \
  --src.feed_cfg.domain $JIRA_URL \
  --src.feed_cfg.personal_access_token $JIRA_TOKEN \
  --src.feed_cfg.projects_query_mode.query_mode "FarosGraph"
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "myjirasrc"

Common Problems & Solutions

Self-signed certificate in certificate chain

You might run into this if you're running your own source system server, e.g., your own GitHub server.

You can disable certificate validation for the server by passing --src.feed_cfg.reject_unauthorized false (for the Faros Feeds source) or --src.reject_unauthorized false(other Faros sources).

Please make sure that the source supports this flag by running the help commands.

If the source doesn't support it you can temporarily work around it by passing --src-docker-options '-e NODE_TLS_REJECT_UNAUTHORIZED=0'. Reach out to us and we'll add support for the flag on the source.

Dealing with Out Of Memory (OOM) errors

In some cases when connectors are processing a large amount of data they might require additional memory. In such cases make sure your environment has enough free memory to offer (at least 16gb) and make sure the connector can utilize it. In the following example we are capping each container to 6144mb of memory and allowing Node.js application within each container to consume 6000mb: 

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --max-memory "6144m" \
  --src-docker-options "-e NODE_OPTIONS=--max_old_space_size=6000" \
  --dst-docker-options "-e NODE_OPTIONS=--max_old_space_size=6000" \     
  // ... additional connector arguments

Updated 6 months ago