Common Airbyte Connection Examples

Suggest Edits

Running the following commands will import data from the given external source into your Faros application. To understand more about how these commands are created and to connect to other sources check out Importing Data in Hybrid Deployment Mode.

📘

Prerequisites

Make sure you have the required bash, docker, jq, and tee installed.

The examples below pull and execute the Airbyte Local CLI bash script directly from its repo. If you prefer to check out the repo and execute from a local copy instead, here's the repo. The airbyte-local.sh script is all you need.

Azure Repos

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src 'farosai/airbyte-azure-repos-source:latest' \
  --src.access_token $AZURE_TOKEN \
  --src.organization "my-azure-org" \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "myazurerepossrc"

Bitbucket

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "bitbucket-feed" \
  --src.feed_cfg.workspaces '["my-bitbucket-workspace-id-1", ...]' \
  --src.feed_cfg.token $BITBUCKET_TOKEN \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mybitbucketsrc"

Buildkite

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \  
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "buildkite-feed" \
  --src.feed_cfg.organization "my-buildkite-org" \
  --src.feed_cfg.token $BUILDKITE_TOKEN \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mybuildkitesrc"

Changeset

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "changeset-feed" \
  --src.feed_cfg.github_cfg.auth.kind "token" \
  --src.feed_cfg.github_cfg.auth.token $GITHUB_TOKEN \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mychangesetsrc"

CircleCI

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-circleci-source:latest \
  --src.token $CIRCLECI_API_TOKEN \
  --src.project_names '["vcs-slug/org-name/repo-name-1", ...]' \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mycirclecisrc"

GitHub

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "github-feed" \
  --src.feed_cfg.auth_cfg.auth "token" \
  --src.feed_cfg.auth_cfg.personal_access_token $GITHUB_TOKEN \
  --src.feed_cfg.org_repo_list '["my-github-org-1/*", "my-github-org-2/repo-1", ...]' \
  --src.feed_cfg.repos_query_mode.query_mode "FarosGraph" \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mygithubsrc"

GitLab

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "gitlab-feed" \
  --src.feed_cfg.token $GITLAB_TOKEN \
  --src.feed_cfg.repos_query_mode.query_mode "FarosGraph" \
  --src.feed_cfg.repos_query_mode.group_names '["groupid1", "groupid2"]' \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mygitlabsrc"

Jenkins

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "jenkins-feed" \
  --src.feed_cfg.server_url $JENKINS_SERVER_URL \ 
  --src.feed_cfg.user $JENKINS_USER \
  --src.feed_cfg.token $JENKINS_TOKEN \
  --src.feed_cfg.jobs_query_mode.query_mode "All" \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "myjenkinssrc"

Jira

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "jira-feed" \
  --src.feed_cfg.domain $JIRA_SERVER_URL \
  --src.feed_cfg.personal_access_token $JIRA_TOKEN \
  --src.feed_cfg.projects_query_mode.query_mode "FarosGraph" \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "myjirasrc"

Additional Resources

Create a GitHub App to Authenticate Faros to Multiple Orgs

You can create a GitHub App and install it on your multiple GitHub organizations to easily enable read-only access from the Faros connector to your desired organizations and repositories.

  1. Navigate to either your profile settings page or your organization's settings page and create a new GitHub app.
  • At the bottom of the creation page, select "Any account" to allow your app to be installed any user or organization.
  • The Webhook option is not required.
  • The app requires the following read-only permissions:
    • Repository permissions: Administration, Contents, Issues, Metadata, Projects, Pull Requests
    • Organization permissions: Members
  1. On your app's settings page, create a private key and a client secret. These values, along with the App Id and Client ID, are required by the Faros connector.
  2. Install the GitHub app on your desired organizations. You can either grant access to all repositories in an org, or only grant access to specific repositories per org.
  3. Run the Faros GitHub connector:
bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "github-feed" \
  --src.feed_cfg.auth_cfg.auth "app" \
  --src.feed_cfg.auth_cfg.app_id $GITHUB_APP_ID \
  --src.feed_cfg.auth_cfg.private_key $GITHUB_APP_PRIVATE_KEY \
  --src.feed_cfg.auth_cfg.app_cfg.auth "client" \
  --src.feed_cfg.auth_cfg.app_cfg.client_secret $GITHUB_APP_CLIENT_SECRET \
  --src.feed_cfg.auth_cfg.app_cfg.client_id $GITHUB_APP_CLIENT_ID \
  --src.feed_cfg.org_repo_list '["my-github-org-1/*", "my-github-org-2/repo-1", ...]' \
  --src.feed_cfg.repos_query_mode.query_mode "FarosGraph"
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mygithubsrc"

Create a GitLab PAT to Authenticate Faros

You can create a GitLab PAT to easily enable read-only access from the Faros connector to your desired projects.

  1. In the top-right corner of your GitLab account, select your avatar.
  2. Select Edit profile.
  3. On the left sidebar, select Access Tokens.
  4. Enter a name and set the expiry date to "no expiration" for the token.
    • The PAT only requires the following read-only permission:read_api
  5. Select Create personal access token.
  6. Run the Faros GitLab connector:
bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "gitlab-feed" \
  --src.feed_cfg.token $GITLAB_TOKEN \
  --src.feed_cfg.repos_query_mode.group_name "mygroup" \
  --src.feed_cfg.repos_query_mode.repositories '["repo1", "repo2"]' \
  --src.feed_cfg.repos_query_mode.query_mode "GitLabGroup" \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mygitlabsrc"

Create a Jira PAT to Authenticate Faros

You can create a Personal Access Token in Jira to enable read-only access from the Faros connector.

  1. In Jira, select your profile picture at the top right of the screen, then choose Profile.
  2. Once you access your profile, select Personal Access Tokens in the left-hand menu.
  3. Select Create token.
    • Give your new token a name.
    • The integration user needs application access to Jira, the 'Browse Users' global permission, and the 'Browse Project' and 'View Development Tools' permissions for each project
    • Set the expiry date to "No Expiration"
  4. Click Create.
  5. Run the Faros Jira connector:
bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "jira-feed" \
  --src.feed_cfg.domain $JIRA_URL \
  --src.feed_cfg.personal_access_token $JIRA_TOKEN \
  --src.feed_cfg.projects_query_mode.query_mode "FarosGraph"
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "myjirasrc"

Additional Notes

📘

Self-signed certificate in certificate chain

You might run into this if you're running your own source system server, e.g., your own GitHub server.

You can disable certificate validation for the server by passing --src.feed_cfg.reject_unauthorized false (for the Faros Feeds source) or --src.reject_unauthorized false(other Faros sources).

Please make sure that the source supports this flag by running the help commands.

If the source doesn't support it you can temporarily work around it by passing --src-docker-options '--env NODE_TLS_REJECT_UNAUTHORIZED=0'. Reach out to us and we'll add support for the flag on the source.

Updated 6 days ago