Guides
Start typing to search…

Common Airbyte Connection Examples

Running the following commands will import data from the given external source into your Faros application. To understand more about how these commands are created and to connect to other sources check out Importing Data in Hybrid Deployment Mode.

To get configuration options for the source, you can run:

docker run -it farosai/IMAGE:latest spec-pretty

To run a wizard to help create that configuration, you can run:

docker run -it farosai/IMAGE:latest airbyte-local-cli-wizard

Images are either airbyte-SOURCENAME-source or airbyte-faros-feeds-source, in which case you usually need to append a --feed argument with the desired feed name.
To configure the upload to Faros, you can use similar commands on the airbyte-faros-destination image.

📘

Prerequisites

Make sure you have the required bash, docker, jq, and tee installed.

The examples below pull and execute the Airbyte Local CLI bash script directly from its repo. If you prefer to check out the repo and execute from a local copy instead, here's the repo. The airbyte-local.sh script is all you need.

Azure Repos

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src 'farosai/airbyte-azure-repos-source:latest' \
  --src.access_token $AZURE_TOKEN \
  --src.organization "my-azure-org" \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "myazurerepossrc"

Bitbucket

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-bitbucket-source:latest \
  --src.token $BITBUCKET_TOKEN \
  --src.workspaces '["my-bitbucket-workspace-1","my-bitbucket-workspace-2"]' \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mybitbucketsrc"

Buildkite

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \  
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "buildkite-feed" \
  --src.feed_cfg.organization "my-buildkite-org" \
  --src.feed_cfg.token $BUILDKITE_TOKEN \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mybuildkitesrc"

Changeset

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "changeset-feed" \
  --src.feed_cfg.github_cfg.auth.kind "token" \
  --src.feed_cfg.github_cfg.auth.token $GITHUB_TOKEN \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mychangesetsrc"

CircleCI

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-circleci-source:latest \
  --src.token $CIRCLECI_API_TOKEN \
  --src.--src.project_slugs '["vcs-slug/org-name/repo-name-1", ...]' \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mycirclecisrc"

GitHub

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-github-source:latest \
  --src.authentication.type "token" \
  --src.authentication.personal_access_token "${GITHUB_TOKEN}" \
  --src.organizations '["apache","airbytehq"]' \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mygithubsrc"

GitLab

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "gitlab-feed" \
  --src.feed_cfg.token $GITLAB_TOKEN \
  --src.feed_cfg.repos_query_mode.query_mode "FarosGraph" \
  --src.feed_cfg.repos_query_mode.group_names '["groupid1", "groupid2"]' \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mygitlabsrc"

Jenkins

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-jenkins-source:latest \
  --src.server_url $JENKINS_SERVER_URL \ 
  --src.user $JENKINS_USER \
  --src.token $JENKINS_TOKEN \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "myjenkinssrc"

Jira

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-jira-source:latest \
  --src.url "https://xyz.atlassian.net" \
  --src.username "${JIRA_USER_EMAIL}" \
  --src.password "${$JIRA_TOKEN}" \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "myjirasrc"

Additional Resources

Create a GitHub App to Authenticate Faros to Multiple Organizations

You can create a GitHub App and install it on multiple GitHub organizations to enable read-only access for the Faros connector to the desired organizations and repositories.

  1. Create a GitHub App:

    • Navigate to your profile settings or organization settings, then go to Developer Settings → GitHub Apps → Create a new GitHub App.
    • On the creation page:
      • At the bottom, select "Any account" to allow your app to be installed on any user or organization.
      • The Webhook option is not required.
      • Set the following read-only permissions:
        • Repository permissions: Administration, Contents, Issues, Metadata, Projects, Pull Requests.
        • Organization permissions: Members.
        • For a detailed list of permissions, see the API token requirements.
      • Optional permissions for specific functionality:
        • To fetch SAML user information (connect GitHub users to Workday identities): read-only Organization permissions: Administration.
        • To fetch Copilot usage information: read and write Organization permissions: GitHub Copilot Business.

  2. Generate a Private Key:

    • After creating the app, go to its settings page and generate a private key. Save the private key along with the App ID, as they are required for the Faros GitHub connector setup.

  3. Install the GitHub App:

    • Install the app on the desired organizations.
    • During installation, you can choose to grant access to all repositories in an organization or only specific repositories per organization.

  4. Run the Faros GitHub connector:

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-github-source:latest \
  --src.authentication.type "app" \
  --src.authentication.app_id $GITHUB_APP_ID \
  --src.authentication.private_key "${$GITHUB_APP_PRIVATE_KEY}" \
  --src.organizations '["apache","airbytehq"]'
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mygithubsrc"

Create a GitLab PAT to Authenticate Faros

You can create a GitLab PAT to easily enable read-only access from the Faros connector to your desired projects.

  1. In the top-right corner of your GitLab account, select your avatar.
  2. Select Edit profile.
  3. On the left sidebar, select Access Tokens.
  4. Enter a name and set the expiry date to "no expiration" for the token.
    • The PAT only requires the following read-only permission:read_api
  5. Select Create personal access token.
  6. Run the Faros GitLab connector:
bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "gitlab-feed" \
  --src.feed_cfg.token $GITLAB_TOKEN \
  --src.feed_cfg.repos_query_mode.group_name "mygroup" \
  --src.feed_cfg.repos_query_mode.repositories '["repo1", "repo2"]' \
  --src.feed_cfg.repos_query_mode.query_mode "GitLabGroup" \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "mygitlabsrc"

Create a Jira PAT to Authenticate Faros

You can create a Personal Access Token in Jira to enable read-only access from the Faros connector.

  1. In Jira, select your profile picture at the top right of the screen, then choose Profile.
  2. Once you access your profile, select Personal Access Tokens in the left-hand menu.
  3. Select Create token.
    • Give your new token a name.
    • The integration user needs application access to Jira, the 'Browse Users' global permission, and the 'Browse Project' and 'View Development Tools' permissions for each project
    • Set the expiry date to "No Expiration"
  4. Click Create.
  5. Run the Faros Jira connector:
bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --src farosai/airbyte-jira-source:latest \
  --src.url $JIRA_URL \
  --src.token $JIRA_TOKEN \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --connection-name "myjirasrc"

Common Problems & Solutions

Self-signed certificate in certificate chain

You might run into this if you're running your own source system server, e.g., your own GitHub server.

You can disable certificate validation for the server by passing --src.feed_cfg.reject_unauthorized false (for the Faros Feeds source) or --src.reject_unauthorized false(other Faros sources).

Please make sure that the source supports this flag by running the help commands.

If the source doesn't support it you can temporarily work around it by passing --src-docker-options '-e NODE_TLS_REJECT_UNAUTHORIZED=0'. Reach out to us and we'll add support for the flag on the source.

Dealing with Out Of Memory (OOM) errors

In some cases when connectors are processing a large amount of data they might require additional memory. In such cases make sure your environment has enough free memory to offer (at least 16gb) and make sure the connector can utilize it. In the following example we are capping each container to 6144mb of memory and allowing Node.js application within each container to consume 6000mb:

bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) \
  --max-memory "6144m" \
  --src-docker-options "-e NODE_OPTIONS=--max_old_space_size=6000" \
  --dst-docker-options "-e NODE_OPTIONS=--max_old_space_size=6000" \     
  // ... additional connector arguments

Updated about 1 month ago