Common Airbyte Connection Examples

Running the following commands will import data from the given external source into your Faros application. To understand more about how these commands are created and to connect to other sources check out Importing Data in Hybrid Deployment Mode doc.

📘

Prerequisites

Download our Airbyte Local CLI script and make sure you have the required bash, docker, jq, and tee installed.

GitHub

./airbyte-local.sh \  
  --src farosai/airbyte-faros-feeds-source:latest \   
  --src.feed_cfg.feed_name github-feed \    
  --src.feed_cfg.auth_cfg.auth token \   
  --src.feed_cfg.auth_cfg.personal_access_token $GITHUB_TOKEN \   
  --src.feed_cfg.org_repo_list '["my-github-org-1/*", "my-github-org-2/repo-1", ...]'  \   
  --src.feed_cfg.repos_query_mode.query_mode FarosGraph \   
  --dst farosai/airbyte-faros-destination:latest \   
  --dst.edition_configs.edition cloud \   
  --dst.edition_configs.api_url https://prod.api.faros.ai \ 
  --dst.edition_configs.api_key $FAROS_API_KEY \   
  --dst.edition_configs.graph default \   
  --state github_state.json

Jira

./airbyte-local.sh \   
  --src farosai/airbyte-faros-feeds-source:latest \   
  --src.feed_cfg.feed_name jira-feed \   
  --src.feed_cfg.feed_path tms/jira-feed \   
  --src.feed_cfg.domain $JIRA_SERVER_URL \   
  --src.feed_cfg.api_token $JIRA_TOKEN \   
  --src.feed_cfg.projects_query_mode.query_mode FarosGraph \   
  --dst farosai/airbyte-faros-destination:latest \   
  --dst.edition_configs.edition cloud \   
  --dst.edition_configs.api_url https://prod.api.faros.ai \ 
  --dst.edition_configs.api_key $FAROS_API_KEY \   
  --dst.edition_configs.graph default \   
  --state jira_state.json

GitLab

./airbyte-local.sh \   
  --src farosai/airbyte-faros-feeds-source:latest \  
  --src.feed_cfg.feed_name gitlab-feed \  
  --src.feed_cfg.feed_path vcs/gitlab-feed \  
  --src.feed_cfg.token $GITLAB_TOKEN \  
  --src.feed_cfg.repos_query_mode.query_mode FarosGraph \  
  --src.feed_cfg.repos_query_mode.group_names '["my-gitlab-group-1", ...]' \  
  --dst farosai/airbyte-faros-destination:latest \  
  --dst.edition_configs.edition cloud \  
  --dst.edition_configs.api_url https://prod.api.faros.ai \
  --dst.edition_configs.api_key $FAROS_API_KEY \  
  --dst.edition_configs.graph default \  
  --state gitlab_state.json

Buildkite

./airbyte-local.sh \   
  --src farosai/airbyte-faros-feeds-source:latest \   
  --src.feed_cfg.feed_name buildkite-feed \   
  --src.feed_cfg.feed_path cicd/buildkite-feed \   
  --src.feed_cfg.organization my-buildkite-org \   
  --src.feed_cfg.token $BUILDKITE_TOKEN \   
  --dst farosai/airbyte-faros-destination:latest \   
  --dst.edition_configs.edition cloud \   
  --dst.edition_configs.api_url https://prod.api.faros.ai \ 
  --dst.edition_configs.api_key $FAROS_API_KEY \   
  --dst.edition_configs.graph default \   
  --state buildkite_state.json

Changeset

./airbyte-local.sh \   
  --src farosai/airbyte-faros-feeds-source:latest \   
  --src.feed_cfg.feed_name changeset-feed \   
  --src.feed_cfg.feed_path cicd/changeset-feed \   
  --src.feed_cfg.faros_cfg.faros_api_url https://prod.api.faros.ai \   
  --src.feed_cfg.faros_cfg.faros_api_key $FAROS_API_KEY \   
  --src.feed_cfg.faros_cfg.graph_name default \   
  --src.feed_cfg.github_cfg.auth.kind token \   
  --src.feed_cfg.github_cfg.auth.token $GITHUB_TOKEN \   
  --dst farosai/airbyte-faros-destination:latest \   
  --dst.edition_configs.edition cloud \   
  --dst.edition_configs.api_url https://prod.api.faros.ai \  
  --dst.edition_configs.api_key $FAROS_API_KEY \   
  --dst.edition_configs.graph default \   
  --state changeset_state.json

Jenkins

./airbyte-local.sh \   
  --src farosai/airbyte-faros-feeds-source:latest \   
  --src.feed_cfg.feed_name jenkins-feed \   
  --src.feed_cfg.feed_path cicd/jenkins-feed \   
  --src.feed_cfg.server_url $JENKINS_SERVER_URL \   
  --src.feed_cfg.user $JENKINS_USER \   
  --src.feed_cfg.token $JENKINS_TOKEN \   
  --src.feed_cfg.jobs_query_mode.query_mode All \   
  --dst farosai/airbyte-faros-destination:latest \   
  --dst.edition_configs.edition cloud \   
  --dst.edition_configs.api_url https://prod.api.faros.ai \  
  --dst.edition_configs.api_key $FAROS_API_KEY \   
  --dst.edition_configs.graph default \   
  --state jenkins_state.json

Bitbucket

./airbyte-local.sh \   
  --src farosai/airbyte-faros-feeds-source:latest \ 
  --src.feed_cfg.feed_name bitbucket-feed \   
  --src.feed_cfg.feed_path vcs/bitbucket-feed \   
  --src.feed_cfg.workspaces '["my-bitbucket-workspace-1", ...]' \   
  --src.feed_cfg.token $BITBUCKET_TOKEN \   
  --dst farosai/airbyte-faros-destination:latest \   
  --dst.edition_configs.edition cloud \   
  --dst.edition_configs.api_url https://prod.api.faros.ai \   
  --dst.edition_configs.api_key $FAROS_API_KEY \   
  --dst.edition_configs.graph default \   
  --state bitbucket_state.json

Additional Resources

Create a GitHub App to Authenticate Faros to Multiple Orgs

You can create a GitHub App and install it on your multiple GitHub organizations to easily enable read-only access from the Faros connector to your desired organizations and repositories.

  1. Navigate to either your profile settings page or your organization's settings page and create a new GitHub app.
  • At the bottom of the creation page, select "Any account" to allow your app to be installed any user or organization.
  • The Webhook option is not required.
  • The app requires the following read-only permissions:
    • Repository permissions: Administration, Contents, Issues, Metadata, Projects, Pull Requests
    • Organization permissions: Members
  1. On your app's settings page, create a private key and a client secret. These values, along with the App Id and Client ID, are required by the Faros connector.
  2. Install the GitHub app on your desired organizations. You can either grant access to all repositories in an org, or only grant access to specific repositories per org.
  3. Run the Faros GitHub connector:
docker pull farosai/github-feed:latest

docker run farosai/github-feed:latest \
    --faros-api-url https://prod.api.faros.ai \
    --faros-api-key $FAROS_API_KEY \
    --repos-query-mode GitHubOrg \
    --app-id $GITHUB_APP_ID \
    --client-id $GITHUB_APP_CLIENT_ID \
    --client-secret $GITHUB_APP_CLIENT_SECRET \
    --private-key $GITHUB_APP_PRIVATE_KEY

Create a GitLab App to Authenticate Faros

You can create a GitLab App and install it to easily enable read-only access from the Faros connector to your desired organizations and repositories.

  1. In the top-right corner of your GitLab account, select your avatar.
  2. Select Edit profile.
  3. On the left sidebar, select Access Tokens.
  4. Enter a name and set the expiry date to "no expiration" for the token.
    • The app only requires the following read-only permission:read_api
  5. Select Create personal access token.
  6. Run the Faros GitLab connector:
docker pull farosai/gitlab-feed:latest

docker run farosai/gitlab-feed:latest \
  --faros-api-key $FAROS_API_KEY \
  --faros-api-url https://prod.api.faros.ai \
  --repos-query-mode GitLabGroup \
  --token $GITLAB_TOKEN \
  --gitlab-api-url https://gitlab.example.com \
  -l mygroup \
  -r 'repo1,repo2,repo3'

Create a Jira PAT to Authenticate Faros

You can create a Personal Access Token in Jira to enable read-only access from the Faros connector

  1. In Jira, select your profile picture at the top right of the screen, then choose Profile.
  2. Once you access your profile, select Personal Access Tokens in the left-hand menu.
  3. Select Create token.
    • Give your new token a name.
    • The integration user needs application access to Jira, the 'Browse Users' global permission, and the 'Browse Project' and 'View Development Tools' permissions for each project
    • Set the expiry date to "No Expiration"
  4. Click Create.
  5. Run the Faros Jira connector:
docker pull farosai/jira-feed:latest

docker run farosai/jira-feed:latest \
   --faros-api-url https://prod.api.faros.ai \
   --faros-api-key $FAROS_API_KEY \
   --jira-project-query-mode FarosGraph \
   --jira-url $JIRA_URL \
   --jira-username $JIRA_USER \
   --jira-token $JIRA_TOKEN \
   --no-jira-pull-requests \
   --origin myjira

Note: if you run into a certificate validation error, you can add the following line to the end of the Jira command in order to bypass certificate validation

--no-jira-reject-unauthorized