Optimizing CI Costs: Manual Jenkins Build Trigger from GitHub Actions

My client needed a solution to trigger jobs in Jenkins efficiently.

We were experiencing high costs and resource consumption when opening PRs or committing because our CI was consuming significant resources and incurring high costs whenever we opened PRs or committed.

To address this, we implemented a solution that allows manually triggering Jenkins builds directly from the GitHub Pull Request (PR) page without granting direct access to Jenkins. This approach leverages GitHub Actions, GitHub Apps, and AWS Lambda to create a seamless and secure workflow.

The flow of our solution is as follows:

1. A GitHub Action creates a button on the PR page.
2. When the button is clicked, a GitHub App is invoked.
3. The GitHub App triggers an AWS Lambda function.
4. The Lambda function initiates the Jenkins build.

This integration ensures that builds are only triggered when necessary, thus optimizing resource usage and reducing costs while maintaining the security of our Jenkins environment.

Configuring the GitHub App

To create Github Apps you can follow this guide via github.com – GitHub DocsRegistering a GitHub App – GitHub Docs​

Basic information:

1. GitHub App name – you will refer to it later in the GitHub action.
2. Homepage URL – Lambda URL
3. Webhook: Lambda URL (events will POST to this URL) – you can put a secret key there for secured access to the API.
4. Private Keys – Generate a private key and put it in your secrets in the organization.

Permissions :

1. Allow your App permissions for the repository.
2. Install the App in your organization (inside your GitHub Apps organization page as well).

Initiating Jenkins Build from AWS Lambda

• Create Lambda with the right access to Jenkins (mine inside a VPC)
• Create API Gateway for POST.
• Inside Lambda, configure what you need to trigger the build, access the Secret Manager for the Jenkins token, URLs, and the code itself to filter for the PR number and which job you trigger.

My Lambda :

import json
import boto3
import requests


def trigger_pr(body,JENKINS_USER,JENKINS_USER_PASS,JENKINS_URL):
    try:
        check_run = body.get('check_run')
        
        pull_requests = body.get('check_run', {}).get('pull_requests', [])
        repository = body.get('repository', {})
        PR_NUMBER = pull_requests[0].get('number')
        MULTI_BRANCH_NAME = check_key(repository.get("name"))

        
        print(f"Triggering : MultiBranchName: {MULTI_BRANCH_NAME} PR NUMBER : {PR_NUMBER}")
        URL = f"{JENKINS_URL}/job/{MULTI_BRANCH_NAME}/view/change-requests/job/PR-{PR_NUMBER}/build"
        print(f"URL: {URL}")
        

        # Perform the POST request with basic authentication and payload
        response =  requests.post(URL, auth=(JENKINS_USER, JENKINS_USER_PASS))

        response.raise_for_status()  # Raise an exception for 4XX or 5XX status codes
        print(response.json)
        return {
            'statusCode': 200,
            'body': "Successfull triggered Build PR"
        }
        
        
        # return "Build triggered successfully"
    except requests.exceptions.RequestException as e:
        print("Error triggering build:", e)
        return {
            'statusCode': 400,
            'body': "Error triggering build FROM BUTTON POST"
        }
            


def trigger_develop(body,event,JENKINS_USER,JENKINS_USER_PASS,JENKINS_URL):
    try:
        MERGED_TO = body.get('MERGED_TO')
        MULTI_BRANCH_NAME = check_key(body.get('MULTI_BRANCH_NAME'))

        
        print(f"Triggering Develop Merge")
        URL = f"{JENKINS_URL}/job/{MULTI_BRANCH_NAME}/job/{MERGED_TO}/build"
        print(f"URL: {URL}")
               

        # Perform the POST request with basic authentication and payload
        response =  requests.post(URL, auth=(JENKINS_USER, JENKINS_USER_PASS))
        response.raise_for_status()  # Raise an exception for 4XX or 5XX status codes
        print(response.json)

    #     # return "Build triggered successfully"
    except requests.exceptions.RequestException as e:
        print("Error triggering build:", e)
        return {
            'statusCode': 400,
            'body': "Error triggering build FROM BUTTON"
        }
            


def lambda_handler(event, context):
    # Initialize Secrets Manager client
    print(f"event: {event}")
    

    # Retrieve Jenkins credentials from Secrets Manager
    try:
        client = boto3.client('secretsmanager')
        response = client.get_secret_value(SecretId='JenkinsCredentials')

        secret_data = json.loads(response['SecretString'])
        JENKINS_USER = secret_data['JENKINS_USER']
        JENKINS_USER_PASS = secret_data['JENKINS_PASSWORD']
        JENKINS_URL = secret_data['JENKINS_URL']
        

    except Exception as e:
        print("Error retrieving Jenkins credentials from Secrets Manager:", e)
        res = {
        "statusCode": 400,
        "headers": {
            "Content-Type": "*/*"
        },
        "body": "Error retrieving Jenkins credentials from Secrets Manager"
        }

        return res
            
    body = json.loads(event['body'])
    # check_suite = body.get('check_run', {}).get('check_suite', [])
    # head_branch = check_suite.get('head_branch')

    if body.get('MERGED_TO') == "develop":
        print("Triggering Merge Develop Build")
        trigger_develop(body=body,event=event,JENKINS_USER=JENKINS_USER,JENKINS_USER_PASS=JENKINS_USER_PASS,JENKINS_URL=JENKINS_URL)
        return {
            'statusCode': 200,
            'body': "Successfull triggered Build Develop Meged"
        }
        

    elif body.get('check_run', {}).get('pull_requests', []):
        print("Triggering PR Build")
        trigger_pr(body=body,JENKINS_USER=JENKINS_USER,JENKINS_USER_PASS=JENKINS_USER_PASS,JENKINS_URL=JENKINS_URL)
        return {
            'statusCode': 200,
            'body': "Successfull triggered Build PR"
        }

Layers that I used for the lambda :

Set up the GitHub Action to create a button

creating : .github/workflows/button.yaml file.

name: Button
on:
  pull_request:
    permissions:
      checks: write
      contents: read
env:
  GITHUB_PR_NUMBER: ${{github.event.pull_request.number}}
  
jobs:
  Test:
    runs-on: ubuntu-latest
    permissions: write-all
    
    steps:
      - name: Generate a token
        id: generate-token
        uses: actions/create-github-app-token@v1
        with:
          app-id: "<APP ID>" #We will create this later
          private-key: ${{ secrets.APP_PRIVATE_KEY }} 


      - uses: LouisBrunner/checks-action@v2.0.0
        if: always()
        with:
          token: ${{ steps.generate-token.outputs.token }}
          name: Trigger Jenkins Job
          conclusion: success
          action_url: "https://github.com/"
          actions: |
            [{"label":"Trigger Jenkins","description":"Click me to trigger jenkins job","identifier":"<LAMBDA APP NAME>"}]

PR UI in GitHub :

What we see here :

1. A button was created in the “Checks” section
2. Trigger Jenkins Job (GitHub Apps that forward the request to lambda)
3. continuous-integration – Jenkins builds CI itself.

PR UI – in the Checks Section :

What we see here :

1. In the left section, we can see 2 workflows: one that created the button and the second that created the button itself.
2. In the right section, we can see the button itself. When we click the “Trigger Jenkins” button, it forwards the request to our Lambda.

Steps in the Job

1. actions/create-github-app-token action to generate a GitHub App token.

• app-id: The ID of the GitHub App, which will be created later.
• private-key: The private key of the GitHub App, stored as a secret (APP_PRIVATE_KEY).

The token generated in this step will be used for authentication in subsequent steps. The step’s ID is generate-token, which allows its outputs to be referenced later.

2. LouisBrunner/checks-action to create a button in the GitHub PR checks section.

• token: The GitHub App token generated in the previous step.
• name: The name of the check (e.g., “Trigger Jenkins Job”).
• conclusion: The conclusion of the check (e.g., success).
• action_url: A URL associated with the check (could be a relevant link or simply a placeholder).
• actions: Defines the button’s label, description, and identifier. The label is the button text (“Trigger Jenkins”), the description provides more information (“Click me to trigger jenkins job”), and the identifier is a unique ID for the button action – Github Apps name.

When clicked, this button triggers further actions (e.g., calling an AWS Lambda function to start a Jenkins build).

In addition to that, I created a Workflow to trigger when merging to the master branch.

name: Trigger Master job
on:
  push:
    branches:
      - master
env:
  BRANCH: ${{ github.ref_name }}
  MULTIBRANCH_JOB_NAME: ${{ github.event.repository.name }}

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Trigger Jenkins build
      shell: bash
      run: |
        PAYLOAD="{\"MERGED_TO\": \"$BRANCH\", \"MULTI_BRANCH_NAME\": \"$MULTIBRANCH_JOB_NAME\"}"
        echo $PAYLOAD
        curl -XPOST <API URL> -d "$PAYLOAD"

Summary:

Every client has a distinct work methodology shaped by their unique business processes, team dynamics, and project requirements. This diversity necessitates tailored solutions to meet specific needs effectively. You can achieve webhook triggers in a few ways like creating labels, making specific comments in the PR and more. In this case, our client has a particular workflow that requires the integration of a manual trigger within their user interface.

By implementing this solution, we achieved significant cost savings and optimized resource usage for our Jenkins builds. The manual triggering of builds from the GitHub PR page ensures that builds are only run when necessary, reducing unnecessary resource consumption. Improvements could include more granular control over build triggers and further automation to enhance efficiency.