Automating Python Scripts with GitHub Actions: API Integration and Email Notifications

Automating Python Scripts with GitHub Actions: API Integration and Email Notifications

Introduction

Automating Python scripts to interact seamlessly with GitHub repositories enhances productivity and ensures consistency in project workflows. This guide delves into setting up a robust automation pipeline using GitHub Actions to execute Python scripts that create new files in a GitHub repository. Additionally, it covers integrating an API to trigger these workflows externally and configuring email notifications to monitor successes and errors effectively.

Table of Contents

  1. Prerequisites
  2. Setting Up the GitHub Repository
  3. Creating the Python Script
  4. Configuring GitHub Actions Workflow
  5. Integrating API Trigger
  6. Setting Up Email Notifications
  7. Testing the Automation
  8. Conclusion
  9. References

Prerequisites

Before diving into the setup, ensure you have the following:

  • GitHub Account: To create and manage repositories.
  • Python Environment: Python 3.x installed on your local machine.
  • Basic Knowledge of GitHub Actions: Familiarity with CI/CD concepts.
  • SMTP Credentials: For configuring email notifications.
  • Hosting Platform for API: Such as Heroku, AWS, or any platform supporting Python Flask applications.

Setting Up the GitHub Repository

  1. Create a New Repository:

    • Navigate to GitHub and create a new repository named python-automation.
    • Initialize the repository with a README.md file.

  2. Clone the Repository Locally:

    git clone https://github.com/your-username/python-automation.git
cd python-automation

Creating the Python Script

Develop a Python script that generates new files. For demonstration, we'll create a script that adds a timestamped log file.

  1. Create generate_file.py:

    # generate_file.py
import os
from datetime import datetime

def create_new_file():
    # Define the directory for generated files
    output_dir = 'generated_files'
    os.makedirs(output_dir, exist_ok=True)

    # Create a new file with the current timestamp
    timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
    filename = f'log_{timestamp}.txt'
    filepath = os.path.join(output_dir, filename)

    with open(filepath, 'w') as file:
        file.write(f'Log created on {datetime.now()}\n')

    print(f'Created {filepath}')

if __name__ == '__main__':
    create_new_file()

  2. Commit and Push the Script:

    git add generate_file.py
git commit -m "Add Python script to generate timestamped log files"
git push origin main

Configuring GitHub Actions Workflow

GitHub Actions will automate the execution of the Python script and handle Git operations.

  1. Create Workflow Directory and File:

    • Create a directory for workflows:

      mkdir -p .github/workflows

    • Create automate_script.yml inside .github/workflows/:

      # .github/workflows/automate_script.yml
name: Automate Python Script

on:
  schedule:
    - cron: '0 0 * * *'  # Runs daily at midnight UTC
  workflow_dispatch:    # Allows manual and API triggering

jobs:
  run-script:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Repository
        uses: actions/checkout@v3
        with:
          persist-credentials: false  # Handle authentication manually

      - name: Set Up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.x'

      - name: Install Dependencies
        run: |
          python -m pip install --upgrade pip
          # Install additional dependencies if needed
          # pip install -r requirements.txt

      - name: Run Python Script
        run: |
          python generate_file.py

      - name: Configure Git
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"

      - name: Commit and Push Changes
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          git add generated_files/
          git commit -m "Automate: Add new log file [skip ci]" || echo "No changes to commit"
          git push origin main

  2. Commit and Push the Workflow:

    git add .github/workflows/automate_script.yml
git commit -m "Add GitHub Actions workflow for automating Python script"
git push origin main

Integrating API Trigger

To allow external systems to trigger the GitHub Actions workflow, we'll set up an API endpoint using Flask.

a. Generate a GitHub Personal Access Token

  1. Create Token:

    • Navigate to GitHub Settings.
    • Click on Developer settings > Personal access tokens > Tokens (classic) > Generate new token.
    • Select scopes:
      • repo: Full control of private repositories.
      • workflow: Update GitHub Action workflows.
    • Generate and securely store the token.

b. Implement the Flask API

  1. Set Up Flask Application:

    • Create a New Directory:

      mkdir trigger-workflow-api
cd trigger-workflow-api

    • Initialize Virtual Environment and Install Dependencies:

      python3 -m venv venv
source venv/bin/activate
pip install Flask requests gunicorn

    • Create app.py:

      # app.py
import os
from flask import Flask, request, jsonify
import requests

app = Flask(__name__)

# Load environment variables
GITHUB_TOKEN = os.getenv('GITHUB_TOKEN')
REPO_OWNER = os.getenv('REPO_OWNER')  # e.g., 'your-username'
REPO_NAME = os.getenv('REPO_NAME')    # e.g., 'python-automation'
WORKFLOW_FILE = os.getenv('WORKFLOW_FILE', 'automate_script.yml')  # Workflow file name
API_KEY = os.getenv('API_KEY')  # For securing the API

@app.route('/trigger-workflow', methods=['POST'])
def trigger_workflow():
    # Authenticate API request
    api_key = request.headers.get('x-api-key')
    if api_key != API_KEY:
        return jsonify({'message': 'Unauthorized'}), 401

    data = request.get_json()
    ref = data.get('ref', 'main')  # Branch to run the workflow on

    url = f'https://api.github.com/repos/{REPO_OWNER}/{REPO_NAME}/actions/workflows/{WORKFLOW_FILE}/dispatches'
    headers = {
        'Accept': 'application/vnd.github.v3+json',
        'Authorization': f'token {GITHUB_TOKEN}'
    }
    payload = {
        'ref': ref,
        # 'inputs': {}  # Add if your workflow uses inputs
    }

    response = requests.post(url, json=payload, headers=headers)

    if response.status_code == 204:
        return jsonify({'message': 'Workflow triggered successfully.'}), 200
    else:
        return jsonify({'message': 'Failed to trigger workflow.', 'details': response.json()}), response.status_code

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

    • Create requirements.txt:

      Flask
requests
gunicorn

    • Create Procfile for Deployment:

      web: gunicorn app:app

  2. Deploy the Flask Application:

    We'll use Heroku for deployment.

    • Initialize Git Repository:

      git init
git add .
git commit -m "Initial commit for workflow trigger API"

    • Create Heroku App and Deploy:

      heroku create your-app-name
heroku config:set GITHUB_TOKEN=your_personal_access_token
heroku config:set REPO_OWNER=your-username
heroku config:set REPO_NAME=python-automation
heroku config:set WORKFLOW_FILE=automate_script.yml
heroku config:set API_KEY=your_secure_api_key
git push heroku main
heroku ps:scale web=1
heroku open

    • Secure the API Endpoint:

      The API_KEY environment variable ensures that only authorized requests can trigger the workflow. When making requests, include the x-api-key header with the correct key.

Setting Up Email Notifications

To monitor the automation process, configure GitHub Actions to send email notifications upon workflow completion or failure.

  1. Configure SMTP Credentials as GitHub Secrets:

    • Navigate to your GitHub repository.
    • Go to Settings > Secrets and variables > Actions.
    • Add the following secrets:
      • EMAIL_USERNAME: Your email address (e.g., your-email@gmail.com).
      • EMAIL_PASSWORD: Your email password or app-specific password.

  2. Update GitHub Actions Workflow:

    Modify automate_script.yml to include email notifications using dawidd6/action-send-mail.

    # .github/workflows/automate_script.yml
name: Automate Python Script

on:
  schedule:
    - cron: '0 0 * * *'  # Runs daily at midnight UTC
  workflow_dispatch:    # Allows manual and API triggering

jobs:
  run-script:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Repository
        uses: actions/checkout@v3
        with:
          persist-credentials: false  # Handle authentication manually

      - name: Set Up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.x'

      - name: Install Dependencies
        run: |
          python -m pip install --upgrade pip
          # Install additional dependencies if needed
          # pip install -r requirements.txt

      - name: Run Python Script
        run: |
          python generate_file.py

      - name: Configure Git
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"

      - name: Commit and Push Changes
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          git add generated_files/
          git commit -m "Automate: Add new log file [skip ci]" || echo "No changes to commit"
          git push origin main

      - name: Send Email Notification
        uses: dawidd6/action-send-mail@v4
        with:
          server_address: smtp.gmail.com
          server_port: 465
          username: ${{ secrets.EMAIL_USERNAME }}
          password: ${{ secrets.EMAIL_PASSWORD }}
          subject: GitHub Actions Workflow - ${{ job.status }}
          body: |
            The GitHub Actions workflow **${{ github.workflow }}** has completed with status: **${{ job.status }}**.

            **Repository:** ${{ github.repository }}
            **Branch:** ${{ github.ref }}
            **Workflow:** ${{ github.workflow }}
            **Run ID:** ${{ github.run_id }}

            Check the logs for more details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}

          to: your-email@example.com
          from: ${{ secrets.EMAIL_USERNAME }}
          secure: true

    Explanation of the Email Notification Step:

    • server_address & server_port: SMTP server details (e.g., Gmail uses smtp.gmail.com and port 465 for SSL).
    • username & password: Retrieved from GitHub Secrets for security.
    • subject & body: Dynamic content incorporating workflow status and details.
    • to & from: Recipient and sender email addresses.
    • secure: Enables SSL for secure email transmission.

  3. Commit and Push the Updated Workflow:

    git add .github/workflows/automate_script.yml
git commit -m "Enhance workflow with email notifications"
git push origin main

Testing the Automation

a. Manual Trigger via API

  1. Send POST Request to API Endpoint:

    curl -X POST https://your-app-name.herokuapp.com/trigger-workflow \
  -H "Content-Type: application/json" \
  -H "x-api-key: your_secure_api_key" \
  -d '{"ref": "main"}'

  2. Verify Workflow Execution:

    • Navigate to the Actions tab in your GitHub repository.
    • Confirm that the Automate Python Script workflow was triggered.
    • Monitor the workflow progress and check for any errors.

  3. Check Email Notifications:

    • Ensure you receive an email detailing the workflow's status.
    • Verify the content includes relevant workflow information and links to logs.

b. Scheduled Trigger

  1. Wait for Scheduled Execution:

    • The workflow is set to run daily at midnight UTC.
    • Alternatively, adjust the cron schedule for testing purposes.

  2. Verify File Creation and Commit:

    • After the workflow runs, check the generated_files/ directory for the new log file.
    • Confirm that a new commit was pushed to the main branch with the appropriate message.

  3. Review Email Notifications:

    • Receive an email notification confirming the workflow's completion and status.

c. Simulate Errors

  1. Introduce an Error in the Python Script:

    • Modify generate_file.py to include a syntax error.

      # generate_file.py
import os
from datetime import datetime

def create_new_file()
    # Missing colon above
    output_dir = 'generated_files'
    os.makedirs(output_dir, exist_ok=True)

    # Rest of the code...

  2. Trigger the Workflow:

    • Use the API or wait for the scheduled run.

  3. Check Email for Error Notification:

    • An email should indicate the workflow failed, including error details.

  4. Revert the Error and Re-run:

    • Correct the syntax error and re-trigger the workflow to ensure successful execution.

Conclusion

Automating Python scripts with GitHub Actions streamlines workflows, ensuring consistent execution and seamless integration with GitHub repositories. By integrating an external API trigger, you gain flexibility to initiate workflows from various platforms or applications. Additionally, configuring email notifications provides real-time monitoring, allowing you to stay informed about the automation process's successes and failures.

Key benefits of this setup include:

  • Efficiency: Reduces manual intervention by automating repetitive tasks.
  • Scalability: Easily extendable to accommodate more complex automation pipelines.
  • Flexibility: API integration allows external systems to interact with GitHub workflows.
  • Monitoring: Email notifications ensure you are promptly informed about workflow statuses.

This comprehensive automation pipeline leverages the strengths of GitHub Actions, Python scripting, and external APIs to create a robust and maintainable system tailored to your project's needs.

References

  1. GitHub Actions Documentation - https://docs.github.com/en/actions
  2. Flask Documentation - https://flask.palletsprojects.com/
  3. dawidd6/action-send-mail GitHub Action - https://github.com/dawidd6/action-send-mail
  4. Heroku Deployment Guide - https://devcenter.heroku.com/articles/getting-started-with-python
  5. SMTP Configuration for Gmail - https://support.google.com/mail/answer/7126229
  6. GitHub Personal Access Tokens - https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token
  7. Flask Limiter for Rate Limiting - https://flask-limiter.readthedocs.io/
  8. Cron Syntax Guide - https://crontab.guru/
  9. Heroku CLI Documentation - https://devcenter.heroku.com/articles/heroku-cli
  10. Securely Storing Secrets in GitHub - https://docs.github.com/en/actions/security-guides/encrypted-secrets

Last updated: January 8, 2025