Continuous integration (CI) is the backbone of modern software development. By automating tests, linting, and builds, you catch bugs early and keep your codebase healthy. GitHub Actions offers a powerful, native way to run CI pipelines directly from your repository, and it works beautifully with Python projects. In this guide, we’ll walk through every step you need to set up a reliable CI workflow, from repository preparation to advanced caching tricks. Whether you’re maintaining a small open‑source library or a large web service, the patterns here will scale with you.
What You’ll Need
- A GitHub account and a repository containing a Python project.
- Basic knowledge of Git and the command line.
- Python 3.8+ installed locally for testing.
- Familiarity with
requirements.txtorpyproject.tomldependency files. - An optional Docker installation if you want container‑based testing.
Step 1: Organize Your Project Structure
Before adding any workflow files, ensure your project follows a conventional layout. A typical Python project looks like this:
my_project/ # Root folder
├─ .git/ # Git metadata (auto‑generated)
├─ .github/ # GitHub‑specific configuration
│ └─ workflows/ # CI/CD workflow files go here
├─ src/ # Your package source code
│ └─ my_package/ # Example package
│ ├─ __init__.py
│ └─ core.py
├─ tests/ # Test suite (pytest preferred)
│ └─ test_core.py
├─ requirements.txt # Runtime dependencies
├─ requirements-dev.txt # Development & testing dependencies
├─ pyproject.toml (optional) # Build system config
└─ README.md
Having a clear separation between src and tests makes it easier for tools like pytest to discover tests and for coverage reports to be accurate.
Step 2: Create a Basic Workflow File
Inside .github/workflows/, create a file named ci.yml. This file tells GitHub Actions what to do on each push or pull request.
name: CI
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install -r requirements-dev.txt
- name: Run tests
run: |
pytest --color=yes
This minimal pipeline checks out the code, installs Python 3.11, installs both runtime and development dependencies, and finally runs pytest. Commit and push this file; you’ll see a new workflow run in the “Actions” tab of your repo.
Step 3: Add Linting and Type Checking
Code quality tools such as flake8 (linting) and mypy (static type checking) help keep the codebase consistent. Extend the requirements-dev.txt file:
pytest
flake8
mypy
Then update the workflow to run these checks after installing dependencies.
- name: Lint with flake8
run: |
flake8 src tests
- name: Type check with mypy
run: |
mypy src
By failing the workflow when linting or typing errors appear, you enforce a clean code standard on every pull request.
Step 4: Cache Dependencies for Faster Builds
Installing dependencies from scratch on each run can waste minutes. GitHub Actions offers a built‑in caching action. Add a cache step before the install step:
- name: Cache pip packages
uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt') }}
restore-keys: |
${{ runner.os }}-pip-
The cache key hashes all requirements files, so any change to dependencies automatically creates a new cache. This can shave 30‑60 seconds off each run.
Step 5: Generate Test Coverage Reports
Coverage data tells you how much of your code is exercised by tests. Install coverage and the pytest-cov plugin, then modify the test step:
- name: Install coverage tools
run: |
pip install coverage pytest-cov
- name: Run tests with coverage
run: |
pytest --cov=src --cov-report=xml
- name: Upload coverage to Codecov (optional)
uses: codecov/codecov-action@v3
with:
files: coverage.xml
If you prefer a free, built‑in solution, you can use the actions/upload-artifact action to store the coverage.xml file as a workflow artifact.
Step 6: Deploy on Successful Builds (Optional)
Many Python projects need to publish a package to PyPI or deploy a web service. Below is a minimal example that pushes a new release to PyPI when a tag matching v* is created.
on:
push:
tags:
- 'v*'
jobs:
publish:
runs-on: ubuntu-latest
needs: build
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: Install build tools
run: |
python -m pip install --upgrade pip
pip install build twine
- name: Build distribution
run: |
python -m build
- name: Publish to PyPI
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
run: |
twine upload dist/*
Remember to store your PyPI API token in the repository’s Settings → Secrets as PYPI_API_TOKEN. This step is optional but demonstrates how CI can become CD (continuous deployment) with just a few extra lines.
Common Mistakes to Avoid
1. Hard‑coding Python versions. Use a matrix strategy if you need to test against multiple versions.
2. Forgetting to cache. Skipping the cache step leads to slower builds and wasted CI minutes.
3. Running tests in the wrong directory. Ensure pytest runs from the repository root so it can locate tests/ and src/ correctly.
4. Missing requirements-dev.txt. Linting and type checking will fail if dev dependencies aren’t installed.
5. Exposing secrets. Never hard‑code API keys; always use GitHub Secrets.
Tips and Tricks
– Matrix builds. Test against Python 3.9, 3.10, and 3.11 in parallel to ensure compatibility.
– Conditional steps. Use if: github.ref == 'refs/heads/main' to run deployment only on the main branch.
– Reusable workflows. Store common steps (like linting) in a separate YAML file and call it with uses to keep your CI DRY.
– Self‑hosted runners. For heavy workloads, consider a self‑hosted runner with more CPU or GPU resources.
Frequently Asked Questions
Do I need a Docker container for Python CI?
No. GitHub’s ubuntu-latest runner already includes Python, pip, and common build tools. Use Docker only when you need a specific OS environment or system‑level dependencies.
How can I test multiple Python versions?
Define a matrix in the workflow:
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ['3.9', '3.10', '3.11']
steps:
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
# rest of steps …
This runs the same job three times, one for each version.
Can I run my own scripts before the CI starts?
Yes. Add a step that executes a shell script or a Python module. For example, you might generate dynamic configuration files or download test data from an external source.
Conclusion
Setting up a continuous integration pipeline with GitHub Actions for a Python project is straightforward once you understand the building blocks: checkout, Python setup, dependency installation, caching, testing, linting, and optional deployment. By following the steps above, you’ll have a fast, reliable CI workflow that catches errors early, enforces code quality, and even automates releases. Remember to iterate on your pipeline—add matrix testing, improve caching, and keep an eye on common pitfalls. Happy coding, and may your builds always be green!
Photo by Rubaitul Azad on Unsplash
