This document is for pytest_celery's development version, which can be significantly different from previous releases. Get the stable docs here: 1.0.
How to release a new version¶
- Release:
1.1
- Date:
Sep 30, 2024
The following guide will describe the steps to release a new version of the pytest-celery plugin. It will explain how does the CI/CD pipeline work and how to trigger a new release.
CI/CD Pipeline¶
Continuous Integration¶
Added in version 1.0.0.
The CI platform is based on GitHub Actions and it is triggered on every push to the repository, and on every pull request, according to the changes made.
The configuration files for the CI pipeline are located in the .github/workflows directory of the repository.
CI Tests¶
Unit, Integration and Smoke tests.
# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
name: pytest-celery
on:
push:
branches: ["main"]
paths:
- ".github/workflows/python-package.yml"
- 'src/pytest_celery/vendors/worker/**'
- "**.py"
- "**.txt"
- "**.toml"
- "tox.ini"
- 'Dockerfile'
- "poetry.lock"
pull_request:
paths:
- ".github/workflows/python-package.yml"
- 'src/pytest_celery/vendors/worker/**'
- "**.py"
- "**.txt"
- "**.toml"
- "tox.ini"
- 'Dockerfile'
- "poetry.lock"
permissions:
contents: read # to fetch code (actions/checkout)
jobs:
Unit:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
os: ["blacksmith-4vcpu-ubuntu-2204"]
steps:
- name: Fetch Docker Images
run: |
docker pull redis:latest
docker pull rabbitmq:latest
docker pull memcached:latest
- name: Install apt packages
run: |
sudo apt update
sudo apt-get install -y procps # Install procps to enable sysctl
sudo sysctl -w vm.overcommit_memory=1
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: useblacksmith/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: '**/setup.py'
- name: Install Poetry
uses: snok/install-poetry@v1.4.1
- name: Install tox
run: |
poetry config virtualenvs.create false
poetry install --only ci
- name: Run tox for "${{ matrix.python-version }}-unit"
timeout-minutes: 5
run: |
tox --verbose --verbose -e "${{ matrix.python-version }}-unit"
- name: Upload coverage reports to Codecov
uses: codecov/codecov-action@v4
with:
fail_ci_if_error: true # optional (default = false)
token: ${{ secrets.CODECOV_TOKEN }}
verbose: true # optional (default = false)
Integration:
needs:
- Unit
if: needs.Unit.result == 'success'
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
os: ["blacksmith-4vcpu-ubuntu-2204"]
steps:
- name: Fetch Docker Images
run: |
docker pull redis:latest
docker pull rabbitmq:latest
docker pull memcached:latest
- name: Install apt packages
run: |
sudo apt update
sudo apt-get install -y procps # Install procps to enable sysctl
sudo sysctl -w vm.overcommit_memory=1
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: useblacksmith/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: '**/setup.py'
- name: Install Poetry
uses: snok/install-poetry@v1.4.1
- name: Install tox
run: |
poetry config virtualenvs.create false
poetry install --only ci
- name: Run tox for "${{ matrix.python-version }}-integration"
timeout-minutes: 20
run: |
tox --verbose --verbose -e "${{ matrix.python-version }}-integration" -- -n auto --reruns 5
Smoke:
needs:
- Integration
if: needs.Integration.result == 'success'
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
os: ["blacksmith-4vcpu-ubuntu-2204"]
steps:
- name: Fetch Docker Images
run: |
docker pull redis:latest
docker pull rabbitmq:latest
docker pull memcached:latest
- name: Install apt packages
run: |
sudo apt update
sudo apt-get install -y procps # Install procps to enable sysctl
sudo sysctl -w vm.overcommit_memory=1
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: useblacksmith/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: '**/setup.py'
- name: Install Poetry
uses: snok/install-poetry@v1.4.1
- name: Install tox
run: |
poetry config virtualenvs.create false
poetry install --only ci
- name: Run tox for "${{ matrix.python-version }}-smoke"
timeout-minutes: 30
run: |
tox --verbose --verbose -e "${{ matrix.python-version }}-smoke" -- -n auto --reruns 5
Parallel Tests¶
These are the parallel and xdist tox environments. The purpose of this CI pipeline is to make sure that the plugin is compatible with parallel running, both in terms of supporting pytest-xdist and functionally (i.e. that the plugin does not break when running in parallel).
# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
name: parallel-support
on:
push:
branches: ["main"]
paths:
- ".github/workflows/parallel-support.yml"
- 'src/pytest_celery/vendors/worker/**'
- "**.py"
- "**.txt"
- "**.toml"
- "tox.ini"
- 'Dockerfile'
- "poetry.lock"
pull_request:
paths:
- ".github/workflows/parallel-support.yml"
- 'src/pytest_celery/vendors/worker/**'
- "**.py"
- "**.txt"
- "**.toml"
- "tox.ini"
- 'Dockerfile'
- "poetry.lock"
permissions:
contents: read # to fetch code (actions/checkout)
jobs:
xdist:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: ["3.12"]
os: ["blacksmith-4vcpu-ubuntu-2204"]
steps:
- name: Fetch Docker Images
run: |
docker pull redis:latest
docker pull rabbitmq:latest
docker pull memcached:latest
- name: Install apt packages
run: |
sudo apt update
sudo apt-get install -y procps # Install procps to enable sysctl
sudo sysctl -w vm.overcommit_memory=1
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: useblacksmith/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: '**/setup.py'
- name: Install Poetry
uses: snok/install-poetry@v1.4.1
- name: Install tox
run: |
poetry config virtualenvs.create false
poetry install --only ci
- name: Run tox for all environments in parallel
timeout-minutes: 30
run: |
tox -e xdist
parallel:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: ["3.12"]
os: ["blacksmith-4vcpu-ubuntu-2204"]
steps:
- name: Fetch Docker Images
run: |
docker pull redis:latest
docker pull rabbitmq:latest
docker pull memcached:latest
- name: Install apt packages
run: |
sudo apt update
sudo apt-get install -y procps # Install procps to enable sysctl
sudo sysctl -w vm.overcommit_memory=1
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: useblacksmith/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: '**/setup.py'
- name: Install Poetry
uses: snok/install-poetry@v1.4.1
- name: Install tox
run: |
poetry config virtualenvs.create false
poetry install --only ci
- name: Run tox for all environments in parallel
timeout-minutes: 30
run: |
tox -e parallel
Linting¶
Standard linting checks.
name: Linter
on: [pull_request]
jobs:
check:
name: ${{ matrix.check }} check
runs-on: blacksmith-4vcpu-ubuntu-2204
strategy:
matrix:
check: [lint, mypy]
steps:
- name: Checkout branch
uses: actions/checkout@v4
- name: Install apt packages
run: |
sudo apt update
- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install Poetry
uses: snok/install-poetry@v1.4.1
- name: Install CI dependencies
run: |
poetry config virtualenvs.create false
poetry install --only ci
- name: Run check
run: tox -e ${{ matrix.check }}
Examples¶
The official plugin examples are tested as part of the standard CI pipeline.
# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
name: examples
on:
push:
branches: ["main"]
paths:
- ".github/workflows/examples.yml"
- 'src/pytest_celery/vendors/worker/**'
- "**.py"
- "**.txt"
- "**.toml"
- "tox.ini"
- 'Dockerfile'
- "poetry.lock"
pull_request:
paths:
- ".github/workflows/examples.yml"
- 'src/pytest_celery/vendors/worker/**'
- "**.py"
- "**.txt"
- "**.toml"
- "tox.ini"
- 'Dockerfile'
- "poetry.lock"
permissions:
contents: read # to fetch code (actions/checkout)
jobs:
celery_bug_report:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: ["3.12"]
os: ["blacksmith-4vcpu-ubuntu-2204"]
steps:
- name: Install apt packages
run: |
sudo apt update
sudo apt-get install -y procps # Install procps to enable sysctl
sudo sysctl -w vm.overcommit_memory=1
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: useblacksmith/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: '**/setup.py'
- name: Install dependencies
working-directory: examples
run: |
python -m pip install --upgrade pip pytest-cov
pip install -U "pytest-celery[all]@git+https://github.com/celery/pytest-celery.git"
- name: Run tests
working-directory: examples
timeout-minutes: 5
run: |
pytest -xsv celery_bug_report.py --no-cov
myworker:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: ["3.12"]
os: ["blacksmith-4vcpu-ubuntu-2204"]
steps:
- name: Install apt packages
run: |
sudo apt update
sudo apt-get install -y procps # Install procps to enable sysctl
sudo sysctl -w vm.overcommit_memory=1
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: useblacksmith/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: '**/setup.py'
- name: Install dependencies
working-directory: examples/myworker
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Run tests
working-directory: examples/myworker
timeout-minutes: 10
run: |
pytest -xsv tests --reruns 3 --reruns-delay 10
range:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: ["3.12"]
os: ["blacksmith-4vcpu-ubuntu-2204"]
steps:
- name: Install apt packages
run: |
sudo apt update
sudo apt-get install -y procps # Install procps to enable sysctl
sudo sysctl -w vm.overcommit_memory=1
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: useblacksmith/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: '**/setup.py'
- name: Install dependencies
working-directory: examples/range
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Run tests
working-directory: examples/range
timeout-minutes: 30
run: |
pytest -xsv tests --reruns 3 --reruns-delay 10
rabbitmq_management:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: ["3.12"]
os: ["blacksmith-4vcpu-ubuntu-2204"]
steps:
- name: Install apt packages
run: |
sudo apt update
sudo apt-get install -y procps # Install procps to enable sysctl
sudo sysctl -w vm.overcommit_memory=1
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: useblacksmith/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: '**/setup.py'
- name: Install dependencies
working-directory: examples/rabbitmq_management
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Run tests
working-directory: examples/rabbitmq_management
timeout-minutes: 10
run: |
pytest -xsv tests --reruns 3 --reruns-delay 10
django:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: ["3.12"]
os: ["blacksmith-4vcpu-ubuntu-2204"]
steps:
- name: Install apt packages
run: |
sudo apt update
sudo apt-get install -y procps # Install procps to enable sysctl
sudo sysctl -w vm.overcommit_memory=1
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: useblacksmith/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: '**/setup.py'
- name: Install dependencies
working-directory: examples/django
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Run Migrations
working-directory: examples/django
run: |
./manage.py migrate
- name: Run tests
working-directory: examples/django
timeout-minutes: 10
run: |
export DJANGO_SETTINGS_MODULE=proj.settings
pytest -xsv tests --reruns 3 --reruns-delay 10
myutils:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: ["3.12"]
os: ["blacksmith-4vcpu-ubuntu-2204"]
steps:
- name: Install apt packages
run: |
sudo apt update
sudo apt-get install -y procps # Install procps to enable sysctl
sudo sysctl -w vm.overcommit_memory=1
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: useblacksmith/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: '**/setup.py'
- name: Install dependencies
working-directory: examples/myutils
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Run tests
working-directory: examples/myutils
timeout-minutes: 10
run: |
pytest -xsv tests --reruns 3 --reruns-delay 10
worker_pool:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: ["3.12"]
os: ["blacksmith-4vcpu-ubuntu-2204"]
steps:
- name: Install apt packages
run: |
sudo apt update
sudo apt-get install -y procps # Install procps to enable sysctl
sudo sysctl -w vm.overcommit_memory=1
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: useblacksmith/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: '**/setup.py'
- name: Install dependencies
working-directory: examples/worker_pool
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Run tests
working-directory: examples/worker_pool
timeout-minutes: 10
run: |
pytest -xsv tests --reruns 3 --reruns-delay 10
hybrid_setup:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: ["3.12"]
os: ["blacksmith-4vcpu-ubuntu-2204"]
steps:
- name: Install apt packages
run: |
sudo apt update
sudo apt-get install -y procps # Install procps to enable sysctl
sudo sysctl -w vm.overcommit_memory=1
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: useblacksmith/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: '**/setup.py'
- name: Install dependencies
working-directory: examples/hybrid_setup
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Run tests
working-directory: examples/hybrid_setup
timeout-minutes: 10
run: |
pytest -xsv tests --reruns 5 --reruns-delay 10
vhost:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: ["3.12"]
os: ["blacksmith-4vcpu-ubuntu-2204"]
steps:
- name: Install apt packages
run: |
sudo apt update
sudo apt-get install -y procps # Install procps to enable sysctl
sudo sysctl -w vm.overcommit_memory=1
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: useblacksmith/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: '**/setup.py'
- name: Install dependencies
working-directory: examples/vhost
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Run tests
working-directory: examples/vhost
timeout-minutes: 10
run: |
pytest -xsv tests --reruns 3 --reruns-delay 10
Docker¶
This pipeline is used to to make sure the provided Dockerfiles from the plugin are built successfully.
name: Docker
on:
pull_request:
branches: [ 'main']
paths:
- '.github/workflows/docker.yml'
- 'src/pytest_celery/vendors/worker/**'
- "**.py"
- "**.txt"
- "**.toml"
- "tox.ini"
- 'Dockerfile'
- "poetry.lock"
push:
branches: [ 'main']
paths:
- '.github/workflows/docker.yml'
- 'src/pytest_celery/vendors/worker/**'
- "**.py"
- "**.txt"
- "**.toml"
- "tox.ini"
- 'Dockerfile'
- "poetry.lock"
jobs:
build-worker:
runs-on: blacksmith-4vcpu-ubuntu-2204
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
- name: Build Celery Worker
run: cd src/pytest_celery/vendors/worker && docker build -t pytest-celery-worker .
Continuous Deployment¶
Added in version 1.0.0.
The CD is configured to deploy a new release to the PyPI package index.
The following release workflow is triggered automatically when a new released is tagged and published on GitHub.
name: Deploy to PyPI
on:
release:
types: [published]
jobs:
deploy:
runs-on: blacksmith-4vcpu-ubuntu-2204
env:
POETRY_VIRTUALENVS_CREATE: "false"
steps:
- uses: actions/checkout@v4
- name: Install poetry
run: |
pipx install poetry
pipx inject poetry poetry-bumpversion
- name: Build
run: |
poetry version ${{ github.ref_name }}
poetry build
- name: Publish
run: |
poetry config pypi-token.pypi ${{ secrets.PYPI_API_TOKEN }}
poetry publish
Note
The poetry version
command should say there’s nothing to change, because this should have been done in the PR that prepared the release.
Release Steps¶
1. Celery Tests¶
The plugin is used as the official testing infrastructure for Celery. Every new release requires manually testing that the new version works as expected with the Celery test suite.
To run the Celery test suite with the new version of the plugin, modify the Celery test environment as follows:
test.txt¶
Comment out the pytest-celery installation line.
tox.ini¶
Add -e "../pytest-celery[all]"
to the deps list.
[testenv]
...
deps=
-e "../pytest-celery[all]"
-r{toxinidir}/requirements/test.txt
...
And then execute with tox:
tox -e 3.12-smoke -- -n auto
This will run the Celery test suite with the new version of the plugin in edit mode, allowing you to test the new version before releasing it and tweaking it if necessary to debug any issues.
Tip
Use the following snippet to run all of the tests with the new version of the plugin:
Pull RabbitMQ & Redis images for the integration tests:
docker run -d -p 6379:6379 --name redis redis:latest
docker run -d -p 5672:5672 -p 15672:15672 --name rabbitmq rabbitmq:management
Unit Tests:
tox -e 3.12-unit
Integration Tests:
docker start rabbitmq redis
tox -e 3.12-integration-rabbitmq_redis
docker stop rabbitmq redis
Unit & Integration & Smoke Tests:
tox -e 3.12-unit && docker start rabbitmq redis && tox -e 3.12-integration-rabbitmq_redis && docker stop rabbitmq redis && tox -e 3.12-smoke -- -n auto
Warning
The instructions above assume you have the pytest-celery and celery repositories cloned in the same root directory.
2. Release PR¶
To make a new release, you need to create a new PR with one of these titles.
Official Release: Prepare for release: vX.Y.Z
Pre-release: Prepare for (pre) release: vX.Y.Z
The PR should contain the following changes:
Version bump using poetry version command.
Changelog update.
This PR will be used as a double check for the CI to make sure everything passes successfully before releasing the new version. Once this PR is merged, the last step is to release a version on GitHub. This will trigger the CD pipeline to deploy the new release to PyPI automatically.
SemVer¶
If you’re not sure how to number the version, consult the SemVer documentation.
3. Post-release¶
After the release is done, you should update the official Celery to use the new version of the plugin, in the same test.txt that you modified earlier.
Future Releases¶
Releases should be planned in the official Milestones of the repository. Each milestone should include in its description what is planned for the release and when is the expected release date.