Deploy your Python package with Github Actions

Credits: © 2021 GitHub, Inc.

You probably know that GitHub offers a service of CI/CD, called GitHub Actions. It’s free, incredibly simple and can automate your tasks for production.

I’m gonna talk about GitHub Actions from a specific perspective : publish a python package in PyPI. The way I’m going to show you is probably not the best one, but it lets you do certain things automatically, to ensure a “good” process. You can course do the same things in several ways. GitHub Actions is very flexible !

In this post, I assume that you are familiar with GitHub & Python packaging, and you have a GitHub repository to follow along.

Step 0 : Defining a workflow

With Github Actions you can create “workflows”.

A workflow is a “pipeline” that will be triggered on events. For example, you can create a workflow triggered on “push”, “pull request”, “release”, you got it : on a specific event.

The syntax is pretty simple. It’s a YAML file. For example, let’s see how we can declare a simple workflow, with a name, and a trigger on “push” events.

Github recommends storing actions in the .github directory, at the root of your repository.

For example, .github/actions/action-a and .github/actions/action-b.

In this example, the workflow file is named “main.yml”, correctly placed at the root of my project folder :

This is the location of the file that we will code.

name: NameOfMyWorkflow
on: push

If i translate this code, it means that my workflow’s name is “NameOfMyWorkflow” and it will be triggered on “push” event. You see ? I told you that it was simple 😉

If you want to trigger your workflow on several events, you can use this syntax :

on: [push, pull_request, ...]

Next, we will focus more on Python package deployment on PyPI.

We are gonna specify to our workflow, that it will have jobs.

jobs:  
  build_test_publish:

In our case, we will have only one job. Why ? Because for example :

If you are building your package on a first job, and deploy your package to PyPI in a second job, your second job will not have access to your builded package (wheel file). So let’s put everything in a single job, to ensure that we have access to it.

Note: You can synchronize your jobs, create dependencies, etc. See GitHub Actions Jobs

on: push
jobs:  
  build_test_publish:
  name: "Build & Test"
  runs-on: ubuntu-latest
  strategy:
    max-parallel: 1
    matrix:
      python-version: [3.6, 3.7, 3.8]

In this part, you can see that i added a “runs-on” element. It define the type of machine to run the job on. The “strategy” creates a build matrix for your jobs. You can define different variations to run each job in.

I set “max-parallel” to 1, because I want to deploy my jobs one by one, in order to wait until all builds are done to push our package.

“Matrix” is the definition of my set of values. In this case, python-version.

With this “Matrix python-version” I will be able to access a variable.

${{ matrix.python-version }}

You will see that “${{ myvar.properties }}” is the syntax to use variable.

My job will iterate on each value of my matrix, and expose a value through this variable.

Let’s create steps for our job now.

STEP 1 : Let’s set up a python environment

on: push
jobs:  
  build_test_publish:
  name: "Build & Test"
  runs-on: ubuntu-latest
  strategy:
    max-parallel: 1
    matrix:
      python-version: [3.6, 3.7, 3.8]
  steps:
    - uses: actions/checkout@v2
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v2
      with:
        python-version: ${{ matrix.python-version }}

“steps” defines…? steps, literally 😛

“uses” instruction tells our script to call an action published in GitHub action, and execute it. “actions/checkout@v2” checkout your repository.

The name of your step will define the node that you see in your Actions section of your repository.

I use one more external action, “actions/setup-python@v2” to set up python on the machine.

Step “Set up Python” matrix.python-version

Finally, the “with” instruction makes it possible to run my step with a var called “python-version”, which is the value of my matrix, on the current iteration.

STEP 2 : Install our dependencies !

The dependencies installation depends on which method you use. In this example, we will use a “extra requires” of our package.

- name: Install dependencies
  run: |
    python -m pip install --upgrade pip wheel setuptools
    if [ -f requirements.dev.txt ]; then pip install -r requirements.dev.txt;  else pip install .[testing]; fi

We created a step, called “Install dependencies”, which run the following commands :

• Upgrade pip, wheel & setuptools
• If requirements.dev.txt is present, then, install the package list that the file “requirements.dev.txt” contains. Else, install the package with testing extra-requires.

STEP 3 : Let’s lint our code with Flake8 !

We don’t want to have a bad code syntax in our package, so let’s check it !

- name: Lint with flake8
  run: |
    # stop the build if there are Python syntax errors or undefined names
    flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
    # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
    flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics

Now, we got our packages, and we have checked our code syntax.

STEP 4 : Testing

We need to test our code, so we need a simple step with pytest to verify that we are good with this push.

- name: Test with pytest
  run: |
    pytest

STEP 5 : Build our Python package

As you probably know, if you want to build your python package, you should have a setup.py, setup.cfg.

- name: Build python package
  run: |
    python setup.py bdist_wheel

After this, we know that our code has a good code quality, it’s tested, and built a wheel file.

STEP 6 : Push our package to PyPI

This last step is particular, because you don’t want your package on PyPI to be updated on every push event.

Moreover, we want to push our package only once. If you push two times a package with version 1.0, PyPI will reject your package push because it already exist.

So the first thing to do is to add a condition to this step.

We want this step to be triggered only :

• If all previous steps are on success
• If our current push event is a tag (to know at which version of package we are)
• If we are in the last iteration of our matrix

Remember ? We created a python-version matrix, and our job will iterate through this matrix. So it will be executed 3 times.

Let’s add this condition.

- name: Deploy to PyPI
  if: success() && startsWith(github.ref, 'refs/tags') && matrix.python-version == 3.8

Now we can add our external action with :

uses: pypa/gh-action-pypi-publish@master

And our variables to inject in this action we use :

with:
  user: __token__
  password: ${{ secrets.PYPI_API_TOKEN }}

To see more details about the deploy to PyPI step, take a look to this action.

As you can see, we call a variable secrets.PYPI_API_TOKEN.
So you will need to :

• create an API_TOKEN in your PyPI project
• Add your token to secrets variable for GitHub Actions (See here documentation)

Great ! We have finished creating our workflow 🥳

You can test it now with a simple push, and try to deploy to PyPI with a tag.

You can find the entire code here

MaxGdr/deploy-python-package-on-pypi

I hope that this post helps you, and if you have some suggestions, leave comments !

This workflow has been implemented initially to manage the CI/CD of an Open source library SHAPASH

Shapash: MAIF releases a new solution for a more transparent AI

MAIF/shapash