Poetry: Running Black and isort with pre-commit hooks

Git Hooks provide “a way to fire off custom scripts when certain important actions occur”, and are typically stored in .git/hooks within your Git repository. For example, the Git pre-commit hook runs just before files are committed (e.g. when you run git commit), and so is a useful time to perform linting.

Related is a tool with the same name: pre-commit. This tool expects there to be a .pre-commit-config.yaml configuration file in the root of your Git repository. After cloning a Git repository, developers must then run pre-commit install: this invokes the pre-commit tool, which then writes the Git pre-commit hooks to .git/hooks. To try and help distinguish between the Git hook and the tool, I will use the phrase pre-commit tool whenever talking about the tool.

Using the pre-commit tool is particularly useful because “client-side hooks are not copied when you clone a repository”¹. The tool introduces the .pre-commit-config.yaml file, which should be commited into the root of the Git repo, so it will be included in a clone². Then, all the developer has to do is run pre-commit install³, assuming that the pre-commit tool is already installed.

Another significant advantage of the pre-commit tool is that, by default, it passes a list of only the files that have changed into your pre-commit hook entrypoint. In other words, if you use the pre-commit tool to run a linter, that linter will only check the files that have changed. This can be a big time saver, since it avoids needlessly linting all the files in your repository each time you commit.

In these notes we will install the pre-commit tool for a Python project using Poetry, and use pre-commit to automatically run black (to format code) and isort (to order imports). Typically, the .pre-commit-config.yaml would list each linting package, including the upstream location and version number, so that the tool can install these for you automatically. To avoid duplicating dependencies by listing them both in .pre-commit-config.yaml and pyproject.toml, we will integrate the pre-commit tool closely with Poetry.⁴

Install development dependencies §

The official documentation for pre-commit provides steps for installing it system-wide, but since it is just a Python package, we can install it as a development dependency within Poetry. This simplifies the setup process for other developers.

Since black and isort will be used by pre-commit, we will install them as development dependencies as well:

poetry add -G dev pre-commit black isort

To create/update the Poetry virtual environment with the development dependencies, run:

poetry install --with dev

Create .pre-commit-config.yaml §

Create a .pre-commit-config.yaml which runs Black using Poetry.⁵

fail_fast: true
repos:
- repo: local
  hooks:
    - id: black
      name: black
      entry: poetry run black
      language: system
      types: [file, python]
      files: '^(src|test)/'

Going through each of these lines in detail:

fail_fast: true

This tells the pre-commit tool to stop after the first failure.

repos:

Begins the list of “repository mapping[s]”, i.e. where to get the code from for each pre-commit command.

- repo: local

Typically, this would specify the URL for the Git repository required for this hook. But the special keyword local can be used when the required code is provided in or by the local repository. In our case, the Poetry virtual environment will provide the required code.

hooks:

Begins a list of hooks provided by this repo.

- id: black

The ID of this hook. Seems to be arbitrary but let’s go with black to reflect the command being run.

name: black

The display name of this hook, as it appears in the terminal when the hook runs.

entry: poetry run black

The command to run. We use poetry run ... to run the command within the Poetry virtual environment. This calls black which executes the Black package, installed within the Poetry virtual environment. Note that we do not pass in any directories or files to include/exclude in the filter. As mentioned earlier, the pre-commit tool automatically passes in the list of files that have changed to our entrypoint. (This can be disabled with pass_filenames: false if necessary).

language: system

This specifies the programming language required for running this hook, of which many are supported. Since our hook does not call any specific language, but just expects the system to provide the poetry command, we use the system option.

types: [file, python]

This filters the files that are automatically passed into the entrypoint, and using types allows us to filter this list by file type. Specifically, the [file, python] type filter means that only Python files are included (e.g. those ending in *.py). It is possible to write a “raw” regex filter using files and exclude, but for most cases using types is sufficient and also less error-prone.

files: '^(src|test)/'

This provides a regex for filtering which files to include. This is not necessary but can be useful if you only want to run the command on certain parts of the code. For example, if you are incrementally adding linting/typing support, you might begin with files: '^(src/my_project/folder_a|src/my_project/folder_b|src/other_project/folder_c)' to initially target these three folders within two sub-packages.

Install the pre-commit hooks §

Now that we have our configuration, first we must add it to Git, and then we can install the pre-commit hooks.

git add .pre-commit-config.yaml
poetry run pre-commit install

Now, whenever you do git commit ..., it will run the poetry run python -m black ... command like this:

$ git commit -m "Test commit"
black....................................................................Passed
[test-branch a4b13de] Test commit
 3 file changed, 2 insertions(+), 3 deletions(-)
 create mode 100644 .pre-commit-config.yaml

Extend configuration to support isort §

We can easily extend our .pre-commit-config.yaml to add isort as a hook:

fail_fast: true
repos:
- repo: local
  hooks:
    - id: black
      name: black
      entry: poetry run black
      language: system
      types: [file, python]
      files: '^(src|test)/'
    - id: isort
      name: isort
      entry: poetry run isort
      language: system
      types: [file, python]
      files: '^(src|test)/'

Since isort is “provided” by the local repo, we do not need to add another repo, but can add it as a second hook within the same repo.

By default, isort and black will “fight” as their default configuration disagrees on how to handle spacing between imports. To make them play nicely, you must enable the black profile in isort by updating your pyproject.toml with the following:

[tool.isort]
profile = "black"

For further isort configuration suggestions, see this note.

Advanced usage §

Normally you always want the pre-commit hooks to run when committing, but in some circumstances you might want to do one without the other. These two scenarios are covered below:

Running the pre-commit hooks without committing §

To run the pre-commit tool hooks without actually comitting anything to Git, use the pre-commit run command within Poetry. If you have not made any changes then you will get a (no files to check)Skipped message, because no files are being passed in. This is useful to verify that the pre-commit tool is indeed only passing in files that have changed:

$ poetry run pre-commit run
black................................................(no files to check)Skipped
isort................................................(no files to check)Skipped

If you want to run the pre-commit tool hooks on all files, regardless of whether they have changed, use the --all-files option:

$ poetry run pre-commit run --all-files
black....................................................................Passed
isort....................................................................Passed

Committing without running the pre-commit hooks §

To commit something to Git without any of the pre-commit hooks running, do:

git commit --no-verify

The --no-verify options tells Git not to invoke anything in .git/hooks.

Using Poetry for pre-commit but not for packaging §

For basic repositories, the above approach assumes you only have a single, top-level pyproject.toml file. For more advanced repositories you may have multiple Python packages each with their own pyproject.toml, or you may have a mixture of languages in use and so your Python package may be nested in a directory somewhere.

In any case, the above approach will fall short because the .pre-commit-config.yaml has entry commands that assume poetry run <blah> will work in the root of your project, but that will not work if pyproject.toml is not in the root.

A simple solution is to create a top-level pyproject.toml with package mode disabled, i.e. use Poetry to manage dependencies but not to create a Python package. This can be a tidy way to handle Python developer dependencies for more complex projects.

To do so, simply add package-mode = false in [tool.poetry] within pyproject.toml. You then cannot have any packages = [...] sections, because we are telling Poetry that this pyproject.toml does not have any packages:

[tool.poetry]
# ...
package-mode = false

# Cannot use `packages` when using `package-mode = false`.
# packages = [...]

If you forget to set package-mode = false and simply omit packages = [...] then Poetry will give a warning (or soon to be an error):

Warning: The current project could not be installed: No file/folder found for package <YOUR_PACKAGE>
If you do not want to install the current project use --no-root.
If you want to use Poetry only for dependency management but not for packaging, you can disable package mode by setting package-mode = false in your pyproject.toml file.
In a future version of Poetry this warning will become an error!

If this structure means your Python code is now nested inside another directory, the files option in .pre-commit-config.yaml may need updating to reflect that new structure, e.g.:

fail_fast: true
repos:
- repo: local
  hooks:
    - id: black
      name: black
      entry: poetry run black
      language: system
      types: [file, python]
      files: '^py_package/(src|test)/'
    - id: isort
      name: isort
      entry: poetry run isort
      language: system
      types: [file, python]
      files: '^py_package/(src|test)/'

Running under CI §

To run the pre-commit hooks under CI, it is as simple as:

poetry install --with dev
poetry run pre-commit run --all-files

The exit code will indicate success or failure.