GitLab CI and poetry-dynamic-versioning

The poetry-dynamic-versioning plugin enables configuring Poetry to automatically generate the package version number from your VCS (e.g. Git), rather than from the version field in pyproject.toml.

For example, you can tag a commit in Git, and the release built from that commit will use the name of the tag.

Using poetry-dynamic-versioning with GitLab §

The following steps assume you are building a Python wheel using Poetry and Tox.

1. Install the plugin for Poetry:

   poetry self add "poetry-dynamic-versioning[plugin]"`

2. Enable the plugin for your project:

   poetry dynamic-versioning enable

3. In your .gitlab-ci.yml, set GIT_DEPTH: 0 for your build job:

   build_wheel:py310:
     stage: build_wheel
     variables:
       GIT_DEPTH: 0
     script:
       - tox -e py310-build
     artifacts:
       paths:
         - dist/*

4. In your .gitlab-ci.yml, pass the current branch (CI_COMMIT_REF_NAME) into Tox as an environment variable:

   build_wheel:py310:
     stage: build_wheel
     variables:
       GIT_DEPTH: 0
     script:
       - MY_BRANCH=${CI_COMMIT_REF_NAME} tox -e py310-build
     artifacts:
       paths:
         - dist/*

   This assumes your tox.ini looks something like this:

   [testenv:py{310,311}-build]
   commands =
       # ...snip...
       poetry build

5. In the tox.ini, pass through the MY_BRANCH environment variable:

   pass_env =
       MY_BRANCH

6. Update pyproject.toml to specify pattern and format-jinja fields. The pattern is used to extract the version number from the tag, and the format-jinja is used to generate the version string. This is what I use:

   [tool.poetry-dynamic-versioning]
   enable = true
   vcs = "git"
   pattern  = "^(?P<base>\\d+\\.\\d+\\.\\d+)(-?((?P<stage>[a-zA-Z]+)\\.?(?P<revision>\\d+)?))?"
   format-jinja = """
       {%- if distance == 0 -%}
           {{ base }}
       {%- elif env["MY_BRANCH"] == "main" -%}
           {{ bump_version(base) }}.alpha{{ distance }}+g{{ commit }}
       {%- else -%}
           {{ base }}.dev{{ distance }}+g{{ commit }}
       {%- endif -%}
   """

That’s it!

If you’d like to know why GIT_DEPTH and MY_BRANCH are necessary, read on.

Ensuring poetry-dynamic-versioning works with GitLab §

Unfortunately, there are a couple of issues that can prevent poetry-dynamic-versioning from fully working with GitLab:

• By default, GitLab performs a “detached HEAD” checkout, which can prevent the branch variable in format-jinja from working.
• By default, GitLab performs a shallow clone, which may prevent other fields from working if the last tag was too many commits ago.

The details for these fixes are documented below.

Using MY_BRANCH §

The branch variable §

The branch variable is the name of the current branch, or if that fails, None:

branch (string or None)

This allows you to change the build version depending upon the branch. For example, to have all main builds use the version hello_from_main, and all branch builds have the version hello_from_branch, you would do:

format-jinja = """
    {%- if branch == "main" -%}
        hello_from_main
    {%- else -%}
        hello_from_branch
    {%- endif -%}
"""

This works locally, but unfortunately under GitLab CI the branch variable returns None, preventing it from working.

Verifying branch is None under GitLab CI §

You can verify this with the following:

format-jinja = """
    TEST_{{- branch -}}_TEST
"""

This will fail locally and on GitLab CI, because TEST_<blah>_TEST is not a valid Python version. But crucially, it will fail differently!

Assuming you are running it on the branch main:

• Locally it will fail with TEST_main_TEST.
• On GitLab CI it will fail with TEST_None_TEST.

Investigating why we get None under GitLab CI §

Looking at the plugin source, the plugin gets the Version object from dunamai. Following through in the dunamai source, it gets the branch variable from Git:

code, msg = _run_cmd("git symbolic-ref --short HEAD", path, codes=[0, 128])
if code == 128:
    branch = None
else:
    branch = msg

If we try this out manually in GitLab CI, we see the issue:

$ git symbolic-ref --short HEAD
fatal: ref HEAD is not a symbolic ref

The Git command is failing, which explains why branch is defaulting to None under GitLab CI.

It fails because Git is checking out a commit, rather than a branch, and so it has a “detached head”.

For example, see this GitLab tutorial:

Checking out 7226fc70 as detached HEAD (ref is main)...

Fixing the branch variable §

Since GitLab already knows the current branch, and poetry-dynamic-versioning provides the env object for reading from the environment, we can avoid the “deatched HEAD” issue by passing the current branch from GitLab into poetry-dynamic-versioning. Since Tox does not pass environment variables by default, it must be added into pass_env for this method to work.

Previously I used this solution, but it has some shortcomings.

Using GIT_DEPTH §

The Dunamai README warns that it needs access to the full version history, and suggests using GIT_DEPTH: 0 in GitLab. This prevents GitLab from doing a shallow clone, and ensures that poetry-dynamic-versioning has enough information to populate variables such as distance.

variables:
  GIT_DEPTH: 0

Appendix §

Attempted fix of detached HEAD with checkout §

An alternate fix is to call the following in GitLab CI:

git checkout "$CI_COMMIT_REF_NAME"

Call it in before_script (or wherever makes more sense), and it will ensure the checkout does not have a detached HEAD. This will allow git symbolic-ref --short HEAD to get the branch name, and in poetry-dynamic-verisoning, the branch variable will be populated correctly.

This can however lead to issues where the wrong commit is used for the build!

Attempted fix with GIT_STRATEGY §

I attempted setting GIT_STRATEGY: clone to avoid the shallow clone issue, but it did not appear to be doing anything.

Debugging GitLab CI §

Using the CI_DEBUG_TRACE GitLab CI variable is very useful for debugging:

variables:
  CI_DEBUG_TRACE: "true"