smhk

Poetry: Offline installation of packages

All notes in this series:

These notes are about how to perform an offline installation of a Python application using poetry install, not about how to install Poetry itself offline.

Currently, Poetry does not support offline installation of packages. As a work-around, you can run a local PyPI server on your offline machine, then configure poetry install to use the local PyPI server. First though, you need to gather all the packages your application needs. The following notes cover all these steps.

Perform offline package installation with Poetry §

The initial setup §

In my case, my Python application has a pyproject.toml with both public and private dependencies. These are available via two Poetry sources: one is an internal mirror of the public PyPI (pypi_mirror), and the other is an internal private PyPI repository (pypi_private):

[[tool.poetry.source]]
name = "pypi_mirror"
url = "<URL to PyPI mirror>"
priority = "default"

[[tool.poetry.source]]
name = "pypi_private"
url = "<URL to private PyPI>"
priority = "supplemental"

I have two computers: an online machine which has Internet access, and can reach both sources; and an offline machine, which is effectively air-gapped.

Download packages §

First, we need to download all the packages necessary for the Python application using the online machine.

Poetry does not provide a way to just download packages, but you can get the same effect by combining Poetry and Pip as follows1:

poetry export --with-credentials > requirements.txt
poetry run pip download -r requirements.txt -d packages/

This will leave you with a packages directory that includes all the packages, from both the public and private sources.

How this method works §

The --with-credentials option is crucial for this method to work with the private PyPI server. This ensures that the requirements.txt includes the credentials required for the pip download command to fetch the private packages. The resulting requirements.txt will look something like:

--extra-index-url <URL to private PyPI>
--index-url <URl to PyPI mirror>

aiofiles==23.2.1 ; python_version >= "3.10" and python_version < "3.13" \
    --hash=sha256:19297512c647d4b27a2cf7c34caa7e405c0d60b5560618a29a9fe027b18b0107 \
    --hash=sha256:84ec2218d8419404abcb9f0c02df3f34c6e0a68ed41072acfb1cef5cbc29051a
annotated-types==0.6.0 ; python_version >= "3.10" and python_version < "3.13" \
    --hash=sha256:0641064de18ba7a25dee8f96403ebc39113d0cb953a01429249d5c7564666a43 \
    --hash=sha256:563339e807e53ffd9c267e99fc6d9ea23eb8443c08f112651963e24e22f84a5d
ansi2html==1.8.0 ; python_version >= "3.10" and python_version < "3.13" \
    --hash=sha256:38b82a298482a1fa2613f0f9c9beb3db72a8f832eeac58eb2e47bf32cd37f6d5 \
    --hash=sha256:ef9cc9682539dbe524fbf8edad9c9462a308e04bce1170c32daa8fdfd0001785
# etc...

If you omit --with-credentials, then the pip download command will work until it hits a package in your private PyPI server (assuming it requires authentication), at which point it will error.

Install the local PyPI server §

On the offline machine, install pypiserver2. It is a single Python wheel with no dependencies, so it is easy to bootstrap if necessary. Copy across the pypiserver wheel, then from the same directory run:

pip install --no-index --find-links=. pypiserver

Run the local PyPI server §

Then copy across the packages directory which was created earlier, and within that directory run:

pypi-server run .

This will start to serve up the packages at http://localhost:8080/simple/. You can visit in your browser to confirm.

Modify your pyproject.toml §

In order to use the local PyPI server on the offline machine, the pyproject.toml sources need editing. Since we now have one PyPI server that provides both the public and private packages, we can change both sources to point at it3:

[[tool.poetry.source]]
name = "pypi_mirror"
url = "http://localhost:8080/simple/"
priority = "default"

[[tool.poetry.source]]
name = "pypi_private"
url = "http://localhost:8080/simple/"
priority = "supplemental"

Perform the offline install §

Perform the offline install by running:

poetry install

This will install your application on the offline machine, fetching all the packages from your local PyPI server.

That’s it! 🎉

A note on the warning §

When running poetry install you will get the following warning:

Warning: poetry.lock is not consistent with pyproject.toml. You may be getting improper dependencies.

In this case, it is safe to ignore the warning. It occurs because the poetry.lock file contains a hash of the pyproject.toml file which is updated when you run poetry lock. We modified pyproject.toml without running poetry lock, so the hash has been broken, but the only change has been to point at our local PyPI server which is serving up the same packages.

Future work §

Read on if you’d like to contribute to a (potentially) better solution.

Buried within this very relevant Poetry issue is this nugget of a WIP PR4 which adds a poetry download command. If this command works it would simplify the download steps listed above. Unfortunately it was never merged and has not been updated in two years. Regardless, I tried to give it a go. To install it into C:\poetry\poetry-dl-fork (to avoid clobbering my main Poetry installation), I ran:

curl -sSL https://install.python-poetry.org | POETRY_HOME=/c/poetry/poetry-dl-fork python - --git https://github.com/nikolaikopernik/poetry.git@fix_2184_download

However, when I tried to run poetry download I got this error:

$ poetry download
C:\poetry\poetry-dl-fork\venv\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
Traceback (most recent call last):
  File "C:\poetry\poetry-dl-fork\venv\lib\site-packages\cleo\application.py", line 329, in run
    exit_code = self._run(io)
  File "C:\poetry\poetry-dl-fork\venv\lib\site-packages\poetry\console\application.py", line 181, in _run
    return super()._run(io)
  File "C:\poetry\poetry-dl-fork\venv\lib\site-packages\cleo\application.py", line 423, in _run
    exit_code = self._run_command(command, io)
  File "C:\poetry\poetry-dl-fork\venv\lib\site-packages\cleo\application.py", line 465, in _run_command
    raise error
  File "C:\poetry\poetry-dl-fork\venv\lib\site-packages\cleo\application.py", line 446, in _run_command
    self._event_dispatcher.dispatch(event, COMMAND)
  File "C:\poetry\poetry-dl-fork\venv\lib\site-packages\cleo\events\event_dispatcher.py", line 23, in dispatch
    self._do_dispatch(listeners, event_name, event)
  File "C:\poetry\poetry-dl-fork\venv\lib\site-packages\cleo\events\event_dispatcher.py", line 84, in _do_dispatch
    listener(event, event_name, self)
  File "C:\poetry\poetry-dl-fork\venv\lib\site-packages\poetry\console\application.py", line 276, in configure_env
    from poetry.utils.env import EnvManager
  File "C:\poetry\poetry-dl-fork\venv\lib\site-packages\poetry\utils\env.py", line 39, in <module>
    from poetry.core.semver.helpers import parse_constraint
ModuleNotFoundError: No module named 'poetry.core.semver'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\sam\.pyenv\pyenv-win\versions\3.10.11\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\sam\.pyenv\pyenv-win\versions\3.10.11\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\poetry\poetry-dl-fork\bin\poetry.exe\__main__.py", line 7, in <module>
  File "C:\poetry\poetry-dl-fork\venv\lib\site-packages\poetry\console\application.py", line 363, in main
    return Application().run()
  File "C:\poetry\poetry-dl-fork\venv\lib\site-packages\cleo\application.py", line 334, in run
    self.render_error(e, io)
  File "C:\poetry\poetry-dl-fork\venv\lib\site-packages\poetry\console\application.py", line 172, in render_error
    self.set_solution_provider_repository(self._get_solution_provider_repository())
  File "C:\poetry\poetry-dl-fork\venv\lib\site-packages\poetry\console\application.py", line 352, in _get_solution_provider_repository
    from poetry.mixology.solutions.providers.python_requirement_solution_provider import (
  File "C:\poetry\poetry-dl-fork\venv\lib\site-packages\poetry\mixology\__init__.py", line 5, in <module>
    from .version_solver import VersionSolver
  File "C:\poetry\poetry-dl-fork\venv\lib\site-packages\poetry\mixology\version_solver.py", line 14, in <module>
    from .failure import SolveFailure
  File "C:\poetry\poetry-dl-fork\venv\lib\site-packages\poetry\mixology\failure.py", line 6, in <module>
    from poetry.core.semver.helpers import parse_constraint
ModuleNotFoundError: No module named 'poetry.core.semver'

I’ve not dug into it much, but I presume this error happened because poetry-core has moved on a lot since this MR.

I attempted to rebase this branch against master in case it was a trivial patch, but there were a lot of conflicts, and it looks like files have moved around a bit since the MR was made. It may be possible to resurrect this branch without too much work, though I’m not familiar enough with the Poetry source to know.

  1. Thanks to this answer, this answer and this answer which provided a good starting point, but were missing the --with-credentials required for private repositories. ↩︎

  2. Instead, the local PyPI server could be installed on a separate machine so long as the offline machine has network access to it, but then you will need to update the hostname/IP in the pyproject.toml file accordingly in the “Modify your pyproject.toml” section. ↩︎

  3. Ideally I’d like to find a way to do this without editing the pyproject.toml. This could be done by messing around with the hosts file, but that doesn’t seem ideal. ↩︎

  4. Thanks to nikolaikopernik for beginning to tackle this issue, even though ultimately it was not merged. ↩︎

All notes in this series: