Alright, there are 100 ways in Python land to do dependency management and it seems the mind share of projects shifts every two years. So, whatever, a summary of how I like to use poetry.

Installation

Poetry should not be installed in the same virtual environment as the project, because otherwise it may break when installing additional dependencies. My two go-to solutions:

1. Use a platform-specific package manager: e.g., on Arch pacman, or brew on MacOS:

   pacman -Syu python-poetry
   

   I prefer this on my local machine.
2. Use pipx. pipx installs tools into their own virtual environments; quite nice and preferrable whenever I need a specific poetry version

   $ sudo pacman -Syu install pipx
   $ pipx install poetry
   

The basic Poetry workflow

1. List the top-level dependnecies in the pyproject.toml file (plus some additional fluff):

   [tool.poetry]
   name = "project-name"
   version = "0.1.0"
   description = ""
   readme = "README.md"
   package-mode = false
   
   [tool.poetry.dependencies]
   python = "^3.11"
   Flask = "^3.0.3"
   pandas = "^2.0"
   
   [build-system]
   requires = ["poetry-core>=1.0.0"]
   build-backend = "poetry.core.masonry.api"
   

   ^x.y means <(x+1, so no major upgrades - okay for an application, bad if you develop a library, because you lock all your consumers out of a version upgrade. For libraries, >=x.y is more appropriate.
2. Let poetry resolve the dependencies: poetry lock. This writes a poetry.lock file that lists the exact dependency versions.
3. Install the dependencies to a virtual environment. See the next section.

Using Poetry for local development

Poetry can manage virtual environments, but because I keep forgetting the details and because I prefer to have the virtual environment located in the project root, this is good to know:

• If poetry detects that it is running in an activated virtual environment (it seems it checks for $VIRTUAL_ENV), it will install the dependencies into this environment.
• If you have an existing virtualen environment in a .venv directory, it uses that.
• If you set virtuelenvs.in-project true, poetry install will create a new virtual environment in ./.venv, if it doesn't exist, yet: poetry config --local virtualenvs.in-project true (will write to poetry.toml).

Because old habits die hard, I usually create the virtual environments manually: python3 -m venv .venv and then poetry can do its thing.

My second requirement is that I mostly want dependency management to be done, but I don't need to install my project as a package ("deployment"). In old poetry versions you would poetry install --no-root, but nowadays you can just define your package-mode = false in your pyproject.toml:

[tool.poetry]
name = "project-name"
version = "0.1.0"
description = ""
readme = "README.md"
package-mode = false

[tool.poetry.dependencies]
python = "^3.11"
Flask = "^3.0.3"
pandas = "^2.0"

[tool.poetry.group.test]
optional = true

[tool.poetry.group.test.dependencies]
pytest = "^8.3.3"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Poetry always resolves the version dependnecies for all groups in combination, but you can still only install a subset with --only, --without and --with:

poetry install --only main --no-root
poetry install --without test --no-root

Poetry and Docker

Let's get our application into a Docker container with poetry. The idea in a nutshell:

• Install poetry in a separate virtual environment
• Create another virtual environment for the project and install the dependencies into it with poetry.
• Move the virtual environment to a clean image.

Some things to keep in mind about docker files:

• Docker caches layers. If we change e.g. an ENV or ARGS definition, everything afterwards will be re-run. Same, if any of of the files used in a command are updated Thus, we try to have commands that invalidate cached layers as late as possible.
• Potentially use a .dockerignore file to only have the required files in the build context.
• ENV environment variable definitions can be referenced across stages. ARG definitions are only available during the build and only in a single stage. If we want to re-use them in later stages, we need to reference them again theree with ARG MY_NAME.

Example:

# General setup:
# Build:
# sudo docker buildx build -t poetry_testing:latest [--progress=plain] .
# Run image and drop in the shell to debug:
# sudo docker run -it poetry_testing:latest bash

ARG SOURCE_IMAGE="python:3.12-slim-bookworm"
ARG PROJECT_VENV_PATH="/opt/venv"
# Poetry binary pipx installs.
ARG POETRY_BIN_PATH="/root/.local/bin/poetry"

# 1. Install poetry as a build tool via pipx
FROM "$SOURCE_IMAGE" AS poetry
RUN apt-get update \
    && apt-get install --no-install-suggests --no-install-recommends --yes pipx \
    && rm -rf /var/lib/apt/lists/*
ARG POETRY_VERSION=1.8.3
RUN pipx install "poetry==${POETRY_VERSION}"

# 2. Install the base dependencies in a virtual environment
FROM poetry AS base
ARG BUILD_DIR="/build"
WORKDIR "$BUILD_DIR"
COPY ./pyproject.toml ./poetry.lock
ENV POETRY_REQUESTS_TIMEOUT=60 \
    PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1
ARG PROJECT_VENV_PATH
RUN python3 -m venv --without-pip "${PROJECT_VENV_PATH}"
# Activate venv in this stage - poetry will use it.
ENV VIRTUAL_ENV="$PROJECT_VENV_PATH"
COPY ./pyproject.toml ./poetry.lock .
ARG POETRY_BIN_PATH
RUN "$POETRY_BIN_PATH" install \
    --no-root \
    --no-cache \
    --no-interaction \
    --no-ansi \
    --without=test

# 3. Final image: Use clean image, just copy the virtual environment.
FROM "$SOURCE_IMAGE"
WORKDIR /opt/app
ARG PROJECT_VENV_PATH
COPY --from=base "$PROJECT_VENV_PATH" "$PROJECT_VENV_PATH"
COPY . .
# Don't create pyc files. If we want them, we should create them during the
# build, so that the image can be read-only.
ENV PYTHONDONTWRITEBYTECODE=1
# Don't buffer stdout/stderr
ENV PYTHONUNBUFFERED=1
ENTRYPOINT ["/opt/venv/bin/python", "/opt/app/main.py"]

Some open/missing points:

• Having .pyc files or not? poetry --compile will create them, but then we should also compile the whole project.
• In the future, use the poetry bundle plugin to create the virtual environment and then just copy it to the final image. At the moment, the plugin does not work with package-mode=false, yet.
• Use a different, non-root user in the final stage. Would require to chmod/chown the files.

Userful resources

• Build cache invalidation
• Building best practices