smhk

Run Poetry command as systemd service

All notes in this series:

These notes cover how to run a Python Poetry command as a systemd service. I’m using Ubuntu 22.04, but the steps should be the same for systemd on other OSes.

Create an example Poetry Python package §

Run poetry new example_service to create a new Poetry Python package named example_service.

Add systemd-python so that we can write to journald (see these notes for more details on installing systemd-python):

poetry add systemd-python

Add a basic main loop §

Create an example_service/service.py file with a basic main loop, which just writes to journald every second:

service.py
from systemd import journal
import time

def main():
    journal.send("starting up")
    try:
        while True:
            t = time.time()
            journal.send("example message", MY_COUNTER=t)
            time.sleep(1)
    except SystemExit:
        # Allow Ctrl-C to gracefully exit
        pass

A real service would be more complex, but this is just a basic example.

Add an entry point in pyproject.toml §

Create an entry point, which starts the above main loop:

pyproject.toml
[tool.poetry.scripts]
service-run = "example_service.service:main"

Find path to entry point §

On the command line we would normally run the service-run command by doing poetry run service-run. However, we cannot do that within a systemd service file because it requires the absolute path to the command.

Fortunately, the service-run command is just a “binary” within the virtual environment created by Poetry. If we enter the virtual environment created by Poetry, we can use which to get the path to the service-run binary:

$ poetry shell
$ which service-run
/home/squirrel/.cache/pypoetry/virtualenvs/example-service-0J3MkRb1-py3.10/bin/service-run

This is the command we will give to systemd to launch our service.

Create a systemd service file §

To add the service to systemd, we need to create a service file. A good starting point is the man pages:

man systemd.unit

Within here, we can find this extremely basic example foo.service file:

[Unit]
Description=Foo

[Service]
ExecStart=/usr/sbin/foo-daemon

[Install]
WantedBy=multi-user.target

Taking this example file, we can update the Description, and change ExecStart to use the path to the entry point:

[Unit]
Description=ExampleService

[Service]
ExecStart=/home/squirrel/.cache/pypoetry/virtualenvs/example-service-0J3MkRb1-py3.10/bin/service-run

[Install]
WantedBy=multi-user.target

Save the service file to /etc/systemd/system/example-service.service.

In summary, this service:

  • Has a description of ExampleService.
  • Is named example-service (based upon the filename). This is the name we will use in systemctl commands later.
  • Calls the service-run command when it starts, using the absolute path.

Test the service §

With the service installed, we can test starting the service:

sudo systemctl start example-service

Then check status to verify it is running:

$ systemctl status example-service
example-service.service - ExampleService
     Loaded: loaded (/etc/systemd/system/example-service.service; disabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-11-29 13:00:51 GMT; 22s ago
   Main PID: 23746 (service-run)
      Tasks: 1 (limit: 4556)
     Memory: 5.3M
        CPU: 41ms
     CGroup: /system.slice/example-service.service
             └─23746 /home/squirrel/.cache/pypoetry/virtualenvs/example-service-0J3MkRb1-py3.10/bin/python /home/squirrel/.cache/pypoetry/virtualenvs/example-service-0J3MkRb1-py>

Nov 29 13:01:04 squirrel-virtual-machine python[23746]: example message
Nov 29 13:01:05 squirrel-virtual-machine python[23746]: example message
Nov 29 13:01:06 squirrel-virtual-machine python[23746]: example message
# ...
Nov 29 13:01:11 squirrel-virtual-machine python[23746]: example message
Nov 29 13:01:12 squirrel-virtual-machine python[23746]: example message
Nov 29 13:01:13 squirrel-virtual-machine python[23746]: example message

All looks good.

Stop the service:

sudo systemctl stop example-service

View the logs §

Our (short lived) service has written some logs to journald.

We can use journalctl to view the log entries for our service:

$ journalctl -u example-service.service
Nov 29 13:00:51 squirrel-virtual-machine systemd[1]: Started ExampleService.
Nov 29 13:00:51 squirrel-virtual-machine python[23746]: starting up
Nov 29 13:00:51 squirrel-virtual-machine python[23746]: example message
Nov 29 13:00:52 squirrel-virtual-machine python[23746]: example message
Nov 29 13:00:53 squirrel-virtual-machine python[23746]: example message
# ...
Nov 29 13:01:26 squirrel-virtual-machine python[23746]: example message
Nov 29 13:01:27 squirrel-virtual-machine python[23746]: example message
Nov 29 13:01:28 squirrel-virtual-machine python[23746]: example message
Nov 29 13:01:29 squirrel-virtual-machine systemd[1]: Stopping ExampleService...
Nov 29 13:01:29 squirrel-virtual-machine systemd[1]: example-service.service: Deactivated successfully.
Nov 29 13:01:29 squirrel-virtual-machine systemd[1]: Stopped ExampleService.

That looks good. We can see the example message lines, but our custom MY_COUNTER field is not shown. We can verify that looking at the output in JSON.

View the logs (in JSON) §

Now we’ll use journalctl to view the logs in JSON format. This will enable us to see the MY_COUNTER field:

journalctl -u example-service.service -o json-pretty

Here are a few observations about the following output:

  • Using JSON (via -o json-pretty or -o json) shows more fields. The following example contains only six of the “example message” entries for brevity.
  • The ordering of the fields is not consistent, with no apparent pattern. For example, in the following six entries, the MESSAGE field is at index 7, 4, 16, 6, 2, 20.
  • The MY_COUNTER field can be seen.
  • There is no comma between each JSON object in the list. It appears each entry is considered a separate object.
  • See here for more details about the JSON format.

Highlighted below are the MESSAGE and MY_COUNTER fields:

{
        "_UID" : "0",
        "_BOOT_ID" : "37e336ad066846788762df9dd3434383",
        "_TRANSPORT" : "journal",
        "_SYSTEMD_UNIT" : "example-service.service",
        "_CAP_EFFECTIVE" : "1ffffffffff",
        "_EXE" : "/home/squirrel/.pyenv/versions/3.10.12/bin/python3.10",
        "__MONOTONIC_TIMESTAMP" : "10244496630",
        "MESSAGE" : "example message",
        "_MACHINE_ID" : "33bc1116507e48e485dd1c2349f34250",
        "MY_COUNTER" : "1701262883.4878783",
        "_SELINUX_CONTEXT" : "unconfined\n",
        "_SYSTEMD_CGROUP" : "/system.slice/example-service.service",
        "CODE_LINE" : "9",
        "_PID" : "23746",
        "_SYSTEMD_INVOCATION_ID" : "b5ed8b3604604ba7b670b17fac332352",
        "_CMDLINE" : "/home/squirrel/.cache/pypoetry/virtualenvs/example-service-0J3MkRb1-py3.10/bin/python /home/squirrel/.cache/pypoetry/virtualenvs/example-service-0J3MkRb1-py3.10/bin/service-run",
        "__REALTIME_TIMESTAMP" : "1701262883488336",
        "SYSLOG_IDENTIFIER" : "python",
        "CODE_FUNC" : "main",
        "_SYSTEMD_SLICE" : "system.slice",
        "_SOURCE_REALTIME_TIMESTAMP" : "1701262883488229",
        "__CURSOR" : "s=6f5532dee89c46d49412e3998509ba23;i=d2b;b=37e336ad066846788762df9dd3434383;m=2629e9cf6;t=60b4a21d70250;x=fbb109a998998c23",
        "CODE_FILE" : "/home/squirrel/projects/example_service/example_service/service.py",
        "_COMM" : "service-run",
        "_HOSTNAME" : "squirrel-virtual-machine",
        "_GID" : "0"
}
{
        "_PID" : "23746",
        "__MONOTONIC_TIMESTAMP" : "10245497960",
        "_CAP_EFFECTIVE" : "1ffffffffff",
        "_SYSTEMD_CGROUP" : "/system.slice/example-service.service",
        "MESSAGE" : "example message",
        "_MACHINE_ID" : "33bc1116507e48e485dd1c2349f34250",
        "_BOOT_ID" : "37e336ad066846788762df9dd3434383",
        "_HOSTNAME" : "squirrel-virtual-machine",
        "_SYSTEMD_UNIT" : "example-service.service",
        "_SYSTEMD_INVOCATION_ID" : "b5ed8b3604604ba7b670b17fac332352",
        "CODE_FUNC" : "main",
        "SYSLOG_IDENTIFIER" : "python",
        "MY_COUNTER" : "1701262884.489381",
        "_TRANSPORT" : "journal",
        "_CMDLINE" : "/home/squirrel/.cache/pypoetry/virtualenvs/example-service-0J3MkRb1-py3.10/bin/python /home/squirrel/.cache/pypoetry/virtualenvs/example-service-0J3MkRb1-py3.10/bin/service-run",
        "__CURSOR" : "s=6f5532dee89c46d49412e3998509ba23;i=d2c;b=37e336ad066846788762df9dd3434383;m=262ade468;t=60b4a21e649c1;x=add90525c73f308",
        "_GID" : "0",
        "_COMM" : "service-run",
        "__REALTIME_TIMESTAMP" : "1701262884489665",
        "_UID" : "0",
        "_SOURCE_REALTIME_TIMESTAMP" : "1701262884489567",
        "_SELINUX_CONTEXT" : "unconfined\n",
        "CODE_LINE" : "9",
        "CODE_FILE" : "/home/squirrel/projects/example_service/example_service/service.py",
        "_SYSTEMD_SLICE" : "system.slice",
        "_EXE" : "/home/squirrel/.pyenv/versions/3.10.12/bin/python3.10"
}
{
        "_BOOT_ID" : "37e336ad066846788762df9dd3434383",
        "_MACHINE_ID" : "33bc1116507e48e485dd1c2349f34250",
        "_SYSTEMD_UNIT" : "example-service.service",
        "_CAP_EFFECTIVE" : "1ffffffffff",
        "_SELINUX_CONTEXT" : "unconfined\n",
        "CODE_LINE" : "9",
        "__MONOTONIC_TIMESTAMP" : "10246499375",
        "CODE_FILE" : "/home/squirrel/projects/example_service/example_service/service.py",
        "_HOSTNAME" : "squirrel-virtual-machine",
        "_EXE" : "/home/squirrel/.pyenv/versions/3.10.12/bin/python3.10",
        "_GID" : "0",
        "_SOURCE_REALTIME_TIMESTAMP" : "1701262885491058",
        "CODE_FUNC" : "main",
        "_SYSTEMD_CGROUP" : "/system.slice/example-service.service",
        "__CURSOR" : "s=6f5532dee89c46d49412e3998509ba23;i=d2d;b=37e336ad066846788762df9dd3434383;m=262bd2c2f;t=60b4a21f59189;x=32adc06622f7ea03",
        "_COMM" : "service-run",
        "MESSAGE" : "example message",
        "SYSLOG_IDENTIFIER" : "python",
        "_UID" : "0",
        "_TRANSPORT" : "journal",
        "_SYSTEMD_SLICE" : "system.slice",
        "_CMDLINE" : "/home/squirrel/.cache/pypoetry/virtualenvs/example-service-0J3MkRb1-py3.10/bin/python /home/squirrel/.cache/pypoetry/virtualenvs/example-service-0J3MkRb1-py3.10/bin/service-run",
        "_PID" : "23746",
        "MY_COUNTER" : "1701262885.4909284",
        "__REALTIME_TIMESTAMP" : "1701262885491081",
        "_SYSTEMD_INVOCATION_ID" : "b5ed8b3604604ba7b670b17fac332352"
}
{
        "__REALTIME_TIMESTAMP" : "1701262886492949",
        "_COMM" : "service-run",
        "CODE_FUNC" : "main",
        "_SYSTEMD_UNIT" : "example-service.service",
        "_SYSTEMD_CGROUP" : "/system.slice/example-service.service",
        "_MACHINE_ID" : "33bc1116507e48e485dd1c2349f34250",
        "MESSAGE" : "example message",
        "_EXE" : "/home/squirrel/.pyenv/versions/3.10.12/bin/python3.10",
        "_PID" : "23746",
        "CODE_FILE" : "/home/squirrel/projects/example_service/example_service/service.py",
        "_SYSTEMD_SLICE" : "system.slice",
        "_GID" : "0",
        "CODE_LINE" : "9",
        "SYSLOG_IDENTIFIER" : "python",
        "_SOURCE_REALTIME_TIMESTAMP" : "1701262886492926",
        "_HOSTNAME" : "squirrel-virtual-machine",
        "_CAP_EFFECTIVE" : "1ffffffffff",
        "_UID" : "0",
        "_TRANSPORT" : "journal",
        "_CMDLINE" : "/home/squirrel/.cache/pypoetry/virtualenvs/example-service-0J3MkRb1-py3.10/bin/python /home/squirrel/.cache/pypoetry/virtualenvs/example-service-0J3MkRb1-py3.10/bin/service-run",
        "_SYSTEMD_INVOCATION_ID" : "b5ed8b3604604ba7b670b17fac332352",
        "MY_COUNTER" : "1701262886.4927928",
        "__MONOTONIC_TIMESTAMP" : "10247501243",
        "_BOOT_ID" : "37e336ad066846788762df9dd3434383",
        "_SELINUX_CONTEXT" : "unconfined\n",
        "__CURSOR" : "s=6f5532dee89c46d49412e3998509ba23;i=d2e;b=37e336ad066846788762df9dd3434383;m=262cc75bb;t=60b4a2204db15;x=85f29ec0a6eb34e5"
}
{
        "_SYSTEMD_INVOCATION_ID" : "b5ed8b3604604ba7b670b17fac332352",
        "__REALTIME_TIMESTAMP" : "1701262887494453",
        "MESSAGE" : "example message",
        "_GID" : "0",
        "__MONOTONIC_TIMESTAMP" : "10248502748",
        "__CURSOR" : "s=6f5532dee89c46d49412e3998509ba23;i=d2f;b=37e336ad066846788762df9dd3434383;m=262dbbddc;t=60b4a22142335;x=9c0646fb4437aa08",
        "CODE_FILE" : "/home/squirrel/projects/example_service/example_service/service.py",
        "_COMM" : "service-run",
        "_HOSTNAME" : "squirrel-virtual-machine",
        "_MACHINE_ID" : "33bc1116507e48e485dd1c2349f34250",
        "_SYSTEMD_UNIT" : "example-service.service",
        "SYSLOG_IDENTIFIER" : "python",
        "_CAP_EFFECTIVE" : "1ffffffffff",
        "MY_COUNTER" : "1701262887.4942982",
        "_SYSTEMD_SLICE" : "system.slice",
        "_UID" : "0",
        "CODE_FUNC" : "main",
        "_TRANSPORT" : "journal",
        "_SYSTEMD_CGROUP" : "/system.slice/example-service.service",
        "CODE_LINE" : "9",
        "_BOOT_ID" : "37e336ad066846788762df9dd3434383",
        "_EXE" : "/home/squirrel/.pyenv/versions/3.10.12/bin/python3.10",
        "_SELINUX_CONTEXT" : "unconfined\n",
        "_SOURCE_REALTIME_TIMESTAMP" : "1701262887494431",
        "_CMDLINE" : "/home/squirrel/.cache/pypoetry/virtualenvs/example-service-0J3MkRb1-py3.10/bin/python /home/squirrel/.cache/pypoetry/virtualenvs/example-service-0J3MkRb1-py3.10/bin/service-run",
        "_PID" : "23746"
}
{
        "__MONOTONIC_TIMESTAMP" : "10249504543",
        "_BOOT_ID" : "37e336ad066846788762df9dd3434383",
        "_HOSTNAME" : "squirrel-virtual-machine",
        "CODE_FUNC" : "main",
        "_GID" : "0",
        "__CURSOR" : "s=6f5532dee89c46d49412e3998509ba23;i=d30;b=37e336ad066846788762df9dd3434383;m=262eb071f;t=60b4a22236c78;x=ef183be68aa4176b",
        "MY_COUNTER" : "1701262888.495936",
        "_PID" : "23746",
        "_COMM" : "service-run",
        "SYSLOG_IDENTIFIER" : "python",
        "_TRANSPORT" : "journal",
        "_SYSTEMD_SLICE" : "system.slice",
        "_SYSTEMD_UNIT" : "example-service.service",
        "_SELINUX_CONTEXT" : "unconfined\n",
        "_EXE" : "/home/squirrel/.pyenv/versions/3.10.12/bin/python3.10",
        "__REALTIME_TIMESTAMP" : "1701262888496248",
        "CODE_FILE" : "/home/squirrel/projects/example_service/example_service/service.py",
        "_CMDLINE" : "/home/squirrel/.cache/pypoetry/virtualenvs/example-service-0J3MkRb1-py3.10/bin/python /home/squirrel/.cache/pypoetry/virtualenvs/example-service-0J3MkRb1-py3.10/bin/service-run",
        "_MACHINE_ID" : "33bc1116507e48e485dd1c2349f34250",
        "CODE_LINE" : "9",
        "MESSAGE" : "example message",
        "_UID" : "0",
        "_SOURCE_REALTIME_TIMESTAMP" : "1701262888496206",
        "_SYSTEMD_INVOCATION_ID" : "b5ed8b3604604ba7b670b17fac332352",
        "_SYSTEMD_CGROUP" : "/system.slice/example-service.service",
        "_CAP_EFFECTIVE" : "1ffffffffff"
}

Updating the service file §

If you make changes to the example-service.service file, you’ll need to run this command for them to take effect:

sudo systemctl daemon-reload

Conclusion §

We now have a very basic Python service, installed in a virtual environment create by Poetry, but run as a systemd service. It is logging to journald, including using custom fields.

As a next step, see these notes on how to read and filter the journald log entries in Python.

All notes in this series: