Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Packaging switch from valid simple unit to notify unit results in abandoned processes. #27935

Open
jboero opened this issue Aug 1, 2024 · 5 comments
Labels
dependencies Pull requests that update a dependency file release/packaging

Comments

@jboero
Copy link

jboero commented Aug 1, 2024

[UPDATE] this problem is reproduced with systemd v255.10.

Hi a while ago the packaging systemd units were apparently changed from simple to notify type but I'm seeing problems with unreaped failed processes. I don't think the notifications are being handled correctly. Does anybody else notice these problems?

Comit by @RickyGrassmuck please advise.
b09f3c0

Failed processes are not handled and restarting a failed service just forks another instance.

> sudo systemctl status vault
× vault.service - "HashiCorp Vault - A tool for managing secrets"
     Loaded: loaded (/usr/lib/systemd/system/vault.service; disabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Thu 2024-08-01 13:08:45 BST; 23min ago
       Docs: https://developer.hashicorp.com/vault/docs
    Process: 3888769 ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl (code=exited, status=1/FAILURE)
   Main PID: 3888769 (code=exited, status=1/FAILURE)
      Tasks: 9 (limit: 18782)
     Memory: 34.4M (peak: 60.3M)
        CPU: 269ms
     CGroup: /system.slice/vault.service
             ├─3830852 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
             ├─3831258 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
             ├─3831624 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
             ├─3875357 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
             ├─3875781 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
             ├─3876153 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
             ├─3888016 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
             ├─3888417 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
             └─3888814 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session

Aug 01 13:08:45 xps systemd[1]: vault.service: Unit process 3830852 (dbus-daemon) remains running after unit stopped.
Aug 01 13:08:45 xps systemd[1]: vault.service: Unit process 3831258 (dbus-daemon) remains running after unit stopped.
Aug 01 13:08:45 xps systemd[1]: vault.service: Unit process 3831624 (dbus-daemon) remains running after unit stopped.
Aug 01 13:08:45 xps systemd[1]: vault.service: Unit process 3875357 (dbus-daemon) remains running after unit stopped.
Aug 01 13:08:45 xps systemd[1]: vault.service: Unit process 3875781 (dbus-daemon) remains running after unit stopped.
Aug 01 13:08:45 xps systemd[1]: vault.service: Unit process 3876153 (dbus-daemon) remains running after unit stopped.
Aug 01 13:08:45 xps systemd[1]: vault.service: Unit process 3888016 (dbus-daemon) remains running after unit stopped.

And the pgrep

> ps -ef | grep vault
vault    3830852       1  0 12:54 ?        00:00:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
vault    3831258       1  0 12:54 ?        00:00:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
vault    3831624       1  0 12:54 ?        00:00:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
vault    3875357       1  0 13:05 ?        00:00:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
vault    3875781       1  0 13:05 ?        00:00:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
vault    3876153       1  0 13:05 ?        00:00:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
vault    3888016       1  0 13:08 ?        00:00:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
vault    3888417       1  0 13:08 ?        00:00:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
vault    3888814       1  0 13:08 ?        00:00:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
user   3989592  503396  0 13:32 pts/1    00:00:00 grep --color=auto vault

When using notify type ExecStart should point to a script that handles dbus hooks, not the raw service binary: https://askubuntu.com/questions/1120023/how-to-use-systemd-notify
[UPDATE] dbus support was added to Vault so this is no longer a problem.

@jboero jboero changed the title Packaging switch from valid simple unit to invalid inotify unit results in abandoned processes. Packaging switch from valid simple unit to invalid notify unit results in abandoned processes. Aug 1, 2024
@heatherezell heatherezell added release/packaging bug Used to indicate a potential bug labels Aug 1, 2024
@jboero
Copy link
Author

jboero commented Aug 5, 2024

Any update on this? This is a serious problem and a very quick solution.

@jboero
Copy link
Author

jboero commented Aug 8, 2024

OK I'll do it myself. The original systemd unit I wrote left default simple but this time it's explicitly set to simple as it should be.

#28029

@heatherezell heatherezell removed the bug Used to indicate a potential bug label Aug 8, 2024
@ryancragun
Copy link
Collaborator

Hello @jboero,

The systemd unit for Vault is type notify as we've made use of systemd notify functionality in Vault for some time now.

The dangling dbus processes you're seeing are likely caused by an upstream dependency which has not been fixed yet.

A workaround that ought to resolve it for you would be to define the DBUS_SESSION_BUS_ADDRESS environment variable before launching vault, e.g

# /etc/vault.d/vault.env
DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus

For further information you can refer to our support article.

If that doesn't resolve it please let us know.

@heatherezell heatherezell added the dependencies Pull requests that update a dependency file label Aug 8, 2024
@jboero
Copy link
Author

jboero commented Aug 9, 2024

Hi @ryancragun thanks for the update. I didn't realize you had added native dbus support to Vault.

I did just try to reproduce this on various versions of systemd and I see it's not a problem on older RHEL 8 environments. This happens since a new systemd v255.10 update came in on my test Fedora environment. I think this may actually be an issue in the future. Setting my own back to type=simple solves the problem for me.

Installed Packages
Name         : systemd
Version      : 255.10
Release      : 3.fc40
Architecture : x86_64
Size         : 16 M
Source       : systemd-255.10-3.fc40.src.rpm
Repository   : @System
From repo    : updates

@jboero jboero changed the title Packaging switch from valid simple unit to invalid notify unit results in abandoned processes. Packaging switch from valid simple unit to notify unit results in abandoned processes. Aug 9, 2024
@jboero
Copy link
Author

jboero commented Sep 21, 2024

Excellent thank you for investigating. Well done Mike Oprea

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Pull requests that update a dependency file release/packaging
Projects
None yet
Development

No branches or pull requests

3 participants