Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Importing puslar client libraries creates zombie dbus processes #1184

Open
desolatorxxl opened this issue Mar 22, 2022 · 12 comments
Open

Importing puslar client libraries creates zombie dbus processes #1184

desolatorxxl opened this issue Mar 22, 2022 · 12 comments
Labels
annoying Benthos is mildly annoying but not quite a bug bug waiting for upstream Blocked on changes needed in an upstream dependency

Comments

@desolatorxxl
Copy link

As per discord chat with @Jeffail and @mihaitodor:

Running any of benthos subcommands in a cronjob will create
/usr/bin/dbus-daemon --session processes for every invocation, slowing down
the system or ultimately crashing it.

This is most likely because there are no dbus sessions available to cron processes.

Way to reproduce:

# cat /etc/cron.d/foo
* * * * * root benthos --version
# Wait a few minutes
# ps aux | grep dbus
message+     605  0.0  0.0   8572  4460 ?        Ss   11:28   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-o
root      302826  0.0  0.0   8268  2480 ?        Ss   19:16   0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
root      303318  0.0  0.0   8268  2480 ?        Ss   19:17   0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
root      303870  0.0  0.0   8268  2504 ?        Ss   19:18   0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
... a few more ...
root      309243  0.0  0.0   8268  2480 ?        Ss   19:28   0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
root      309737  0.0  0.0   8268  2540 ?        Ss   19:29   0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
root      310353  0.0  0.0   8268  2484 ?        Ss   19:30   0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
root      310842  0.0  0.0   8268  2484 ?        Ss   19:31   0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session

This also happens on every invocation in a regular shell on a system where dbus
is disabled / not existent (e.g. OpenRC systems).

The root cause is most likely this:
https://github.com/99designs/keyring/blob/81fed19d324843057dab56ba67dd0db54d1d4626/secretservice.go#L19

Workaround is to set DBUS_SESSION_BUS_ADDRESS=/dev/null as environment
variable in the cron file or from where ever else you might run it.

@Jeffail Jeffail added bug annoying Benthos is mildly annoying but not quite a bug waiting for upstream Blocked on changes needed in an upstream dependency labels Mar 22, 2022
@Jeffail
Copy link
Collaborator

Jeffail commented Mar 22, 2022

So I've removed pulsar from the default benthos builds: 1792ca4, and the first release of V4 (unless this gets fixed) will come without it. However, the option to compile Benthos with the standard pulsar input and output is still there, I've added an explanation and example on how to do that in the migration guide.

Ideally we need to let the pulsar client lib maintainers know. To me it seems odd that a dependency like this is even being drawn into a client library by default, but I'm not sure whether the solution would be to chase upstream and then upgrade or to bring out their authentication plugins into separate packages.

Either way I'm not comfortable importing these packages by default, we've had multiple issues with the dependencies being brought in by the pulsar client library over the last year or so and my gut feeling now is to hold back on using it.

@Jeffail Jeffail changed the title Running benthos via cron creates zombie dbus processes Importing puslar client libraries creates zombie dbus processes Mar 23, 2022
@Jeffail
Copy link
Collaborator

Jeffail commented Jun 8, 2022

Related: 99designs/keyring#103

@teabot
Copy link

teabot commented Sep 1, 2022

Did this get raised with the Pulsar client lib project?

@tspannhw
Copy link

tspannhw commented Dec 8, 2022

Any updates?

@Jeffail
Copy link
Collaborator

Jeffail commented Dec 8, 2022

Hey @tspannhw, unfortunately I'm not convinced this will be addressed by the client authors any time soon so I don't think there's any convenient way in which we can progress this ourselves.

@mihaitodor
Copy link
Collaborator

For now, if you wish to use the Pulsar components, you should import Benthos as a library as explained here and build it using Go 1.18+. The steps would be the following:

  • Download & install Go 1.18 (or use the official Docker container)
  • Create a folder somewhere and place in it main.go with the contents that I linked above
  • Run in that folder go build -o benthos .
  • You should now have a fully-featured benthos binary including Pulsar support

@srenatus
Copy link
Contributor

Could it have made a reappearance? Judging from this,

% go mod why github.com/99designs/keyring
# github.com/99designs/keyring
github.com/something/nondisclosed
github.com/benthosdev/benthos/v4/public/components/sql
github.com/snowflakedb/gosnowflake
github.com/99designs/keyring

So it's come back through gosnowflake, I suppose? 😱

@Jeffail
Copy link
Collaborator

Jeffail commented Apr 12, 2023

hecckkkkkk, 99designs more like 99attemptstoruinyourproduct

@Jeffail
Copy link
Collaborator

Jeffail commented Apr 12, 2023

I guess we need to give up on the idea that libraries are going to migrate away from this package, pulsar still has it as far as I can tell. I believe there's a way to manually nuke it from our deps with a replacement but needs investigation.

Either that or we add a lint check specifically for this package to try and avoid future imports.

@srenatus
Copy link
Contributor

srenatus commented Apr 12, 2023

Replacements will do fine when it comes to building benthos. They will, however, not be honored when building packages that use benthos as golang library, but they'll need to be reproduced in the go.mod files there. (Here's an example, open-policy-agent/opa@0fa850b, FWIW)

@Jeffail
Copy link
Collaborator

Jeffail commented Apr 12, 2023

Okay I'm bored of waiting so I've made a fork that nukes all dbus functionality from keyring and replaced usage of it in our mod: 7453f87

Next steps are:

  • Verify that actually fixes the problem
  • Consider re-adding pulsar back into the official build
  • Add documentation for custom binary builders explaining that this is an issue if they choose to import pulsar or snowflake stuff and recommending they do the same replacement directive

@Jeffail
Copy link
Collaborator

Jeffail commented May 10, 2023

I've added pulsar back into the main distribution of benthos: 01aca87, hopefully we're still good with the dependency swap. I'm still not going to be happy until we can get a non-hacky fix from upstream so leaving this open until that happens. Which it will. Surely it will. Yes, it will. Nah probably not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
annoying Benthos is mildly annoying but not quite a bug bug waiting for upstream Blocked on changes needed in an upstream dependency
Projects
None yet
Development

No branches or pull requests

6 participants