Tasks not starting

[danog]

Component

server

Describe the bug

No pipelines are starting on my https://ci.daniil.it instance, trace logs on the server contain this:

{"level":"trace","time":"2023-11-11T15:01:43Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/cron/cron.go:48","message":"Cron: fetch next crons"}
{"level":"trace","time":"2023-11-11T15:01:53Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/cron/cron.go:48","message":"Cron: fetch next crons"}
{"level":"trace","time":"2023-11-11T15:01:55Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/router.go:43","message":"[POST] /api/repos/65/pipelines/1469/cancel"}
{"level":"debug","time":"2023-11-11T15:01:55Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/session/repo.go:150","message":"danog granted &{UserID:1 RepoID:65 Repo:<nil> Pull:true Push:true Admin:true Synced:1699714227 Created:1673777736 Updated:1699714688} permission to danog/MadelineProto"}
{"level":"debug","time":"2023-11-11T15:01:55Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:383","message":"queue: trying to remove 3193"}
{"level":"debug","time":"2023-11-11T15:01:55Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:383","message":"queue: trying to remove 3194"}
{"level":"debug","time":"2023-11-11T15:01:55Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:383","message":"queue: trying to remove 3195"}
{"level":"debug","time":"2023-11-11T15:01:55Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:389","message":"queue: 3195 is removed from pending"}
{"level":"debug","ip":"172.18.0.1","latency":240.618109,"method":"POST","path":"/api/repos/65/pipelines/1469/cancel","status":204,"time":"2023-11-11T15:01:55Z","user-agent":"Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/119.0","time":"2023-11-11T15:01:55Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"}
{"level":"trace","time":"2023-11-11T15:01:55Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/api/stream.go:89","message":"pubsub subscribe recover return: <nil>"}
{"level":"trace","time":"2023-11-11T15:01:56Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/router.go:43","message":"[POST] /api/repos/65/pipelines/1469?fork=true"}
{"level":"debug","time":"2023-11-11T15:01:56Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/session/repo.go:150","message":"danog granted &{UserID:1 RepoID:65 Repo:<nil> Pull:true Push:true Admin:true Synced:1699714227 Created:1673777736 Updated:1699714688} permission to danog/MadelineProto"}
{"level":"trace","time":"2023-11-11T15:01:56Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/api/stream.go:89","message":"pubsub subscribe recover return: <nil>"}
{"level":"debug","ip":"172.18.0.1","latency":1190.694914,"method":"POST","path":"/api/repos/65/pipelines/1469","status":200,"time":"2023-11-11T15:01:57Z","user-agent":"Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/119.0","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3190"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3196"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3197"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3198"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3193"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3190"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3196"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3197"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:288","message":"queue: waiting due to unmet dependencies 3196"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3190"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3196"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3197"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3198"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3193"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3190"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3196"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3197"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3198"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3193"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3190"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3196"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3197"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3198"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:333","message":"queue: pending right now: 3193"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:305","message":"queue: trying to assign task: 3190 with deps [3191 3192]"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:305","message":"queue: trying to assign task: 3197 with deps []"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:305","message":"queue: trying to assign task: 3198 with deps []"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:305","message":"queue: trying to assign task: 3193 with deps [3194 3195]"}
{"level":"trace","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/router.go:43","message":"[GET] /api/repos/65/pipelines/1470"}
{"level":"debug","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/session/repo.go:150","message":"danog granted &{UserID:1 RepoID:65 Repo:<nil> Pull:true Push:true Admin:true Synced:1699714227 Created:1673777736 Updated:1699714688} permission to danog/MadelineProto"}
{"level":"debug","ip":"172.18.0.1","latency":27.48223,"method":"GET","path":"/api/repos/65/pipelines/1470","status":200,"time":"2023-11-11T15:01:57Z","user-agent":"Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/119.0","time":"2023-11-11T15:01:57Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"}

Trace logs for agents are empty:

{"level":"info","time":"2023-11-11T15:01:35Z","caller":"/src/cmd/common/logger.go:100","message":"LogLevel = trace"}
{"level":"info","time":"2023-11-11T15:01:35Z","caller":"/src/cmd/agent/config.go:45","message":"no agent config found at '/etc/woodpecker/agent.conf', start with defaults"}
{"level":"trace","time":"2023-11-11T15:01:35Z","caller":"/src/agent/rpc/auth_interceptor.go:110","message":"token refreshed"}
{"level":"debug","time":"2023-11-11T15:01:35Z","caller":"/src/cmd/agent/agent.go:172","message":"loaded docker backend engine"}
{"level":"error","error":"open /etc/woodpecker/agent.conf: no such file or directory","time":"2023-11-11T15:01:35Z","caller":"/src/cmd/agent/config.go:74","message":"could not persist agent config at '/etc/woodpecker/agent.conf'"}
{"level":"debug","time":"2023-11-11T15:01:35Z","caller":"/src/cmd/agent/agent.go:200","message":"Agent registered with ID 42109"}
{"level":"info","time":"2023-11-11T15:01:35Z","caller":"/src/cmd/agent/agent.go:242","message":"Starting Woodpecker agent with version 'next-08f47afbd1' and backend 'docker' using platform 'linux/aarch64' running up to 1 pipelines in parallel"}
{"level":"debug","time":"2023-11-11T15:01:35Z","caller":"/src/cmd/agent/agent.go:225","message":"created new runner 0"}
{"level":"debug","time":"2023-11-11T15:01:35Z","caller":"/src/cmd/agent/agent.go:233","message":"polling new steps"}
{"level":"debug","time":"2023-11-11T15:01:35Z","caller":"/src/agent/runner.go:54","message":"request next execution"}

System Info

{"source":"https://github.com/woodpecker-ci/woodpecker","version":"next-08f47afbd1"}

Additional context

No response

Validations

• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Checked that the bug isn't fixed in the next version already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use]
• Check that this is a concrete bug. For Q&A join our Discord Chat Server or the Matrix room.

[zc-devs]

https://woodpecker-ci.org/docs/next/usage/workflow-syntax#labels

#2780

[danog]

This was already fixed in danog/MadelineProto@2b13ba5, but jobs are still broken, plus this is a deprecation, not a full removal, so stuff should still work even if I hadn't fixed the definitions

[zc-devs]

Maybe because of arm64 != aarch64 then.