Skip to content

Daemon re-runs "start" task when restarting after "on-failure: shutdown" #527

Open
@benhoyt

Description

If a service is configured with on-failure: shutdown and it fails, pebble will shut down, but then when pebble runs again it'll try to redo that "start" task and immediately shut down again. The task is in status 3 (DoingStatus). For example

services:
  svc1:
    override: replace
    command: sleep x
    on-failure: shutdown
# First run (on "pebble start")
$ go run ./cmd/pebble/ run
2024-11-27T22:57:35.035Z [pebble] Started daemon.
2024-11-27T22:57:35.036Z [pebble] POST /v1/services 244.475µs 400
2024-11-27T22:57:35.036Z [pebble] Cannot start default services: no default services
# "pebble start" executed on another termina
2024-11-27T22:57:37.853Z [pebble] POST /v1/services 33.159477ms 202
2024-11-27T22:57:37.871Z [pebble] Service "svc1" starting: sleep x
2024-11-27T22:57:37.873Z [pebble] Service "svc1" stopped unexpectedly with code 1
2024-11-27T22:57:37.873Z [pebble] Service "svc1" on-failure action is "shutdown", triggering failure shutdown
2024-11-27T22:57:37.873Z [pebble] Server exiting! Reason: <nil>
exit status 10

# Second run
$ go run ./cmd/pebble/ run
2024-11-27T22:57:59.137Z [pebble] Started daemon.
2024-11-27T22:57:59.154Z [pebble] Service "svc1" starting: sleep x
2024-11-27T22:57:59.154Z [pebble] POST /v1/services 15.91363ms 400
2024-11-27T22:57:59.154Z [pebble] Cannot start default services: no default services
2024-11-27T22:57:59.156Z [pebble] Service "svc1" stopped unexpectedly with code 1
2024-11-27T22:57:59.156Z [pebble] Service "svc1" on-failure action is "shutdown", triggering failure shutdown
2024-11-27T22:57:59.156Z [pebble] Server exiting! Reason: <nil>
exit status 10

# Third run
$ go run ./cmd/pebble/ run
2024-11-27T22:58:19.848Z [pebble] Started daemon.
2024-11-27T22:58:19.873Z [pebble] Service "svc1" starting: sleep x
2024-11-27T22:58:19.873Z [pebble] POST /v1/services 23.920321ms 400
2024-11-27T22:58:19.873Z [pebble] Cannot start default services: no default services
2024-11-27T22:58:19.874Z [pebble] Service "svc1" stopped unexpectedly with code 1
2024-11-27T22:58:19.874Z [pebble] Service "svc1" on-failure action is "shutdown", triggering failure shutdown
2024-11-27T22:58:19.874Z [pebble] Server exiting! Reason: <nil>
exit status 10

This does not seem great. Pebble should probably transition the task to done or error.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    BugAn undesired feature ;-)

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions