Skip to content

Run a long-lived service

By the end of this, mytool serve runs a service that stays up until something stops it. Ctrl-C or a SIGTERM shuts it down in order, and a service that keeps failing ends the process with an exit code an orchestrator can act on. You won't write any signal handling yourself.

This is the first part of the web-service series. Later parts put gRPC, HTTP and a gateway on the same command, and every one of them relies on what you set up here.

Allow about twenty minutes.

Before you start

  • The mytool project from Build your first CLI. Any project generated by gtb works the same way.
  • The same gtb release. Everything below is verified against v0.50.0.

Add the command

A long-running process is still a command. Generate one:

gtb generate command --name serve --short "Run the service until stopped"

As with hello, that writes a generated pkg/cmd/serve/cmd.go and a pkg/cmd/serve/main.go for you to fill in.

Run a service under a controller

The services run under a controller from the go/controls module. You register each service with three things: how to start it, how to stop it, and how to tell whether it's healthy. The controller starts them, stops them in order when the command's context ends, and keeps track of their health.

Replace pkg/cmd/serve/main.go with a controller running one service, a heartbeat that logs a line every second:

package serve

import (
    "context"
    "sync/atomic"
    "time"

    "gitlab.com/phpboyscout/go-tool-base/pkg/logger"
    "gitlab.com/phpboyscout/go-tool-base/pkg/props"
    "gitlab.com/phpboyscout/go/controls"
    "gitlab.com/phpboyscout/go/errors"
)

const beatInterval = time.Second

func RunServe(ctx context.Context, props *props.Props, _ *ServeOptions, _ []string) error {
    controller := controls.NewController(ctx,
        controls.WithLogger(logger.ToSlog(props.Logger)),
        controls.WithStopOnGiveUp(),
    )

    registerHeartbeat(controller, props)

    controller.Start()
    <-controller.Done()

    outcome, _ := controller.Outcome()
    if outcome.Cause == controls.CauseGaveUp {
        return errors.Wrapf(outcome.Err, "service %s gave up", outcome.Service)
    }

    return nil
}

func registerHeartbeat(controller *controls.Controller, props *props.Props) {
    var beats atomic.Int64

    controller.Register("heartbeat",
        controls.WithStart(func(ctx context.Context) error {
            ticker := time.NewTicker(beatInterval)
            defer ticker.Stop()

            for {
                select {
                case <-ctx.Done():
                    return nil
                case <-ticker.C:
                    props.Logger.Info("beat", "count", beats.Add(1))
                }
            }
        }),
        controls.WithStop(func(context.Context) {
            props.Logger.Info("heartbeat stopping", "total_beats", beats.Load())
        }),
        controls.WithStatus(func() error { return nil }),
    )
}

The three options are the service's three verbs:

  • WithStart runs the heartbeat. It blocks for as long as the service runs, and returns nil when its context ends.
  • WithStop runs during shutdown, for cleanup.
  • WithStatus answers "is it healthy?". nil means yes.

ctx is the command's own context. The controller is built on it, so when the command is told to stop, the controller stops too. controller.Done() closes once everything has stopped, and Outcome() says why. WithStopOnGiveUp comes into play in the last step.

main.go now imports go/controls directly, so tidy the module before you build:

go mod tidy
just build
./bin/mytool serve

INFO  beat count=1
INFO  beat count=2
INFO  beat count=3

It keeps going until you stop it.

Shut it down cleanly

Press Ctrl-C, or send the process a SIGTERM the way Kubernetes or systemd would:

WARN  received signal — shutting down gracefully (press again to force quit) signal=terminated
INFO  Stopping Services component=controller cause=parent
INFO  heartbeat stopping total_beats=3
INFO  Stopped component=controller

Nothing you wrote handles a signal. The root command gtb generated catches SIGINT and SIGTERM and cancels the command's context. The controller sees its parent context end (cause=parent), cancels the context every service's start is watching, runs each WithStop, and returns. A second Ctrl-C forces the process to quit straight away, for a service that's stuck.

That order (cancel first, then stop) is what lets an HTTP server finish the requests it's in the middle of when the pod rolls. You get it for the heartbeat now, and for every transport you add later.

Don't add controls.WithSignals() to the controller. The root already owns the signals, and a second handler would race it on a single Ctrl-C.

Check the exit code after a SIGTERM:

./bin/mytool serve &
kill -TERM $!
wait $!; echo $?

143

That's 128 plus the signal's number (15 for SIGTERM, 2 for SIGINT, so 130). It's the code a shell or a process manager expects from a process a signal stopped.

Give up on a service that keeps failing

A service that fails is restarted, if its restart policy allows. Once the restarts run out, the service gives up. WithStopOnGiveUp makes that stop the controller, so the command ends and the tool exits non-zero. Without it, the process would stay up but report itself not ready.

Make the heartbeat fail on every third beat, and allow two restarts. Add two constants:

const (
    beatInterval = time.Second
    missEvery    = 3
    maxRestarts  = 2
)

Then return an error from the ticker case, and add a restart policy beside WithStatus:

                case <-ticker.C:
                    count := beats.Add(1)
                    if count%missEvery == 0 {
                        return errors.Newf("missed beat %d", count)
                    }

                    props.Logger.Info("beat", "count", count)
                }

        controls.WithStatus(func() error { return nil }),
        controls.WithRestartPolicy(controls.RestartPolicy{MaxRestarts: maxRestarts}),

The scaffold's linter rejects bare numbers like 3 in code, which is why they're named.

Build and run it, and leave it alone for about ten seconds:

just build
./bin/mytool serve; echo "exit $?"

INFO  beat count=1
INFO  beat count=2
ERROR service failed; restarting component=controller service_name=heartbeat kind=retrying restarts=1 error="missed beat 3"
INFO  beat count=4
INFO  beat count=5
ERROR service failed; restarting component=controller service_name=heartbeat kind=retrying restarts=2 error="missed beat 6"
INFO  beat count=7
INFO  beat count=8
ERROR a registered service will never start; the controller is now reporting unready component=controller service_name=heartbeat error="max restarts exceeded: missed beat 9"
ERROR service unable to start component=controller service_name=heartbeat kind=unable_to_start restarts=2 error="max restarts exceeded: missed beat 9"
INFO  Stopping Services component=controller cause=gave_up
INFO  heartbeat stopping total_beats=9
INFO  Stopped component=controller
ERROR service heartbeat gave up: max restarts exceeded: missed beat 9
exit 1

Two restarts, then a give-up (cause=gave_up). RunServe turned that outcome into an error, and the root reported it and exited 1. That's the code systemd's Restart=on-failure, or a pod's restart policy, acts on.

Take the failure back out before you carry on: delete the count%missEvery check and the restart policy, and set the ticker case back to logging every beat.

Where this leaves you

mytool serve runs a service until it's told to stop, stops it in order, and says through its exit code how it ended. The heartbeat is a placeholder. Every service the next parts add registers on this same controller in the same way, and gets the same shutdown order without any extra work.

The controller also tracks liveness and readiness separately from status, which is what health endpoints read. That gets used once there's a transport to serve them from.