Get pinged for the right events, and only those

Every monitoring setup goes the same way. On day one, everything notifies. By day ten you've muted the channel. On day thirty, the one notification that mattered arrives in the muted channel and nobody sees it.

Butters is built to stop that happening, which is why notifications are opt-in per event. This post covers how the notify flag works, how to connect ntfy so notified events reach your phone, and a rule for deciding what deserves it.

What notify does

Every event takes an optional notify boolean. It defaults to false.

When it's true, two things happen. The event is highlighted in the feed so it stands out from the routine traffic around it, and it's sent to every enabled ntfy destination on the project. Events without the flag never leave the dashboard, however many of them you send.

That means the decision about what's urgent is made in your code, at the moment you know the most about the event. The failed payment handler knows it's on the third retry. The cron wrapper knows the job exited non-zero. The dashboard doesn't have to guess.

What ntfy is

ntfy is an open-source push notification service. You subscribe to a topic in its phone or desktop app, and anything published to that topic shows up as a notification. You can use the public server at ntfy.sh or run your own. There's no account needed to get started, which makes it a good fit for a small team.

Connecting a destination

Destinations are set per project, in the project's settings. Each one has:

• A server. https://ntfy.sh by default, or the address of your own ntfy server.
• A topic. Up to 64 characters of letters, numbers, - and _.
• A priority from 1 to 5, which controls how insistently the phone alerts. The default is 5, the maximum.
• Authentication: none, an access token, or a username and password, for servers that require it. Credentials are encrypted at rest and never sent back to the browser.
• An optional icon. It must be a PNG or JPEG URL, not an emoji.

The settings page shows when each destination last sent successfully and the last error if it didn't, so a mistyped topic or an expired token shows up there rather than as silence.

On the public ntfy.sh server, a topic name works like a password. Anyone who knows or guesses it can subscribe and read your notifications. Use something long and random, like acme-alerts-7f3k9x2m4q, rather than acme-alerts, or use a server with authentication.

What arrives on your phone

The notification is built from the event:

• The event title becomes the notification title.
• The description becomes the body, with **bold** and links rendered. With no description, the title is used again, because ntfy drops a notification with an empty body.
• The category shows as a tag, so you can tell billing from deploys at a glance.
• If the event has a url, tapping the notification opens it.

That last point is the one to use. A notification that opens the failed deploy run, the Stripe invoice, or the admin page for the broken order saves you the "now where do I look" step when you're away from your desk.

The event's emoji icon isn't sent to ntfy, which only accepts image URLs. Use the destination's icon setting if you want one.

More than one destination

A project can have several destinations, and every notified event goes to all of them. Two common setups:

A personal topic and a team topic. You get the alert on your phone; the team topic is there for whoever's on call this week.

One destination per environment. Separate topics, or separate self-hosted servers, for staging and production.

Priority belongs to the destination, not the event. If you want some alerts to buzz loudly and others to arrive quietly, send them to different projects, each with its own destinations at a different priority. Production failures go to a project whose destination has priority 5; things you want to know about today, but not at 2am, go to one set at 2 or 3.

The rule

Here's a test that holds up well: notify only when a person needs to act within the hour.

Passes the test:

• A deploy failed.
• A scheduled backup or billing job exited non-zero.
• A payment failed for the third time.
• The process crashed with an unhandled exception.
• A large order came in that someone should check by hand.

Fails it:

• A new signup. Nice to know, and the feed is where you'll see it.
• A single 500 from one route. Watch the error chart; notify if it becomes a pattern.
• A successful deploy or job run. Record it, but don't announce it.
• Anything that fires more than a few times a day in normal operation.

If a notified event arrives and you find yourself thinking "fine, I'll look later", take the flag off that call site. Every notification you dismiss without acting on makes the next one easier to dismiss.

Starting out

Connect one destination to one project, and set notify: true in exactly two places: failed deploys and crashed processes. Live with that for a week. Then add the next thing you wished you'd been told about, and only that.

The goal isn't to know everything the moment it happens. It's to trust that when your phone buzzes, it matters.