All posts Operations

Deploys, cron jobs, and the things that fail quietly

Operational failures don't announce themselves. A backup exits with code 1 at 3am and the only record is a log file that gets rotated away. Here is how to put deploys and scheduled jobs in the same feed as everything else.

4 min read
Cron Deploys GitHub Actions Incidents
A yellow clock and a blue rocket above a timeline of blue check marks with one coral cross

Product failures are loud. Someone can't check out, and they tell you.

Operational failures are quiet. A nightly backup exits with code 1 at 3am. A report job runs, finds nothing, and sends nothing. A deploy half-succeeds. The only record is a log file on a box nobody logs into, and it gets rotated away before anyone thinks to look.

The fix is to make every deploy and every scheduled job say what happened, somewhere you already look. This post covers both, with copy-paste versions for GitHub Actions and cron.

Deploys belong in the feed

The reason to send deploy events isn't the deploy itself. It's the moment two days later when errors spike and you want to know what changed. If deploys land in the same project as your errors, the feed answers that question by itself: the spike starts two minutes after Deploy success: main @ 4f2c1ab.

In GitHub Actions, add a final step that runs whatever happened before it: 

- name: Report deploy to Butters
  if: always()
  env:
    BUTTERS_API_KEY: ${{ secrets.BUTTERS_API_KEY }}
    BUTTERS_PROJECT: ${{ vars.BUTTERS_PROJECT }}
    STATUS: ${{ job.status }}
    RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
  run: |
    jq -n \
      --arg project "$BUTTERS_PROJECT" \
      --arg status "$STATUS" \
      --arg ref "$GITHUB_REF_NAME" \
      --arg sha "${GITHUB_SHA::7}" \
      --arg actor "$GITHUB_ACTOR" \
      --arg url "$RUN_URL" \
      '{project: $project, category: "deploys",
        title: "Deploy \($status): \($ref) @ \($sha)",
        icon: (if $status == "success" then "🚀" else "💥" end),
        url: $url,
        tags: {ref: $ref, sha: $sha, actor: $actor},
        notify: ($status != "success")}' \
    | curl -fsS -X POST https://app.getbutters.com/api/events \
        -H "Authorization: Bearer $BUTTERS_API_KEY" \
        -H "Content-Type: application/json" \
        --data @- > /dev/null

if: always() makes the step run whether the deploy passed or failed, which is the whole point. jq builds the JSON, so a branch name with a quote in it can't break the request. The event title links straight back to the Actions run, and the short SHA and author end up as searchable tags. A failed deploy sends a notification; a successful one doesn't.

A wrapper for scheduled jobs

For cron jobs, the simplest approach is a small wrapper that runs the real command, times it, and reports the result: 

#!/usr/bin/env bash
# /usr/local/bin/butters-run: run a command and report how it went.
# Usage: butters-run <job-name> <command> [args...]
set -u
name="$1"; shift
start=$(date +%s)
output=$("$@" 2>&1); code=$?
seconds=$(( $(date +%s) - start ))

jq -n \
  --arg project "$BUTTERS_PROJECT" \
  --arg name "$name" \
  --argjson code "$code" \
  --argjson seconds "$seconds" \
  --arg tail "$(printf '%s' "$output" | tail -n 20)" \
  '{project: $project, category: "jobs",
    title: (if $code == 0 then "\($name) ok" else "\($name) failed (exit \($code))" end),
    icon: (if $code == 0 then "✅" else "❌" end),
    tags: {job: $name, seconds: ($seconds | tostring)},
    metadata: {exit: $code, seconds: $seconds, tail: ($tail | split("\n"))},
    notify: ($code != 0)}' \
| curl -fsS -X POST https://app.getbutters.com/api/events \
    -H "Authorization: Bearer $BUTTERS_API_KEY" \
    -H "Content-Type: application/json" \
    --data @- > /dev/null || true

exit "$code"

Then change each crontab line from running the job to running the wrapper: 

0 3 * * * . /etc/butters.env && /usr/local/bin/butters-run nightly-backup /opt/scripts/backup.sh

A few choices in there are worth explaining.

The last 20 lines of output go in metadata, not the title. They're the part you need when a job fails, and metadata keeps them out of the search index. Twenty lines also stays comfortably under the 64 KB metadata limit however chatty the job is.

The job name is a tag. Searching nightly-backup shows its full history in one list.

The report can't change the result. || true means that if Butters is unreachable, the wrapper still exits with the job's real exit code, so anything else that watches the job still sees the truth.

The environment is sourced explicitly. Cron runs with an almost empty environment, so BUTTERS_API_KEY and BUTTERS_PROJECT won't be there unless you load them. Keep them in a file readable only by the user the job runs as.

Report success, not just failure

It's tempting to report only failures: less noise. Don't.

Butters tells you about events that arrive. It can't tell you about an event that never arrived. If the server running your backup is switched off, nothing fails and nothing reports, and a failures-only feed looks exactly the same as a healthy one.

Reporting every successful run turns the feed into a record you can check. The chart for the jobs category should show the same count every day. A day that comes up one short means something didn't run, and searching the job name tells you which one.

If you need a phone alert for a job that stays silent, pair this with a dedicated heartbeat monitor that alerts when it doesn't hear from you. Butters is the log of what happened; a heartbeat monitor is the alarm for what didn't.

Incidents are events too

When something does go wrong, post the incident itself as it unfolds: 

curl -X POST https://app.getbutters.com/api/events \
  -H "Authorization: Bearer $BUTTERS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"project": "q4q8nb18qc2i", "category": "incidents", "title": "Checkout degraded", "description": "Payment provider timing out, retries succeeding slowly", "icon": "🚨"}'

Post one when it starts, one when you find the cause, one when it's resolved. Because deploys, errors and jobs are already in the same project, the feed becomes the incident timeline. When you write the postmortem, the sequence of events is already there with timestamps.

If you only get round to logging it afterwards, set created_at (unix seconds) to when it actually started. The event slots into the right place in the timeline rather than appearing at the top.

The pattern

Every automated thing you run should end by saying what happened, with success and failure alike. Put it in the same project as your errors, send a notification only on failure, and keep the detail in metadata. The feed then answers "what changed?" and "did it run?", which are most of the questions that come up at 3am.

Continue reading