Adding alerts using Service-Level Objectives (SLOs) in Go

Autometrics makes it easy to add Prometheus alerts using Service-Level Objectives (SLOs) to a function or group of functions.

This works using pre-defined Prometheus alerting rules, which can be loaded via the rule_files field in your Prometheus configuration. By default, most of the recording rules are dormant. They are enabled by specific metric labels that can be automatically attached by autometrics.

Pre-requisites

1. Make sure you have setup and initialized Autometrics libraries as described in Quickstart section.

2. Add the pre-configured Prometheus rules file to your Prometheus configuration. This file is located in the Autometrics shared repository (opens in a new tab)

Usage

Taking our example Autometrics-instrumented function from earlier:

//autometrics:inst
func RouteHandler(args interface{}) (err error) {
        // ...
        return nil
}

We can add an SLO to this function with the following comment:

//autometrics:inst --slo "Api" --success-target 90
func RouteHandler(args interface{}) (err error) {
        // ...
        return nil
}

This will create an SLO named Api with a target of 90% success rate. The SLO will be associated with the function RouteHandler and will be reported to Prometheus. You can have multiple functions associated with the same SLO.

The valid arguments for alert generation are:

• --slo (MANDATORY for alert generation): name of the service for which the objective is relevant
• --success-rate : target success rate of the function, between 0 and 100 (you must name the error return value of the function for detection to work.)
• --latency-ms : maximum latency allowed for the function, in milliseconds.
• --latency-target : latency target for the threshold, between 0 and 100 (so X% of calls must last less than latency-ms milliseconds). You must specify both latency options, or none.