prometheus

Best Practices

• Set appropriate scrape intervals based on metric cardinality (15s default, 5s-60s range)
• Use relabeling for dynamic service discovery and target filtering
• Define recording rules to precompute expensive PromQL expressions (rate, histogram_quantile)
• Group related alert rules by service or severity in named groups
• Set meaningful alert labels (severity: critical/warning/info) and annotations (summary, description)
• Use honor_labels and honor_timestamps appropriately for federated/push metrics
• Configure appropriate retention and storage settings in prometheus.yml
• Alert rules must have severity labels (critical, warning, info)
• Alert annotations should include summary and description with template variables ({{ $labels.instance }})
• Use job_name to clearly identify each scrape target
• Recording rule names must follow level:metric:operations convention
• Use for duration in alerts to avoid flapping (5m minimum for non-critical)
• Separate alert rules from recording rules into distinct files
• Configure Alertmanager with proper routing tree, group_by, group_wait, repeat_interval

Documentation Sources