Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
138 changes: 138 additions & 0 deletions api/openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,52 @@ servers:

paths:
/rules:
patch:
operationId: BulkUpdateAlertRules
summary: Bulk update alert rules
description: >
Updates one or more alert rules by their stable IDs. Each rule is
updated independently; per-rule status is returned in the response
so partial success is visible to the caller.
Supports label overrides, drop/restore toggles (platform rules only),
and classification label updates.
requestBody:
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/BulkUpdateAlertRulesRequest"
responses:
"200":
description: Update results (may include per-rule errors)
content:
application/json:
schema:
$ref: "#/components/schemas/BulkUpdateAlertRulesResponse"
"400":
description: Invalid request body
content:
application/json:
schema:
$ref: "#/components/schemas/ErrorResponse"
"401":
description: Missing or invalid authorization token
content:
application/json:
schema:
$ref: "#/components/schemas/ErrorResponse"
"413":
description: Request body exceeds the 1 MB limit
content:
application/json:
schema:
$ref: "#/components/schemas/ErrorResponse"
"500":
description: Unexpected server error
content:
application/json:
schema:
$ref: "#/components/schemas/ErrorResponse"
delete:
operationId: BulkDeleteUserDefinedAlertRules
summary: Bulk delete user-defined alert rules
Expand Down Expand Up @@ -203,6 +249,9 @@ components:
description: The stable alert rule ID that was processed.
statusCode:
type: integer
format: int32
minimum: 100
maximum: 599
description: HTTP status code for this rule's deletion result.
message:
type: string
Expand All @@ -219,6 +268,95 @@ components:
$ref: "#/components/schemas/DeleteAlertRuleResult"
description: Per-rule deletion results.

AlertRuleClassificationUpdate:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unrelated to the topic of this PR?

type: object
description: >
Partial update for alert rule classification labels.
Each field supports three states: omitted (leave unchanged),
null (clear the override), or a string value (set the override).
The three-state semantics require a custom JSON decoder; the Go
type AlertRuleClassificationPatch is used at runtime instead of
the generated struct.
x-go-type: AlertRuleClassificationPatch
properties:
openshift_io_alert_rule_component:
type: string
nullable: true
description: Component classification label override.
openshift_io_alert_rule_layer:
type: string
nullable: true
description: Layer classification label override.
openshift_io_alert_rule_component_from:
type: string
nullable: true
description: Dynamic component source label key.
openshift_io_alert_rule_layer_from:
type: string
nullable: true
description: Dynamic layer source label key.

BulkUpdateAlertRulesRequest:
type: object
required:
- ruleIds
properties:
ruleIds:
type: array
minItems: 1
maxItems: 100
items:
type: string
description: List of stable alert rule IDs to update (at most 100 per request).
labels:
type: object
additionalProperties:
type: string
nullable: true
description: >
Label key/value pairs to set. A null or empty-string value removes
the label. Omitting this field leaves existing labels unchanged.
alertingRuleEnabled:
type: boolean
nullable: true
description: >
When false, drops the alert rule via an AlertRelabelConfig Drop
action — the rule no longer appears in Prometheus query results.
When true, restores a previously dropped rule.
Only supported for platform alert rules.
classification:
$ref: "#/components/schemas/AlertRuleClassificationUpdate"

UpdateAlertRuleResult:
type: object
required:
- id
- statusCode
properties:
id:
type: string
description: The stable alert rule ID that was processed.
statusCode:
type: integer
format: int32
minimum: 100
maximum: 599
description: HTTP status code for this rule's update result.
message:
type: string
description: Error message if update failed; omitted on success.

BulkUpdateAlertRulesResponse:
type: object
required:
- rules
properties:
rules:
type: array
items:
$ref: "#/components/schemas/UpdateAlertRuleResult"
description: Per-rule update results.

ErrorResponse:
type: object
required:
Expand Down
19 changes: 19 additions & 0 deletions docs/alert-management.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,3 +39,22 @@ OpenShift supports routing user workload alerts to:
This is a cluster configuration choice and does not change the plugin API shape. The plugin reads alerts from Alertmanager (for firing/silenced) and Prometheus (for pending), then merges platform and user workload results when available.

The plugin intentionally reads from only the in-cluster Alertmanager endpoints. Supporting multiple external Alertmanagers would introduce ambiguous alert state and silencing outcomes because each instance can apply different routing, inhibition, and silence configurations.

### Managing user-defined alert rules

| Rule ownership | Editable? | Classification? | Drop/Restore? |
|---|---|---|---|
| User-owned | Yes (direct PR mutation) | Yes (set labels directly) | No (ARC not supported) |
| Operator-managed | No (reconciled) | No | No |
| GitOps-managed | No (reconciled) | No | No |

**User-owned** rules can be fully edited (labels, severity, expr, annotations)
via the update API, which mutates the PrometheusRule directly.

**Operator-managed** and **GitOps-managed** user-defined rules cannot be edited
because the owning controller would reconcile the change. These alerts can only
be **silenced** via Alertmanager silences.

ARC-based operations (classification overrides, drop/restore) are not available
for any user-defined rule because the user workload stack does not process
AlertRelabelConfigs. If this capability is needed, open an RFE against CMO.
36 changes: 18 additions & 18 deletions docs/alert-rule-classification.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ The backend classifies Prometheus alerting rules into a "component" and an "impa
- Computes an `openshift_io_alert_rule_id` per alerting rule.
- Determines component/layer based on matcher logic and rule labels.
- Allows operator-managed classification overrides via AlertRelabelConfigs (ARCs) for platform
rules. Operator-managed classification overrides of user-defined workload rules require the `ENABLE_USER_WORKLOAD_ARCS` feature flag.
rules.
- Enriches the Alerts API response with `openshift_io_alert_rule_id`, `openshift_io_alert_component`, and `openshift_io_alert_layer`.

This document explains how it works, how to override, and how to test it.
Expand Down Expand Up @@ -111,19 +111,22 @@ management status — this endpoint never writes directly to `AlertingRule` CRs.
(Other management endpoints, such as severity updates, may write to unmanaged
`AlertingRule` CRs directly, but classification is ARC-only.)

### User-defined workload rules → blocked by default, ARC when enabled
### User-defined workload rules

Classification updates for operator-managed user-defined workload rules are **not
allowed by default**. The API returns a `NotAllowedError` when the feature flag is
disabled.
For **user-owned** rules, classification labels can be set directly on the rule
(via `UpdateAlertRuleLabels` or `UpdateUserDefinedAlertRule`) — the same way any
other label is updated. This mutates the PrometheusRule directly without needing
ARCs.

### Feature flag: `ENABLE_USER_WORKLOAD_ARCS`
The ARC-based `UpdateAlertRuleClassification` endpoint returns `NotAllowedError`
for user-defined rules because CMO does not process AlertRelabelConfigs in the
user workload stack.

Setting the environment variable `ENABLE_USER_WORKLOAD_ARCS=true` enables full
alert management for operator-managed user-defined workload rules, including
classification overrides, label updates, and rule disable/enable (Drop/Restore).
When enabled, these rules use the same ARC-based path as platform rules, with
ARCs stored in the `openshift-user-workload-monitoring` namespace.
**Operator-managed** user-defined rules cannot be edited at all (the operator
would reconcile the change). Those alerts can only be **silenced** via
Alertmanager silences. If ARC-based management for these rules is needed, please
open an RFE against CMO to add AlertRelabelConfig support in the user workload
stack.

### Dynamic classification (`_from` labels)

Expand Down Expand Up @@ -223,8 +226,6 @@ APIs:
Direct K8s (supported for power users/GitOps):
- For platform rules: create or update the `AlertRelabelConfig` CR in `openshift-monitoring`
with the appropriate relabel configs (respect `resourceVersion` for optimistic concurrency).
- For user-defined rules (requires `ENABLE_USER_WORKLOAD_ARCS=true`): create or update the
`AlertRelabelConfig` CR in `openshift-user-workload-monitoring`.
- UI should check update permissions with SelfSubjectAccessReview before showing an editor.

Notes:
Expand All @@ -235,16 +236,15 @@ Notes:
Clients that need to update both should issue two requests. The combined operation is not atomic.

## Security Notes
- Classification overrides are stored in AlertRelabelConfig CRs (`openshift-monitoring`
for platform rules, `openshift-user-workload-monitoring` for user-defined rules when
enabled), subject to standard Kubernetes RBAC.
- Classification overrides are stored in AlertRelabelConfig CRs in `openshift-monitoring`,
subject to standard Kubernetes RBAC.
- No secrets or sensitive data are persisted in classification metadata.

## Testing and Ops
Unit tests:
- `pkg/management/update_classification_test.go`
- ARC-based classification for platform rules, blocked-by-default for user-defined
rules, ARC in user-workload namespace when flag enabled, dynamic `_from` label resolution.
- ARC-based classification for platform rules, not-allowed for user-defined
rules, dynamic `_from` label resolution.
- `pkg/management/get_alerts_test.go`
- Alert enrichment with classification labels, `_from` label behavior, fallback behavior.

Expand Down
Loading