Observability

Prometheus metrics, status fields, and monitoring the Remediator Agent in production.

Prometheus Metrics

The agent serves Prometheus metrics on container port 8080 (metrics.port), exposed through the service-agent-metrics-service Service on port 8443. The endpoint is plain HTTP by default (metrics.secure=false).

Available Metrics

Reconcile and GitOps metrics are shared by all Service Agent controllers and carry a controller_type label — filter on controller_type="remediator" for the Remediator Agent. All of these also carry namespace and k8s_uid, identifying the custom resource the run belongs to.

Metric	Type	Labels	Description
`agent_reconcile_total`	Counter	`controller_type`, `result="success\|error"`	Reconciliation runs
`agent_reconcile_duration_seconds`	Histogram	`controller_type`, `result="success\|error"`	Duration of each reconciliation run
`agent_outcome_total`	Counter	`controller_type`, `outcome_type`	Generic outcomes. `outcome_type="REMEDIATION_PLANS_GENERATED"` counts plans produced
`agent_gitops_outcome_total`	Counter	`controller_type`, `outcome_type`, `repo`, `source_branch`, `target_branch`	GitOps outcomes: `PR_OPENED`, `PR_MERGED`, `ISSUE_CREATED`
`agent_gitops_outcome_latency_seconds`	Histogram	`controller_type`, `outcome_type`, `repo`	Time to reach a GitOps outcome. Currently observed for `PR_MERGED` (PR open to merge)
`remediator_actions_executed_total`	Counter	`type`, `status="success\|error"`	Action executions by action type
`remediator_violations_active`	Gauge	`instance_id`, `severity`	Active violations by severity
`remediator_resources_skipped_unchanged_total`	Counter	`collector_id`	Resources skipped because they have not changed since the last run

Standard controller-runtime, workqueue, leader-election, and Kubernetes client metrics are exposed on the same endpoint.

Enable ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nirmata-agent-metrics
  namespace: nirmata
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: nirmata-agent
  endpoints:
    - port: metrics
      path: /metrics
      scheme: http

Access Metrics Directly

kubectl -n nirmata port-forward svc/service-agent-metrics-service 8443:8443

curl -s http://localhost:8443/metrics | grep '^agent_\|^remediator_'

Example Queries

# Success rate over the last hour
sum(rate(agent_reconcile_total{controller_type="remediator",result="success"}[1h]))
/ sum(rate(agent_reconcile_total{controller_type="remediator"}[1h]))

# P95 reconciliation latency
histogram_quantile(0.95,
  sum by (le) (rate(agent_reconcile_duration_seconds_bucket{controller_type="remediator"}[1h]))
)

# PRs opened per day, by repository
sum by (repo) (
  increase(agent_gitops_outcome_total{controller_type="remediator",outcome_type="PR_OPENED"}[1d])
)

# Active critical violations
sum(remediator_violations_active{severity="critical"})

Remediator Status

The Remediator resource reports detailed status about each run.

# View full status
kubectl get remediator remediator-argo-hub -n nirmata -o yaml

# View just the last run summary
kubectl get remediator remediator-argo-hub -n nirmata \
  -o jsonpath='{.status.lastRunSummary}' | jq

Status Fields

Field	Description
`phase`	Current operational phase: `Running`, `Idle`, or `Failed`
`lastScheduleTime`	When the last remediation was scheduled
`lastSuccessfulTime`	When the last successful run completed
`nextScheduledTime`	When the next run is scheduled
`conditions`	Step-by-step workflow tracking with collector information
`lastRunSummary.startTime` / `endTime`	Run duration timestamps
`lastRunSummary.status`	Success or failure
`lastRunSummary.message`	Human-readable outcome
`lastRunSummary.targetsProcessed`	Number of targets scanned
`lastRunSummary.violationsFound`	Total violations discovered
`lastRunSummary.remediationPlans`	Number of remediation plans produced
`lastRunSummary.actionsExecuted`	Number of actions taken (PRs created, etc.)
`lastRunSummary.errors`	Any errors encountered

Example Status Query

kubectl get remediator remediator-argo-hub -n nirmata \
  -o jsonpath='{.status.lastRunSummary}' | jq '{
  status: .status,
  violations: .violationsFound,
  plans: .remediationPlans,
  actions: .actionsExecuted,
  errors: .errors
}'
```bash

---

## Logs

```bash
# Follow live logs
kubectl logs -n nirmata -l app.kubernetes.io/name=nirmata-agent -f

# Last 100 lines
kubectl logs -n nirmata -l app.kubernetes.io/name=nirmata-agent --tail=100
```yaml

---

## Support Matrix

| Component | Supported |
|-----------|-----------|
| **Kubernetes** | All CNCF-compliant distributions v1.20+, including on-prem |
| **AI providers** | Nirmata AI (default), AWS Bedrock, Azure OpenAI, Anthropic API — or no AI provider at all in `prescriptive` remediation mode |
| **GitOps** | ArgoCD, FluxCD |
| **VCS** | GitHub (App & PAT), GitLab (Enterprise & SaaS) |
| **Manifests** | YAML files, simple Helm charts |