REL-850418: Add rotate-api-key user guide for Environment Watch

elastic-stack-setup/elastic-stack-setup-02-environment-watch/elastic-stack-rotate-api-key-environment-watch.md

@@ -0,0 +1,243 @@
+# Rotate an Elasticsearch API Key using the Relativity Server CLI
+
+The `rotate-api-key` command creates a new Elasticsearch API key for the specified cluster, persists it to the Relativity Secret Store, and invalidates the old key. Run this command periodically to rotate expiring keys or as part of a scheduled security practice.
+
+> [!NOTE]
+> It is recommended to run the CLI from the Primary SQL Server.
+
+> This guide assumes the Relativity Server bundle was extracted to `C:\Server.Bundle.x.y.z` or a similar directory chosen by the user.
+
+## Prerequisites
+
+- The Server-bundle zip file has been downloaded and extracted to `C:\Server.Bundle.x.y.z`
+- Access to the Relativity Secret Store (Whitelisted for Secret Store access. Please see [here](https://help.relativity.com/Server2025/Content/System_Guides/Secret_Store/Secret_Store.htm#Configuringclients) for information on whitelisting.)
+- Elasticsearch is running and accessible
+- The initial Environment Watch setup has been completed. See [Set up Environment Watch using the Relativity Server CLI](./elastic-stack-setup-02-environment-watch.md)
+
+## Options
+
+| Flag | Short alias | Description | Default |
+|------|-------------|-------------|---------|
+| `--cluster <value>` | `-c` | Target Elasticsearch cluster. Valid values: `rel-cluster-infrawatch`, `rel-cluster-datagrid` | Prompted interactively |
+| `--quiet` | | Suppress confirmation and expiry prompts; auto-confirms with the default 180-day expiry. Use together with `--cluster` for fully unattended execution | `false` |
+| `--dryrun` | | Preview what would happen without making any changes to Elasticsearch or the Secret Store | `false` |
+
+## Usage
+
+### Interactive
+
+Running `rotate-api-key` without any flags launches an interactive session. The CLI prompts you to select a target cluster.
+
+```
+C:\Server.Bundle.x.y.z\relsvr.exe rotate-api-key
+
+Relativity Server CLI - 102.1.18
+Copyright (c) 2026, Relativity ODA LLC
+
+Select the Elasticsearch cluster to rotate the API key for:
+
+> rel-cluster-infrawatch
+  rel-cluster-datagrid
+```
+
+After selecting a cluster, the CLI displays the current API key ID and the number of days remaining before expiry, then asks you to confirm the rotation. Entering `n` aborts with no changes made:
+
+```
+Key rotation aborted. No changes were made.
+```
+
+Entering `y` continues to prompt for a validity period in days, then performs the rotation.
+
+**InfraWatch cluster:**
+
+```
+Current API key ID: g9tMUp8BVmBEyCHxvRon  (name: rel-infrawatch)
+Days remaining before expiry: 179
+Rotate API key for cluster 'rel-cluster-infrawatch'? This will invalidate the current key. [y/n] (n): y
+How many days should the new key be valid? (180): 180
+
+API key rotation completed ----------------------------------------- 100%
+
+Successfully rotated the Elasticsearch API key and persisted the new key to the Secret Store.
+```
+
+**DataGrid cluster:**
+
+```
+Current API key ID: idtNUp8BVmBEyCHxYC02  (name: rel-datagrid)
+Days remaining before expiry: 179
+Rotate API key for cluster 'rel-cluster-datagrid'? This will invalidate the current key. [y/n] (n): y
+How many days should the new key be valid? (180): 180
+
+API key rotation completed ----------------------------------------- 100%
+
+Successfully rotated the Elasticsearch API key and persisted the new key to the Secret Store.
+```
+
+### Rotate with a pre-selected cluster
+
+Use `--cluster` to target a specific cluster directly, skipping the cluster selection menu. The CLI still displays the current key information and asks for confirmation before rotating.
+
+```
+C:\Server.Bundle.x.y.z\relsvr.exe rotate-api-key --cluster rel-cluster-infrawatch
+
+Relativity Server CLI - 102.1.18
+Copyright (c) 2026, Relativity ODA LLC
+
+Current API key ID: GUgcV58BbZgP437js08h  (name: rel-infrawatch)
+Days remaining before expiry: 179
+Rotate API key for cluster 'rel-cluster-infrawatch'? This will invalidate the current key. [y/n] (n): y
+How many days should the new key be valid? (180): 180
+
+API key rotation completed ----------------------------------------- 100%
+
+Successfully rotated the Elasticsearch API key and persisted the new key to the Secret Store.
+```
+
+```
+C:\Server.Bundle.x.y.z\relsvr.exe rotate-api-key --cluster rel-cluster-datagrid
+
+Relativity Server CLI - 102.1.18
+Copyright (c) 2026, Relativity ODA LLC
+
+Current API key ID: gUkdV58BbZgP437jHQNQ  (name: rel-datagrid)
+Days remaining before expiry: 179
+Rotate API key for cluster 'rel-cluster-datagrid'? This will invalidate the current key. [y/n] (n): y
+How many days should the new key be valid? (180): 180
+
+API key rotation completed ----------------------------------------- 100%
+
+Successfully rotated the Elasticsearch API key and persisted the new key to the Secret Store.
+```
+
+Use the `-c` short alias to achieve the same result:
+
+```
+C:\Server.Bundle.x.y.z\relsvr.exe rotate-api-key -c rel-cluster-infrawatch
+
+Relativity Server CLI - 102.1.18
+Copyright (c) 2026, Relativity ODA LLC
+
+Current API key ID: GUgcV58BbZgP437js08h  (name: rel-infrawatch)
+Days remaining before expiry: 179
+Rotate API key for cluster 'rel-cluster-infrawatch'? This will invalidate the current key. [y/n] (n): y
+How many days should the new key be valid? (180): 180
+
+API key rotation completed ----------------------------------------- 100%
+
+Successfully rotated the Elasticsearch API key and persisted the new key to the Secret Store.
+```
+
+### Quiet mode (automated / scripted rotation)
+
+Combining `--quiet` with `--cluster` suppresses all prompts, auto-confirms the rotation, and uses the default 180-day expiry. This is suitable for scheduled or unattended scripts.
+
+```
+C:\Server.Bundle.x.y.z\relsvr.exe rotate-api-key --quiet --cluster rel-cluster-infrawatch
+
+Relativity Server CLI - 102.1.18
+Copyright (c) 2026, Relativity ODA LLC
+
+Current API key ID: 5UkeV58BbZgP437jfiJ8  (name: rel-infrawatch)
+Days remaining before expiry: 179
+
+API key rotation completed ----------------------------------------- 100%
+
+Successfully rotated the Elasticsearch API key and persisted the new key to the Secret Store.
+```
+
+```
+C:\Server.Bundle.x.y.z\relsvr.exe rotate-api-key --quiet --cluster rel-cluster-datagrid
+
+Relativity Server CLI - 102.1.18
+Copyright (c) 2026, Relativity ODA LLC
+
+Current API key ID: gUkdV58BbZgP437jHQNQ  (name: rel-datagrid)
+Days remaining before expiry: 179
+
+API key rotation completed ----------------------------------------- 100%
+
+Successfully rotated the Elasticsearch API key and persisted the new key to the Secret Store.
+```
+
+### Dry run
+
+Use `--dryrun` to simulate the rotation without making any changes. The CLI displays the current key information, accepts the same prompts as a normal rotation, then confirms the simulation without writing to Elasticsearch or the Secret Store.
+
+```
+C:\Server.Bundle.x.y.z\relsvr.exe rotate-api-key --cluster rel-cluster-infrawatch --dryrun
+
+Relativity Server CLI - 102.1.18
+Copyright (c) 2026, Relativity ODA LLC
+
+Current API key ID: 5UkeV58BbZgP437jfiJ8  (name: rel-infrawatch)
+Days remaining before expiry: 179
+Rotate API key for cluster 'rel-cluster-infrawatch'? This will invalidate the current key. [y/n] (n): y
+How many days should the new key be valid? (180): 180
+DryRun mode: No changes were committed. The API key rotation was simulated successfully.
+```
+
+### Invalid cluster value
+
+If an unrecognized value is passed to `--cluster`, the CLI rejects it immediately and lists the valid options.
+
+```
+C:\Server.Bundle.x.y.z\relsvr.exe rotate-api-key --cluster infrawatch
+
+Relativity Server CLI - 102.1.18
+Copyright (c) 2026, Relativity ODA LLC
+
+Invalid --cluster value 'infrawatch'. Valid values are: rel-cluster-infrawatch, rel-cluster-datagrid
+```
+
+## Verify the rotation
+
+### Kibana API keys
+
+1. In Kibana, navigate to **Stack Management** > **Security** > **API keys**.
+2. Confirm a new key for the rotated cluster appears at the top of the list with a recent creation timestamp and an expiry approximately 180 days in the future.
+
+![Kibana API keys page showing newly rotated rel-infrawatch and rel-datagrid keys](../../resources/RotateApiKey-Kibana-After.png)
+
+### Secret Store
+
+To confirm the new API key was persisted, use the Secret Store CLI to read the secret for the rotated cluster. To compare values, read the secret both before and after running `rotate-api-key` — the `api-key` value should differ between the two reads.
+
+From an elevated PowerShell on the Secret Store server, navigate to `C:\Program Files\Relativity Secret Store\Client\` and run the read command for the rotated cluster:
+
+**InfraWatch:**
+
+```
+.\secretstore.exe secret read /database/elasticsearch/clusters/rel-cluster-infrawatch/security/api-keys/rel-infrawatch
+```
+
+**DataGrid:**
+
+```
+.\secretstore.exe secret read /database/elasticsearch/clusters/rel-cluster-datagrid/security/api-keys/rel-datagrid
+```
+
+### Elasticsearch Dev Tools (optional)
+
+To confirm that the old key has been invalidated and the new key is active, query the Elasticsearch security API in Kibana Dev Tools using the key ID.
+
+1. In Kibana, navigate to **Dev Tools** > **Console**.
+2. Run the following query, replacing `<key_id>` with the ID of the key to inspect:
+
+    ```
+    GET /_security/api_key?id=<key_id>
+    ```
+
+3. Verify the results:
+    - The old key shows `"invalidated": true`.
+    - The new key shows `"invalidated": false`.
+
+**Old key — invalidated:**
+
+![Old API key showing invalidated: true in Elasticsearch Dev Tools](../../resources/RotateApiKey-InfraWatch-DevTools-OldKey.png)
+
+**New key — active:**
+
+![New API key showing invalidated: false in Elasticsearch Dev Tools](../../resources/RotateApiKey-InfraWatch-DevTools-NewKey.png)
+
+If the Audit tab does not load after rotating the DataGrid API key, refer to [Data Grid Audit Troubleshooting](../troubleshooting/datagrid-audit-troubleshooting.md).

elastic-stack-setup/elastic-stack-setup-03-audit.md

@@ -191,3 +191,17 @@ Follow these steps to set up Data Grid Audit using the Relativity Server CLI. Al
    - Recent audit events are visible and populating (new user actions appear within a few minutes).
    - No error banners or "Elasticsearch connection failed" messages are displayed.
    - Audit search returns results without errors.
+
+## API Key Expiry and Rotation
+
+The Elasticsearch API key created during setup is valid for 180 days by default. Rotate the key before it expires to avoid authentication failures on the Audit tab.
+
+To rotate the DataGrid API key, use the `rotate-api-key` command from the Relativity Server CLI:
+
+```
+C:\Server.Bundle.x.y.z\relsvr.exe rotate-api-key --cluster rel-cluster-datagrid
+```
+
+See [Rotate an Elasticsearch API Key using the Relativity Server CLI](./elastic-stack-setup-02-environment-watch/elastic-stack-rotate-api-key-environment-watch.md) for full instructions, including quiet mode for scripted rotation and how to verify the new key in Kibana and the Secret Store.
+
+Refer to the [Data Grid Audit Troubleshooting Guide](./troubleshooting/datagrid-audit-troubleshooting.md) if the Audit tab does not load data after rotating the key.

elastic-stack-setup/troubleshooting/datagrid-audit-troubleshooting.md

@@ -0,0 +1,77 @@
+# Data Grid Audit Troubleshooting
+
+This document provides troubleshooting guidance for common issues that cause the Data Grid Audit tab in Relativity to fail or display errors. These issues can occur after initial setup, following an API key rotation, or when an existing API key has expired.
+
+## Audit Tab Shows Authentication Error
+
+### Symptoms
+
+The Audit tab may display one of the following errors:
+
+- **"Authentication failed for user to access Elasticsearch. Please check the permissions for the user."** — Relativity cannot authenticate against Elasticsearch with the current API key.
+
+  ![Relativity Audit tab showing authentication failed error](../../resources/troubleshooting-images/authentication-error.png)
+
+- **"This chart did not return any results."** — Relativity can reach Elasticsearch but data has not loaded, typically seen after services have been restarted but Elasticsearch is still initialising.
+
+  ![Relativity Audit tab showing no results](../../resources/troubleshooting-images/charts-did-not-return-results.png)
+
+These errors can be caused by:
+- An Elasticsearch API key that has **expired** or been **invalidated**
+- Relativity services not yet restarted after an API key rotation
+
+---
+
+## Step 1: Check the Elasticsearch Logs
+
+On the Elasticsearch server, open the log file at:
+
+```
+C:\elastic\elasticsearch\logs\elasticsearch.log
+```
+
+Look for repeated `WARN` entries matching this pattern:
+
+```
+[WARN ][o.e.x.s.a.ApiKeyAuthenticator] [<node>] Authentication using apikey failed - api key [<key_id>] has been invalidated
+```
+
+![Elasticsearch log showing repeated invalidated API key warnings](../../resources/troubleshooting-images/elasticsearch-errors.png)
+
+- **If this warning is present** — the API key Relativity uses to connect to the DataGrid Elasticsearch cluster has expired or been invalidated. Proceed to [Step 2: Rotate the Expired API Key](#step-2-rotate-the-expired-api-key).
+- **If this warning is not present** — the key is valid but services may need a restart. Proceed to [Step 3: Restart Services](#step-3-restart-services).
+
+---
+
+## Step 2: Rotate the Expired API Key
+
+When Elasticsearch logs confirm the API key has been invalidated, issue a new key using the Relativity Server CLI.
+
+Follow the [Rotate an Elasticsearch API Key using the Relativity Server CLI](../elastic-stack-setup-02-environment-watch/elastic-stack-rotate-api-key-environment-watch.md) guide, targeting `rel-cluster-datagrid`.
+
+After the rotation completes, continue with [Step 3: Restart Services](#step-3-restart-services) to restore the Audit tab.
+
+---
+
+## Step 3: Restart Services
+
+Restarting Relativity services forces them to re-read the API key from the Secret Store. In some cases, the Elasticsearch service also requires a restart.
+
+1. Open **Services** (`services.msc`) and restart the following Relativity services on **all servers** in the Relativity instance:
+   - `kCura Edds Agent Manager`
+   - `kCura Edds Web Processing Manager`
+   - `kCura Service Host Manager`
+
+2. Navigate to **Audit** > **Audit** in Relativity and refresh the page.
+
+   The error may change to *"This chart did not return any results."* — this indicates Relativity can now reach Elasticsearch but data has not yet loaded.
+
+3. On the Elasticsearch server, open **Services** and restart the **Elasticsearch** service.
+
+4. Wait **1–5 minutes** for Elasticsearch to fully restart and for Relativity to re-establish the connection.
+
+5. Reload the Relativity UI and navigate back to the **Audit** tab. Audit data should now display correctly.
+
+> [!NOTE]
+> If the Audit tab still does not load after following these steps, verify that the new API key was successfully persisted to the Secret Store. See [Rotate an Elasticsearch API Key using the Relativity Server CLI](../elastic-stack-setup-02-environment-watch/elastic-stack-rotate-api-key-environment-watch.md#verify-the-rotation) for Secret Store and Kibana verification steps.
+

elastic-stack-setup/troubleshooting/monitoring-agent-and-otel-collector.md

@@ -234,6 +234,31 @@ If the above steps do not resolve the issue, verify the following access and con
 For service account requirements and troubleshooting, see [Environment_Watch_Installer](../elastic-stack-setup-02-environment-watch/ew-01-install-monitoring-agents.md)
 
 
+## Environment Watch Not Working After API Key Rotation
+
+If the InfraWatch cluster API key has been newly created or rotated using the `rotate-api-key` command and Environment Watch dashboards are no longer updating or showing data, follow these steps.
+
+See [Rotate an Elasticsearch API Key using the Relativity Server CLI](../elastic-stack-setup-02-environment-watch/elastic-stack-rotate-api-key-environment-watch.md) for instructions on rotating the API key.
+
+1. Restart the **Relativity Environment Watch** Windows service on each monitored server:
+    ```powershell
+    Restart-Service -Name "Relativity Environment Watch"
+    ```
+
+2. Wait **2–5 minutes** for the service to reinitialize and begin sending data.
+
+3. Open Kibana and check the **Monitoring Agents** dashboard to confirm hosts are reporting. If data is still not appearing, proceed to step 4.
+
+4. If the issue persists, restart the Elastic Stack services on the Elasticsearch server:
+    1. **Elasticsearch** — open **Services** (`services.msc`) and restart the Elasticsearch service.
+    2. **Kibana** — restart the Kibana service.
+    3. **APM Server** — restart the APM Server service.
+
+5. Wait a few minutes and reload the Kibana dashboards. Environment Watch data should resume populating.
+
+> [!NOTE]
+> If data still does not appear after restarting all services, verify that the rotated API key was successfully persisted to the Secret Store. See the [Verify the rotation](../elastic-stack-setup-02-environment-watch/elastic-stack-rotate-api-key-environment-watch.md#verify-the-rotation) section for Secret Store and Kibana verification steps.
+
 ## Installer and Service Errors
 
 This section covers issues related to the Environment Watch installer and the underlying Windows services it manages.