diff --git a/docs/custom-fields.md b/docs/custom-fields.md new file mode 100644 index 0000000000..a8412eba15 --- /dev/null +++ b/docs/custom-fields.md @@ -0,0 +1,178 @@ +# Custom Fields Architecture + +Custom fields let organizations index arbitrary event data properties for use in filters and search. This document covers the full lifecycle, slot system internals, deletion policy, and operator support. + +## Overview + +When an event is processed, the pipeline handler inspects the event's `Data` dictionary and writes typed values into the `Idx` sub-document using a **pooled slot** model. Rather than creating a unique Elasticsearch field per organization per field name (which would cause mapping explosion in a multi-tenant index), all organizations share a small pool of physical ES fields like `idx.keyword-1`, `idx.double-2`, etc. Each organization gets its own independent slot namespace: "department" for Org A and "department" for Org B both map to `idx.keyword-1` but are isolated by tenant-scoped queries. + +### Foundatio Integration + +The custom fields system is built on [Foundatio.Repositories.Elasticsearch custom fields](https://repositories.foundatio.dev/guide/custom-fields). Key components: + +- `IHaveVirtualCustomFields` — implemented by `PersistentEvent` to control how field values are read/written +- `ICustomFieldDefinitionRepository` — stores field definitions with slot assignments per `(EntityType, TenantKey, IndexType)` +- `EventCustomFieldService` — wires the document-changing pipeline hook and handles system field provisioning +- `EventIndex` — registers the 8 standard custom field types via `AddStandardCustomFieldTypes()` + +### Supported Field Types + +| Type | ES Mapping | Physical Slot Pattern | Filter Operators | +|------|-----------|----------------------|-----------------| +| `keyword` | Keyword (exact match) | `idx.keyword-{n}` | equals, not-equals, exists, missing | +| `string` | Text + `.keyword` sub-field | `idx.string-{n}` | contains/search, exists, missing | +| `int` | Integer | `idx.int-{n}` | equals, gt, gte, lt, lte, range, exists, missing | +| `long` | Long | `idx.long-{n}` | equals, gt, gte, lt, lte, range, exists, missing | +| `double` | Double | `idx.double-{n}` | equals, gt, gte, lt, lte, range, exists, missing | +| `float` | Float | `idx.float-{n}` | equals, gt, gte, lt, lte, range, exists, missing | +| `bool` | Boolean | `idx.bool-{n}` | true, false, exists, missing | +| `date` | Date | `idx.date-{n}` | equals, range, gt, gte, lt, lte, exists, missing | + +> **Note on `string` cost**: Each `string` slot creates **two** Elasticsearch field mappers (the `text` field and its `.keyword` sub-field), making it twice as expensive as other types toward Elasticsearch's `index.mapping.total_fields.limit` (default 1,000). + +## Slot System + +### How Slots Are Assigned + +Slots are assigned **sequentially** per `(EntityType, TenantKey, IndexType)` scope: + +``` +Org A: "department" → keyword slot 1 → idx.keyword-1 +Org A: "region" → keyword slot 2 → idx.keyword-2 +Org B: "department" → keyword slot 1 → idx.keyword-1 ← same physical field, different tenant +Org B: "priority" → int slot 1 → idx.int-1 +``` + +Slot assignment is protected by a **distributed lock** per scope to prevent duplicate allocation under concurrent writes. + +### System Fields + +Three system fields are provisioned automatically per organization and are **protected from deletion**: + +| Field Name | Type | Slot | Purpose | +|-----------|------|------|---------| +| `sessionend` | `date` | `date-1` | Session end timestamp (session tracking) | +| `haserror` | `bool` | `bool-1` | Whether the session has an associated error | +| `@ref:session` | `keyword` | `keyword-1` | Session reference identifier | + +Because system fields are provisioned via `EnsureSystemFieldsAsync` **before** any user-defined fields, they always occupy slot 1 of their type. User fields for `date`, `bool`, and `keyword` start at slot 2. + +If `EnsureSystemFieldsAsync` is not called (for example, for a legacy organization created before custom fields were introduced), the first user to create a `date`, `bool`, or `keyword` field could claim slot 1. The API handler calls `EnsureSystemFieldsAsync` before every field creation to prevent this. + +### Slot Exhaustion and Elasticsearch Field Limits + +Elasticsearch's default `index.mapping.total_fields.limit` is **1,000 field mappers**. Physical slot fields are only created in the index mapping the first time a document with that slot is indexed. The maximum Elasticsearch fields from custom fields is bounded by the highest slot number ever used, multiplied by number of types, multiplied by 2 (for `string` types). + +With a hard limit of 20 active fields per organization and slot recycling deferred beyond the retention window, slot numbers grow slowly over time. For a typical organization cycling through fields over years, the slot high-water mark remains very low (well under 100 per type). Across many organizations sharing the same physical ES index, the absolute maximum slot number approaches the highest number ever assigned to any organization — still bounded in practice. + +**Field churn analysis**: The worst-case scenario is an organization that continuously creates and deletes the maximum 20 fields. Each create/delete cycle permanently consumes one slot. For any single `(EntityType, TenantKey, IndexType)` combination, the slot high-water mark is bounded by the number of distinct fields ever created. Across 8 types and 20 active fields with unlimited churn, the theoretical maximum is `8 types × unlimited cycles` — but since fields created in Elasticsearch are shared across all tenants in the index, the practical concern is the *total* unique slot number across all active organizations, not per-organization. The active per-organization cap of 20 limits how many fields a single organization can consume in any given period. + +There is no application-level per-type slot ceiling — the framework relies on Elasticsearch's mapping limit (default 1,000 field mappers) as the ultimate guard. Retention-aware hard deletion and slot reclamation are not implemented; until they are, deleted definitions continue to reserve their slots. + +## Field Lifecycle + +### Creating a Field + +1. User calls `POST /organizations/{id}/event-custom-fields` +2. API validates: premium plan check, reserved name check, active quota check, duplicate name check +3. `EnsureSystemFieldsAsync` provisions `sessionend`, `haserror`, and `@ref:session` if they are not yet present +4. `AddFieldAsync` assigns the next available slot and persists the definition +5. **From this point on, new events with matching data keys are indexed into the slot** +6. **Existing events are NOT backfilled** — they retain their original `Idx` content unchanged + +> **Search semantics on creation**: Custom field indexing applies only to events processed **after** the field definition is created. Historical events are not re-indexed. If you need historical data, you must re-ingest events or use data-level queries (`data.fieldname:value`) instead of slot queries. + +### Updating a Field + +Only `Description` and `DisplayOrder` are mutable. `Name`, `IndexType`, and `IndexSlot` are immutable once created (enforced by Foundatio's repository at save time). + +### Deleting a Field + +Deletion is a two-phase process designed to prevent **slot reuse corruption** — where a recycled slot causes historical events for a deleted field to appear in queries for a new field with the same slot. + +**Phase 1 — Soft Delete (synchronous):** +1. API checks for usage in saved view filters — returns 409 Conflict if found +2. API marks `IsDeleted = true` and calls `SaveAsync` +3. The field name is freed from the slot system (a new field can use the same name) +4. The slot number is **not** freed — it remains occupied +5. New events no longer index data into this slot +6. API returns 202 Accepted; the field disappears from the management UI +7. A `RemoveCustomFieldWorkItem` is enqueued + +**Phase 2 — Slot Cleanup (deferred):** +The `RemoveCustomFieldWorkItemHandler` currently **acknowledges** the soft-delete without hard-deleting the definition record. This is intentional: + +> **Slot Reuse Safety**: If a slot is freed and immediately recycled for a new field, historical events within the retention window that had data for the old field will appear in queries for the new field. For example: delete "customer_id" (keyword-3), create "project_id" (gets keyword-3), then searching `project_id:acme` returns historical events where `customer_id` was `acme`. This is a data integrity violation. + +Hard-delete (slot freeing) is deferred until retention-aware cleanup is implemented. Until then, the slot number grows monotonically and is never reused for a different field. + +> **Search semantics on deletion**: After Phase 1, no new events write to the deleted slot and the field name no longer resolves in queries. Existing slot values remain in historical event documents until those events age out, but direct raw-slot queries are blocked by the repository query resolver. + +### Slot Recycling + +Slot recycling (reusing a freed slot number for a new field) is **currently deferred** to prevent data contamination within the retention window. See "Deleting a Field" above. Deleted definitions continue reserving their slots until retention-aware cleanup is implemented. + +**Name reuse is always safe**: After soft-deletion, the same field *name* can be immediately reused. The new definition gets the *next available* slot number (monotonically increasing), not the old slot. This means: + +``` +Field A created → keyword slot 1 (active) +Field B created → keyword slot 2 (active) +Field A deleted → soft-delete (slot 1 still occupied, name freed) +Field C created with same name → keyword slot 3 (new slot, no contamination) +Query for "Field C" → only returns events since Field C was created +``` + + + +The active field limit (`MaxFieldsPerOrganization`, default 20) counts only fields that are: +- Not soft-deleted (`IsDeleted = false`) +- Not system fields (`sessionend`, `haserror`, `@ref:session`) + +Soft-deleted fields awaiting cleanup do **not** count toward the active quota. A user who has 20 active fields can delete some and immediately create replacements — the quota check reflects the current active state. + +### Deletion Blocked by Saved Views + +If a custom field is referenced in any saved view filter for the organization, deletion is blocked with HTTP 409 Conflict. The filter is checked using a regex that matches `idx.{fieldName}` tokens in the filter string. Users must remove the field from all saved view filters before deletion proceeds. + +## Plan Restrictions + +Custom fields require a paid plan. Organizations on the free plan receive HTTP 426 Upgrade Required when attempting to create a custom field. Existing fields are unaffected if an organization downgrades — they remain indexed but the management UI is read-only. + +## Security Model + +- All custom field API endpoints require authentication and verify organization ownership before any operation +- Field names are validated against a strict allowlist (`[a-zA-Z0-9_.\-]`, max 100 chars, no `@` prefix) +- Names starting with `@` are reserved for Exceptionless internal data keys (`@error`, `@request`, etc.) +- Users cannot access or modify custom fields belonging to other organizations (tenant isolation is enforced by the API handler's organization-access checks) +- System fields (`@ref:session`, `sessionend`, `haserror`) cannot be created, modified, or deleted via the API + +## Elasticsearch Mapping Considerations + +- Custom field slot templates are registered via `AddStandardCustomFieldTypes()` in `EventIndex` +- Templates use the pattern `idx.{type}-*` (e.g., `idx.keyword-*`, `idx.double-*`) +- Elasticsearch creates field mappings dynamically on first document write — unused slots have zero mapping cost +- Monitor total field count relative to `index.mapping.total_fields.limit` (default 1,000) in high-volume deployments +- The `string` type creates 2 field mappers per slot; all other types create 1 field mapper per slot + +## Common Questions + +**Can I reuse a field name after deleting it?** +Yes, immediately. After soft-deletion, the field name is freed and can be used for a new field. The new field gets a **new** slot number (not the old one), which prevents historical events for the deleted field from appearing in queries for the new field. Slot numbers grow monotonically and are not recycled while retention-aware cleanup remains unimplemented. + +**Does the 20-field quota include soft-deleted fields?** +No. The quota counts only *active* (non-deleted, non-system) fields. Soft-deleted fields awaiting cleanup are excluded. You can delete fields and immediately create replacements up to the quota. + +**Will deleting a field break existing queries?** +Saved view filters that reference the field are blocked at deletion time. Custom code cannot query `idx.keyword-N:value` directly because raw slot access is blocked. The Exceptionless query builder translates active field names to slot paths automatically. + +**Is there a per-type field limit?** +No. The active quota (`MaxFieldsPerOrganization = 20`) is a total across all types. There is no separate limit per type. + +**What happens if I downgrade my plan?** +Existing field definitions and indexed data are preserved. The custom fields management UI becomes read-only. New field creation requires re-upgrading. + +**Can I have more than 20 fields?** +The default limit is 20 per organization. This can be increased via the `MaxFieldsPerOrganization` configuration key for self-hosted deployments. + +**Can slot numbers grow unboundedly from field churn?** +Yes. Each delete-then-recreate cycle consumes another slot number because reclamation is not currently implemented. Operators should monitor Elasticsearch's total-field mapping limit, particularly for deployments whose organizations churn through many custom-field definitions. diff --git a/src/Exceptionless.Core/Bootstrapper.cs b/src/Exceptionless.Core/Bootstrapper.cs index efcfae9efb..20a2143fea 100644 --- a/src/Exceptionless.Core/Bootstrapper.cs +++ b/src/Exceptionless.Core/Bootstrapper.cs @@ -38,6 +38,7 @@ using Foundatio.Queues; using Foundatio.Repositories.Elasticsearch; using Foundatio.Repositories.Elasticsearch.Configuration; +using Foundatio.Repositories.Elasticsearch.CustomFields; using Foundatio.Repositories.Elasticsearch.Jobs; using Foundatio.Repositories.Migrations; using Foundatio.Resilience; @@ -74,6 +75,7 @@ public static void RegisterServices(IServiceCollection services, AppOptions appO services.AddSingleton(); services.AddSingleton(s => s.GetRequiredService().Client); services.AddSingleton(s => s.GetRequiredService()); + services.AddSingleton(s => s.GetRequiredService().CustomFieldDefinitionRepository!); services.AddStartupAction(); services.AddSingleton(); @@ -93,6 +95,7 @@ public static void RegisterServices(IServiceCollection services, AppOptions appO handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); + handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); @@ -172,6 +175,8 @@ public static void RegisterServices(IServiceCollection services, AppOptions appO services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); + services.AddSingleton(); + services.AddStartupAction(); services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); diff --git a/src/Exceptionless.Core/Configuration/AppOptions.cs b/src/Exceptionless.Core/Configuration/AppOptions.cs index ff6217829a..a9d2a3124c 100644 --- a/src/Exceptionless.Core/Configuration/AppOptions.cs +++ b/src/Exceptionless.Core/Configuration/AppOptions.cs @@ -70,6 +70,7 @@ public class AppOptions public int BulkBatchSize { get; internal set; } public CacheOptions CacheOptions { get; internal set; } = null!; + public CustomFieldOptions CustomFieldOptions { get; internal set; } = null!; public MessageBusOptions MessageBusOptions { get; internal set; } = null!; public QueueOptions QueueOptions { get; internal set; } = null!; public StorageOptions StorageOptions { get; internal set; } = null!; @@ -123,6 +124,7 @@ public static AppOptions ReadFromConfiguration(IConfiguration config) catch { } options.CacheOptions = CacheOptions.ReadFromConfiguration(config, options); + options.CustomFieldOptions = CustomFieldOptions.ReadFromConfiguration(config, options); options.MessageBusOptions = MessageBusOptions.ReadFromConfiguration(config, options); options.QueueOptions = QueueOptions.ReadFromConfiguration(config, options); options.StorageOptions = StorageOptions.ReadFromConfiguration(config, options); diff --git a/src/Exceptionless.Core/Configuration/CustomFieldOptions.cs b/src/Exceptionless.Core/Configuration/CustomFieldOptions.cs new file mode 100644 index 0000000000..b944507919 --- /dev/null +++ b/src/Exceptionless.Core/Configuration/CustomFieldOptions.cs @@ -0,0 +1,16 @@ +using Microsoft.Extensions.Configuration; + +namespace Exceptionless.Core.Configuration; + +public class CustomFieldOptions +{ + public int MaxFieldsPerOrganization { get; internal set; } + + public static CustomFieldOptions ReadFromConfiguration(IConfiguration config, AppOptions appOptions) + { + return new CustomFieldOptions + { + MaxFieldsPerOrganization = config.GetValue(nameof(MaxFieldsPerOrganization), 20) + }; + } +} diff --git a/src/Exceptionless.Core/Extensions/PersistentEventExtensions.cs b/src/Exceptionless.Core/Extensions/PersistentEventExtensions.cs index 6a869d0425..9fa183befc 100644 --- a/src/Exceptionless.Core/Extensions/PersistentEventExtensions.cs +++ b/src/Exceptionless.Core/Extensions/PersistentEventExtensions.cs @@ -10,77 +10,6 @@ public static class PersistentEventExtensions { private static readonly char[] _commaSeparator = [',']; - public static void CopyDataToIndex(this PersistentEvent ev, string[]? keysToCopy = null) - { - if (ev.Data is null) - return; - - ev.Idx ??= new DataDictionary(); - - keysToCopy = keysToCopy?.Length > 0 ? keysToCopy : ev.Data.Keys.ToArray(); - - foreach (string key in keysToCopy.Where(k => !String.IsNullOrEmpty(k) && ev.Data.ContainsKey(k))) - { - string field = key.Trim().ToLowerInvariant(); - - if (field.StartsWith("@ref:")) - { - field = field.Substring(5); - if (!field.IsValidFieldName()) - continue; - - ev.Idx[field + "-r"] = ev.Data[key]?.ToString(); - continue; - } - - if (field.StartsWith('@') || ev.Data[key] is null) - continue; - - if (!field.IsValidFieldName()) - continue; - - var dataType = ev.Data[key]?.GetType(); - if (dataType is null) - continue; - - if (dataType == typeof(bool)) - { - ev.Idx[field + "-b"] = ev.Data[key]; - } - else if (dataType.IsNumeric()) - { - ev.Idx[field + "-n"] = ev.Data[key]; - } - else if (dataType == typeof(DateTime) || dataType == typeof(DateTimeOffset)) - { - ev.Idx[field + "-d"] = ev.Data[key]; - } - else if (dataType == typeof(string)) - { - string? input = ev.Data[key]?.ToString(); - if (String.IsNullOrEmpty(input) || input.Length >= 1000) - continue; - - if (input.GetJsonType() != JsonType.None) - continue; - - if (input[0] == '"') - input = input.TrimStart('"').TrimEnd('"'); - - if (Boolean.TryParse(input, out bool value)) - ev.Idx[field + "-b"] = value; - else if (DateTimeOffset.TryParse(input, out var dtoValue)) - ev.Idx[field + "-d"] = dtoValue; - else if (Decimal.TryParse(input, out decimal decValue)) - ev.Idx[field + "-n"] = decValue; - else if (Double.TryParse(input, out double dblValue) && !Double.IsNaN(dblValue) && !Double.IsInfinity(dblValue)) - ev.Idx[field + "-n"] = dblValue; - else - ev.Idx[field + "-s"] = input; - } - } - } - public static string? GetEventReference(this PersistentEvent ev, string name) { if (String.IsNullOrEmpty(name) || ev.Data is null) @@ -147,7 +76,7 @@ public static bool HasSessionEndTime(this PersistentEvent ev) return null; } - public static bool UpdateSessionStart(this PersistentEvent ev, DateTime lastActivityUtc, bool isSessionEnd = false) + public static bool UpdateSessionStart(this PersistentEvent ev, DateTime lastActivityUtc, bool isSessionEnd = false, bool hasError = false) { if (!ev.IsSessionStart()) return false; @@ -168,18 +97,19 @@ public static bool UpdateSessionStart(this PersistentEvent ev, DateTime lastActi if (isSessionEnd) { ev.Data[Event.KnownDataKeys.SessionEnd] = lastActivityUtc; - ev.CopyDataToIndex([Event.KnownDataKeys.SessionEnd]); } else { ev.Data.Remove(Event.KnownDataKeys.SessionEnd); - ev.Idx?.Remove(Event.KnownDataKeys.SessionEnd + "-d"); } + if (hasError) + ev.Data[Event.KnownDataKeys.SessionHasError] = true; + return true; } - public static PersistentEvent ToSessionStartEvent(this PersistentEvent source, ITextSerializer serializer, ILogger logger, DateTime? lastActivityUtc = null, bool? isSessionEnd = null, bool hasPremiumFeatures = true, bool includePrivateInformation = true) + public static PersistentEvent ToSessionStartEvent(this PersistentEvent source, ITextSerializer serializer, ILogger logger, DateTime? lastActivityUtc = null, bool? isSessionEnd = null, bool includePrivateInformation = true, bool hasError = false) { var startEvent = new PersistentEvent { @@ -238,10 +168,7 @@ public static PersistentEvent ToSessionStartEvent(this PersistentEvent source, I } if (lastActivityUtc.HasValue) - startEvent.UpdateSessionStart(lastActivityUtc.Value, isSessionEnd.GetValueOrDefault()); - - if (hasPremiumFeatures) - startEvent.CopyDataToIndex([]); + startEvent.UpdateSessionStart(lastActivityUtc.Value, isSessionEnd.GetValueOrDefault(), hasError); return startEvent; } diff --git a/src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationMaintenanceWorkItemHandler.cs b/src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationMaintenanceWorkItemHandler.cs index 73e886505c..625e13f0db 100644 --- a/src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationMaintenanceWorkItemHandler.cs +++ b/src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationMaintenanceWorkItemHandler.cs @@ -2,6 +2,7 @@ using Exceptionless.Core.Models; using Exceptionless.Core.Models.WorkItems; using Exceptionless.Core.Repositories; +using Exceptionless.Core.Services; using Foundatio.Jobs; using Foundatio.Lock; using Foundatio.Repositories; @@ -13,13 +14,15 @@ public class OrganizationMaintenanceWorkItemHandler : WorkItemHandlerBase { private readonly IOrganizationRepository _organizationRepository; private readonly BillingManager _billingManager; + private readonly EventCustomFieldService _eventCustomFieldService; private readonly TimeProvider _timeProvider; private readonly ILockProvider _lockProvider; - public OrganizationMaintenanceWorkItemHandler(IOrganizationRepository organizationRepository, ILockProvider lockProvider, BillingManager billingManager, TimeProvider timeProvider, ILoggerFactory loggerFactory) : base(loggerFactory) + public OrganizationMaintenanceWorkItemHandler(IOrganizationRepository organizationRepository, ILockProvider lockProvider, BillingManager billingManager, EventCustomFieldService eventCustomFieldService, TimeProvider timeProvider, ILoggerFactory loggerFactory) : base(loggerFactory) { _organizationRepository = organizationRepository; _billingManager = billingManager; + _eventCustomFieldService = eventCustomFieldService; _timeProvider = timeProvider; _lockProvider = lockProvider; } @@ -34,7 +37,8 @@ public override async Task HandleItemAsync(WorkItemContext context) const int LIMIT = 100; var wi = context.GetData()!; - Log.LogInformation("Received upgrade organizations work item. Upgrade Plans: {UpgradePlans}", wi.UpgradePlans); + Log.LogInformation("Received organization maintenance work item. UpgradePlans: {UpgradePlans} RemoveOldUsageStats: {RemoveOldUsageStats} EnsureSystemCustomFields: {EnsureSystemCustomFields}", + wi.UpgradePlans, wi.RemoveOldUsageStats, wi.EnsureSystemCustomFields); var results = await _organizationRepository.GetAllAsync(o => o.PageLimit(LIMIT)); while (results.Documents.Count > 0 && !context.CancellationToken.IsCancellationRequested) @@ -53,6 +57,18 @@ public override async Task HandleItemAsync(WorkItemContext context) foreach (var usage in organization.Usage.Where(u => u.Date < utcNow.Subtract(TimeSpan.FromDays(366))).ToList()) organization.Usage.Remove(usage); } + + if (wi.EnsureSystemCustomFields) + { + try + { + await _eventCustomFieldService.EnsureSystemFieldsAsync(organization.Id); + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + Log.LogError(ex, "Error ensuring system custom fields for organization {OrganizationId}", organization.Id); + } + } } if (wi.UpgradePlans || wi.RemoveOldUsageStats) diff --git a/src/Exceptionless.Core/Jobs/WorkItemHandlers/RemoveCustomFieldWorkItemHandler.cs b/src/Exceptionless.Core/Jobs/WorkItemHandlers/RemoveCustomFieldWorkItemHandler.cs new file mode 100644 index 0000000000..1aaf6cb420 --- /dev/null +++ b/src/Exceptionless.Core/Jobs/WorkItemHandlers/RemoveCustomFieldWorkItemHandler.cs @@ -0,0 +1,74 @@ +using Exceptionless.Core.Models.WorkItems; +using Foundatio.Jobs; +using Foundatio.Lock; +using Foundatio.Repositories.Elasticsearch.CustomFields; +using Microsoft.Extensions.Logging; + +namespace Exceptionless.Core.Jobs.WorkItemHandlers; + +public class RemoveCustomFieldWorkItemHandler : WorkItemHandlerBase +{ + private readonly ICustomFieldDefinitionRepository _customFieldDefinitionRepository; + private readonly ILockProvider _lockProvider; + + public RemoveCustomFieldWorkItemHandler( + ICustomFieldDefinitionRepository customFieldDefinitionRepository, + ILockProvider lockProvider, + ILoggerFactory loggerFactory) : base(loggerFactory) + { + _customFieldDefinitionRepository = customFieldDefinitionRepository; + _lockProvider = lockProvider; + } + + public override Task GetWorkItemLockAsync(object workItem, CancellationToken cancellationToken = default) + { + string cacheKey = $"{nameof(RemoveCustomFieldWorkItem)}:{((RemoveCustomFieldWorkItem)workItem).CustomFieldDefinitionId}"; + return _lockProvider.TryAcquireAsync(cacheKey, TimeSpan.FromMinutes(15), cancellationToken); + } + + public override async Task HandleItemAsync(WorkItemContext context) + { + var workItem = context.GetData()!; + + using (Log.BeginScope(new ExceptionlessState().Organization(workItem.OrganizationId))) + { + Log.LogInformation("Processing custom field removal work item for definition: {DefinitionId}, field: {FieldName}", + workItem.CustomFieldDefinitionId, workItem.FieldName); + + await context.ReportProgressAsync(0, "Acknowledging custom field soft-deletion..."); + + // GetByIdAsync INCLUDES soft-deleted records (by-ID lookups bypass the soft-delete filter). + // After the API handler soft-deletes and enqueues this work item, GetByIdAsync returns the + // definition with IsDeleted=true. A null result means the record has been physically removed. + var definition = await _customFieldDefinitionRepository.GetByIdAsync(workItem.CustomFieldDefinitionId); + + if (definition is null) + { + Log.LogWarning( + "Custom field definition {DefinitionId} ('{FieldName}') no longer exists. " + + "It may have been hard-deleted externally.", + workItem.CustomFieldDefinitionId, workItem.FieldName); + } + else if (!definition.IsDeleted) + { + Log.LogWarning( + "Custom field definition {DefinitionId} ('{FieldName}') is unexpectedly active. " + + "It should have been soft-deleted before this work item was enqueued.", + workItem.CustomFieldDefinitionId, workItem.FieldName); + } + else + { + // Normal path: definition is soft-deleted as expected. + // Hard-deletion (slot reclamation) is intentionally deferred to prevent slot-reuse corruption: + // recycling a slot before the organization's retention window expires would cause historical events + // indexed under the old field to appear in queries for a new field assigned the same slot. + Log.LogInformation( + "Custom field definition {DefinitionId} ('{FieldName}') is soft-deleted. " + + "Slot remains reserved because retention-aware cleanup is not implemented.", + workItem.CustomFieldDefinitionId, workItem.FieldName); + } + + await context.ReportProgressAsync(100, $"Custom field '{workItem.FieldName}' acknowledged."); + } + } +} diff --git a/src/Exceptionless.Core/Models/PersistentEvent.cs b/src/Exceptionless.Core/Models/PersistentEvent.cs index 968a4acd37..a1e3426bff 100644 --- a/src/Exceptionless.Core/Models/PersistentEvent.cs +++ b/src/Exceptionless.Core/Models/PersistentEvent.cs @@ -2,12 +2,13 @@ using System.Diagnostics; using Exceptionless.Core.Attributes; using Exceptionless.Core.Extensions; +using Foundatio.Repositories.Elasticsearch.CustomFields; using Foundatio.Repositories.Models; namespace Exceptionless.Core.Models; [DebuggerDisplay("Id: {Id}, Type: {Type}, Date: {Date}, Message: {Message}, Value: {Value}, Count: {Count}")] -public class PersistentEvent : Event, IOwnedByOrganizationAndProjectAndStackWithIdentity, IHaveCreatedDate, IValidatableObject +public class PersistentEvent : Event, IOwnedByOrganizationAndProjectAndStackWithIdentity, IHaveCreatedDate, IValidatableObject, IHaveVirtualCustomFields { /// /// Unique id that identifies an event. @@ -52,6 +53,28 @@ public class PersistentEvent : Event, IOwnedByOrganizationAndProjectAndStackWith [MiniValidation.SkipRecursion] public DataDictionary? Idx { get; set; } + // IHaveVirtualCustomFields explicit implementation + IDictionary IHaveVirtualCustomFields.Idx => (IDictionary)(Idx ??= new DataDictionary()); + + public string GetTenantKey() => OrganizationId; + + public IDictionary GetCustomFields() + { + if (Data is null) return new DataDictionary(); + var result = new Dictionary(StringComparer.OrdinalIgnoreCase); + foreach (var kvp in Data.Where(kvp => !String.IsNullOrEmpty(kvp.Key) + && (!kvp.Key.StartsWith('@') || kvp.Key.StartsWith("@ref:", StringComparison.OrdinalIgnoreCase)) + && kvp.Value is string or bool or int or long or float or double or decimal or DateTime or DateTimeOffset)) + { + result[kvp.Key] = kvp.Value; + } + return result; + } + + public object GetCustomField(string name) => Data is not null && Data.TryGetValue(name, out var v) && v is not null ? v : null!; + public void SetCustomField(string name, object value) { Data ??= new DataDictionary(); Data[name] = value; } + public void RemoveCustomField(string name) => Data?.Remove(name); + public IEnumerable Validate(ValidationContext validationContext) { if (Date == DateTimeOffset.MinValue) diff --git a/src/Exceptionless.Core/Models/SavedView.cs b/src/Exceptionless.Core/Models/SavedView.cs index 1c3bcc9393..27aed9d7cc 100644 --- a/src/Exceptionless.Core/Models/SavedView.cs +++ b/src/Exceptionless.Core/Models/SavedView.cs @@ -87,6 +87,9 @@ public partial record SavedView : IOwnedByOrganizationWithIdentity, IHaveDates /// Schema version for future filter definition migrations. public int Version { get; set; } = 1; + /// True when the filter references at least one custom field or other premium feature. + public bool UsesPremiumFeatures { get; set; } + /// Dashboard page identifier: "events", "stacks", or "stream". [Required] [RegularExpression("^(events|stacks|stream)$")] diff --git a/src/Exceptionless.Core/Models/WorkItems/OrganizationMaintenanceWorkItem.cs b/src/Exceptionless.Core/Models/WorkItems/OrganizationMaintenanceWorkItem.cs index d619d1a3dd..2bd1d04ca6 100644 --- a/src/Exceptionless.Core/Models/WorkItems/OrganizationMaintenanceWorkItem.cs +++ b/src/Exceptionless.Core/Models/WorkItems/OrganizationMaintenanceWorkItem.cs @@ -4,4 +4,5 @@ public class OrganizationMaintenanceWorkItem { public bool UpgradePlans { get; set; } public bool RemoveOldUsageStats { get; set; } + public bool EnsureSystemCustomFields { get; set; } } diff --git a/src/Exceptionless.Core/Models/WorkItems/RemoveCustomFieldWorkItem.cs b/src/Exceptionless.Core/Models/WorkItems/RemoveCustomFieldWorkItem.cs new file mode 100644 index 0000000000..25485a1e07 --- /dev/null +++ b/src/Exceptionless.Core/Models/WorkItems/RemoveCustomFieldWorkItem.cs @@ -0,0 +1,8 @@ +namespace Exceptionless.Core.Models.WorkItems; + +public record RemoveCustomFieldWorkItem +{ + public required string OrganizationId { get; init; } + public required string CustomFieldDefinitionId { get; init; } + public required string FieldName { get; init; } +} diff --git a/src/Exceptionless.Core/Pipeline/035_CopySimpleDataToIdxAction.cs b/src/Exceptionless.Core/Pipeline/035_CopySimpleDataToIdxAction.cs deleted file mode 100644 index b85723efa2..0000000000 --- a/src/Exceptionless.Core/Pipeline/035_CopySimpleDataToIdxAction.cs +++ /dev/null @@ -1,29 +0,0 @@ -using Exceptionless.Core.Plugins.EventProcessor; -using Microsoft.Extensions.Logging; - -namespace Exceptionless.Core.Pipeline; - -[Priority(40)] -public class CopySimpleDataToIdxAction : EventPipelineActionBase -{ - public CopySimpleDataToIdxAction(AppOptions options, ILoggerFactory loggerFactory) : base(options, loggerFactory) { } - - public override Task ProcessAsync(EventContext ctx) - { - if (!ctx.Organization.HasPremiumFeatures) - return Task.CompletedTask; - - // TODO: Do we need a pipeline action to trim keys and remove null values that may be sent by other native clients. - ctx.Event.CopyDataToIndex([]); - int fieldCount = ctx.Event.Idx?.Count ?? 0; - AppDiagnostics.EventsFieldCount.Record(fieldCount); - if (fieldCount > 20 && _logger.IsEnabled(LogLevel.Warning)) - { - var ev = ctx.Event; - using (_logger.BeginScope(new ExceptionlessState().Organization(ctx.Organization.Id).Property("Event", new { ev.Date, ev.StackId, ev.Type, ev.Source, ev.Message, ev.Value, ev.Geo, ev.ReferenceId, ev.Tags, ev.Idx }))) - _logger.LogWarning("Event has {FieldCount} indexed fields", fieldCount); - } - - return Task.CompletedTask; - } -} diff --git a/src/Exceptionless.Core/Plugins/EventProcessor/Default/70_SessionPlugin.cs b/src/Exceptionless.Core/Plugins/EventProcessor/Default/70_SessionPlugin.cs index 580831e116..3ae40afa97 100644 --- a/src/Exceptionless.Core/Plugins/EventProcessor/Default/70_SessionPlugin.cs +++ b/src/Exceptionless.Core/Plugins/EventProcessor/Default/70_SessionPlugin.cs @@ -89,7 +89,8 @@ private async Task ProcessManualSessionsAsync(ICollection contexts }); // try to update an existing session - string? sessionStartEventId = await UpdateSessionStartEventAsync(projectId, session.Key, lastSessionEvent.Event.Date.UtcDateTime, sessionEndEvent is not null); + bool sessionHasError = session.Any(ctx => String.Equals(ctx.Event.Type, Event.KnownTypes.Error, StringComparison.Ordinal)); + string? sessionStartEventId = await UpdateSessionStartEventAsync(projectId, session.Key, lastSessionEvent.Event.Date.UtcDateTime, sessionEndEvent is not null, sessionHasError); // do we already have a session start for this session id? if (!String.IsNullOrEmpty(sessionStartEventId) && sessionStartEvent is not null) @@ -100,7 +101,7 @@ private async Task ProcessManualSessionsAsync(ICollection contexts else if (String.IsNullOrEmpty(sessionStartEventId) && sessionStartEvent is not null) { // no existing session, session start is in the batch - sessionStartEvent.Event.UpdateSessionStart(lastSessionEvent.Event.Date.UtcDateTime, sessionEndEvent is not null); + sessionStartEvent.Event.UpdateSessionStart(lastSessionEvent.Event.Date.UtcDateTime, sessionEndEvent is not null, sessionHasError); sessionStartEvent.SetProperty("SetSessionStartEventId", true); } else if (String.IsNullOrEmpty(sessionStartEventId)) @@ -116,7 +117,7 @@ private async Task ProcessManualSessionsAsync(ICollection contexts } // create a new session start event - await CreateSessionStartEventAsync(firstSessionEvent, lastSessionEvent.Event.Date.UtcDateTime, sessionEndEvent is not null); + await CreateSessionStartEventAsync(firstSessionEvent, lastSessionEvent.Event.Date.UtcDateTime, sessionEndEvent is not null, sessionHasError); } } } @@ -179,16 +180,17 @@ private async Task ProcessAutoSessionsAsync(ICollection contexts) session.ForEach(s => s.Event.SetSessionId(sessionId)); + bool identitySessionHasError = session.Any(ctx => String.Equals(ctx.Event.Type, Event.KnownTypes.Error, StringComparison.Ordinal)); if (isNewSession) { if (sessionStartEvent is not null) { - sessionStartEvent.Event.UpdateSessionStart(lastSessionEvent.Event.Date.UtcDateTime, lastSessionEvent.Event.IsSessionEnd()); + sessionStartEvent.Event.UpdateSessionStart(lastSessionEvent.Event.Date.UtcDateTime, lastSessionEvent.Event.IsSessionEnd(), identitySessionHasError); sessionStartEvent.SetProperty("SetSessionStartEventId", true); } else { - await CreateSessionStartEventAsync(firstSessionEvent, lastSessionEvent.Event.Date.UtcDateTime, lastSessionEvent.Event.IsSessionEnd()); + await CreateSessionStartEventAsync(firstSessionEvent, lastSessionEvent.Event.Date.UtcDateTime, lastSessionEvent.Event.IsSessionEnd(), identitySessionHasError); } if (!lastSessionEvent.Event.IsSessionEnd()) @@ -203,7 +205,7 @@ private async Task ProcessAutoSessionsAsync(ICollection contexts) sessionStartEvent.IsCancelled = true; } - await UpdateSessionStartEventAsync(projectId, sessionId, lastSessionEvent.Event.Date.UtcDateTime, lastSessionEvent.Event.IsSessionEnd()); + await UpdateSessionStartEventAsync(projectId, sessionId, lastSessionEvent.Event.Date.UtcDateTime, lastSessionEvent.Event.IsSessionEnd(), identitySessionHasError); } } } @@ -284,9 +286,9 @@ private Task SetIdentitySessionIdAsync(string projectId, string identity, return _cache.SetAsync(GetIdentitySessionIdCacheKey(projectId, identity), sessionId, _sessionTimeout); } - private async Task CreateSessionStartEventAsync(EventContext startContext, DateTime? lastActivityUtc, bool? isSessionEnd) + private async Task CreateSessionStartEventAsync(EventContext startContext, DateTime? lastActivityUtc, bool? isSessionEnd, bool hasError = false) { - var startEvent = startContext.Event.ToSessionStartEvent(_serializer, _logger, lastActivityUtc, isSessionEnd, startContext.Organization.HasPremiumFeatures, startContext.IncludePrivateInformation); + var startEvent = startContext.Event.ToSessionStartEvent(_serializer, _logger, lastActivityUtc, isSessionEnd, startContext.IncludePrivateInformation, hasError); var startEventContexts = new List { new(startEvent, startContext.Organization, startContext.Project) }; diff --git a/src/Exceptionless.Core/Repositories/Configuration/ExceptionlessElasticConfiguration.cs b/src/Exceptionless.Core/Repositories/Configuration/ExceptionlessElasticConfiguration.cs index 7fd742b21b..175deb290b 100644 --- a/src/Exceptionless.Core/Repositories/Configuration/ExceptionlessElasticConfiguration.cs +++ b/src/Exceptionless.Core/Repositories/Configuration/ExceptionlessElasticConfiguration.cs @@ -12,6 +12,7 @@ using Foundatio.Queues; using Foundatio.Repositories.Elasticsearch; using Foundatio.Repositories.Elasticsearch.Configuration; +using Foundatio.Repositories.Elasticsearch.CustomFields; using Foundatio.Repositories.Elasticsearch.Queries.Builders; using Foundatio.Repositories.Serialization; using Foundatio.Resilience; @@ -41,6 +42,7 @@ ILoggerFactory loggerFactory _logger.LogInformation("All new indexes will be created with {ElasticsearchNumberOfShards} Shards and {ElasticsearchNumberOfReplicas} Replicas", _appOptions.ElasticsearchOptions.NumberOfShards, _appOptions.ElasticsearchOptions.NumberOfReplicas); AddIndex(Stacks = new StackIndex(this)); AddIndex(Events = new EventIndex(this, serviceProvider, appOptions)); + AddCustomFieldIndex(_appOptions.ElasticsearchOptions.ScopePrefix + "customfields", appOptions.ElasticsearchOptions.NumberOfReplicas); AddIndex(Migrations = new MigrationIndex(this, _appOptions.ElasticsearchOptions.ScopePrefix + "migrations", appOptions.ElasticsearchOptions.NumberOfReplicas)); AddIndex(Organizations = new OrganizationIndex(this)); AddIndex(OAuthApplications = new OAuthApplicationIndex(this)); diff --git a/src/Exceptionless.Core/Repositories/Configuration/Indexes/EventIndex.cs b/src/Exceptionless.Core/Repositories/Configuration/Indexes/EventIndex.cs index 9c7b77b4cd..52ec423723 100644 --- a/src/Exceptionless.Core/Repositories/Configuration/Indexes/EventIndex.cs +++ b/src/Exceptionless.Core/Repositories/Configuration/Indexes/EventIndex.cs @@ -28,6 +28,8 @@ public sealed class EventIndex : DailyIndex public EventIndex(ExceptionlessElasticConfiguration configuration, IServiceProvider serviceProvider, AppOptions appOptions) : base(configuration, configuration.Options.ScopePrefix + "events", 1, doc => ((PersistentEvent)doc).Date.UtcDateTime) { + AddStandardCustomFieldTypes(); + _configuration = configuration; _serviceProvider = serviceProvider; @@ -53,12 +55,6 @@ public override void ConfigureIndexMapping(TypeMappingDescriptor dt - .Add("idx_bool", t => t.Match("*-b").Mapping(m => m.Boolean(s => { }))) - .Add("idx_date", t => t.Match("*-d").Mapping(m => m.Date(s => { }))) - .Add("idx_number", t => t.Match("*-n").Mapping(m => m.DoubleNumber(s => { }))) - .Add("idx_reference", t => t.Match("*-r").Mapping(m => m.Keyword(s => s.IgnoreAbove(256)))) - .Add("idx_string", t => t.Match("*-s").Mapping(m => m.Keyword(s => s.IgnoreAbove(1024))))) .Properties(p => p .SetupDefaults() .Keyword(e => e.OrganizationId) @@ -80,7 +76,6 @@ public override void ConfigureIndexMapping(TypeMappingDescriptor e.Count) .Boolean(e => e.IsFirstOccurrence) .FieldAlias(Alias.IsFirstOccurrence, a => a.Path(f => f.IsFirstOccurrence)) - .Object(e => e.Idx, o => o.Dynamic(DynamicMapping.True)) .Object(e => e.Data, o => o.Properties(p2 => p2 .AddVersionMapping() .AddLevelMapping() @@ -153,7 +148,6 @@ protected override void ConfigureQueryParser(ElasticQueryParserConfiguration con EventIndexExtensions.DataPath(Event.KnownDataKeys.UserInfo, u => u.Identity), EventIndexExtensions.DataPath(Event.KnownDataKeys.UserInfo, u => u.Name) ]) - .AddQueryVisitor(new EventFieldsQueryVisitor()) .UseFieldMap(new Dictionary { { Alias.BrowserVersion, EventIndexExtensions.DataDictionaryPath(Event.KnownDataKeys.RequestInfo, r => r.Data, RequestInfo.KnownDataKeys.BrowserVersion) }, { Alias.BrowserMajorVersion, EventIndexExtensions.DataDictionaryPath(Event.KnownDataKeys.RequestInfo, r => r.Data, RequestInfo.KnownDataKeys.BrowserMajorVersion) }, diff --git a/src/Exceptionless.Core/Repositories/EventRepository.cs b/src/Exceptionless.Core/Repositories/EventRepository.cs index 3fba37a69d..ff82fd390d 100644 --- a/src/Exceptionless.Core/Repositories/EventRepository.cs +++ b/src/Exceptionless.Core/Repositories/EventRepository.cs @@ -1,22 +1,33 @@ using Elastic.Clients.Elasticsearch.QueryDsl; using Exceptionless.Core.Models; using Exceptionless.Core.Repositories.Configuration; +using Exceptionless.Core.Repositories.Options; using Exceptionless.Core.Repositories.Queries; +using Exceptionless.Core.Services; using Exceptionless.Core.Validation; using Exceptionless.DateTimeExtensions; +using Foundatio.Parsers.LuceneQueries.Visitors; using Foundatio.Repositories; +using Foundatio.Repositories.Elasticsearch.CustomFields; +using Foundatio.Repositories.Elasticsearch.Extensions; using Foundatio.Repositories.Models; +using Foundatio.Repositories.Options; namespace Exceptionless.Core.Repositories; public class EventRepository : RepositoryOwnedByOrganizationAndProject, IEventRepository { + private const string LegacySessionEndIdxField = "sessionend-d"; private readonly TimeProvider _timeProvider; + private readonly IProjectRepository _projectRepository; + private readonly IStackRepository _stackRepository; - public EventRepository(ExceptionlessElasticConfiguration configuration, AppOptions options, MiniValidationValidator validator) + public EventRepository(ExceptionlessElasticConfiguration configuration, AppOptions options, MiniValidationValidator validator, IProjectRepository projectRepository, IStackRepository stackRepository) : base(configuration.Events, validator, options) { _timeProvider = configuration.TimeProvider; + _projectRepository = projectRepository; + _stackRepository = stackRepository; DisableCache(); // NOTE: If cache is ever enabled, then fast paths for patching/deleting with scripts will be super slow! BatchNotifications = true; @@ -35,7 +46,14 @@ public Task> GetOpenSessionsAsync(DateTime createdB { var query = new RepositoryQuery() .FieldEquals(e => e.Type, Event.KnownTypes.Session) - .ElasticFilter(new BoolQuery { MustNot = [new ExistsQuery { Field = $"idx.{Event.KnownDataKeys.SessionEnd}-d" }] }); + .ElasticFilter(new BoolQuery + { + MustNot = + [ + new ExistsQuery { Field = $"idx.{EventCustomFieldService.SessionEndIdxField}" }, + new ExistsQuery { Field = $"idx.{LegacySessionEndIdxField}" } + ] + }); if (createdBeforeUtc.Ticks > 0) query = query.DateRange(null, createdBeforeUtc, (PersistentEvent e) => e.Date); // No lower bound, upper bound is exclusive @@ -52,7 +70,7 @@ public async Task UpdateSessionStartLastActivityAsync(string id, DateTime if (ev is null) return false; - if (!ev.UpdateSessionStart(lastActivityUtc, isSessionEnd)) + if (!ev.UpdateSessionStart(lastActivityUtc, isSessionEnd, hasError)) return false; await SaveAsync(ev, o => o.Notifications(sendNotifications)); @@ -65,7 +83,7 @@ public Task RemoveAllAsync(string organizationId, string? clientIpAddress, var query = new RepositoryQuery().Organization(organizationId); if (utcStart.HasValue && utcEnd.HasValue) - query = query.DateRange(utcStart, utcEnd, InferField(e => e.Date)).Index(utcStart, utcEnd); + query = query.DateRange(utcStart, utcEnd, InferField(e => e.Date)); else if (utcEnd.HasValue) query = query.DateRange(null, utcEnd, (PersistentEvent e) => e.Date); else if (utcStart.HasValue) @@ -196,4 +214,102 @@ public Task RemoveAllByStackIdsAsync(string[] stackIds) return RemoveAllAsync(q => q.Stack(stackIds)); } + + /// + /// Override to prevent the base from clearing idx (which would destroy slot values populated by EventCustomFieldService). + /// Custom field indexing is handled externally by EventCustomFieldService. + /// + protected override Task OnCustomFieldsDocumentsChanging(object sender, DocumentsChangeEventArgs args) + => Task.CompletedTask; + + /// + /// Resolve the tenant key from the query's organization filter. + /// + protected override string? GetTenantKey(IRepositoryQuery query) + { + var organizationIds = query.GetOrganizations(); + return organizationIds.Count == 1 ? organizationIds.First() : null; + } + + /// + /// Custom field query resolution: resolves idx.fieldName and data.fieldName to idx.{type}-{slot}. + /// Blocks raw slot access (e.g., idx.keyword-7) to prevent querying deleted or other tenants' fields. + /// Returns null for non-idx/data fields so the global resolver (field aliases) still works. + /// + // Well-known system field slot mappings (deterministic — always slot 1 for each type). + private static readonly Dictionary _systemFieldSlots = new(StringComparer.OrdinalIgnoreCase) + { + ["@ref:session"] = EventCustomFieldService.SessionReferenceIdxField, + [Event.KnownDataKeys.SessionEnd] = EventCustomFieldService.SessionEndIdxField, + [Event.KnownDataKeys.SessionHasError] = EventCustomFieldService.SessionHasErrorIdxField, + }; + + protected override async Task OnCustomFieldsBeforeQuery(object sender, BeforeQueryEventArgs args) + { + var tenantKey = await ResolveTenantKeyAsync(args.Query); + + var definitionRepo = ElasticIndex.Configuration.CustomFieldDefinitionRepository; + + // Lazy-load field mapping only when a query actually references idx.* or data.* fields. + // Most queries (count, date histograms, simple filters) never hit custom fields, + // so deferring this avoids a cache/ES lookup on every hot-path query. + Dictionary? mapping = null; + + args.Options.QueryFieldResolver(async (field, _) => + { + string? fieldName = null; + if (field.StartsWith("idx.", StringComparison.OrdinalIgnoreCase)) + fieldName = field.Substring(4); + else if (field.StartsWith("data.", StringComparison.OrdinalIgnoreCase)) + fieldName = field.Substring(5); + else if (field.StartsWith("ref.", StringComparison.OrdinalIgnoreCase)) + fieldName = $"@ref:{field.Substring(4)}"; + + if (fieldName is null) + return null; + + // System fields have deterministic slots that don't require tenant resolution. + if (_systemFieldSlots.TryGetValue(fieldName, out var systemSlot)) + return $"idx.{systemSlot}"; + + // Non-system fields require a tenant key to look up their slot assignment. + if (String.IsNullOrEmpty(tenantKey) || definitionRepo is null) + { + // Without tenant context, block raw idx access; data.* fields fall through. + return field.StartsWith("idx.", StringComparison.OrdinalIgnoreCase) ? "idx.__blocked__" : null; + } + + mapping ??= (await definitionRepo.GetFieldMappingAsync(EntityTypeName, tenantKey)) + .ToDictionary(kvp => kvp.Key, kvp => kvp.Value.GetIdxName(), StringComparer.OrdinalIgnoreCase); + + if (mapping.TryGetValue(fieldName, out var idxName)) + return $"idx.{idxName}"; + + // Block raw slot access (e.g., idx.keyword-7) and unknown idx fields. + // Redirect to a non-existent field so the clause matches nothing. + if (field.StartsWith("idx.", StringComparison.OrdinalIgnoreCase)) + return "idx.__blocked__"; + + // For data.* and ref.* fields that don't map to a custom field, return null to let + // other resolvers handle legitimate data paths (e.g., data.@version). + return null; + }); + } + + private async Task ResolveTenantKeyAsync(IRepositoryQuery query) + { + var organizationIds = query.GetOrganizations(); + if (organizationIds.Count == 1) + return organizationIds.Single(); + + var projectIds = query.GetProjects(); + if (projectIds.Count == 1) + return (await _projectRepository.GetByIdAsync(projectIds.Single()))?.OrganizationId; + + var stackIds = query.GetStacks(); + if (stackIds.Count == 1) + return (await _stackRepository.GetByIdAsync(stackIds.Single()))?.OrganizationId; + + return null; + } } diff --git a/src/Exceptionless.Core/Repositories/Queries/Visitors/EventFieldsQueryVisitor.cs b/src/Exceptionless.Core/Repositories/Queries/Visitors/EventFieldsQueryVisitor.cs deleted file mode 100644 index 43065cc189..0000000000 --- a/src/Exceptionless.Core/Repositories/Queries/Visitors/EventFieldsQueryVisitor.cs +++ /dev/null @@ -1,135 +0,0 @@ -using Exceptionless.Core.Extensions; -using Exceptionless.Core.Models; -using Foundatio.Parsers.LuceneQueries.Nodes; -using Foundatio.Parsers.LuceneQueries.Visitors; - -namespace Exceptionless.Core.Repositories.Queries; - -public class EventFieldsQueryVisitor : ChainableQueryVisitor -{ - public override async Task VisitAsync(GroupNode node, IQueryVisitorContext context) - { - var childTerms = new List(); - if (node.Left is TermNode { Field: null, Term: not null } leftTermNode) - childTerms.Add(leftTermNode.Term); - - if (node.Left is TermRangeNode { Field: null } leftTermRangeNode) - { - if (leftTermRangeNode.Min is not null) - childTerms.Add(leftTermRangeNode.Min); - if (leftTermRangeNode.Max is not null) - childTerms.Add(leftTermRangeNode.Max); - } - - if (node.Right is TermNode { Field: null, Term: not null } rightTermNode) - childTerms.Add(rightTermNode.Term); - - if (node.Right is TermRangeNode { Field: null } rightTermRangeNode) - { - if (rightTermRangeNode.Min is not null) - childTerms.Add(rightTermRangeNode.Min); - if (rightTermRangeNode.Max is not null) - childTerms.Add(rightTermRangeNode.Max); - } - - node.Field = GetCustomFieldName(node.Field, childTerms.ToArray()) ?? node.Field; - foreach (var child in node.Children) - await child.AcceptAsync(this, context); - } - - public override Task VisitAsync(TermNode node, IQueryVisitorContext context) - { - // using all fields search - if (String.IsNullOrEmpty(node.Field)) - { - return Task.CompletedTask; - } - - node.Field = GetCustomFieldName(node.Field, [node.Term]); - return Task.CompletedTask; - } - - public override Task VisitAsync(TermRangeNode node, IQueryVisitorContext context) - { - node.Field = GetCustomFieldName(node.Field, [node.Min, node.Max]); - return Task.CompletedTask; - } - - public override Task VisitAsync(ExistsNode node, IQueryVisitorContext context) - { - node.Field = GetCustomFieldName(node.Field, []); - return Task.CompletedTask; - } - - public override Task VisitAsync(MissingNode node, IQueryVisitorContext context) - { - node.Field = GetCustomFieldName(node.Field, []); - return Task.CompletedTask; - } - - private string? GetCustomFieldName(string? field, string?[] terms) - { - if (String.IsNullOrEmpty(field)) - return null; - - string[] parts = field.Split('.'); - if (parts.Length != 2 || (parts.Length == 2 && parts[1].StartsWith("@"))) - return field; - - if (String.Equals(parts[0], "data", StringComparison.OrdinalIgnoreCase)) - { - string termType; - if (String.Equals(parts[1], Event.KnownDataKeys.SessionEnd, StringComparison.OrdinalIgnoreCase)) - termType = "d"; - else if (String.Equals(parts[1], Event.KnownDataKeys.SessionHasError, StringComparison.OrdinalIgnoreCase)) - termType = "b"; - else - termType = GetTermType(terms); - - field = $"idx.{parts[1].ToLowerInvariant()}-{termType}"; - } - else if (String.Equals(parts[0], "ref", StringComparison.OrdinalIgnoreCase)) - { - field = $"idx.{parts[1].ToLowerInvariant()}-r"; - } - - return field; - } - - private static string GetTermType(string?[] terms) - { - string termType = "s"; - - var trimmedTerms = terms.OfType().Distinct().ToList(); - foreach (string term in trimmedTerms) - { - if (term.StartsWith('*')) - continue; - - if (Boolean.TryParse(term, out bool _)) - termType = "b"; - else if (term.IsNumeric()) - termType = "n"; - else if (DateTime.TryParse(term, out DateTime _)) - termType = "d"; - - break; - } - - // Some terms can be a string date range: [now TO now/d+1d} - if (String.Equals(termType, "s") && trimmedTerms.Count > 0 && trimmedTerms.All(t => String.Equals(t, "now", StringComparison.OrdinalIgnoreCase) || t.StartsWith("now/", StringComparison.OrdinalIgnoreCase))) - termType = "d"; - - return termType; - } - - public static Task RunAsync(IQueryNode node, IQueryVisitorContext? context = null) - { - return new EventFieldsQueryVisitor().AcceptAsync(node, context ?? new QueryVisitorContext()); - } - - public static IQueryNode? Run(IQueryNode node, IQueryVisitorContext? context = null) - { - return RunAsync(node, context).GetAwaiter().GetResult(); - } -} diff --git a/src/Exceptionless.Core/Services/EventCustomFieldService.cs b/src/Exceptionless.Core/Services/EventCustomFieldService.cs new file mode 100644 index 0000000000..7458dc2264 --- /dev/null +++ b/src/Exceptionless.Core/Services/EventCustomFieldService.cs @@ -0,0 +1,431 @@ +using System.Globalization; +using Exceptionless.Core.Models; +using Exceptionless.Core.Repositories; +using Foundatio.Extensions.Hosting.Startup; +using Foundatio.Lock; +using Foundatio.Repositories.Elasticsearch.CustomFields; +using Foundatio.Repositories.Models; +using Microsoft.Extensions.Logging; + +namespace Exceptionless.Core.Services; + +public class EventCustomFieldService : IStartupAction +{ + private readonly IEventRepository _eventRepository; + private readonly ICustomFieldDefinitionRepository _customFieldDefinitionRepository; + private readonly ILockProvider _lockProvider; + private readonly ILogger _logger; + + private const int MaxKeywordLength = 256; + + /// + /// System fields that are auto-provisioned per organization and cannot be deleted. + /// These support the session system and core event metadata. + /// Because they are always provisioned first in their respective types, their slot numbers are deterministic. + /// + public static readonly IReadOnlyList<(string Name, string IndexType)> SystemFields = + [ + ("@ref:session", "keyword"), + (Event.KnownDataKeys.SessionEnd, "date"), + (Event.KnownDataKeys.SessionHasError, "bool") + ]; + + /// + /// Well-known idx field names for system fields. These are deterministic because system fields + /// are always provisioned first via EnsureSystemFieldsAsync (slot 1 for each type). + /// + public const string SessionReferenceIdxField = "keyword-1"; + public const string SessionEndIdxField = "date-1"; + public const string SessionHasErrorIdxField = "bool-1"; + + /// + /// The set of index types registered by AddStandardCustomFieldTypes() in EventIndex. + /// Only these types are supported for custom field definitions; any other type string would result + /// in an un-indexed, unqueryable field. + /// + public static readonly IReadOnlySet SupportedIndexTypes = new HashSet(StringComparer.OrdinalIgnoreCase) + { + "bool", "date", "double", "float", "int", "keyword", "long", "string" + }; + + public EventCustomFieldService( + IEventRepository eventRepository, + ICustomFieldDefinitionRepository customFieldDefinitionRepository, + ILockProvider lockProvider, + ILoggerFactory loggerFactory) + { + _eventRepository = eventRepository; + _customFieldDefinitionRepository = customFieldDefinitionRepository; + _lockProvider = lockProvider; + _logger = loggerFactory.CreateLogger(); + } + + public Task RunAsync(CancellationToken shutdownToken = default) + { + _eventRepository.DocumentsChanging.AddHandler(OnDocumentsChangingAsync); + return Task.CompletedTask; + } + + /// + /// Ensures system fields exist for the given organization. + /// + public async Task EnsureSystemFieldsAsync(string organizationId) + { + var existing = await _customFieldDefinitionRepository.GetFieldMappingAsync(nameof(PersistentEvent), organizationId); + + foreach (var (name, indexType) in SystemFields) + { + if (existing.ContainsKey(name)) + continue; + + await _customFieldDefinitionRepository.AddFieldAsync( + nameof(PersistentEvent), organizationId, name, indexType, + description: $"System field: {name}"); + } + } + + /// + /// Returns true if the given field name is a system/reserved field that cannot be deleted. + /// + public static bool IsSystemField(string fieldName) + { + return SystemFields.Any(f => String.Equals(f.Name, fieldName, StringComparison.OrdinalIgnoreCase)); + } + + /// + /// Creates a new custom field definition under a distributed lock so concurrent requests + /// from the same organization cannot race past the quota check. + /// Returns null when the field cannot be created (quota exceeded or duplicate name). + /// + public async Task CreateFieldAsync( + string organizationId, + string name, + string indexType, + int maxFieldsPerOrganization, + string? description = null, + int? displayOrder = null, + CancellationToken cancellationToken = default) + { + // Ensure system fields are provisioned before user-defined fields so they occupy slot 1 of their type. + await EnsureSystemFieldsAsync(organizationId); + + await using var fieldLock = await _lockProvider.AcquireAsync($"custom-field-create:{organizationId}", TimeSpan.FromSeconds(30), cancellationToken: cancellationToken); + if (fieldLock is null) + { + _logger.LogWarning("Could not acquire custom field creation lock for organization {OrganizationId}", organizationId); + return null; + } + + // Re-read the field mapping inside the lock for an accurate count. + var existingPage = await _customFieldDefinitionRepository.FindByTenantAsync(nameof(PersistentEvent), organizationId); + var allActive = new List(existingPage.Documents); + while (await existingPage.NextPageAsync()) + allActive.AddRange(existingPage.Documents); + + // System fields are not counted against the user quota. + var userDefinedActiveCount = allActive.Count(f => !IsSystemField(f.Name)); + if (userDefinedActiveCount >= maxFieldsPerOrganization) + return null; + + // Case-insensitive duplicate check inside the lock. + if (allActive.Any(f => String.Equals(f.Name, name, StringComparison.OrdinalIgnoreCase))) + return null; + + return await _customFieldDefinitionRepository.AddFieldAsync( + nameof(PersistentEvent), organizationId, name, indexType, description, displayOrder ?? 0); + } + + private async Task OnDocumentsChangingAsync(object sender, DocumentsChangeEventArgs args) + { + if (args.ChangeType == ChangeType.Removed) + return; + + if (args.Documents is null || args.Documents.Count == 0) + return; + + var documentsByOrganization = args.Documents + .Select(d => d.Value) + .OfType() + .GroupBy(d => d.OrganizationId) + .Where(g => !String.IsNullOrEmpty(g.Key)); + + foreach (var organizationGroup in documentsByOrganization) + { + IDictionary? fieldMapping = null; + try + { + fieldMapping = await _customFieldDefinitionRepository.GetFieldMappingAsync(nameof(PersistentEvent), organizationGroup.Key); + + // Lazily ensure all system fields are provisioned for this organization. + // Check each system field individually to handle partial-provisioning failures. + if (SystemFields.Any(f => !fieldMapping.ContainsKey(f.Name))) + { + await EnsureSystemFieldsAsync(organizationGroup.Key); + fieldMapping = await _customFieldDefinitionRepository.GetFieldMappingAsync(nameof(PersistentEvent), organizationGroup.Key); + } + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + _logger.LogError(ex, "Error loading custom field definitions for organization {OrganizationId}", organizationGroup.Key); + continue; + } + + foreach (var document in organizationGroup) + { + try + { + ProcessEventCustomFields(document, fieldMapping); + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + _logger.LogError(ex, "Error processing custom fields for event {EventId}", document.Id); + } + } + } + } + + private void ProcessEventCustomFields(PersistentEvent ev, IDictionary fieldMapping) + { + ClearCustomFieldSlots(ev); + + if (fieldMapping.Count == 0 || ev.Data is null || ev.Data.Count == 0) + return; + + var idx = ((IHaveVirtualCustomFields)ev).Idx; + + // Iterate the field mapping (max ~20 entries) rather than all of ev.Data + // to avoid allocating an intermediate dictionary for events with large payloads. + // DataDictionary uses OrdinalIgnoreCase so the lookup is case-insensitive. + foreach (var (fieldName, definition) in fieldMapping) + { + if (definition.IsDeleted) + continue; + + if (!ev.Data.TryGetValue(fieldName, out var rawValue) || rawValue is null) + continue; + + // Only primitive types are indexable (mirrors GetCustomFields filtering). + if (rawValue is not (string or bool or int or long or float or double or decimal or DateTime or DateTimeOffset)) + continue; + + try + { + var value = ConvertValue(rawValue, definition.IndexType); + if (value is not null) + idx[definition.GetIdxName()] = value; + } + catch (Exception ex) when (ex is FormatException or InvalidCastException or OverflowException) + { + _logger.LogDebug(ex, "Skipping custom field {FieldName}: type mismatch for index type {IndexType}", fieldName, definition.IndexType); + } + } + + if (ev.Idx?.Count == 0) + ev.Idx = null; + } + + private static void ClearCustomFieldSlots(PersistentEvent ev) + { + if (ev.Idx is null || ev.Idx.Count == 0) + return; + + // Only clear new-format managed slot keys (type-N, e.g. keyword-1, date-2). + // Legacy keys (e.g. sessionend-d from pre-PR data) are preserved so that ES queries + // that check both formats (GetOpenSessionsAsync) remain backward-compatible. + // Client-injected new-format slots are therefore stripped before server re-population. + foreach (var idxKey in ev.Idx.Keys.Where(IsManagedCustomFieldSlotKey).ToArray()) + ev.Idx.Remove(idxKey); + + if (ev.Idx.Count == 0) + ev.Idx = null; + } + + private static bool IsManagedCustomFieldSlotKey(string idxKey) + { + if (String.IsNullOrWhiteSpace(idxKey)) + return false; + + int separatorIndex = idxKey.LastIndexOf('-'); + if (separatorIndex <= 0 || separatorIndex == idxKey.Length - 1) + return false; + + return SupportedIndexTypes.Contains(idxKey[..separatorIndex]) + && Int32.TryParse(idxKey.AsSpan(separatorIndex + 1), out _); + } + + /// + /// Strictly converts a value to the target index type. Returns null if conversion + /// is not possible (value is skipped rather than failing event ingestion). + /// + public static object? ConvertValue(object? value, string indexType) + { + if (value is null) + return null; + + return indexType switch + { + "keyword" => ConvertToKeyword(value), + "string" => ConvertToString(value), + "bool" => ConvertToBool(value), + "int" => ConvertToInt(value), + "long" => ConvertToLong(value), + "float" => ConvertToFloat(value), + "double" => ConvertToDouble(value), + "date" => ConvertToDate(value), + _ => null + }; + } + + private static object? ConvertToKeyword(object value) + { + string? str = FormatInvariant(value); + if (str is null || str.Length > MaxKeywordLength) + return null; + return str; + } + + private static object? ConvertToString(object value) + { + string? str = FormatInvariant(value); + if (str is null || str.Length > 8192) + return null; + return str; + } + + /// + /// Formats a primitive value to a culture-invariant string suitable for keyword/string ES fields. + /// Using without a format provider would produce locale-dependent + /// output for float/double/decimal (e.g., "1,5" on German servers) and non-ISO DateTime strings. + /// + private static string? FormatInvariant(object value) + { + return value switch + { + string s => s, + bool b => b.ToString(), // "True"/"False" + int i => i.ToString(CultureInfo.InvariantCulture), + long l => l.ToString(CultureInfo.InvariantCulture), + float f => f.ToString(CultureInfo.InvariantCulture), + double d => d.ToString(CultureInfo.InvariantCulture), + decimal m => m.ToString(CultureInfo.InvariantCulture), + DateTime dt when dt.Kind != DateTimeKind.Unspecified => dt.ToUniversalTime().ToString("O", CultureInfo.InvariantCulture), + DateTime dt => DateTime.SpecifyKind(dt, DateTimeKind.Utc).ToString("O", CultureInfo.InvariantCulture), + DateTimeOffset dto => dto.UtcDateTime.ToString("O", CultureInfo.InvariantCulture), + _ => null + }; + } + + private static object? ConvertToBool(object value) + { + return value switch + { + bool b => b, + int i => i != 0, + long l => l != 0, + string s when s.Equals("true", StringComparison.OrdinalIgnoreCase) || s == "1" => (object)true, + string s when s.Equals("false", StringComparison.OrdinalIgnoreCase) || s == "0" => (object)false, + _ => null + }; + } + + private static object? ConvertToInt(object value) + { + return value switch + { + int i => i, + short s => (int)s, + byte b => (int)b, + sbyte sb => (int)sb, + long l when l is >= Int32.MinValue and <= Int32.MaxValue => (int)l, + double d when Double.IsFinite(d) && d is >= Int32.MinValue and <= Int32.MaxValue => (int)d, + float f when Single.IsFinite(f) && f is >= Int32.MinValue and <= Int32.MaxValue => (int)f, + decimal m when m is >= Int32.MinValue and <= Int32.MaxValue => (int)m, + string s when Int32.TryParse(s, NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsed) => parsed, + _ => null + }; + } + + private static object? ConvertToLong(object value) + { + return value switch + { + long l => l, + int i => (long)i, + short s => (long)s, + byte b => (long)b, + sbyte sb => (long)sb, + double d when Double.IsFinite(d) && d >= (double)Int64.MinValue && d < (double)Int64.MaxValue => (long)d, + float f when Single.IsFinite(f) && f >= (float)Int64.MinValue && f < (float)Int64.MaxValue => (long)f, + decimal m when m is >= Int64.MinValue and <= Int64.MaxValue => (long)m, + string s when Int64.TryParse(s, NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsed) => parsed, + _ => null + }; + } + + private static object? ConvertToFloat(object value) + { + return value switch + { + float f when Single.IsFinite(f) => f, + int i => (float)i, + // long range must be checked at runtime; Single.MinValue/MaxValue don't fit in long constants + long l => l >= -16777216L && l <= 16777216L ? (float)l : (object?)null, + double d when Double.IsFinite(d) && d is >= Single.MinValue and <= Single.MaxValue => (float)d, + decimal m => (float)m, + string s when Single.TryParse(s, NumberStyles.Float, CultureInfo.InvariantCulture, out var parsed) && Single.IsFinite(parsed) => parsed, + _ => null + }; + } + + private static object? ConvertToDouble(object value) + { + return value switch + { + double d when Double.IsFinite(d) => d, + float f when Single.IsFinite(f) => (double)f, + int i => (double)i, + long l => (double)l, + decimal m => (double)m, + string s when Double.TryParse(s, NumberStyles.Float, CultureInfo.InvariantCulture, out var parsed) && Double.IsFinite(parsed) => parsed, + _ => null + }; + } + + private static object? ConvertToDate(object value) + { + return value switch + { + DateTime dt when dt.Kind != DateTimeKind.Unspecified => dt.ToUniversalTime(), + DateTime dt => DateTime.SpecifyKind(dt, DateTimeKind.Utc), + DateTimeOffset dto => dto.UtcDateTime, + // AssumeUniversal treats strings without explicit timezone info as UTC, avoiding + // silent server-local-time interpretation. Strings with explicit offsets use those offsets. + string s when DateTimeOffset.TryParse(s, CultureInfo.InvariantCulture, + DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal, out var parsed) => parsed.UtcDateTime, + _ => null + }; + } + + /// + /// Validates that a custom field name meets requirements: + /// - Not empty, max 100 chars + /// - Any name starting with '@' is reserved + /// - Only ASCII letters, digits, underscore, dot, dash allowed (no Unicode) + /// + public static bool IsValidFieldName(string name) + { + if (String.IsNullOrWhiteSpace(name)) + return false; + + if (name.Length > 100) + return false; + + // Any @-prefixed name is reserved + if (name.StartsWith('@')) + return false; + + // Only ASCII alphanumeric, underscore, dot, and dash — no Unicode identifiers + return name.All(c => Char.IsAsciiLetterOrDigit(c) || c == '_' || c == '.' || c == '-'); + } + +} diff --git a/src/Exceptionless.Web/Api/Endpoints/OrganizationEndpoints.cs b/src/Exceptionless.Web/Api/Endpoints/OrganizationEndpoints.cs index b90082f8d6..f1663d7af9 100644 --- a/src/Exceptionless.Web/Api/Endpoints/OrganizationEndpoints.cs +++ b/src/Exceptionless.Web/Api/Endpoints/OrganizationEndpoints.cs @@ -87,6 +87,42 @@ public static IEndpointRouteBuilder MapOrganizationEndpoints(this IEndpointRoute } }); + group.MapGet("organizations/{id:objectid}/event-custom-fields", async (string id, HttpContext httpContext, IMediator mediator, IMediatorResultMapper resultMapper) + => (await mediator.InvokeAsync>>(new OrganizationMessages.GetEventCustomFields(id, httpContext))).ToHttpResult(resultMapper)) + .Produces>() + .ProducesProblem(StatusCodes.Status404NotFound) + .WithSummary("Get event custom fields"); + + group.MapPost("organizations/{id:objectid}/event-custom-fields", async (string id, HttpContext httpContext, IMediator mediator, IMediatorResultMapper resultMapper, [FromBody] NewCustomFieldDefinition field) + => (await mediator.InvokeAsync>(new OrganizationMessages.CreateEventCustomField(id, field, httpContext))).ToHttpResult(resultMapper)) + .Accepts("application/json", "application/*+json") + .Produces(StatusCodes.Status201Created) + .ProducesProblem(StatusCodes.Status400BadRequest) + .ProducesProblem(StatusCodes.Status404NotFound) + .ProducesProblem(StatusCodes.Status409Conflict) + .ProducesProblem(StatusCodes.Status422UnprocessableEntity) + .ProducesProblem(StatusCodes.Status426UpgradeRequired) + .WithSummary("Create event custom field"); + + group.MapPatch("organizations/{id:objectid}/event-custom-fields/{fieldId:objectid}", async (string id, string fieldId, HttpContext httpContext, IMediator mediator, IMediatorResultMapper resultMapper, [FromBody] UpdateCustomFieldDefinition field) + => (await mediator.InvokeAsync>(new OrganizationMessages.UpdateEventCustomField(id, fieldId, field, httpContext))).ToHttpResult(resultMapper)) + .Accepts("application/json", "application/*+json") + .Produces() + .ProducesProblem(StatusCodes.Status400BadRequest) + .ProducesProblem(StatusCodes.Status404NotFound) + .ProducesProblem(StatusCodes.Status422UnprocessableEntity) + .ProducesProblem(StatusCodes.Status426UpgradeRequired) + .WithSummary("Update event custom field"); + + group.MapDelete("organizations/{id:objectid}/event-custom-fields/{fieldId:objectid}", async (string id, string fieldId, HttpContext httpContext, IMediator mediator, IMediatorResultMapper resultMapper) + => (await mediator.InvokeAsync(new OrganizationMessages.DeleteEventCustomField(id, fieldId, httpContext))).ToHttpResult(resultMapper)) + .Produces(StatusCodes.Status202Accepted) + .ProducesProblem(StatusCodes.Status400BadRequest) + .ProducesProblem(StatusCodes.Status404NotFound) + .ProducesProblem(StatusCodes.Status409Conflict) + .ProducesProblem(StatusCodes.Status426UpgradeRequired) + .WithSummary("Delete event custom field"); + group.MapPatch("organizations/{id:objectid}", async (string id, HttpContext httpContext, IMediator mediator, IMediatorResultMapper resultMapper, [FromBody] Delta? changes) => { if (changes is null) diff --git a/src/Exceptionless.Web/Api/Handlers/AdminHandler.cs b/src/Exceptionless.Web/Api/Handlers/AdminHandler.cs index bb05f1f940..ef7b09641e 100644 --- a/src/Exceptionless.Web/Api/Handlers/AdminHandler.cs +++ b/src/Exceptionless.Web/Api/Handlers/AdminHandler.cs @@ -184,6 +184,9 @@ public async Task Handle(AdminRunMaintenance message) { switch (message.Name.ToLowerInvariant()) { + case "ensure-system-custom-fields": + await workItemQueue.EnqueueAsync(new OrganizationMaintenanceWorkItem { EnsureSystemCustomFields = true }); + break; case "fix-stack-stats": var effectiveUtcStart = message.UtcStart ?? timeProvider.GetUtcNow().UtcDateTime.AddDays(-90); diff --git a/src/Exceptionless.Web/Api/Handlers/EventHandler.cs b/src/Exceptionless.Web/Api/Handlers/EventHandler.cs index eb77c82c92..1956e4c237 100644 --- a/src/Exceptionless.Web/Api/Handlers/EventHandler.cs +++ b/src/Exceptionless.Web/Api/Handlers/EventHandler.cs @@ -697,6 +697,9 @@ private async Task> CountInternalAsync(AppFilter sf, TimeInf sf.UsesPremiumFeatures = pr.UsesPremiumFeatures || far.UsesPremiumFeatures; + if (sf.UsesPremiumFeatures && sf.Organizations.Count > 0 && sf.Organizations.All(organization => !organization.HasPremiumFeatures)) + return PlanLimitResult("Searching with custom fields requires a paid plan. Please upgrade to use this filter."); + if (mode == "stack_new") filter = AddFirstOccurrenceFilter(ti.Range, filter); @@ -755,6 +758,9 @@ private async Task>> GetInternalAsync(AppFilter sf, T sf.UsesPremiumFeatures = pr.UsesPremiumFeatures || usesPremiumFeatures; + if (sf.UsesPremiumFeatures && sf.Organizations.Count > 0 && sf.Organizations.All(organization => !organization.HasPremiumFeatures)) + return PlanLimitResult>("Searching with custom fields requires a paid plan. Please upgrade to use this filter."); + try { FindResults events; diff --git a/src/Exceptionless.Web/Api/Handlers/OrganizationHandler.cs b/src/Exceptionless.Web/Api/Handlers/OrganizationHandler.cs index 9c910c721c..377f67e5bc 100644 --- a/src/Exceptionless.Web/Api/Handlers/OrganizationHandler.cs +++ b/src/Exceptionless.Web/Api/Handlers/OrganizationHandler.cs @@ -1,3 +1,5 @@ +using System.ComponentModel.DataAnnotations; +using System.Text.RegularExpressions; using Exceptionless.Core; using Exceptionless.Core.Authorization; using Exceptionless.Core.Billing; @@ -6,6 +8,7 @@ using Exceptionless.Core.Messaging.Models; using Exceptionless.Core.Models; using Exceptionless.Core.Models.Billing; +using Exceptionless.Core.Models.WorkItems; using Exceptionless.Core.Repositories; using Exceptionless.Core.Repositories.Queries; using Exceptionless.Core.Services; @@ -17,7 +20,11 @@ using Exceptionless.Web.Models; using Foundatio.Caching; using Foundatio.Messaging; +using Foundatio.Jobs; +using Foundatio.Queues; using Foundatio.Repositories; +using Foundatio.Repositories.Elasticsearch.CustomFields; +using Foundatio.Repositories.Exceptions; using Foundatio.Repositories.Models; using Exceptionless.Web.Utility; using Stripe; @@ -30,6 +37,7 @@ namespace Exceptionless.Web.Api.Handlers; public class OrganizationHandler( OrganizationService organizationService, + EventCustomFieldService eventCustomFieldService, IOrganizationRepository repository, ICacheClient cacheClient, IEventRepository eventRepository, @@ -41,6 +49,9 @@ public class OrganizationHandler( IStripeBillingClient stripeBillingClient, IMailer mailer, IMessagePublisher messagePublisher, + ICustomFieldDefinitionRepository customFieldDefinitionRepository, + ISavedViewRepository savedViewRepository, + IQueue workItemQueue, ApiMapper mapper, AppOptions options, TimeProvider timeProvider, @@ -543,6 +554,144 @@ await Task.WhenAll( return new ChangePlanResult { Success = true }; } + public async Task>> Handle(GetEventCustomFields message) + { + var organization = await GetModelAsync(message.Id, useCache: false); + if (organization is null) + return Result.NotFound("Organization not found."); + + var results = await customFieldDefinitionRepository.FindByTenantAsync(nameof(PersistentEvent), message.Id); + var fields = new List(); + do + { + fields.AddRange(results.Documents + .Where(field => !EventCustomFieldService.IsSystemField(field.Name)) + .Select(CustomFieldDefinitionResponse.FromDefinition)); + } while (await results.NextPageAsync()); + + return fields; + } + + public async Task> Handle(CreateEventCustomField message) + { + var organization = await GetModelAsync(message.Id, useCache: false); + if (organization is null) + return Result.NotFound("Organization not found."); + + if (!organization.HasPremiumFeatures) + return Result.Invalid(ValidationError.Create(ApiValidationErrorIdentifiers.PlanLimit, "Custom fields require a paid plan. Please upgrade to add custom fields.")); + + if (EventCustomFieldService.IsSystemField(message.Field.Name)) + return Result.BadRequest($"'{message.Field.Name}' is a reserved system field and cannot be created manually."); + + CustomFieldDefinition? definition; + try + { + definition = await eventCustomFieldService.CreateFieldAsync( + message.Id, + message.Field.Name, + message.Field.IndexType.ToLowerInvariant(), + options.CustomFieldOptions.MaxFieldsPerOrganization, + message.Field.Description, + message.Field.DisplayOrder, + message.Context.RequestAborted); + } + catch (Exception ex) when (ex is ArgumentException or ValidationException or InvalidOperationException or DocumentValidationException) + { + return Result.Invalid(ValidationError.Create("general", ex.Message)); + } + + if (definition is null) + { + var existingPage = await customFieldDefinitionRepository.FindByTenantAsync(nameof(PersistentEvent), message.Id); + var allActive = new List(); + do + { + allActive.AddRange(existingPage.Documents); + } while (await existingPage.NextPageAsync()); + + if (allActive.Any(field => String.Equals(field.Name, message.Field.Name, StringComparison.OrdinalIgnoreCase))) + return Result.BadRequest($"A custom field named '{message.Field.Name}' already exists for this organization."); + + return Result.BadRequest($"Maximum of {options.CustomFieldOptions.MaxFieldsPerOrganization} custom fields per organization has been reached."); + } + + var response = CustomFieldDefinitionResponse.FromDefinition(definition); + return Result.Created(response, $"/api/v2/organizations/{message.Id}/event-custom-fields"); + } + + public async Task> Handle(UpdateEventCustomField message) + { + var organization = await GetModelAsync(message.Id, useCache: false); + if (organization is null) + return Result.NotFound("Organization not found."); + + if (!organization.HasPremiumFeatures) + return Result.Invalid(ValidationError.Create(ApiValidationErrorIdentifiers.PlanLimit, "Custom fields require a paid plan. Please upgrade to manage custom fields.")); + + var definition = await customFieldDefinitionRepository.GetByIdAsync(message.FieldId); + if (definition is null || definition.IsDeleted || definition.TenantKey != message.Id || definition.EntityType != nameof(PersistentEvent)) + return Result.NotFound("Custom field not found."); + + if (EventCustomFieldService.IsSystemField(definition.Name)) + return Result.BadRequest($"'{definition.Name}' is a reserved system field and cannot be modified."); + + bool changed = false; + if (message.Field.Description is not null) + { + definition.Description = message.Field.Description.Length == 0 ? null : message.Field.Description; + changed = true; + } + + if (message.Field.DisplayOrder.HasValue) + { + definition.DisplayOrder = message.Field.DisplayOrder.Value; + changed = true; + } + + if (changed) + await customFieldDefinitionRepository.SaveAsync(definition); + + return CustomFieldDefinitionResponse.FromDefinition(definition); + } + + public async Task Handle(DeleteEventCustomField message) + { + var organization = await GetModelAsync(message.Id, useCache: false); + if (organization is null) + return Result.NotFound("Organization not found."); + + if (!organization.HasPremiumFeatures) + return Result.Invalid(ValidationError.Create(ApiValidationErrorIdentifiers.PlanLimit, "Custom fields require a paid plan. Please upgrade to manage custom fields.")); + + var definition = await customFieldDefinitionRepository.GetByIdAsync(message.FieldId); + if (definition is null || definition.IsDeleted || definition.TenantKey != message.Id || definition.EntityType != nameof(PersistentEvent)) + return Result.NotFound("Custom field not found."); + + if (EventCustomFieldService.IsSystemField(definition.Name)) + return Result.BadRequest($"'{definition.Name}' is a reserved system field and cannot be deleted."); + + var fieldNamePattern = BuildFieldNameRegex(definition.Name); + var savedViews = await savedViewRepository.GetByOrganizationIdAsync(message.Id, o => o.SearchAfterPaging().PageLimit(1000)); + do + { + if (savedViews.Documents.Any(view => IsCustomFieldUsedInFilter(view.Filter, fieldNamePattern))) + return Result.Conflict($"Custom field '{definition.Name}' is used in one or more saved filters and cannot be deleted. Remove it from all filters first."); + } while (await savedViews.NextPageAsync()); + + definition.IsDeleted = true; + await customFieldDefinitionRepository.SaveAsync(definition); + + await workItemQueue.EnqueueAsync(new RemoveCustomFieldWorkItem + { + OrganizationId = message.Id, + CustomFieldDefinitionId = definition.Id, + FieldName = definition.Name + }); + + return Result.Accepted(); + } + public async Task> Handle(AddOrganizationUser message) { if (String.IsNullOrEmpty(message.Id) || !message.Context.Request.CanAccessOrganization(message.Id) || String.IsNullOrEmpty(message.Email)) @@ -764,6 +913,17 @@ private async Task AddModelAsync(Organization value, HttpContext h var organization = await repository.AddAsync(value, o => o.Cache()); + try + { + await eventCustomFieldService.EnsureSystemFieldsAsync(organization.Id); + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + // Custom fields are also provisioned lazily during event processing and before user-field creation. + // Do not leave a newly persisted organization inaccessible to its creator if this best-effort step fails. + _logger.LogError(ex, "Error provisioning system custom fields for new organization {OrganizationId}", organization.Id); + } + user.OrganizationIds.Add(organization.Id); await userRepository.SaveAsync(user, o => o.Cache()); await messagePublisher.PublishAsync(new UserMembershipChanged @@ -942,4 +1102,13 @@ private static Result PermissionToResult(PermissionResult permission) private static User GetCurrentUser(HttpContext httpContext) => httpContext.Request.GetUser(); private static bool IsStatsMode(string? mode) => !String.IsNullOrEmpty(mode) && String.Equals(mode, "stats", StringComparison.OrdinalIgnoreCase); private static bool messageIsGlobalAdmin(HttpContext httpContext) => httpContext.Request.IsGlobalAdmin(); + + private static Regex BuildFieldNameRegex(string fieldName) + { + var escapedName = Regex.Escape(fieldName); + return new Regex($@"(? !String.IsNullOrWhiteSpace(filter) && fieldNamePattern.IsMatch(filter); } diff --git a/src/Exceptionless.Web/Api/Handlers/SavedViewHandler.cs b/src/Exceptionless.Web/Api/Handlers/SavedViewHandler.cs index c698e33fac..3193ca421e 100644 --- a/src/Exceptionless.Web/Api/Handlers/SavedViewHandler.cs +++ b/src/Exceptionless.Web/Api/Handlers/SavedViewHandler.cs @@ -6,6 +6,7 @@ using Exceptionless.Core.Models; using Exceptionless.Core.Models.WorkItems; using Exceptionless.Core.Repositories; +using Exceptionless.Core.Queries.Validation; using Exceptionless.Core.Seed; using Exceptionless.Web.Api.Infrastructure; using Exceptionless.Web.Api.Messages; @@ -28,6 +29,7 @@ public partial class SavedViewHandler( IOrganizationRepository organizationRepository, ILockProvider lockProvider, IQueue workItemQueue, + PersistentEventQueryValidator eventQueryValidator, ApiMapper mapper, LinkGenerator linkGenerator, IHttpContextAccessor httpContextAccessor) @@ -225,6 +227,12 @@ public async Task> Handle(UpdateSavedViewMessage message) original.UpdatedByUserId = GetCurrentUserId(); + if (changedNames.Contains(nameof(UpdateSavedView.Filter))) + { + var validationResult = await eventQueryValidator.ValidateQueryAsync(original.Filter); + original.UsesPremiumFeatures = validationResult.UsesPremiumFeatures; + } + await repository.SaveAsync(original, o => o.Cache()); return MapToViewModel(original); } @@ -279,6 +287,9 @@ private async Task> PostImplAsync(NewSavedView value) mapped.CreatedByUserId = GetCurrentUserId(); mapped.Version = 1; + var validationResult = await eventQueryValidator.ValidateQueryAsync(mapped.Filter); + mapped.UsesPremiumFeatures = validationResult.UsesPremiumFeatures; + var model = await repository.AddAsync(mapped, o => o.Cache()); var viewModel = MapToViewModel(model); string location = linkGenerator.GetUriByName(HttpContext, "GetSavedViewById", new { id = model.Id }) diff --git a/src/Exceptionless.Web/Api/Handlers/StackHandler.cs b/src/Exceptionless.Web/Api/Handlers/StackHandler.cs index b61206d1d1..c66998702e 100644 --- a/src/Exceptionless.Web/Api/Handlers/StackHandler.cs +++ b/src/Exceptionless.Web/Api/Handlers/StackHandler.cs @@ -409,6 +409,9 @@ private async Task>> GetInternalAsync(AppFilter sf, T sf.UsesPremiumFeatures = pr.UsesPremiumFeatures; + if (sf.UsesPremiumFeatures && sf.Organizations.Count > 0 && sf.Organizations.All(organization => !organization.HasPremiumFeatures)) + return PlanLimitResult>("Searching with custom fields requires a paid plan. Please upgrade to use this filter."); + try { var results = await stackRepository.FindAsync(q => q.AppFilter(ShouldApplySystemFilter(sf, filter, httpContext.Request) ? sf : null).FilterExpression(filter).SortExpression(sort).DateRange(ti.Range.UtcStart, ti.Range.UtcEnd, ti.Field), o => o.PageNumber(page).PageLimit(limit)); diff --git a/src/Exceptionless.Web/Api/Messages/OrganizationMessages.cs b/src/Exceptionless.Web/Api/Messages/OrganizationMessages.cs index 672bd6eff9..6a9c06db2e 100644 --- a/src/Exceptionless.Web/Api/Messages/OrganizationMessages.cs +++ b/src/Exceptionless.Web/Api/Messages/OrganizationMessages.cs @@ -19,6 +19,10 @@ public record GetInvoice(string Id, HttpContext Context); public record GetInvoices(string Id, string? Before, string? After, int Limit, HttpContext Context); public record GetPlans(string Id, HttpContext Context); public record ChangeOrganizationPlan(string Id, ChangePlanRequest? Model, string? PlanId, string? StripeToken, string? Last4, string? CouponId, HttpContext Context); +public record GetEventCustomFields(string Id, HttpContext Context); +public record CreateEventCustomField(string Id, NewCustomFieldDefinition Field, HttpContext Context); +public record UpdateEventCustomField(string Id, string FieldId, UpdateCustomFieldDefinition Field, HttpContext Context); +public record DeleteEventCustomField(string Id, string FieldId, HttpContext Context); public record AddOrganizationUser(string Id, string Email, HttpContext Context); public record RemoveOrganizationUser(string Id, string Email, HttpContext Context); public record SuspendOrganization(string Id, SuspensionCode Code, string? Notes, HttpContext Context); diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/extended-data-item.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/extended-data-item.svelte index 1c30d0a441..b40743a06c 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/extended-data-item.svelte +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/extended-data-item.svelte @@ -3,6 +3,7 @@ import { Code, CodeBlock, H4 } from '$comp/typography'; import { Button } from '$comp/ui/button'; import * as DropdownMenu from '$comp/ui/dropdown-menu'; + import IndexAsCustomFieldAction from '$features/organizations/custom-fields/components/index-as-custom-field-action.svelte'; import { isJSONString, isObject, isString, isXmlString } from '$features/shared/typing'; import { UseClipboard } from '$lib/hooks/use-clipboard.svelte'; import ArrowDown from '@lucide/svelte/icons/arrow-down'; @@ -36,6 +37,21 @@ title }: Props = $props(); + function isPrimitiveData(value: unknown): boolean { + if (value === null || value === undefined) { + return false; + } + + const type = typeof value; + return type === 'string' || type === 'number' || type === 'boolean'; + } + + function isIndexableFieldName(name: string): boolean { + return !name.startsWith('@') && /^[a-zA-Z0-9_.-]+$/.test(name) && name.length <= 100; + } + + const canIndex = $derived(isPrimitiveData(data) && isIndexableFieldName(title)); + function transformData(data: unknown): unknown { if (isJSONString(data)) { try { @@ -153,6 +169,9 @@ {/if} {/if} + {#if canIndex} + + {/if} diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/organization-defaults-faceted-filter-builder.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/organization-defaults-faceted-filter-builder.svelte index c4da6c8e32..568858e0bd 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/organization-defaults-faceted-filter-builder.svelte +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/organization-defaults-faceted-filter-builder.svelte @@ -4,6 +4,8 @@ } const { includeDateFacets = true }: Props = $props(); + import CustomFieldFilterBuilders from '$features/organizations/custom-fields/components/custom-field-filter-builders.svelte'; + import * as FacetedFilter from './index'; type KnownTermsFilterConfig = { @@ -106,3 +108,5 @@ {#each eventsVersionFilters as { priority, term, title } (term)} {/each} + + diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/api.svelte.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/api.svelte.ts new file mode 100644 index 0000000000..848e019802 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/api.svelte.ts @@ -0,0 +1,150 @@ +import type { ProblemDetails } from '@exceptionless/fetchclient'; + +import { accessToken } from '$features/auth/index.svelte'; +import { useFetchClient } from '@exceptionless/fetchclient'; +import { createMutation, createQuery, useQueryClient } from '@tanstack/svelte-query'; + +import type { CustomFieldDefinition, NewCustomFieldDefinition, UpdateCustomFieldDefinition } from './models'; + +interface ApiCustomFieldDefinition { + created_utc: string; + description?: string; + display_order: number; + id: string; + index_type: string; + name: string; + updated_utc: string; +} + +interface ApiNewCustomFieldDefinition { + description?: string; + display_order?: number; + index_type: string; + name: string; +} + +interface ApiUpdateCustomFieldDefinition { + description?: string; + display_order?: number; +} + +export const queryKeys = { + customFields: (organizationId: string | undefined) => ['Organization', organizationId, 'custom-fields'] as const, + type: ['CustomField'] as const +}; + +export interface CreateCustomFieldRequest { + route: { + organizationId: string; + }; +} + +export interface DeleteCustomFieldRequest { + route: { + fieldId: string; + organizationId: string; + }; +} + +export interface GetCustomFieldsRequest { + route: { + organizationId: string | undefined; + }; +} + +export interface UpdateCustomFieldRequest { + route: { + fieldId: string; + organizationId: string; + }; +} + +export function createCustomFieldMutation(request: CreateCustomFieldRequest) { + const queryClient = useQueryClient(); + return createMutation(() => ({ + enabled: () => !!accessToken.current && !!request.route.organizationId, + mutationFn: async (data: NewCustomFieldDefinition) => { + const client = useFetchClient(); + const response = await client.postJSON( + `organizations/${request.route.organizationId}/event-custom-fields`, + mapNewFieldRequest(data) + ); + return mapApiDefinition(response.data!); + }, + onSuccess: () => { + queryClient.invalidateQueries({ queryKey: queryKeys.customFields(request.route.organizationId) }); + } + })); +} + +export function deleteCustomFieldMutation(request: DeleteCustomFieldRequest) { + const queryClient = useQueryClient(); + return createMutation(() => ({ + enabled: () => !!accessToken.current && !!request.route.organizationId && !!request.route.fieldId, + mutationFn: async () => { + const client = useFetchClient(); + await client.delete(`organizations/${request.route.organizationId}/event-custom-fields/${request.route.fieldId}`); + }, + onSuccess: () => { + queryClient.invalidateQueries({ queryKey: queryKeys.customFields(request.route.organizationId) }); + } + })); +} + +export function getCustomFieldsQuery(request: GetCustomFieldsRequest) { + return createQuery(() => ({ + enabled: () => !!accessToken.current && !!request.route.organizationId, + queryFn: async ({ signal }: { signal: AbortSignal }) => { + const client = useFetchClient(); + const response = await client.getJSON(`organizations/${request.route.organizationId}/event-custom-fields`, { signal }); + return response.data?.map(mapApiDefinition) ?? []; + }, + queryKey: queryKeys.customFields(request.route.organizationId) + })); +} + +export function updateCustomFieldMutation(request: UpdateCustomFieldRequest) { + const queryClient = useQueryClient(); + return createMutation(() => ({ + enabled: () => !!accessToken.current && !!request.route.organizationId && !!request.route.fieldId, + mutationFn: async (data: UpdateCustomFieldDefinition) => { + const client = useFetchClient(); + const response = await client.patchJSON( + `organizations/${request.route.organizationId}/event-custom-fields/${request.route.fieldId}`, + mapUpdateFieldRequest(data) + ); + return mapApiDefinition(response.data!); + }, + onSuccess: () => { + queryClient.invalidateQueries({ queryKey: queryKeys.customFields(request.route.organizationId) }); + } + })); +} + +function mapApiDefinition(definition: ApiCustomFieldDefinition): CustomFieldDefinition { + return { + createdUtc: definition.created_utc, + description: definition.description, + displayOrder: definition.display_order, + id: definition.id, + indexType: definition.index_type, + name: definition.name, + updatedUtc: definition.updated_utc + }; +} + +function mapNewFieldRequest(data: NewCustomFieldDefinition): ApiNewCustomFieldDefinition { + return { + description: data.description, + display_order: data.displayOrder, + index_type: data.indexType, + name: data.name + }; +} + +function mapUpdateFieldRequest(data: UpdateCustomFieldDefinition): ApiUpdateCustomFieldDefinition { + return { + description: data.description, + display_order: data.displayOrder + }; +} diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/components/custom-field-filter-builders.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/components/custom-field-filter-builders.svelte new file mode 100644 index 0000000000..691abe8495 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/components/custom-field-filter-builders.svelte @@ -0,0 +1,51 @@ + + +{#each activeFields as field (field.id)} + {@const filterType = getFilterType(field)} + {@const term = `idx.${field.name}`} + {@const title = field.description || field.name} + + {#if filterType === 'boolean'} + + {:else if filterType === 'number'} + + {:else if filterType === 'date'} + + {:else} + + {/if} +{/each} diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/components/index-as-custom-field-action.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/components/index-as-custom-field-action.svelte new file mode 100644 index 0000000000..6a95556537 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/components/index-as-custom-field-action.svelte @@ -0,0 +1,140 @@ + + +{#if organizationId && canCreate} + (showDialog = true)} title="Index this field for filtering"> + + Index as Custom Field + + + + + + Index "{fieldName}" as Custom Field + + This will start indexing "{fieldName}" from future events, making it available for filtering and search. Existing events will not be + retroactively indexed. + + + +
{ + e.preventDefault(); + handleCreate(); + }} + > + {#if error} + + {/if} + + + Index Type + (selectedType = (v ?? 'keyword') as IndexType)}> + + {INDEX_TYPE_LABELS[selectedType]} + {INDEX_TYPE_DESCRIPTIONS[selectedType]} + + + + {#each INDEX_TYPES as type (type)} + + {INDEX_TYPE_LABELS[type]} + {INDEX_TYPE_DESCRIPTIONS[type]} + + {/each} + + + + Choose the type that best matches this field's data. + + + + + + + +
+ +{/if} diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/index.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/index.ts new file mode 100644 index 0000000000..301556fc49 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/index.ts @@ -0,0 +1,3 @@ +export { createCustomFieldMutation, deleteCustomFieldMutation, getCustomFieldsQuery, queryKeys, updateCustomFieldMutation } from './api.svelte'; +export type { CustomFieldDefinition, IndexType, NewCustomFieldDefinition, UpdateCustomFieldDefinition } from './models'; +export { INDEX_TYPE_DESCRIPTIONS, INDEX_TYPE_LABELS, INDEX_TYPES } from './models'; diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/models.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/models.ts new file mode 100644 index 0000000000..8602560e06 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/models.ts @@ -0,0 +1,46 @@ +export interface CustomFieldDefinition { + createdUtc: string; + description?: string; + displayOrder: number; + id: string; + indexType: string; + name: string; + updatedUtc: string; +} + +export interface NewCustomFieldDefinition { + description?: string; + displayOrder?: number; + indexType: string; + name: string; +} + +export interface UpdateCustomFieldDefinition { + description?: string; + displayOrder?: number; +} + +export const INDEX_TYPES = ['keyword', 'string', 'int', 'long', 'float', 'double', 'bool', 'date'] as const; +export type IndexType = (typeof INDEX_TYPES)[number]; + +export const INDEX_TYPE_LABELS: Record = { + bool: 'Boolean', + date: 'Date/Time', + double: 'Double', + float: 'Float', + int: 'Integer', + keyword: 'Keyword', + long: 'Long', + string: 'Text' +}; + +export const INDEX_TYPE_DESCRIPTIONS: Record = { + bool: 'True/false values.', + date: 'ISO 8601 date or timestamp.', + double: '64-bit decimal. Higher precision.', + float: '32-bit decimal. Lower precision.', + int: '32-bit whole number (-2B to 2B).', + keyword: 'Exact-match string. Best for IDs, codes, tags.', + long: '64-bit whole number. For very large integers.', + string: 'Full-text search. Best for messages and descriptions.' +}; diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/saved-views/use-saved-views.test.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/saved-views/use-saved-views.test.ts index 0c1b301792..3eef01b8bf 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/saved-views/use-saved-views.test.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/saved-views/use-saved-views.test.ts @@ -55,6 +55,7 @@ function buildSavedView({ id, name, ...overrides }: Partial & Pick; +export const CustomFieldDefinitionResponseSchema = object({ + id: string().min(1, "Id is required"), + name: string().min(1, "Name is required"), + description: string().min(1, "Description is required").nullable().optional(), + index_type: string().min(1, "Index type is required"), + display_order: int32(), + created_utc: iso.datetime(), + updated_utc: iso.datetime(), +}); +export type CustomFieldDefinitionResponseFormData = Infer< + typeof CustomFieldDefinitionResponseSchema +>; + export const ExternalAuthInfoSchema = object({ clientId: string().min(1, "Client id is required"), code: string().min(1, "Code is required"), @@ -158,6 +171,24 @@ export const LoginSchema = object({ }); export type LoginFormData = Infer; +export const NewCustomFieldDefinitionSchema = object({ + name: string() + .min(1, "Name is required") + .max(100, "Name must be at most 100 characters"), + index_type: string() + .min(1, "Index type is required") + .max(20, "Index type must be at most 20 characters"), + description: string() + .min(1, "Description is required") + .max(500, "Description must be at most 500 characters") + .nullable() + .optional(), + display_order: int32().nullable().optional(), +}); +export type NewCustomFieldDefinitionFormData = Infer< + typeof NewCustomFieldDefinitionSchema +>; + export const NewOrganizationSchema = object({ name: string().min(1, "Name is required"), }); @@ -565,6 +596,18 @@ export const TokenResultSchema = object({ }); export type TokenResultFormData = Infer; +export const UpdateCustomFieldDefinitionSchema = object({ + description: string() + .min(1, "Description is required") + .max(500, "Description must be at most 500 characters") + .nullable() + .optional(), + display_order: int32().nullable().optional(), +}); +export type UpdateCustomFieldDefinitionFormData = Infer< + typeof UpdateCustomFieldDefinitionSchema +>; + export const UpdateEmailAddressResultSchema = object({ is_verified: boolean(), }); @@ -828,6 +871,7 @@ export const ViewSavedViewSchema = object({ time: string().min(1, "Time is required").nullable().optional(), sort: string().min(1, "Sort is required").nullable().optional(), version: int32(), + uses_premium_features: boolean(), view_type: string().min(1, "View type is required"), created_utc: iso.datetime(), updated_utc: iso.datetime(), diff --git a/src/Exceptionless.Web/ClientApp/src/routes/(app)/(components)/layouts/sidebar.svelte b/src/Exceptionless.Web/ClientApp/src/routes/(app)/(components)/layouts/sidebar.svelte index 2b09f91d2f..394053c230 100644 --- a/src/Exceptionless.Web/ClientApp/src/routes/(app)/(components)/layouts/sidebar.svelte +++ b/src/Exceptionless.Web/ClientApp/src/routes/(app)/(components)/layouts/sidebar.svelte @@ -11,6 +11,7 @@ import ChevronRight from '@lucide/svelte/icons/chevron-right'; import Settings from '@lucide/svelte/icons/settings-2'; import Wrench from '@lucide/svelte/icons/wrench'; + import Zap from '@lucide/svelte/icons/zap'; import { onDestroy } from 'svelte'; import type { NavigationItem } from '../../../routes.svelte'; @@ -258,6 +259,13 @@ {savedItem.title} + {#if savedItem.usesPremiumFeatures} + + {/if} {/each} @@ -308,6 +316,13 @@ {...subProps} > {savedItem.title} + {#if savedItem.usesPremiumFeatures} + + {/if} {/snippet} diff --git a/src/Exceptionless.Web/ClientApp/src/routes/(app)/+layout.svelte b/src/Exceptionless.Web/ClientApp/src/routes/(app)/+layout.svelte index 9deb9cb990..dd6eb02a48 100644 --- a/src/Exceptionless.Web/ClientApp/src/routes/(app)/+layout.svelte +++ b/src/Exceptionless.Web/ClientApp/src/routes/(app)/+layout.svelte @@ -414,7 +414,8 @@ const children = [ ...sortedViews.map((savedView) => ({ href: buildSavedViewHref(savedView), - title: savedView.name + title: savedView.name, + usesPremiumFeatures: savedView.uses_premium_features })), ...(route.children ?? []) ]; diff --git a/src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/custom-fields/+page.svelte b/src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/custom-fields/+page.svelte new file mode 100644 index 0000000000..ccece144c8 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/custom-fields/+page.svelte @@ -0,0 +1,436 @@ + + +
+
+
+

Custom Event Fields

+ Define indexed fields from event data for filtering and search. +
+ +
+ + {#if customFieldsQuery.isPending} +
+ + + +
+ {:else if customFieldsQuery.isError} + + {:else if !hasFields} +
+ +

No custom fields yet

+ Add a field to start indexing event data properties for use in filters and search. + +
+ {:else} +
+ + + + Name + Type + Description + Added + Actions + + + + {#each allFields as field (field.id)} + + {field.name} + + + {INDEX_TYPE_LABELS[field.indexType as IndexType] ?? field.indexType} + + + {field.description ?? '—'} + + + + +
+ + +
+ + + {/each} + + +
+ {/if} +
+ + + + + + Add Custom Field + + Define a new indexed field from event data. The field name must match a key in your event's extended data. + Indexing applies to new events only — historical events are not backfilled. + + + +
{ + e.preventDefault(); + handleCreate(); + }} + > + {#if createError} + + {/if} + + + Field Name * + + {#if newFieldName && !isValidFieldName} +

Only letters, digits, underscore, dot, and dash. Cannot start with @.

+ {:else} + Must match a key in event data. Letters, digits, _ . - only. + {/if} + + + + Index Type * + (newFieldIndexType = (v ?? 'keyword') as IndexType)}> + + {INDEX_TYPE_LABELS[newFieldIndexType]} + {INDEX_TYPE_DESCRIPTIONS[newFieldIndexType]} + + + + {#each INDEX_TYPES as type (type)} + + {INDEX_TYPE_LABELS[type]} + {INDEX_TYPE_DESCRIPTIONS[type]} + + {/each} + + + + Determines how the field is stored and what filter operators are available. + + + + Description (optional) +