feat(scheduler): surface failed and at-risk scheduled tasks - #202
Open
renada-jacob wants to merge 7 commits into
Open
feat(scheduler): surface failed and at-risk scheduled tasks#202renada-jacob wants to merge 7 commits into
renada-jacob wants to merge 7 commits into
Conversation
A scheduled task whose command threw was written to storage as Completed. $State is set to 'Failed' in the executor's inner catch, but it was only ever read for orchestrator-based commands, so every other failing task fell through to the success path and the run was logged as "Successfully executed task". Partial failures were worse. A scheduled user creation whose licence assignment failed never threw at all, because Set-CIPPUserLicense returns the Graph error as a string rather than raising it. The user was created unlicensed and the task reported success, so nothing surfaced it. - Honour $State when writing terminal task state. Recurring tasks still reschedule, via 'Failed - Planned'. - Collect Error and Critical log entries against the running task using the existing scheduled-task AsyncLocal context, and record them as HasErrors and ErrorSummary. This catches steps that fail without throwing, which is the licence case. - Flag multi-tenant tasks with partial failures. They previously reported Failed only when every tenant failed, so 3 failures out of 50 looked like a clean run. - Add Start-CIPPTaskPreflightCheck, a timer that flags planned tasks whose licence is no longer available before their scheduled time arrives. - Add State, HasErrors, AtRisk and NeedsAttention filters to ListScheduledItems, and Failed Queue and Pending with Issues tabs to the scheduler page. - Add Scheduler_UserTasks to the notification log types so task failures can raise an alert instead of waiting to be noticed. HasErrors, ErrorSummary, AtRisk and AtRiskReason are new properties and default to false/empty on existing rows, so the endpoint's output is unchanged for callers that do not ask for them.
The preflight check marks a Planned task AtRisk when the licence it needs is no longer available. That flag is a prediction about a run that has not happened yet - but nothing cleared it afterwards, and the preflight only re-examines tasks still in the Planned state. So a flagged task that was run anyway (RunNow, or on schedule) kept AtRisk forever and sat in the Pending with Issues view as a Completed task. Every write that takes a task out of the Planned state now clears AtRisk and AtRiskReason: the three early failure paths (missing command, module not allow-listed, blocked command), the executor's outer catch, the failed orchestrator dispatch, and both terminal writes. Two paths deliberately keep the flag: the DeltaQuery early-return puts the task back to Planned without running it, so the prediction still stands, and the multi-tenant post-execution aggregate is untouched because AllTenants and tenant-group tasks are never preflighted in the first place.
…heduler-failure-visibility
New-CIPPUserTask reports a failed creation by throwing a hashtable that
carries its partial results (throw @{'Results' = ...}). A thrown
hashtable's Exception.Message is the literal type name, so the stored
task result read "Task Failed: System.Collections.Hashtable" instead of
the actual error - surfaced by the failed-task views, which make the
Results column far more visible than it used to be.
Unwrap the thrown object's Results when it is a dictionary, and fall back
to Exception.Message for ordinary throws.
…le at-risk reasons Closes the operational gaps in the failed and at-risk task views. Acknowledge: a task that completed with errors could only leave the needs-attention view by being deleted, even after the underlying problem was fixed out of band. The new Acknowledge Errors action records who acknowledged and when, and the needs-attention filter skips acknowledged rows. HasErrors and ErrorSummary stay on the row on purpose - the task did fail and the history should say so - and every run that records a failure resets the acknowledgement, so a recurring problem resurfaces. The acknowledged filter is applied in memory: an OData clause like "Acknowledged ne true" drops rows that lack the property entirely, which is every task written before this existed. Re-check Now: the preflight runs six-hourly, so after remediating a licence shortage the at-risk view could lag by hours. The new endpoint runs the check on demand at CIPP.Scheduler.ReadWrite - ExecCippFunction can already do this but requires SuperAdmin - and runs it synchronously so the caller's refresh shows the updated flags. At-risk notifications get their own log type (Scheduler_Preflight, "Tasks pending with issues") instead of sharing Scheduler_UserTasks: at-risk is a prediction about a task that has not run, and admins may want those separately from actual failures. At-risk reasons now resolve SKUs through Convert-SKUname, so the view says "no licences available for Microsoft 365 E5 Developer" rather than DEVELOPERPACK_E5, falling back to the part number for unmapped SKUs. The at-risk mechanism itself is check-agnostic: licence availability is the first preflight check, and further predicates (username already taken, target mailbox or group deleted, tenant unreachable) can append to the same problems list without touching the views or notifications.
…heduler-failure-visibility # Conflicts: # backend/Config/CIPPTimers.json
renada-jacob
marked this pull request as ready for review
August 6, 2026 14:54
Adds pages for the Failed Queue and Pending with Issues tabs, covering what lands in each view, the acknowledge workflow, the six-hourly availability check and the Re-check Now button. Updates the scheduler overview for the new tabs, columns and action, and lists the two new notification log types on the notifications page.
Contributor
Author
|
Docs added in new commit: new pages for the Failed Queue and Pending with Issues tabs, covering what lands in each view, the acknowledge workflow, and the six-hourly availability check with the Re-check Now button. The scheduler overview picks up the new tabs, columns and action, and the notifications page lists the two new log types. Both pages are registered in SUMMARY.md |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I've run into a problem many times over where for example: if you schedule a new user creation using available licences based on CIPP's license count check, and by the time the scheduled task runs the licence has been used or otherwise is no longer available, the user is created unlicensed the task shows Completed (successfully), but nothing tells you that creating that user without a license comes with its own problems.. the user wasnt added to the mail-enabled groups you wanted because it doesnt have a mailbox, and oh yeah the user doesnt have a mailbox - you wanted that right?... and the worst bit is nothing anywhere tells you.
Set-CIPPUserLicensereturns the Graph error as a string rather than throwing, so the failure only exists as text inside Results on a row that might not get looked at. Outright failures aren't much better. The executor sets$State = 'Failed'when a command throws, but only ever read it for orchestrator-based commands, so every other failing task was written as Completed and logged as "Successfully executed task".This PR aims to fix this problem and does the following:
Failed tasks now report as failed.
$Stateis honoured in the terminal writes, with recurring tasks going to Failed - Planned so they still reschedule. Errors logged during a run are collected against the task through the same AsyncLocal context that already stamps ScheduledTaskId onto log rows, so a task that finishes while a step inside it failed getsHasErrorsand anErrorSummarynaming what went wrong. That's what catches the licence case. Multi-tenant tasks flag partial failures too; previously 3 failed tenants out of 50 aggregated to a clean Completed.Planned tasks that would fail get flagged before they run. A six-hourly preflight check walks planned licence-dependent tasks and compares what they need against what the tenant has left. Tasks that would fail today are marked at risk with a readable reason, "no licences available for Microsoft 365 E5 Developer" rather than a bare SKU id (resolved through
Convert-SKUname), and the flag clears on its own when the licence comes back, when the task is edited, or when the task runs. Sherweb-backed creations are skipped since they buy the licence at run time. Licence availability is the first check, but the mechanism doesn't care what the predicate is: a username already taken or a deleted target mailbox can append to the same problems list later without touching anything downstream. The timer runs at :07 past its hours so its writes never race the orchestrator's ETag claim on the */15 boundary.Two new scheduler tabs, using the same route-based tab pattern as the applications and alert-configuration groups. Failed Queue is everything needing attention, including completed-with-errors. Pending with Issues is the flagged planned tasks, with a Re-check Now button for after you've fixed the underlying problem. The endpoint runs the check synchronously so the refreshed table shows the updated flags, and it's exposed at Scheduler.ReadWrite because the only existing way to run it on demand is
ExecCippFunction, which needs SuperAdmin.Acknowledge Errors on failed rows, for failures remediated out of band (previously the only way out of the view was deleting the task). It records who and when, and the row stops demanding attention but keeps
HasErrorsand the summary: the task did fail, and the history should say so. Any later run that fails clears the acknowledgement, so a recurring problem resurfaces rather than staying dismissed.Followed by the confirmation modal after selection:
Both sides can notify: two new log types in notification settings, "Scheduled task failures" and "Tasks pending with issues", routing through the existing pipeline to email, webhook and PSA. They're separate on purpose. One is a thing that happened, the other is a forecast, and you may well want one without the other.
Nothing existing changes shape. The new columns are additive and default to false/empty on existing rows, the new ListScheduledItems parameters are all optional, and the parameterless response is unchanged. Filter input goes through
ConvertTo-CIPPODataFilterValue. The acknowledged exclusion is applied in memory rather than in the OData filter on purpose:Acknowledged ne truedrops rows that don't carry the property at all, which is every task written before this, so it would have silently emptied the view.Also unwraps hashtable throws in the executor's catch.
New-CIPPUserTaskreports a failed creation withthrow @{'Results' = ...}, and a thrown hashtable's Exception.Message is the literal type name, so the stored result read "Task Failed: System.Collections.Hashtable" instead of what actually went wrong.Worth flagging for review: the multi-tenant aggregation decides success by string-matching results for "Failed" and "Error". That's pre-existing, but
HasErrorsnow makes its false positives more visible, so a success message containing the word "Error" will flag the task. Fixing it properly means changing how per-tenant results are shaped, which felt like its own PR.Tested by scheduling real tasks through the API and letting the unmodified timer cadence run them over two days against a sandbox tenant:
PS - this is rather large commit of mine I have spent a decent amount of time working on, but with that being said, I would appreciate insight that anyone has to things that should be adjusted/updated/removed etc.