Alerts: eliminate the false positives, and stay silent on dispatch - #26
Open
OpenSource-For-Freedom wants to merge 1 commit into
Open
Alerts: eliminate the false positives, and stay silent on dispatch#26OpenSource-For-Freedom wants to merge 1 commit into
OpenSource-For-Freedom wants to merge 1 commit into
Conversation
The first version of the droplet monitor would have made its own channel
worthless. Three collectors fired during normal operation:
every sudo command ordinary admin work buried the signal
RestartCount > 0 that counter is CUMULATIVE, so after any deploy it
alerted on EVERY run forever until the container was
recreated. A permanent alert is the same as no alert.
whitelist edits adding a player is routine, not an incident
What is watched now is deliberately narrow: rare things a person would
want to know about within minutes.
SSH logins every one, and an UNKNOWN username escalates to
high. Only two accounts should ever log in here.
Failed auth burst 10+ since the last check. Port 22 is closed to all
but DigitalOcean console ranges, so genuine
failures are ~0 and a burst means something changed.
Sensitive sudo only useradd/usermod/passwd/visudo/sudoers/
authorized_keys/sshd_config/iptables/ufw/
systemctl enable-disable-mask. Measured on the live
box: 1 sudo command in 24h, of which 0 sensitive.
fail2ban bans
Container not running, or restarted since the last check, compared by
StartedAt rather than the cumulative counter, so
exactly one alert per real restart
ops.json changed granting operator is privilege escalation in-game
sudoers, sshd_config, authorized_keys changed
New user accounts
Disk at 90%
Deliberately not watched: routine sudo, whitelist add/remove, cumulative
restart counts.
The health workflow also stops posting a status card on manual dispatch.
A dispatch is for checking the server, not for producing a message; tick
always_post when deliberately confirming the wiring.
The script committed here was read back OFF the running droplet rather
than from a local copy, so the repo and the live box are identical by
construction rather than by assumption. Verified before committing: two
consecutive runs both reported nothing, which is correct on a quiet box
and the exact case the old version got wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebuilt the event catalogue around zero false positives, and stopped manual dispatches posting status cards.
The three collectors that would have ruined the channel
RestartCount > 0What is watched now
StartedAt, so one alert per real restartops.jsonchangedDeliberately not watched: routine sudo, whitelist add/remove, cumulative restart counts.
Dispatch no longer posts a status card
always_postdefaults to false. A dispatch is for checking the server, not for generating a message.Verified on the live droplet
The script in this PR was read back off the running droplet, not from a local copy, so the repo and the live box are identical by construction rather than by assumption.