Account lockout stops counting instead of falling back to memory when the identity store cannot hold the counter

[vharseko]

Summary

Account lockout keeps its failure counter in the user entry when Store Invalid Attempts in Data Store (sunStoreInvalidAttemptsInDS, default true) is on. Where that write cannot be made, lockout does not degrade to the in-memory counters the same class already implements — it stops counting altogether, so no account is ever locked and the only sign is an error line per failed attempt.

Two distinct cases hit this:

1. an identity repository that refuses the write — a bind DN without write access, a directory without the sunAMAuthAccountLockout object class, or a read-only replica;
2. SpecialRepo, which backs amadmin, dsameuser and amService-UrlAccessAgent: SpecialRepo.setAttributes() refuses any token that is not amAdmin, and the authentication service writes with the administrative token, so the counter can never be persisted for these identities.

Where

openam-core/src/main/java/com/sun/identity/common/ISAccountLockout.java, invalidPasswd():

if (storeInvalidAttemptsInDS) {
    try {
        setLockoutObjectClass(amIdentity);
        amIdentity.setAttributes(attrMap);
        amIdentity.store();
    } catch (Exception e) {
        debug.error("ISAccountLockout.invalidPasswd", e);
        return -1;                 // <-- returns before the counters are updated
    }
}
acInfo.setLastFailTime(now);
acInfo.setFailCount(fail_count);
acInfo.setLockoutAt(lockedAt);

The return -1 happens before acInfo is updated, so the count does not advance even in memory. On the next attempt getAcInfo() reads the (absent) sunAMAuthInvalidAttemptsData attribute back and returns a fresh zeroed state, so the failure count is permanently 1.

The in-memory mode exists, but is only reachable through the configuration flag — getAcInfo() reads loginFailHash in its else branch, that is only when sunStoreInvalidAttemptsInDS is false.

Physical locking cannot work in such a store either: inactivateUserAccount() writes inetuserstatus=Inactive through the same identity, so with iplanet-am-auth-lockout-duration=0 the account is never inactivated.

Effect

• An installation configured with account lockout, against a store that cannot hold the counter, has no lockout at all. Nothing reports this except a repeating ISAccountLockout.invalidPasswd error in the Authentication debug log, which looks like noise rather than a disabled control.
• amadmin and the other special users are exempt from lockout — by accident, through the refused write, rather than by a documented rule. Setting sunStoreInvalidAttemptsInDS=false moves the counters into memory and makes them lockable, so the same deployment can be denied its administrator by five bad passwords.

Expected

When the store cannot hold the counter, fall back to the in-memory counters rather than stop counting:

• mark the repository (or realm) as unable to persist the counter on the first failed write, and use loginFailHash for both the read and the write from then on — patching only the write path is not enough, since getAcInfo() would keep reading a zeroed state out of the directory;
• log that once, at error level, instead of once per failed attempt, and say plainly that the failure count is now per server and lost on restart;
• force memory locking while in that mode regardless of the configured lockout duration, because physical locking needs the same write;
• decide explicitly what happens to the special users, rather than leaving it to the refused write. If they are meant to be exempt from lockout, exempt them by name — otherwise adding the fallback above turns amadmin into a target that anyone who knows the name can lock out.

Steps to reproduce

1. Point a realm at an identity store whose bind DN cannot modify user entries, or one without the sunAMAuthAccountLockout object class.
2. In Realms > Realm Name > Authentication > Settings > Account Lockout, enable lockout, leave Login Failure Lockout Count at 5 and Store Invalid Attempts in Data Store at its default.
3. Authenticate as a user of that realm with a wrong password more than five times.

The account is not locked, and Authentication debug shows one ISAccountLockout.invalidPasswd error per attempt. The same happens for amadmin against any store, since its entry lives in SpecialRepo.