Server Clock Skew: Why SSH Logins Fail and TOTP Codes Get Rejected

Yesterday the server accepted your SSH key without complaint; today the same connection is refused, and every six-digit two-factor code returns "verification failed." Nothing was changed. The cause in a surprising number of these cases is the least visible one: the server clock has drifted out of sync. Clock drift is a classic "silent failure" — it produces no error, fires no alert, and every process keeps running, while every time-dependent authentication mechanism fails at once. This article explains why authentication leans on time, how drift breaks TOTP and SSH certificates one by one, and how to diagnose and prevent it.

Why authentication depends on time

Many mechanisms that "just work" carry a hidden time dependency that stays invisible as long as the clock is right. List the dependency and it becomes clear why a wrong clock rejects you in different ways:

• TOTP codes are derived from a hash of "the current time divided by 30 seconds" — wrong time, wrong code.
• SSH certificates carry "valid from / to" fields; drift outside the window returns "not yet valid" or "expired."
• Kerberos tickets have lifetimes; a client–KDC clock gap beyond the default five minutes is refused.
• TLS certificates are judged against notBefore / notAfter.

Time isn't auxiliary here — it's an input to authentication. The two that bite most often are TOTP and SSH certificates.

TOTP: a few dozen seconds of drift is enough

TOTP (RFC 6238) verifies offline thanks to a shared secret, but the flip side of that design is second-level sensitivity to time. The code is computed as:

counter = floor(unix_time / 30)
digest  = HMAC_SHA1(secret, counter)
code    = dynamic_truncate(digest) % 1000000

The server takes the six digits you type, runs the same algorithm with its copy of the secret, and accepts if they match. Phone and server each compute independently with no communication, so if their clocks disagree by more than one 30-second window, the same secret yields different codes.

Implementations usually allow ±1 window of tolerance — roughly ±30 seconds, sometimes ±90. That sounds generous, but real-world drift sails past it easily: a server that never ran NTP drifts minutes in months on hardware clock alone; a VM that gets paused, rolled back from a snapshot, or live-migrated jumps instantly. The result is the classic symptom — the password is right, the key is right, but the code keeps failing, and the phone is visibly showing the "correct" code.

Also remember: drift is relative, and either end can be wrong. A phone that disables "automatic time" to save battery, a new phone that never finished its Authenticator time calibration, or an old device with an offline clock can all push the phone's codes out of sync with the server. So check both ends — timedatectl on the server, and whether "set time automatically" got turned off on the phone. Past one time window of gap, whoever is wrong, the result is the same: verification fails.

The first signal it's a TOTP clock problem: the same account fails on every host at once, and the error is "code invalid" rather than "password incorrect." Don't reset the secret, don't suspect the phone — check the server clock first.

SSH certificates: the validity window turns time into a hard gate

If the team uses CA-signed SSH certificates (ssh-keygen -s), the time dependency gets even stricter. Unlike a plain public key, a certificate has a validity window baked in at signing time:

Valid: from 2026-09-12T00:00:00 to 2026-09-13T00:00:00

During verification the server rejects the certificate whenever its local time falls outside that window, no matter how valid the content is. That produces two confusing errors: a slow server clock marks the cert "not yet valid," a fast one marks a freshly issued cert "expired." The ops instinct is "did we sign it wrong, do we re-sign?" — but the certificate is fine; the verifier's clock is wrong.

The same logic applies to host certificates. It gets worse when verification is centralized: with an SSH proxy server fronting many assets, one bad clock on the entry point rejects certificates across the fleet at once. The takeaway: when troubleshooting SSH auth failures, put "is the certificate in its validity window" and "is the server clock correct" on the same checklist. To see the window a certificate has signed in, ssh-keygen -L -f cert.pub prints a Valid: line with the exact range — compare it against date and you can tell at a glance whether the cert actually expired or the verifying host's clock is off. For the broader SSH authentication model, see /blog/ssh-authentication; for how the TOTP second factor layers on top, see /blog/ssh-mfa-two-factor.

Not just logins: wrong time breaks audit and troubleshooting too

The nastiest part of clock drift is that the damage goes far beyond "can't log in."

• Audit timestamps lose integrity. Compliance and internal review require records that answer "who did what and when." If the machine recording the operation has a wrong clock, the sequence scrambles — even absurdities like "the replay exists before the login." Bastion session auditing and replay (Audit Logs) depend heavily on an accurate timeline; a drifting clock erodes the credibility of the evidence.
• Cross-server logs stop aligning. Reconstructing an incident means lining up logs from several machines by time; one box drifting a few minutes breaks the causal chain.
• Event ordering becomes undecidable. In a security event, "which came first" is often the conclusion itself. Wrong timestamps make the conclusion indefensible.

Time sync, then, isn't cosmetic tidiness — it's a precondition for audit and forensics to be usable at all.

Why clocks drift in the first place

Know the causes, and the fix follows. In rough order of frequency:

1. NTP was never running. The server boots on the hardware clock (RTC) alone; crystal error adds seconds a day, minutes over months. This is the most common root cause.
2. VM pause / snapshot / migration. After a VM is suspended, rolled back, or live-migrated, the clock jumps instantly, and unstable vCPU scheduling keeps the drift accumulating.
3. chrony or ntpd runs on one box, or never enabled at boot. Many environments have an "NTP server" but miss the "every server syncs" step — or install chrony without enabling it, so it's bare after a reboot.
4. System suspend / sleep. On laptops or some bare-metal hosts, waking from sleep skips the wall clock forward by the sleep duration.

There's also a subtler source of offset — leap seconds. Earth's rotation and atomic time don't line up, so UTC periodically inserts a leap second to reconcile them. Most modern systems absorb leap seconds through a continuously running NTP service, but old ntpd versions caused well-documented mass outages over leap-second handling (2012 and 2017 both had public incidents). For production machines the takeaway is simple: use a continuously online chrony, not one-shot time setting.

A common confusion worth clearing up: timezone is not time. Setting the timezone wrong in timedatectl changes display only, not the actual instant. What breaks authentication is a wrong "absolute time," which has nothing to do with timezone. Plenty of people see a wrong-looking time and go straight for the TZ — the wrong direction.

Diagnosis: measure the offset first, then judge the impact

Diagnosis is two steps: confirm how far the clock is off, then judge whether that offset exceeds the authentication tolerance.

Step one, look at system state. timedatectl is the fastest single glance:

$ timedatectl
               Local time: Mon 2026-09-12 09:03:11 CST
           Universal time: Mon 2026-09-12 01:03:11 UTC
                 RTC time: Mon 2026-09-12 01:02:48 UTC
                Time zone: Asia/Shanghai (CST, +0800)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: no

System clock synchronized: no and NTP service: inactive settle it: nothing is syncing. On a chrony host, follow with chronyc tracking and read System time and Last offset:

$ chronyc tracking
Reference ID    : 0A1B2C3D (192.168.1.1)
Stratum         : 3
System time     : 0.000003421 seconds fast of NTP time
Last offset     : +0.000018901 seconds
RMS offset      : 0.000012345 seconds

If System time shows tens of seconds or minutes off while your TOTP tolerance is ±30 seconds, the conclusion writes itself: the clock is the failure.

Step two, compare against tolerance. Put symptoms against likely causes:

Symptom	Most likely cause	Check first
TOTP code rejected but password correct	Server clock past ±30s	chronyc tracking System time
SSH cert "not yet valid / expired" but cert is fine	Verifier clock slow / fast	timedatectl synchronized
Logs across hosts don't align by time	Inconsistent sync state	NTP status per host
Failures right after snapshot / migration	VM clock jump	Host–guest time sync config

Step three, confirm the fix actually took. Once the clock is corrected, System time in chronyc tracking converges toward zero, but logins may need another window to settle — codes rotate every 30 seconds, so near a boundary you get "right this second, wrong the next" jitter. And don't consider it done after a manual date -s: that only masks the problem; if NTP isn't running or enabled at boot, it drifts right back after a reboot. synchronized: yes with the offset converging to milliseconds is what "fixed" looks like.

Once the offset is confirmed, the fix is to bring the clock back to NTP time. One easily missed pitfall: stepping the clock and slewing it are different things. chrony steps large offsets by default and slews small ones smoothly; forcibly winding the clock backward can produce duplicate log timestamps and confuse processes that depend on a monotonic clock (timers, databases). So before correcting time, think about whether this host's workload tolerates a time jump — don't just date -s for speed.

Three things that keep a clock in line

Prevention beats repeated diagnosis. It comes down to three things:

1. Run chrony on every host and enable it at boot. Don't maintain one "NTP server" and stop there — synchronization has to take effect per host. On systemd, systemctl enable --now chronyd once; VMs should prefer syncing with the host.
2. Understand the slew/step boundary, and don't yank the clock. Let chrony handle offsets by its own policy, and evaluate business impact before any backward step.
3. Alert on offset. Set a monitoring threshold on NTP offset (say, above one second) — far earlier than discovering it via locked-out users.

Of the three, the first matters most: the vast majority of these auth failures trace back to "NTP not running or not enabled at boot." Check that before fiddling with any parameters.

A bastion host is the kind of "authentication entry point" that's especially sensitive to time: it verifies TOTP at login (OTP) and stamps session audit records, so a drifting clock locks users out and distorts the audit at the same time. Open-source Next Terminal follows exactly this pattern — keep the bastion node itself well synchronized and login verification plus operation audit (Audit Logs) stay dependable; the same method applies to any comparable product.

To close on something actionable: when you hit an auth failure where nothing was changed, credentials are right, and multiple systems fail together, check the clock first. If the error reads "verification failed / not yet valid / expired" and timedatectl shows synchronized: no or an offset beyond tolerance, the problem is almost certainly the clock. Setting the clock straight gets closer to the root cause than resetting secrets or re-signing certificates.