Skip to content

Retries and fail-safes ​

Atlas retries throttling, server errors, and dropped connections inside a run, saves nothing that points at data it has not stored, and reports anything it could not finish through the exit code. A scheduler needs nothing more than this:

bash
status=0
atlas onedrive backup -o user@example.com || status=$?
case "$status" in
  0) ;;                                   # complete
  2) echo "Partial run: the next run retries the failed items" >&2 ;;
  3|12) echo "Transient: schedule another attempt later" >&2 ;;
  *) echo "Needs an operator: read stderr before rerunning" >&2 ;;
esac
exit "$status"

The exit code table says which codes a rerun can fix.

At a glance ​

FailureWhat Atlas doesSection
Microsoft Graph 429, 500, 502, 503, 504, network errorRetries up to 12 times with backoff, honouring Retry-AfterMicrosoft Graph requests
A large file transfer stalls or its URL expiresRetries the range, refreshes the URL, resumes at the failed rangeFile transfers
S3 request failsThe AWS SDK retries. Writes that must not collide are conditionalStorage
You press Ctrl+CCommits what finished, exits 2Ctrl+C and signals
Process killed mid-runStorage only references what was fully written. The next run resumes from the last saved cursorCrash safety and resume
Two runs on one ownerThe second one exits 12 and writes nothingConcurrency guards
Some items fail, others succeedThe run keeps going, records each failure, and exits 2Partial runs
Stored data altered, or wrong passphraseDecryption fails and nothing is restored from itIntegrity

What Atlas retries on its own ​

Microsoft Graph requests ​

Every Microsoft Graph call goes through one retry loop. The Graph SDK's own retry handler is disabled, so retries are never multiplied by a second layer.

ParameterValue
Retried HTTP statuses429, 500, 502, 503, 504
Retried transport errorsNode and undici codes such as ECONNRESET, ETIMEDOUT, ENOTFOUND, EAI_AGAIN, UND_ERR_SOCKET, and aborted or timed-out transfers
Maximum retries12, so at most 13 attempts per call
Backoff without Retry-Aftermin(1 s × 2^attempt + random(0 to 1 s), 5 min)
Retry-AfterHonoured up to 1 hour, plus a release delay of up to a tenth of the wait, kept between 1 and 5 s. Both RFC 9110 forms are read: seconds or an HTTP date
Retry-After missing, unreadable, or in the pastExponential backoff as above
Retry-After below 1 sRaised to 1 s
Retry-After above 1 hourThe call fails at once with ThrottledError naming the value
429 without Retry-After30 s cooldown for the pool
Waiting budget per call24 minutes of backoff, Retry-After, and cooldown waits. A wait already started runs in full
Per-attempt timeout60 s by default. A timed-out request is aborted before the next attempt is sent

500 and 502 are retried because Microsoft Graph returns them for load and gateway faults that clear on their own. 501 and every other 4xx are not retried: the same request returns the same answer. Atlas decides from the HTTP status and structured error codes only, never from message text, which carries request IDs and item IDs whose digits could pass for a status.

The 24-minute budget sits just above the full exponential schedule (1 + 2 + 4 + ... capped at 5 minutes comes to about 23.5 minutes), so backoff alone always gets all 12 retries. Time spent inside a request does not count, so a slow download keeps its retries. One call therefore waits at most about 24 minutes plus one cooldown.

Cooldowns shared across requests ​

A 429 starts a cooldown for every request to the same budget, not only a wait for the one call. Outlook and Teams cooldowns cover one mailbox or team, because Microsoft Graph meters them that way. SharePoint and OneDrive share one tenant-wide cooldown. Overlapping cooldowns do not add up: the longest one wins. When a cooldown ends, waiting requests resume one by one over the release delay instead of in a single burst.

A host coordinator passed through the SDK fails open: if its shared store is unreachable, requests proceed on their own backoff and a warning is logged, so a coordinator outage never stops a backup. Throttle handling covers the budgets and the coordinator.

Pacing ​

Atlas paces requests below Microsoft's published limits rather than waiting for the first 429:

WorkloadPacing
OutlookPer mailbox: 9,600 requests per sliding 10-minute window (96% of Microsoft Graph's 10,000) and 4 concurrent requests
Teams80% of the Microsoft Graph Teams limits, per tenant, per team, and per channel

Concurrency is fixed, not adaptive. A 429 does not lower it; the shared cooldown is what slows the instance down. atlas teams backup --concurrency sets how many channels run at once (default 4, maximum 16).

When retries run out ​

What ran outInside a folder, drive or library, file download, or Teams channelAnywhere else, such as resolving the mailbox or site before any item is read
A 429 (12 retries or the 24-minute budget)ThrottledError (ATLAS_THROTTLED) with the last Retry-After attached. That unit fails and the run continuesThe command ends with exit 3
A 500, 502, 503, 504, or network errorGraphUnavailableError (ATLAS_GRAPH_UNAVAILABLE). That unit fails, the run continues with the rest, and the command exits 2The command ends with exit 3

An exhausted Outlook folder is marked as failed and keeps its previous sync checkpoint, so the next run reads those changes again. See Microsoft Graph unavailable after retries for the message and what to do.

Writes that are sent once ​

A restore request that creates something (a message, attachment, event, contact, folder, or a drive file under --conflict rename or --conflict fail) is retried only on a 429, or a 503 carrying Retry-After. Microsoft Graph sends those instead of processing the request.

A 500, 502, 504, timeout, or dropped connection after a create may arrive after Microsoft 365 committed it, so repeating the request could leave a duplicate. The item is reported as failed and the restore continues. See Restore writes are sent once for what to check before restoring it again.

Once a restore write is on the wire, Ctrl+C lets it finish, up to its timeout, instead of aborting it. An aborted write might exist in the target while Atlas reports it as not written.

File transfers (OneDrive and SharePoint) ​

File content comes from pre-authenticated download URLs rather than through the Graph client, so it has its own retries. Transfers share the sharepoint_onedrive cooldown: each waits for it before it starts, and a 429, or a 503 with Retry-After, raises it.

ParameterValue
File-level attempts3, with backoff from 2 s capped at 30 s
Ranged downloadFiles over 4 MiB are read in 4 MiB ranges
Range timeoutmax(30 s, size in bytes / 256), about 33 s for a 4 MiB range
Range retriesUp to 5 per range, backoff from 1 s plus up to 1 s of jitter, capped at 30 s. Retry-After honoured on 429
Streaming threshold64 MiB and above: streamed to storage through a multipart upload
Expired URL (401 or 403 from the CDN)Looked up again. Files of 64 MiB and more resume at the failed range, until 3 refreshed URLs in a row deliver nothing
Truncation checkEach range's length and Content-Range, and the total size, must match what Microsoft Graph listed

A file that still fails is added to the drive's list of failed files, and the drive's sync checkpoint moves on. Each later run fetches the listed files again before processing new changes, up to 5 attempts per file. Failures that are not the file's fault, such as an expired URL, do not use an attempt. A missing Microsoft Graph permission (403 accessDenied) stops the drive instead of costing one file per item. Snapshot health describes the list and the 403 cases.

A file version Microsoft Graph reports as gone (404 or 410, purged by retention) is counted as unavailable, not as an error. Only the numeric status decides this.

During a restore, a file over 4 MiB is uploaded through an upload session in 10 MiB chunks. Each chunk is sent up to 3 times in all: a 429, 500, 502, 503, or 504 on either of the first two attempts sends it again after Retry-After or 1 s. A failed session is not resumed: Atlas deletes it, giving the delete 10 s, and the next restore starts a new one. The final chunk is the request that creates the file, so a checksum mismatch found while streaming abandons the session before anything is created.

Storage ​

Atlas uses the AWS SDK for JavaScript v3 with its default retry strategy: standard mode, 3 attempts in total, with the SDK's own backoff for throttling (SlowDown) and 5xx responses. Atlas sets no request timeout of its own on S3. Storage request events reports the time each request spent in SDK retries.

SafeguardBehaviour
Content-MD5Sent on every PutObject, so the backend rejects a body damaged in transit
Create-only writesManifests, Teams indexes, version indexes, the data key, and new leases are written with If-None-Match: *. An identical object already there is treated as this run's own retried write. A different one fails the run with ATLAS_STORAGE_FAILURE and is never overwritten
Compare-and-swapLease renewal and takeover, the identity registry, and key re-wraps use If-Match on the ETag Atlas read
Read-backManifests, indexes, and delta cursors are read back after upload and compared byte for byte
Identity registry contentionUp to 8 attempts, waiting 50 ms × attempt + random(0 to 200 ms) between them

Conditional writes must be enforced

The leases, the data key bootstrap, and the identity registry are only safe on a backend that enforces these conditions. atlas storage-check writes a probe object and reports a backend that ignores them in its Conditional writes row.

See Create-only snapshot objects and the read-back and Metadata is checked before and after it is stored.

Abandoned multipart uploads ​

Large files are streamed to a staging object through a multipart upload, then promoted. An upload abandoned by a crash keeps its parts, and the bucket keeps billing for them. Three mechanisms collect them:

  1. Each OneDrive and SharePoint backup removes staging objects older than 24 hours, and aborts uploads that started more than 24 hours ago and have had no part written since. The second condition keeps a sweep from aborting a slow upload another run is still feeding.
  2. Buckets Atlas creates get a lifecycle rule that aborts incomplete multipart uploads after 7 days and expires noncurrent versions after 30 days. The rule is best-effort: a backend that refuses it logs a warning.
  3. Deleting an owner's backup aborts all of that owner's uploads at once.

Object Lock retention is set per object at write time, and a delete it refuses exits 8. See Immutability and Object Lock.

What happens when a run stops ​

Ctrl+C and signals ​

SignalResult
First Ctrl+C (SIGINT)Soft stop. Microsoft Graph requests, retries, and cooldown waits end at once; restore writes already sent finish. Completed work is committed, the lease is released, and the command exits 2
Second Ctrl+CExits at once with 130. Nothing more is committed and the lease is left to expire, so a run on the same owner within 5 minutes exits 12
SIGTERMAtlas installs no handler, so Node ends the process at once, as with a second Ctrl+C. The shell reports 143

The soft stop covers the long-running commands, such as backup, restore, restore-version, and save. A command without it ends at once on the first Ctrl+C. A cooldown raised during the run is dropped on Ctrl+C, so an hour-long Retry-After does not hold the process open. Stopping a run describes what a soft-stopped backup saves.

Soft stop under systemd

systemctl stop sends SIGTERM by default. Set KillSignal=SIGINT in the service unit so a stopped backup commits what it finished and releases its lease, rather than leaving the next run to wait out the 5-minute lease.

Crash safety and resume ​

Every workload writes a run in the same order: content first, then the snapshot manifest or index, then the delta cursor. A run that stops anywhere leaves storage pointing only at content that exists.

WorkloadCommit order
OutlookContent, manifest, cursor
OneDrive, SharePointContent, manifest, version index, cursor
Contacts, CalendarContent, manifest, cursor. An interrupted run writes neither
TeamsContent, team index, channel cursors

What an interrupted run leaves behind:

  • Content uploaded before the stop. Each unique item is stored once, so the next run finds it already stored and skips the upload.
  • No cursor past the last manifest. The next run starts from the previous cursor and reads the same changes again. A crash between manifest and cursor costs a second snapshot of the same interval and the Microsoft 365 reads, nothing else.
  • Staging objects and multipart parts for large files, collected as described in Abandoned multipart uploads.
  • The owner lease, if the process was killed. It expires 5 minutes after its last renewal.

Commit order for OneDrive and SharePoint explains why the cursor must come last.

SituationWhat the next run does
An Outlook folder fails, or a message in it failsThe folder keeps its previous sync checkpoint (delta link), so the run reads those changes again
A drive or library failsThat drive keeps its previous checkpoint. Drives that finished advance
Microsoft Graph rejects the checkpoint (syncStateNotFound, resyncRequired, syncStateInvalid)Reads the folder, drive, or calendar in full
Stale-delta safeguard (Outlook)A saved checkpoint returns nothing while the previous manifest stored nothing: the checkpoint is discarded and the folder read in full
The cursor's snapshot was deleted or quarantinedThe checkpoints would skip what that snapshot held, so they are ignored and the owner is read in full
A stored cursor is damagedThe run stops before writing. A damaged cursor is never treated as absent

Atlas has no deletion threshold. A large deletion in Microsoft 365 is recorded as deletions in the next snapshot, and earlier snapshots keep every item. Delta sync covers each case.

Partial runs ​

A failure that belongs to one item is recorded against that item and the run continues. The command then exits 2 and prints each failure.

Scope of the failureRecorded as
One Outlook message (MIME or attachments)A message error. The folder keeps its entries but not its new checkpoint
One Outlook folderA folder error. The other folders run
One OneDrive or SharePoint fileA failed file, retried by the next run up to 5 times
One drive or library (for example its delta query)That drive's error. Other drives run
One Teams channelA channel error. Other channels run. With --all, one team's failure does not stop the others
One calendar or contact folderThat folder's error

Some failures are not the item's fault. Atlas stops the larger unit instead of failing every remaining item the same way:

FailureWhat stops
AuthError (ATLAS_AUTH_DENIED) or MailboxNotLicensedError in Calendar or ContactsThe command, with exit 4
AuthError in a Teams channelSibling channels are cancelled and the command exits 4. With --all, the remaining teams are not started
AuthError or a missing permission inside a OneDrive drive or SharePoint libraryThat drive or library, recorded as its error. Other drives still run and the command exits 2
AuthError while capturing an Outlook messageThat folder, recorded as a folder error. The next folder runs
AuthError before any item is read, such as resolving the ownerThe command, with exit 4
Wrong passphrase, unreadable cursor, storage failure while committingThe command, before anything is written past the last commit

Concurrency guards ​

Atlas runs one operation at a time per mailbox, drive owner, site, or team. It enforces this with a lease object in storage.

GuardTriggerResult
Owner leaseA backup, restore, delete, or quarantine starts while another holds the same mailbox, drive owner, site, or teamATLAS_OWNER_BUSY, exit 12, nothing written
Tenant leaseatlas delete --purge or a keys command runs while any owner lease is live, or the reverseExit 12, or busy, skipped and exit 2 for keys
Lost leaseThis run could not renew its lease, or another host took it overStops before its next commit, exit 12
Snapshot not newestdelete -s names a snapshot newer ones are built onATLAS_SNAPSHOT_NOT_NEWEST, exit 11
Refused metadata writeA manifest, index, or cursor Atlas built fails its own reader's checks or does not decrypt backATLAS_WRITE_REFUSED, exit 1, nothing written
Newer formatA stored object was written by a newer AtlasATLAS_FORMAT_UNSUPPORTED, exit 9
Lease parameterValue
Lifetime5 minutes after the last renewal
RenewalEvery 60 s, by compare-and-swap on the lease's ETag
Commit marginA commit is refused once 30 s or less of the lease remain
Acquire attempts3

A storage error during renewal is logged and retried on the next tick. Only a lease that is gone, replaced, or expired counts as lost. A restore whose storage credentials cannot write the lease (403) runs without one and warns that concurrent-run protection is off. Another operation holds the owner lists what to do for each case.

Keep host clocks in sync

Hosts that share a bucket must keep their clocks within a few minutes of each other. A host running ahead sees a live lease as expired and takes it over.

Integrity ​

CheckWhat it catches
AES-256-GCM tagAny change to a stored object. One flipped bit fails decryption
Associated dataThe header and the object's directory are authenticated, so a ciphertext copied to another owner's path does not decrypt
SHA-256 checksumChecked against the manifest before a restored item is written to Microsoft 365
Wrapped data keyA wrong passphrase fails to unwrap the key before any content is read: ATLAS_WRONG_PASSPHRASE, exit 5
verifyDownloads, decrypts, and checks the checksum of every object in a snapshot. Outlook, Contacts, and Calendar also offer --fast, which only checks that objects exist

A restore skips an item whose object is missing or fails its tag, names it, restores the rest, and exits 2. A content object is only read after the snapshot's manifests decrypted under the same key, so a single failing tag means damage to that object, not a wrong key. See Integrity validation for what the tag does and does not prove.

Key and configuration safety ​

  • atlas keys rewrap replaces _meta/dek.enc by compare-and-swap, reads the new wrapper back, and proves it opens the same data key. It writes the new passphrase to the configuration only when every tenant succeeded; otherwise it exits 2 and leaves the configuration unchanged. The previous wrapper stays as a noncurrent version on a versioned bucket until atlas keys purge-old-wrappers removes it. See Rewrapping the passphrase.
  • The data key is created with a create-only write. Two runs bootstrapping one tenant agree on one key, and Atlas never generates a key over an existing one.
  • The encrypted config store (~/.atlas/config.enc) is written to a temporary file with mode 0600, flushed to disk, and renamed into place, so a crash leaves the previous file intact. Updates take ~/.atlas/config.lock, and a second update waits up to 10 s for it. A stale lock is removed only when its process no longer exists on the same host.

Keep a copy of the passphrase elsewhere

Atlas keeps no copy of the config store or its key. Keep a copy of the passphrase and the S3 credentials somewhere other than the backup host. Without the passphrase the backups cannot be read.

Restore safety ​

WorkloadDefault
OutlookRestores into a new Restore-<timestamp> folder, or the folder named by --destination. There is no in-place restore, and a well-known folder such as Inbox is refused as a destination
OneDrive, SharePointRestores under /Restore-<timestamp> with --conflict rename, so no existing file is overwritten. --in-place and --conflict replace must be asked for
Contacts, CalendarItems are tagged with their source ID, and a rerun skips those already restored. Nothing is overwritten
Cross-tenant--target-profile requires an explicit target, checked before any Microsoft Graph or storage call

Atlas has no dry-run mode. To check a restore first, restore into the default Restore-<timestamp> location, or extract the content with save, which writes to a local file and never touches Microsoft 365.

What to check first ​

When a scheduled run did not exit 0:

  1. Read the exit code against the table below. It tells you whether a rerun can fix it.
  2. Read stderr. A classified failure prints an Atlas code line, such as ATLAS_OWNER_BUSY, which Troubleshooting indexes.
  3. For exit 3, check the Microsoft 365 service health page, and schedule the rerun for later rather than immediately.
  4. For exit 2 after a restore, check the target for the reported items before restoring them again, because their create may have committed.
  5. For exit 12, wait for the other operation, or for the lease to expire, before rerunning.

Exit codes and whether to retry ​

ExitMeaningRetry unchanged?
0Complete
1Unclassified failure, storage failure, or ATLAS_WRITE_REFUSEDOnce, after reading stderr. A persistent 1 needs a person
2Partial run, soft interrupt, or items skippedBackup: yes, the next backup retries failed items. Restore: first check the target for items whose create may have committed, see Restore writes are sent once
3Throttled, network failure, or Microsoft Graph 5xx after retriesYes, later
4Authentication, permission, or licenceNo. Fix consent or licensing
5Wrong passphraseNo. Supply the original passphrase
6Invalid configurationNo
7Not foundNo
8Object Lock retention or legal holdNo, not before the retention ends
9Unreadable or newer-format stored contentNo. Quarantine, save, or upgrade Atlas
11Snapshot is not the newestNo. Delete newer snapshots first
12Owner or tenant busy, or lease lostYes, after the other operation ends or within 5 minutes
130Second Ctrl+CYes

StorageError (ATLAS_STORAGE_FAILURE) covers many causes, so its cause decides the exit: 4 for a 401 or 403, 7 for a 404, 3 for a network error or a transient status, and 1 otherwise. The full contract, including how scripts should branch, is in CLI exit codes.

See also ​