Mounting btrfs with an atime, bad idea?

[TheEvilSkeleton]

There was this post from 2012 I read a couple of weeks ago called Atime and btrfs: a bad combination?, stating btrfs' issues with mounting with something like relatime.

It was an interesting read, but I'd like to ask if the issues still hold true to this day, and if mounting btrfs with an atime is a bad idea or not.

I found two relevant sections in the btrfs wiki so far:

1. https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs(5)#NOTES_ON_GENERIC_MOUNT_OPTIONS
2. https://btrfs.wiki.kernel.org/index.php/FAQ#Why_I_experience_poor_performance_during_file_access_on_filesystem.3F

[Zygo]

I would not say the combination of snapshots and atime is "bad", since the combination is required by some use cases and the costs are reasonable on some combinations of workload and storage hardware; however, If the user is not committed to spending a comparatively large amount of time, space, and IO on both snapshots and atime updates, and managing the extra space requirements for the combination of the two, then they should use noatime or avoid using snapshots. Because the costs are relatively high for atime and the benefits relatively low (and the users who require those benefits relatively rare), I'd recommend making noatime the default unless the user has provided some hint otherwise (e.g. by installing a package known to depend strongly on atime).

The issues are pretty fundamental to both snapshots and atime. XFS and LVM snapshots with other filesystems have similar problems. In all cases, running out of space during CoW is bad enough to at least halt all further writes. btrfs can throw errors on individual write operations before space is completely exhausted, while LVM requires active monitoring of snapshot LV usage to avoid failures that can only be detected after write commands are issued. btrfs might be forced into a read-only state that can only be recovered by umount and mount, while LVM throws IO errors on writes to the LV after free space is exhausted. LVM must either reject all writes to original and snapshot LV (forcing both to be umounted), or allow writes to the origin LV to continue while the snapshot LV becomes unrecoverably broken. I haven't tested XFS's out-of-space behavior, but it looks very similar to the LVM case structurally, so I would not expect substantively different results. We can say "btrfs is quantitatively better than the XFS/LVM+other filesystem experience" here, but most users don't have the worse experience to compare btrfs to, and may not be expecting to have to cope with CoW space exhaustion issues at all.

Snapshots on btrfs are lazy reflink copies, which have different cost tradeoffs than block-level snapshots. Snapshots on btrfs are extremely cheap if you never modify the shared pages before deleting the snapshot, but they can be extremely expensive if you modify all of the shared pages (most use cases fall somewhere in between). A full reflink copy (e.g. cp -a --reflink=always) must create parent references for every individual extent item referenced in the new tree, so a reflink copy of a subvol will create slightly more new metadata pages in total than were present in the original subvol's metadata tree. A snapshot has the same worst-case IO and storage cost as a full reflink copy, but those costs are spread out over time, and hidden in other write operations that modify metadata pages. atime updates are one of those other write operations.

A snapshot can be fully converted to a reflink copy when as little as 1% of its content is modified, so a recursive grep or even a find with any form of atime update enabled can force such a conversion. IO latency and metadata space costs must be paid at the time the copied pages are made, which will be during or shortly after the read access operation (there is also a second extra cost to delete the copied metadata pages when the snapshot is deleted). btrfs's cost model is different from XFS and LVM here: on LVM, the original subvol and snapshot are structured such that the snapshot can be trivially deleted at any time without (non-constant) cost, while the original subvol is very expensive to remove (other than the special case where both original and snapshot volumes are deleted). On btrfs there is no such distinction between original and snapshot subvol, so the deletion cost of either subvol is proportional to how much reflink conversion has occurred. On XFS the deletion costs are equivalent to btrfs, but the scalars for XFS are two orders of magnitude smaller than btrfs due to the much larger size of btrfs metadata. An XFS snapshot is not a lazy reflink copy (it is literally a reflink copy, all costs paid at snapshot creation time), so atime updates on XFS don't incur additional costs after the snapshot was created as they do on btrfs.

It may be much less convenient for the user to have IO latency and metadata space usage elevated during read-intensive times (e.g. while the user is working with the machine interactively) compared to the snapshot creation time (i.e. during overnight maintenance windows or at non-peak service times). It may also be inconvenient for the user to increase the time required to free space by deleting snapshots (making it take longer to recover their space when needed).

relatime may make this effect worse for some users, by batching up a lot of atime updates at close to the same time each day (e.g. when the user logs in at the start of the work day, triggering the once-every-24-hours update behavior of relatime). On the other hand, if the user is making more than one snapshot per day, then relatime will limit the reflink conversion effect to only one of the snapshots each day.

The impact of all this is relative. Metadata updates on random pages are extremely expensive on big, slow spinners, and not at all expensive on small, fast NVME devices. All writers of the filesystem will be blocked while the metadata updates are committed in a transaction, and the memory pressure from blocked writers can block readers and even processes that aren't touching the filesystem (i.e. those allocating memory or referencing lazily mapped memory). This can result in the entire system freezing up for some time--from a fraction of a second on fresh installs to NVME devices, to several minutes on larger and older filesystems on slower spinning disks. This is true of any large metadata update--atime is just one special and easily avoidable case. A similar effect can occur when running git clean on a snapshotted source tree, or chmod -R on a big tree, but these are not as surprising when triggered by write operations as they are when triggered by read operations.

"ENOSPC on read" mentioned in the LWN article is a real outcome but not entirely accurate. Reads don't fail due to lack of space, but the atime updates that succeed will consume metadata space if there is a snapshot, and in the extreme case, metadata growth will fill up the filesystem so no further allocations are possible. Once that point is reached, the only possible writes are those that reduce metadata usage or increase available space: deleting or truncating unshared files, removing entire snapshots, adding disks, or resizing devices larger. Every other write operation decreases available metadata space, and is not possible when no remaining space is available for metadata. Normally this is not a problem, because btrfs allocates space for metadata generously. Usually, as long as metadata is never balanced, there will be enough space for common metadata expansion cases; however, there are still exceptional cases, especially on filesystems where raid profile conversions, device resizes, or metadata balances have deviated from the usual space allocation patterns.

Some of the bad effects can be reduced by creative scheduling: e.g. make a snapshot for backups at 3 AM, run plocate at 4 AM, and by 8 AM all of the atime-related snapshot conversion costs are fully paid, so the machine is ready to be used without unexpected IO multiplications. plocate must never scan the snapshots or the number of snapshots must be limited; otherwise, plocate will take more than 4 hours to run. If metadata grows slowly and continuously over time, it is less likely to be a problem than rapid ad-hoc growth.

The opposite case (i.e. worst-case creative scheduling) would be e.g. running find / for the first time on a large, full filesystem with a snapshot, when no process has ever walked the filesystem tree before, while atime or relatime is enabled. This could force the filesystem into no free metadata space with a sudden new demand for metadata space that had never been needed before.

[polarathene]

With the strictatime / relatime concerns mentioned in this issue, does this likely make the lazytime option a worse choice? ArchWiki has some good info about it (depending on the trade-offs):

lazytime reduces writes to disk by maintaining changes to inode timestamps (access, modification and creation times) only in memory. The on-disk timestamps are updated only when either:

1. The file inode needs to be updated for some change unrelated to file timestamps
2. A sync to disk occurs
3. An undeleted inode is evicted from memory
4. If more than 24 hours passed since the the last time the in-memory copy was written to disk.

Warning: In the event of a system crash, the access and modification times on disk might be out of date by up to 24 hours.
Note: The lazytime option works in combination with the aforementioned *atime options, not as an alternative. That is relatime by default, but can be even strictatime with the same or less cost of disk writes as the plain relatime option.

That might not suffer the same bulk relatime update (potentially updating atime on first access and then possibly again 24 hours later if any subsequent accesses), depending on context, though the bulk probably would still be applied at that 24 hour window or during a sync that applies to all affected files (eg shutting down?).

As it also affects mtime and ctime, would that further strain the I/O concerns, or would their inclusion be minimal?

As long as no new snapshots are created in-between, it'd seem that depending on activity it might be more staggered of an operation in practice?

[Zygo]

Some other filesystems separate inodes from file contents by some physical distance, so lazytime can prevent an expensive head movement or a write to a separate flash zone between data block flushes. Other filesystems that write data in-place usually don't have requirements to update inode metadata other than timestamps when only file contents are changed.

On btrfs, file contents are described in reference items that are interleaved with the inodes in metadata pages. When file contents change in a normal (datacow) file, the data is stored in a new location and the file's metadata is updated to point to that location. In that process, the inode is usually written out to disk because it is located in the same metadata block, so there is very little additional cost to update the timestamps on disk and therefore very little saving by not updating the timestamps. When a snapshot is present, every file is datacow because all blocks are shared between the original subvol and snapshot (even nodatacow files). Thus, lazytime does not save very much work compared to relatime (though it will save a little because sometimes the inode is in a different metadata page from the data references, e.g. if an inode happens to be one of the ~1% of all filesystem items that occupies the last slot on a metadata page, then all of the inode's data reference items are necessarily on a separate page where lazytime could provide some benefit. Also an unshared nodatacow file has no need to update anything but the data blocks so lazytime can help those as well).

lazytime reduces write loads from all timestamp updates compared to cases where it is not used, though it may not reduce very much in common cases on btrfs. It wasn't part of the original discussion. The original discussion contains concerns about the adverse effects of dropping atime writes or having any non-default mount option. There is far more existing software that is adversely affected by dropping mtime and ctime writes than atime, and given the stated preference for default options, lazytime would be a step in the wrong direction for an installer default.

[polarathene]

There is far more existing software that is adversely affected by dropping mtime and ctime writes than atime

lazytime isn't dropping mtime and ctime like noatime does for atime...?

It's paired with strictatime or relatime (if neither specified paired with the default), the changes to atime, mtime, ctime are more accurate but remain in memory instead of flushed to disk (until one of the conditions are met that would do so).

I don't follow with how that negatively impacts software, other than the warning scenario of power loss or crash failing to persist any in-memory updates.

The original discussion contains concerns about the adverse effects of dropping atime writes or having any non-default mount option.

While lazytime does affect mtime and ctime, it still impacts atime updates differently than strictatime or relatime options without lazytime? Seems relevant to the discussion. I only bring up mtime and ctime additions as lazytime additionally affects those too, and I wasn't sure if lazytime would be appropriate for BTRFS vs just relatime by itself, if choosing not to use noatime.

Since you seem to have good insights into the impact, this seemed like a relevant related question that someone coming across this issue may also have thought about.

---

On btrfs, file contents are described in reference items that are interleaved with the inodes in metadata pages. When file contents change in a normal (datacow) file, the data is stored in a new location and the file's metadata is updated to point to that location. In that process, the inode is usually written out to disk because it is located in the same metadata block, so there is very little additional cost to update the timestamps on disk and therefore very little saving by not updating the timestamps.

Is this referring to a file write operation and thus stating that there is minimal benefit to lazytime because mtime/ctime would be updated anyway?

I was referring to reads with atime. If the IO operation is going to trigger one of the sync timestamps to disk conditions, I'm assuming that with write operations it's less of a concern. But this issue was about read operations causing writes, and the impact of that risking notable disk usage increase as shared extents from a snapshot become unshared due to writes for new atime right?

Thus, lazytime does not save very much work compared to relatime

My understanding of the difference was in timing of when atime updates were persisted to disk.

relatime can update atime on first file access straight away to disk, and then delay further atime updates until 24 hours pass?

lazytime (when paired with relatime) would not update to disk like relatime could do (if the condition is met), but would instead keep that change in-memory until one of the lazytime conditions is met. That reduces/delays updates to disk until relevant, but keeps the atime change available in memory for other software to be aware of, as if it was the atime for that file on disk no?

Depending on system usage, some files atime may be updated on disk within that 24 hour window earlier, unlike relatime AFAIK? Or have I misunderstood, and even in read-only situations which this issue discusses, lazytime is not providing much benefit over relatime either? (while the mtime and ctime updates also being handled differently isn't of any concern with lazytime and BTRFS?)

---

lazytime reduces write loads from all timestamp updates compared to cases where it is not used, though it may not reduce very much in common cases on btrfs.

That's good to know, I'd just like clarification about the perspective on reads too vs relatime without lazytime.

Presumably there is also benefit for those caring about atime to pair strictatime with lazytime, which would otherwise from my understanding be less desirable than relatime on BTRFS?

[Zygo]

OK, now that I've read the actual code, I see that lazytime is not the feature I was thinking of (the "turn mtime completely off" feature for Ceph and similar upper storage layers that have no need for mtime/ctime at all).

lazytime changes the timestamp update function so it sets the I_DIRTY_TIME bit but not the I_DIRTY_SYNC bit on timestamp updates. Inodes are written to disk if I_DIRTY_SYNC is set but not I_DIRTY_TIME. The last iput (i.e. when the file is closed by the last process to have it open) checks for I_DIRTY_TIME and flushes the inode to disk if it is set when the inode is closed (and not deleted). sync and fsync turn I_DIRTY_TIME into I_DIRTY_SYNC. umount is only possible after every inode has been released by iput. That covers the 4 cases in the descriptions of the feature's behavior.

So lazytime only reduces the number of writes for files that are held open for a long time (e.g. database/VM image files, or binaries in /usr if they fault in new pages while running). Otherwise, it only delays the inode write until the file is closed. There is apparently also a sysctl which can be used to force inode updates to happen more often when lazytime is enabled.

In the find and grep cases, most of the opened files are immediately closed, so the effect of lazytime is negligible on inode updates--these tools will follow whatever policy was set by strictatime, relatime, nodiratime, or noatime. The total number of writes will only be reduced if the grep is still reading a file while the writeback timer expires--lazytime would drop an inode update in that case, since it won't mark the inode dirty before the file is closed.

lazytime will adversely affect ctime/mtime timestamp accuracy on disk, because it separates timestamp updates from other updates at the VFS level, above the filesystem. So lazytime can lead to a situation where the inode's data contents are flushed and persisted on disk by btrfs, but the VFS layer didn't trigger an inode update at the same time, so the timestamp in the inode on disk is much older than the data. For ext4 this is excusable, since ext4 declines to make file content consistency guarantees after a crash; however, btrfs datacow files are expected to be consistent after a crash (the file contents and metadata are to be consistent as of the latest transaction commit or fsync, all modifications after that point have no effect on the file metadata or contents after the crash), so this could adversely affect workloads on btrfs that expect the post-crash consistency.

[polarathene]

In the find and grep cases, most of the opened files are immediately closed, so the effect of lazytime is negligible on inode updates--these tools will follow whatever policy was set by strictatime, relatime, nodiratime, or noatime.

Ah.. so I've misunderstood (and perhaps many other less experienced users in this area from what I can see online), the atime update in the cited cases where you might do a large read-only operation that causes a bulk metadata update and breaks sharing extents.. is going to update to disk rather quickly?

I was under the impression the update would be delayed and remain in the file cache (page-cache?) in RAM (provided there is sufficient memory spare), but that's not what happening 😞

Your description doesn't make it sound useful to a desktop or workstation user then? It doesn't really act as an in-memory cache for a meaningful amount of time for the majority of files where atime updates is the concern..

btrfs datacow files are expected to be consistent after a crash (the file contents and metadata are to be consistent as of the latest transaction commit or fsync, all modifications after that point have no effect on the file metadata or contents after the crash), so this could adversely affect workloads on btrfs that expect the post-crash consistency.

Making lazytime ill-advised for BTRFS?

So lazytime only reduces the number of writes for files that are held open for a long time (e.g. database/VM image files, or binaries in /usr if they fault in new pages while running).

Database and VM would often be using nodatacow, but as you mentioned can still benefit from snapshots. I'm not really following how it helps reduce the atime updates to disk here compared to just relatime (unless it does delay that potential update on first read?), it's not due to mtime is it? (I thought I had read that doesn't update until a file is closed as well)

---

There was a discussion with the Fedora Working Group regarding defaulting to noatime for BTRFS where Lennart of systemd chimed in about atime being relevant for systemd-tmpfiles to function reliably/optimally, oddly he was in encouraging strictatime over relatime for that specific usecase.

While those involved were mostly in favor initially of noatime they decided by the end to retain the relatime default. They do note that they don't presently make much use of snapshots with their default BTRFS install, and some don't seem to think relatime pragmatically causes much concern.

They do care about responsiveness though, and may not have the same insights that @Zygo has expressed here with I/O pressure? 😅

---

So to sum up the discussion..

noatime is likely preferable unless you know you need atime for something despite the trade-off.

noatime avoids running into some surprises if a large amount of files are accessed and the bulk of them update atime:

• Reading from a BTRFS filesystem as source, and copying to another filesystem such as external disk can negatively impact I/O performance? (reference: BTRFS requires noatime (2017))
• Performing a read-only operation such as a file transfer or searching through many files (eg grep or find on / or some other location with N files) can break the sharing of a snapshots extents for metadata due to CoW, likely increasing disk usage unintentionally from the users perspective.
  -The disk usage issue compounds with frequency of snapshots assuming similar read heavy operations occur between them. This can muddy the atime updates in the snapshots with data that may be valued in the snapshots, requiring extra effort to remedy if wanting to retain snapshots while freeing up space by removing the atime metadata updates? (which for most software where atime actually matters, the history in snapshots isn't likely to be relevant to retain)

If choosing to ignore the concerns that noatime prevents because atime is important to the user (systemd-tmpfiles may the most relevant beyond more niche software for a desktop/workstation user), they can use:

• relatime: Ideal for minimizing atime updates which is useful if taking regular snapshots (daily, hourly?), as to avoid the compounding effect between snapshots. While the number of atime writes is minimized from subsequent reads (eg recurrent reads for: a nodeJS project with a large node_modules directory, a Docker image build with many layers, apps/plugins that scan through project files, etc?_), a delayed batch write to disk may be less convenient UX wise depending on the timing.
• strictatime avoids the delayed writes, but all reads incur writes regardless. If snapshots are regularly occurring the negative impact on disk size usage will be more pronounced, and each snapshot incurs the CoW cost on metadata for atime (unlike relatime AFAIK?)
• lazytime paired with either isn't likely to be that beneficial in practice and it's trade-off when a power loss or crash event occurs adds more risk on data with BTRFS than it does other filesystems?

TL;DR

noatime is likely preferable unless you know you need atime for something despite the trade-off.

• noatime avoids unexpected disk usage allocations when paired with snapshots from read operations, as well as potentially poor I/O performance due to each file read incurring a write.
• If you need atime updates:
  • relatime for systems with frequent snapshots.
  • strictatime with minimal to no snapshot usage has less I/O pressure than relatime.
  • lazytime should be avoided, it risks data consistency in power loss or crash event.

[Zygo]

Ah.. so I've misunderstood (and perhaps many other less experienced users in this area from what I can see online), the atime update in the cited cases where you might do a large read-only operation that causes a bulk metadata update and breaks sharing extents.. is going to update to disk rather quickly?

strictatime marks the inode dirty during each read() call. lazytime marks the inode specially, so the inode only becomes dirty when it is closed after a read (the inode may become dirty for other reasons--if that happens, the atime is flushed with the rest of the inode). relatime marks the inode dirty during a read() call, but only if this would change the order of atime vs mtime/ctime, or if the old atime is 24+ hours old. noatime doesn't change anything about an inode during a read().

Once an inode is marked dirty, the VFS layer may direct the filesystem to flush the inode to disk at any time. This doesn't necessarily happen immediately, the inode just becomes eligible to be flushed. A dirty inode can't be simply discarded when no process has the inode's file open and memory runs low, so when memory runs low, memory allocations must be delayed until flushes are completed. Normal writeback is based on a timer, but umounting the filesystem, running low on memory, calling fsync(), etc will make it happen sooner.

If the user hasn't changed the writeback expire timeouts, we can assume any dirty inode is on its way to disk within 30 seconds or so. That's not always the case, e.g. laptop mode can set the writeback time to minutes or hours, if the user prefers longer battery runtime over updating trivial data.

'm not really following how [lazytime] helps reduce the atime updates to disk here compared to just relatime

lazytime doesn't flush the inode for an atime update while the file is open--the atime update happens when the last process closes the file. If one process holds the file open while other processes open and close, then the atime updates won't trigger a flush to disk.

The disk usage issue compounds with frequency of snapshots assuming similar read heavy operations occur between them. This can muddy the atime updates in the snapshots with data that may be valued in the snapshots, requiring extra effort to remedy if wanting to retain snapshots while freeing up space by removing the atime metadata updates?

I'm not sure what the question is here. Snapshots act like independent filesystems. If you read the files in one subvol, it doesn't affect the atimes of files in any other subvol. It just takes up more space on disk, because you have two copies of the tree with different timestamps on the inodes. The atime updates can't be deleted separately from the snapshot that holds them.

e.g. 1: if you make hourly snapshots and read files continuously in the original subvol, and you mounted with strictatime, then you'll have one snapshot where the atimes are from 3 to 4 o'clock, another snapshot where the atimes are between 4 and 5 o'clock, a third snapshot where the atimes are from 5 to 6 o'clock...each will be a mostly complete copy of the original subvol's metadata, with different timestamp values on all the inodes.

e.g. 2: If you have hourly snapshots with relatime and read all the files every minute, then you'll have 23 snapshots with shared metadata (all the same inode timestamps), then one snapshot where all the atimes are updated for that 24-hour period (a copy of most metadata), then 23 more sharing space with the 24th. Every 24 hours there will be a new set of atime timestamps, so a new full set of metadata, then 23 smaller snapshots that share metadata with the first.

systemd-tmpfiles may the most relevant beyond more niche software for a desktop/workstation user

For tmpfiles and automatic file reaping you should not use a snapshot. The snapshot will prevent the file reaper from freeing any space (reaping activity will allocate more space instead as nothing is freed, and metadata pages are CoW-modified).

For a tmp reaper there is not much practical difference between ctime and atime for selecting old files. If you need a file to persist for a long time, why is it in /tmp? If you don't need long persistence, then atime extending the persistence doesn't matter. The main exceptions to this rule are authentication tokens, like kerberos ticket files or .Xauthority, which might not be stored in $HOME for various reasons. Reaping those while they're still in use would be bad.

lazytime paired with either isn't likely to be that beneficial in practice and it's trade-off when a power loss or crash event occurs adds more risk on data with BTRFS than it does other filesystems?

An example of where this matters: suppose you have a build server which checks out a directory, runs a big recursive make. The server is set up with the following constraints:

1. btrfs filesystem mounted -o flushoncommit
2. good power management, stops CPU and PCI before RAM (so no corruption can escape during power failure)
3. good drive firmware, no write reordering bugs or reneging on write ACKs on power failure
4. the build tools and Makefile rules handle SIGKILL well
5. all dependencies in the Makefiles are correct

then the following events should have identical effects on the content of the build directory:

1. at some random time T during the build, kill all build-related processes simultaneously with SIGKILL.
2. at some random time a few seconds later than T, kill power to the machine.

In both cases, it should be possible to restart the build iwth 'make' again, and always produce correct build output. This should remain true even if source files are modified. If precondition 3 fails then an error shall be detected and reported to the build process.

With lazytime, the timestamps may not be updated at the same times as the file contents, so scenario 1 and 2 may produce different results. (They'll produce different results every time because the results are nondeterministic; however, the set of results from event 2 should contain no result that cannot also be obtained from event 1).

For comparison, ext4 doesn't meet precondition 1, so after a power failure the build directory is almost always garbage, and the build must be restarted from git clean. Or you have to run ext4 in sync mode to make this work, and get very bad performance.

Now there are a lot of preconditions to be met before we can consider this example relevant. As a practical matter, I'd never believe that precondition 5 had been met unless the build project was small enough to audit personally, so I'd always run git clean -dfx after either a power failure event or a SIGKILL event even though theoretically I don't need to. Also a lot of legacy / proprietary code fails precondition 4, which means SIGKILL breaks the build directory even if nothing goes wrong with the filesystem.

btrfs does have cases where it's at least possible to resume a build after a power failure, and lazytime breaks those.

[polarathene]

lazytime marks the inode specially, so the inode only becomes dirty when it is closed after a read (the inode may become dirty for other reasons--if that happens, the atime is flushed with the rest of the inode).

The quote you were responding to with this was specifically regarding my misunderstanding regarding lazytime with reads impact on atime updates.

Apologies, I see I wasn't particularly clear about that in the quoted paragraph and just assumed we'd be on the same page about discussing lazytime which I was responding to.

---

lazytime doesn't flush the inode for an atime update while the file is open--the atime update happens when the last process closes the file. If one process holds the file open while other processes open and close, then the atime updates won't trigger a flush to disk.

You covered the VM or database type of data as an example for when a file is likely open for a longer duration.

The quote you were responding to was asking what tangible benefit was lazytime adding for atime updates with scenarios like find / opening and closing a large amount of files with reads, which you noted earlier as closing the file triggering the write a lot sooner than I and perhaps others thought.

You clarified that the inode is marked as dirty and from that point is written to disk when a condition is met (dirty time expiring, low memory, etc). Provided nothing else is keeping the file open, lazytime is not reducing atime updates to disk that much more than relatime alone would, given the find / bulk read example?

Your descriptions of relatime vs lazytime seem to convey that relatime will update atime upon read if appropriate, while lazytime would delay that disk write. Is this what you meant by lazytime potentially reducing writes for atime with reads? (even if the delay may not offset a write by much longer)

---

I'm not sure what the question is here.

I was seeking confirmation if the statement was correct. I tried to be terse as our responses are already a tad verbose 😅

Snapshots act like independent filesystems. If you read the files in one subvol, it doesn't affect the atimes of files in any other subvol. It just takes up more space on disk, because you have two copies of the tree with different timestamps on the inodes.

I understood this.

The atime updates can't be deleted separately from the snapshot that holds them.

I was trying to highlight this issue, and that dealing with the issue after the it's polluted snapshots would be frustrating to workaround. One could presumably recreate the snapshots or some other workaround after the fact that excludes the atime metadata, it just isn't likely a desirable workaround to apply.

This was intended in favor of noatime to avoid the situation, since opting for strictatime or relatime for any software that relies on atime to function or work optimally is what causes this to occur from the read scenarios discussed.

In contrast, if you default to noatime and notice that your software isn't working well due to a lack of atime updates... you're not tasked with a frustrating cleanup (or opting for data loss by not trying to retain affected snapshots data), just replace noatime and get the desired atime updates.

noatime is thus the better default to prefer.

---

e.g. 1 strictatime
each will be a mostly complete copy of the original subvol's metadata, with different timestamp values on all the inodes.

e.g. 2 relatime
Every 24 hours there will be a new set of atime timestamps, so a new full set of metadata, then 23 smaller snapshots that share metadata with the first.

Yes. This is how I understood them as well and tried to communicate.

strictatime:

• Is not ideal with hourly snapshots if you're going to have a recurring find / like task. A more likely scenario may be daily snapshots with some scheduled task or usage pattern that would update atime for many files between each snapshot.
• On the other hand, if you're not frequently creating snapshots.. then it may be better than relatime? (which you described an I/O pressure scenario and delayed writes)

relatime:

• Better when snapshots are created frequently as reads are less frequent, as low as once per day? So regardless of number of snapshots over a duration, reads alone after each snapshot won't result in new atime updates wasting disk space unexpectedly (from a typical user perspective).
• The benefit from strictatime is reduced once snapshot frequency is daily or greater.

In your initial comment you mentioned:

relatime may make this effect worse for some users, by batching up a lot of atime updates at close to the same time each day (e.g. when the user logs in at the start of the work day, triggering the once-every-24-hours update behavior of relatime).

Which I thought was comparing to strictatime, and a misunderstanding of delaying an atime update being written up to 24 hours... you meant the delay as a throttle (perform straight away but no more than X times within Y duration), not delaying the write until the duration expires (which lazytime appears to if no other conditions are met earlier).

So relatime or strictatime, when either would write an atime update, they do so upon the read and the mentioned IO pressure concern for certain disks can occur. lazytime doesn't really help, only noatime?:

The impact of all this is relative. Metadata updates on random pages are extremely expensive on big, slow spinners, and not at all expensive on small, fast NVME devices.

This can result in the entire system freezing up for some time--from a fraction of a second on fresh installs to NVME devices, to several minutes on larger and older filesystems on slower spinning disks. This is true of any large metadata update--atime is just one special and easily avoidable case.

---

For tmpfiles and automatic file reaping you should not use a snapshot. The snapshot will prevent the file reaper from freeing any space (reaping activity will allocate more space instead as nothing is freed, and metadata pages are CoW-modified).

I just mention it as an example of something that can be configured on a system. The user may not even be aware of it when considering if they should use strictatime or relatime.

systemd-tmpfiles --cat-config | grep -v ^# | sort -u lists many locations and files that it affects.. If you want to selectively avoid snapshotting such locations you're probably going to have a more nuanced subvol layout/config? Or not use snapshots much if atime matters to you.

For a tmp reaper there is not much practical difference between ctime and atime for selecting old files. If you need a file to persist for a long time, why is it in /tmp?

It's not just /tmp which is often on tmpfs anyway and a non-issue. The command I shared above shows various locations nested in /var, /run, /etc /dev, even /srv and /home are listed... although I haven't quite looked into how to interpret what those config lines are doing for the service.

One example lennart mentioned was coredump files. Without atime updates to determine age, it will also fall back to mtime or ctime, however you may access something older that an atime update would have provided a younger age and avoided reaping.

One case that comes to mind for that would perhaps be caches such as file browser thumbnail previews. If you were using this service, you'd perhaps want to retain frequently visited locations thumbnails, rather than generate those again when atime for less common thumbnail previews could be reaped instead.

Not the best example, and perhaps of little value to snapshot, but you might have that location as part of a broader subvolume. Perhaps there are some scenarios where you may still want snapshots of that subvol, but clean those up on some scheduled basis to discard older cache?

systemd-tmpfiles could be beneficial for that, so long as the user is aware of it and enabling atime for that, they can go through the extra effort to minimize any drawbacks from that content being captured in a subvol snapshot (if any).

---

btrfs does have cases where it's at least possible to resume a build after a power failure, and lazytime breaks those.

To clarify, that's due to mtime and ctime not having been updated on disk right, not atime? Or also an issue with atime?

For mtime and ctime updates, these would get written to disk not long after on a standard system (eg 30 seconds dirty writeback?), which is still a risk, but the window for it being out of sync would be small.

I know that it's advised to perform a scrub after a power loss event (and presumably crash like a kernel panic), is BTRFS not able to detect the problem and provide a last known good state in this situation when lazytime is involved and causes a consistency issue?

Pragmatically how much of a meaningful data loss is risked? (this make operation example isn't losing anything important in practice?)

With lazytime, the timestamps may not be updated at the same times as the file contents, so scenario 1 and 2 may produce different results.

When you say produce different results, do you mean with the metadata difference alone, or the state of data that this example was in the process of generating/performing?

I'm having a bit of difficulty seeing it as a pragmatic example, since from my understanding you'd generally restart such a process from a clean state if a failure botched it previously.

Likewise something like a system update which could presumably be a related scenario would perform in the same manner (often with extra effort on BTRFS to have pre and post snapshots)

[Zygo]

Provided nothing else is keeping the file open, lazytime is not reducing atime updates to disk that much more than relatime alone would, given the find / bulk read example?
is this what you meant by lazytime potentially reducing writes for atime with reads? (even if the delay may not offset a write by much longer)

Yes and yes.

One could presumably recreate the snapshots or some other workaround after the fact that excludes the atime metadata, it just isn't likely a desirable workaround to apply.

It is very hard to get the space back without deleting every snapshot. If even one old snapshot still exists, it will keep a full copy of the old metadata on disk. btrfs provides no way to recombine metadata between subvols once it has been unshared.

On xfs, subvol metadata is parent volume data, so it's possible to dedupe the metadata, maybe also take the snapshot offline for external editing...but probably still quite impractical.

If you want to selectively avoid snapshotting such locations you're probably going to have a more nuanced subvol layout/config? Or not use snapshots much if atime matters to you.

The size also matters. Most users would not be bothered by 1500 16K thumbnail files, even if 100 snapshots contain different sets of thumbnails. They might be bothered by 100 snapshots of 160MB core files as they will use non-trivial space, and recovering that space requires deleting every reference in a snapshot (either by mutating the snapshot or discarding the entire snapshot). But that also depends on the filesystem size--it could be a disaster for a 128GB filesystem, and utterly trivial on a 16TB disk.

I'm mostly thinking about free-space-demand-driven tmpreaper use cases. We have various large caches where we retain data as long as free space allows, and we do have to make sure those aren't ever snapshotted because we need to free a lot of space very quickly. We build the system around these caches, so they have properly non-overlapping subvols set up.

It would be nice to have an inheritable attribute (as we do for datacow and compression) that controlled atime, so it could be selectively enabled in the specific places where it is needed, and disabled by default. We currently have an attribute that can only disable atime on specific inodes (so we can't mount noatime and then turn atime on for specific directories) and is not inheritable (i.e. files created in noatime directories do not get the noatime property, it has to be set for every inode individually).

To clarify, that's due to mtime and ctime not having been updated on disk right, not atime? Or also an issue with atime?

mtime and ctime. atime usually doesn't matter to a build. lazytime in theory affects all 3 timestamps, so it's quite different from the other atime-related options.

For mtime and ctime updates, these would get written to disk not long after on a standard system (eg 30 seconds dirty writeback?), which is still a risk, but the window for it being out of sync would be small.

The problem is that the risk is not exactly zero.

There is normally no separation between data and metadata update on btrfs for a single write() call. Both data and timestamp are updated, or neither, because the write() call holds a lock that prevents btrfs from committing a transaction between the two updates. lazytime would cause the mtime/ctime update to occur after the return of the write() call, possibly in a separate transaction from the data update, so the risk of the timestamp being out of sync with the data becomes greater than zero.

(Note that all of the above also applies to the i_version field too. NFSv4 relies on every filesystem to persistently update i_version to keep client-side data caches consistent. It is updated almost as often as the mtime field)

It turns out that this is moot on btrfs. btrfs has its own update_time implementation, and that implementation does not implement lazytime for writes. As I mentioned earlier, the inode and data references frequently occupy the same metadata pages, so very little IO saving is possible; however, there are benefits in some cases, so it's worth analyzing the effect of a hypothetical btrfs lazytime implementation in case the feature is implemented some day.

btrfs uses all generic VFS code to handle atime updates, so lazytime does affect atime updates on btrfs, just like all the other atime options; however, since a read() updates atime and nothing else, no inode/data inconsistency is possible.

I know that it's advised to perform a scrub after a power loss event (and presumably crash like a kernel panic). is BTRFS not able to detect the problem and provide a last known good state in this situation when lazytime is involved and causes a consistency issue?

Quite the opposite. The normal post-power-failure recovery procedure for btrfs is to turn the power back on and resume normal work immediately. Regular scheduled scrubs are important for discovering disk failures, but there is no benefit from scrub that is specific to power failure or crashes. It won't hurt to start a scrub after a power failure, but the post-power-on scrub won't have any useful effect unless the disk happened to start failing some time after the previous scrub.

btrfs uses transactions to ensure data consistency (not just metadata) after crashes and some hardware failures. This is a requirement of the data csum feature, where data and metadata are updated in lock step. btrfs does this without the performance hit of -o data=journal or -o sync so it is the default behavior. After a crash, applications get a point-in-time snapshot of their state on disk with no reordering of operations (-o flushoncommit) or no reordering except delalloc writes (-o noflushoncommit).

With flushoncommit, an application that stores persistent data on only one btrfs filesystem can't tell whether the application was previously terminated by SIGKILL or a host power failure/crash event, unless the application can access storage outside of btrfs filesystem, or the application intentionally bypasses the transaction mechanism on btrfs with nodatacow, or the application exploits a bug in btrfs.

lazytime would introduce a detectable deviation from the normal btrfs behavior, as it would allow data to become inconsistent with its ctime/mtime timestamps where this is currently not possible. Applications would then be able to behave differently if they are restarted after a power failure vs after being killed by SIGKILL.

From an ext4 user's point of view this concern may seem trivial, as ext4 users can expect far more data loss, inconsistency, and corruption to occur after every crash, and this is easily detectable by applications that modify data without using fsync. Indeed, on ext4 it requires skill and discipline to build applications that can avoid catastrophic failure after a crash.

On btrfs, a single-bit post-crash data/metadata inconsistency or reordering of the effects of mutating syscalls would be considered a serious bug. The same tree update code is used for snapshots in btrfs, which are supposed to be atomic, suitable for making consistent backups. If the post-crash filesystem content is wrong, the backups are probably wrong too, so the post-crash behavior is a first-class feature that is thoroughly regression-tested.

When you say produce different results, do you mean with the metadata difference alone, or the state of data that this example was in the process of generating/performing?

The output could be different due to inconsistencies between data and timestamps after the crash. If we have a dependency chain where A depends on B, and B depends on C, and C is updated, then lazytime results in a possible post-crash state where B's contents are updated on disk, but B's timestamp is not. After the crash, A may not be rebuilt with the new contents of B, because A's timestamp is newer than B's (old due to lazytime and the crash) timestamp.

Note that properly written Makefile rules will simply rebuild B again after the crash, since B's timestamp is older than C, so we can't have incorrect output with just Make and lazytime by themselves. If C is updated by some other tool, such as rsync or git checkout, just before the crash, then the update may not be repeated because other tool notices the file contents (not timestamp) are up to date. Make relies on file timestamp (not contents) to determine whether to execute build rules, so after the crash, Make sees that C is older than B, and B is older than A, so A is not rebuilt when it should be.

A more complicated dependency chain might result in a binary being built with .o files that contain code built from two different versions of a header file that defines the interface between the .o files (the make run before the crash updates one .o file, but the crash occurs, the timestamp on the header is lost, and make run after the crash does not update the other .o file using the same header). In that case the resulting binaries would have undefined behavior, which is generally considered very bad.

Backups are a better example. Tools like rsync and file indexers tend to rely on mtime as a kind of unique content identifier. They assume equal mtime means equal file contents, and will not propagate updates to backup without a mtime change. If lazytime drops a mtime change, then the backups don't get updated, and there is potential data loss if the backups need to be restored. (For clarity to those reading this text out of context: this would not affect btrfs send backups, which compare the file metadata directly, and do not rely on timestamps.)

Since btrfs doesn't implement lazytime for writes, the above is purely theoretical.

Pragmatically how much of a meaningful data loss is risked? (this make operation example isn't losing anything important in practice?)

One problem with my Make example is easy to analyze and predict, and the problems are easy to work around--and that makes it a bad example.

Real life isn't this predictable. A big app box running a dozen apps for an enterprise typically has a dependency on unsynced write ordering somewhere. The people running the box most likely don't know where that dependency is until something breaks on a power outage--and may still not know, even after fixing the resulting problems.

Pragmatically, over the last 5 years we spent about 30 days per year performing post-crash recovery actions on ext4 and xfs filesystems (comparing backups, scrubbing, auditing databases, restoring backups, manually fixing critical services, purging and repeating any large file tree updates that were in progress at the time of the crash, and generally trying to understand what went wrong with each individual application), compared to 0 on btrfs (we do all the checks, but don't find any problems to fix). We are very conservative about changes that make btrfs behave even a little bit like ext4.

[polarathene]

@Zygo thanks for the awesome insights and examples! Very informative and helpful :)

---

I will try to tersely summarize the problem and advice for someone making a decision:

Is atime a bad idea with BTRFS? Should I prefer noatime?

noatime is often a better choice.

This is only relevant when a file is read and the metadata is shared with at least one snapshot. Updating the atime in metadata then results in requiring disk space for a new copy of metadata to accommodate the change, while keeping the old atime for any snapshots that still reference it.

With enough files read or snapshots over time, it can become a noticeable amount of disk space wasted. Using noatime can avoid this. Although if the affected snapshots are disposable / short-lived, you can keep the default relatime and just delete older snapshots when you need the space.

It is easier to switch from noatime to relatime/strictatime (avoid lazytime as it risks data consistency guarantees) than it is to switch the other way around (due to snapshots "polluted" with metadata containing only unwanted atime updates).

More verbose version

The atime + snapshot disk usage problem

• Metadata for files on BTRFS can be shared across multiple snapshots efficiently when it's identical.
• Reading a file can trigger an atime update write to the metadata on disk.
• Modifying the metadata loses the benefit of sharing with snapshots, CoW causes new writes to update the metadata, using up extra disk space.
• If the file metadata to be updated is not shared with any snapshot(s), no extra disk space is needed to write the update.
• The disk space can be recovered when all snapshots referencing the same copy of metadata are deleted.
• The problem is the old metadata is tied to any other content in your snapshots that may be of value to retain, there is no way to clean up space by removing only the redundant metadata in snapshots.
• Thus the problem compounds if the frequency of atime updates affects many snapshots over time (as in each snapshot has different metadata for a file due to different atimes).

relatime, strictatime, lazytime vs noatime

• relatime minimizes atime updates (and thus their precision) which is beneficial with frequent use of snapshots.
• If snapshot intervals are daily or greater, strictatime can provide more atime precision at the expense of writing to disk with each file read.
• Many atime updates for the same file(s) does not compound the disk usage increase if there is no new snapshots taken. Only the first atime update for a file incurs the penalty from no longer being able to share the same data with a snapshot(s).
• lazytime behaviour does not affect writes on BTRFS, and does not provide much benefit over relatime for reads. Additionally it can risk consistency guarantees that BTRFS is intended to provide for filesystem state.

noatime avoids the problem since no atime updates occur. Some software like systemd-tmpfiles may not function as optimally/accurately, but otherwise most users aren't likely to notice.

---

Let me know if this sounds right and we can wrap up the issue? :)

[Zygo]

I think it's a good summary, but it's mostly summarizing stuff I wrote, so I'm biased. ;)

[TheEvilSkeleton]

@polarathene thank you so much for the TL;DR and @Zygo thank you so much for not only answering my question, but also going in depth. The whole discussion was worth reading.

I'll wait for a member of the repo to close the issue in case they want to add/correct something.