ZFS, Actually Storage you can finally reason about.

The ARC

concept
depth 3/4 − deeper + surface

You check a ZFS machine's system monitor, see nearly all the memory in use on a quiet day, and conclude something is broken. Actually, that's the ARC—ZFS's reading cache—and it is supposed to be huge. The ARC keeps recently and frequently read data parked in memory, where fetching it again is thousands of times faster than going back to the disks. Memory sitting empty helps no one; ZFS puts it to work—and gives it back when your programs genuinely need it.

The correction under the correction: the memory bar you've read as a fuel gauge all your life is the wrong instrument here. Full is the healthy state.

Reading it honestly: arc_summary (or arcstat) instead of htop. The number that matters is the hit rate—a warm homelab ARC serving in the high-90s percent is why the pool feels faster than its spinning disks have any right to be.

The one legitimate reason to cap it: VM hosts. The ARC's give-it-back dance with the Linux kernel is imperfect and slightly laggy, and VMs allocating big chunks of RAM can collide with it. That's why Proxmox installations conventionally set zfs_arc_max to a fixed budget (Proxmox itself defaults to a capped ARC on new installs). On a pure storage box, leave it alone—capping ARC "to keep RAM free" is buying nothing with your cache.

Its neighbors, and the traps between them:

  • The page cache—Linux's own file cache, which ordinary filesystems use and which htop doesn't alarm you about (it hides under "available"). ZFS's separate ARC accounting is why identical caching reads as a crisis on ZFS and as normal on ext4. Same behavior, scarier bookkeeping.
  • L2ARC—the ARC's spillover tier on SSD. Not more ARC: every L2ARC block costs real ARC RAM for its index (see L2ARC before buying).
  • The ZIL/SLOG—the write-side sibling in name only; it's a crash log, not a cache. "ARC for reads, SLOG for writes" is a symmetry your brain wants and the system doesn't have.