The ARC
conceptYou check a ZFS machine's system monitor, see nearly all the memory in use on a quiet day, and conclude something is broken. Actually, that's the ARC—ZFS's reading cache—and it is supposed to be huge. The ARC keeps recently and frequently read data parked in memory, where fetching it again is thousands of times faster than going back to the disks. Memory sitting empty helps no one; ZFS puts it to work—and gives it back when your programs genuinely need it.
The correction under the correction: the memory bar you've read as a fuel gauge all your life is the wrong instrument here. Full is the healthy state.
Reading it honestly: arc_summary (or arcstat) instead of htop. The number
that matters is the hit rate—a warm homelab ARC serving in the high-90s
percent is why the pool feels faster than its spinning disks have any right
to be.
The one legitimate reason to cap it: VM hosts. The ARC's give-it-back
dance with the Linux kernel is imperfect and slightly laggy, and VMs
allocating big chunks of RAM can collide with it. That's why Proxmox
installations conventionally set zfs_arc_max to a fixed budget (Proxmox
itself defaults to a capped ARC on new installs). On a pure storage box,
leave it alone—capping ARC "to keep RAM free" is buying nothing with your
cache.