Solaris 11.4 and user_reserve_hint_pct
A long time ago, i wrote about the /etc/system-parameter user_reserve_hint_pct. It limits the size of the ZFS ARC cache by proxy by essentially limiting the amount of the memory the kernel uses (to simplify it).
By proxy, because the ARC is more or less the only large structure inside the kernel that can shrink and grow signficantly in a kernel in a short time.You set the amount of memory that you want to “reserve” for applications, and whats left is for the kernel. The nice thing about this parameter is the fact, that you can change it while the system is running. But the “by proxy” has an important implication.
This implication is related to the reason why the behaviour of this parameter was changed with Solaris 11.4. It now has an uper limit of 75%.
You may ask why. I saw myself customers setting this parameter to values of 90 to 95% - as the parameter was allowed to be set to 99% - because sometimes they misunderstood the parameter as directly related (and only impacting it) to the ZFS cache size whereas the relation was only indirect and that you have to keep in mind that the kernel excluding the ARC cache needs space as well. So whats left over by user_reserve_hint_pct must fit the kernel as well.
The implication of limiting the kernel is, that you can can starve off ZFS by a ARC cache that is forced into to the absolute minimum as the system tried to obey the user_reserve_hint_pct hint, as sometimes with smaller speced system not even the kernel was fitting inside the hinted limit when the hint was set to an extremely high limit. When you starve down the system on the ARC cache, it may have severe performance implications and give you some performance headaches.
Out of this reason since 11.4 whenever you set this parameter to something larger than 75, it accept this parameter but behaves like like the parameter was set to 75. It’s internally capped. So you can’t totally starve off your system.
If you have problems with the 75% limit, you can still use the old method zfs_arc_max or in the case of kernel zones think about memory reservation pools, which solved the problem why you limit the ARC cache by reserving memory for the kernel zones early in the bootup of the system before the ARC cache can really grow up.