What have L2ARC and a furniture shop in common ....
In the last week i´ve got some questions how the L2ARC works. I thought a moment about it and came up with the following description. The whole model of caching with two layers is similar to IKEA. Okay, i don´t know if IKEA works exactly this way, but it´s a good global example. I once visited an IKEA in Singapore (it´s really strange to see the blue and yellow logo of IKEA in Asia) and i visited an IKEA in Sunnyvale.
At first you have the bestsellers … you place them in the show room or in the self service area. You customers can access them with short latency. The just take some spoons, a new plaid or something like that. After a while you see, that you don´t sell that kitchen so often, and it doesn´t make sense to have it in direct access for your customers. Normally you would say: Okay, i need the room in my show room for better selling stuff and send it back to your central storehouse and order it on-demand from there. But that introduces a large latency to the customer. The customer would have to wait for it´s new kitchen a few days, as you have to transmit the order and you have to wait for the next delivery to you local IKEA storehouse. So there are some people that take some of the parts into you local storehouse. You can´t go into it directly but you can ask the people at IKEA to fetch it for you after purchasing it. The shopping latency isn´t so small as in the case of the spoons but nevertheless you get it shortly afterwards and try to put it in your car.
It´s important to know, that the stuff isn´t put to the local storehouse as a intermediate step. When your people responsible to fill your showroom get a large bunch of new stuff, they send the least selling stuff directly back to the central storehouse. Why? Well … when your people get new stuff, they cant wait for the people in the storehouse to free the room. They take the least selling list and fill the truck with the new stuff with the least selling stuff for his trip back.
The people in the local storehouse work a little bit differently: They look at the least selling list at a regular schedule and put one or two items of both into their local storehouse before the rest is send back to the central store house.
Why do the concepts works this way: When the people are not fast enough and your IKEA need the room for something differently really urgently (a magazine talked about some of your products), you don´t have to wait for local storehouse people. This way, the latency of getting new stuff in your showroom is much shorter.
ZFS, ARC, L2ARC and the pool disks works exactly this way: The ARC is the showroom, the L2ARC is your local storehouse, the pool is the central storehouse. When you want to know more, you can look in the source code, as it contains a really good documentation: /onnv/onnv-gate/usr/src/uts/common/fs/zfs/arc.c.