So there’s lots of excitement around Varnish, which is a caching proxy that is built to be first and foremost a reverse-proxy, as opposed to squid, which does both forward and reverse. Acceleration (reverse-proxying) is obviously important to us at Flickr, as we use squid extensively.
I’m hoping to do some testing with Varnish once it’s stable and has the ability to manage a constantly full cache. After emailing with Poul Henning-Kamp (one of the main developers) he says that object replacement/eviction is indeed on the roadmap, so we shall see.
From what I can tell, Varnish sounds a little like the COSS filesystem that squid can use, in that it uses one big file to store objects in. In varnish, this is mmap’d into the process and the kernel does all of the disk work. Since replacement/eviction isn’t done yet, not sure if the mechanism is “cyclical” like COSS, but however it will work, it’ll probably see some big performance increases when compared to the standard ‘nested directories’ way that aufs does things in squid currently.
Woohoo!