Using squid as a caching server for drush module downloads

Adrian Rollett bio photo By Adrian Rollett

Intro

Due to popular request, I’ve decided to quickly document how we at UNT Web support use squid as a proxy caching server for drush. The main goal was to speed up module updates, and reduce the load on drupal.org and UNT’s internet connection. The instructions are based on my memory, and are for Debian, so there may be gaps here and there depending on your experience/OS.

Squid Setup

Installation was super-simple. (gotta love apt)

apt-get install squid3

Configuration was also quite simple. Basically, the debian package for squid is setup fairly nicely as a proxy caching server out of the box, and you need only configure access. First, add an acl in /etc/squid3/squid.conf for your local network, something like the following:

acl localnet src 192.168.0.0/16 # RFC1918 possible internal network

(make sure to change the subnet to match your network) Next, add a rule allowing access from the local network you just defined:

http_access allow localnet

(If your squid server is on the same box as your drush installation, you just need to allow localhost)

Restart squid, and away you go.

/etc/init.d/squid3 restart

Using drush with squid

To make drush use squid, simply do the following:

http_proxy="http://squid-host.example.com:3128/" php /usr/local/drush/drush.php dl cck

(You’ll probably want to add an alias)

Is it working?

Let’s try and download the inlinetags module.

drush-host # http_proxy="<a href="http://squid-host.example.com:3128/"">http://squid-host.example.com:3128/"</a> php /usr/local/drush/drush.php dl inlinetags
squid-host # tail -f /var/log/squid3
1267120330.674    273 192.168.208.29 TCP_MISS/200 8952 GET <a href="http://ftp.drupal.org/files/projects/inlinetags-6.x-1.1.tar.gz">http://ftp.drupal.org/files/projects/inlinetags-6.x-1.1.tar.gz</a> - DIRECT/140.211.166.142 application/x-gzip

So, we can tell from the log that the request came into squid, but did not find the object cached. So far so good. Let’s try again:

drush-host # http_proxy="<a href="http://squid-host.example.com:3128/"">http://squid-host.example.com:3128/"</a> php /usr/local/drush/drush.php dl inlinetags
squid-host # tail -f /var/log/squid3
1267121152.216      0 192.168.208.29 TCP_HIT/200 8960 GET <a href="http://ftp.drupal.org/files/projects/inlinetags-6.x-1.1.tar.gz">http://ftp.drupal.org/files/projects/inlinetags-6.x-1.1.tar.gz</a> - NONE/- application/x-gzip

Good news! This time the request never went out to drupal.org. That’s all I have for now, hope it helps someone. Please feel free to comment if you have difficulties with these instructions. (or success!)

Thanks for taking the time to write this. I just did the setup on ubuntu 9.10, the only thing different was the log files, I had to do sudo tail -f /var/log/squid3/access.log.

Seems to be working. Now to try it with drush. :)

Thanks for this post. It took me a while to get it to automatically work for drush (eg, w/o having to type the full http_proxy=… bit above). The missing piece was to add http_proxy=http://localhost:3128 ftp_proxy=http://localhost:3128 use_proxy=on to my .wgetrc file.

Greetings …

This is an awesome idea, but if you using Aegir, you can’t really pass proxy details to drush and you can’t alias it either, because the aegir user account is not really a shell, for security reasons.

So, what I did was setup iptables, to forward all outbound http traffic to squid, using the following commands …

iptables -t nat -F # clear table

normal transparent proxy

iptables -t nat -A PREROUTING -p tcp -i eth0 –dport 80 -j REDIRECT –to-port 3127

handle connections on the same box (SQUIDIP is a loopback instance)

gid=id -g proxy iptables -t nat -A OUTPUT -p tcp –dport 80 -m owner –gid-owner $gid -j ACCEPT iptables -t nat -A OUTPUT -p tcp –dport 80 -j DNAT –to-destination SQUIDIP:3127

Complete details at … http://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxLocalhost

Hope that helps other people.

The excellent and free SquidMan makes this very easy to set up on a Mac. Make sure you select a cache size on the SquidMan Preferences/General pane.

Note that OS X doesn’t come with wget so proxy settings go in ~/.curlrc - e.g.: proxy localhost:3128

I got this setup working under OS X with MacPorts like this:

sudo port install squid3

wait a bit while it builds then make squid load as a daemon with

sudo port load squid3

and you can get wget (and hence drush) to use squid by creating a .wgetrc file with the suggested

http_proxy=http://localhost:3128 ftp_proxy=http://localhost:3128 use_proxy=on

You can see all the action with a quick:

sudo tail /opt/local/var/squid/logs/access.log

thanks, drush was hanging until i added this to my .wgetrc file.

for me drush make wanted to use cURL, so when aegir tried to set up new platforms for me they’d hang or fail.

i made a .curlrc file that was basically the same as the .wgetrc file suggested by jhedstrom

more info here: http://drupal.org/node/270963#comment-2544452

Hmmm… in the squid3 access.log, I seem to be getting TCP_CLIENT_REFRESH_MISS for all modules when I use the alias: drush=”http_proxy=http://localhost:3128 drush”

Funny, cause when we specify a direct git repo in the make file, then we get a TCP_HIT, but it misses for everything else…

but hey, drush dl works fine, it’s just drush make that’s giving issue :S

Just posting in case someone else gets the same problem!