Using squid as a caching server for drush module downloads

25 Feb 2010

Posted by acrollet

Intro

Due to popular request, I've decided to quickly document how we at UNT Web support use squid as a proxy caching server for drush. The main goal was to speed up module updates, and reduce the load on drupal.org and UNT's internet connection. The instructions are based on my memory, and are for Debian, so there may be gaps here and there depending on your experience/OS.

Squid Setup

Installation was super-simple. (gotta love apt)

  1. apt-get install squid3

Configuration was also quite simple. Basically, the debian package for squid is setup fairly nicely as a proxy caching server out of the box, and you need only configure access. First, add an acl in /etc/squid3/squid.conf for your local network, something like the following:

  1. acl localnet src 192.168.0.0/16 # RFC1918 possible internal network

(make sure to change the subnet to match your network) Next, add a rule allowing access from the local network you just defined:

  1. http_access allow localnet

(If your squid server is on the same box as your drush installation, you just need to allow localhost)

Restart squid, and away you go.

  1. /etc/init.d/squid3 restart

Using drush with squid

To make drush use squid, simply do the following:

  1. http_proxy="<a href="http://squid-host.example.com:3128/"">http://squid-host.example.com:3128/"</a> php /usr/local/drush/drush.php dl cck

(You'll probably want to add an alias)

Is it working?

Let's try and download the inlinetags module.

  1. drush-host # http_proxy="<a href="http://squid-host.example.com:3128/"">http://squid-host.example.com:3128/"</a> php /usr/local/drush/drush.php dl inlinetags
  2. squid-host # tail -f /var/log/squid3
  3. 1267120330.674    273 192.168.208.29 TCP_MISS/200 8952 GET <a href="http://ftp.drupal.org/files/projects/inlinetags-6.x-1.1.tar.gz">http://ftp.drupal.org/files/projects/inlinetags-6.x-1.1.tar.gz</a> - DIRECT/140.211.166.142 application/x-gzip

So, we can tell from the log that the request came into squid, but did not find the object cached. So far so good. Let's try again:

  1. drush-host # http_proxy="<a href="http://squid-host.example.com:3128/"">http://squid-host.example.com:3128/"</a> php /usr/local/drush/drush.php dl inlinetags
  2. squid-host # tail -f /var/log/squid3
  3. 1267121152.216      0 192.168.208.29 TCP_HIT/200 8960 GET <a href="http://ftp.drupal.org/files/projects/inlinetags-6.x-1.1.tar.gz">http://ftp.drupal.org/files/projects/inlinetags-6.x-1.1.tar.gz</a> - NONE/- application/x-gzip

Good news! This time the request never went out to drupal.org. That's all I have for now, hope it helps someone. Please feel free to comment if you have difficulties with these instructions. (or success!)

Blog Tags: 

Comments

Thanks for taking the time to write this. I just did the setup on ubuntu 9.10, the only thing different was the log files, I had to do sudo tail -f /var/log/squid3/access.log.

Seems to be working. Now to try it with drush. :)


Thanks for this post. It took me a while to get it to automatically work for drush (eg, w/o having to type the full http_proxy=... bit above). The missing piece was to add

http_proxy=http://localhost:3128
ftp_proxy=http://localhost:3128
use_proxy=on

to my .wgetrc file.


ah, thanks for filling in that piece of information - I just use an alias for drush, so I haven't thought about it again...


apt-get install squid3
sudo edit /etc/environment

>add the following line:
http_proxy="http://localhost:3128"

>done


I got this setup working under OS X with MacPorts like this:

sudo port install squid3

wait a bit while it builds then make squid load as a daemon with

sudo port load squid3

and you can get wget (and hence drush) to use squid by creating a .wgetrc file with the suggested


http_proxy=http://localhost:3128
ftp_proxy=http://localhost:3128
use_proxy=on

You can see all the action with a quick:

sudo tail /opt/local/var/squid/logs/access.log


thanks, drush was hanging until i added this to my .wgetrc file.

for me drush make wanted to use cURL, so when aegir tried to set up new platforms for me they'd hang or fail.

i made a .curlrc file that was basically the same as the .wgetrc file suggested by jhedstrom

more info here: http://drupal.org/node/270963#comment-2544452


Greetings ...

This is an awesome idea, but if you using Aegir, you can't really pass proxy details to drush and you can't alias it either, because the aegir user account is not really a shell, for security reasons.

So, what I did was setup iptables, to forward all outbound http traffic to squid, using the following commands ...

iptables -t nat -F # clear table

# normal transparent proxy
iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 80 -j REDIRECT --to-port 3127

# handle connections on the same box (SQUIDIP is a loopback instance)
gid=`id -g proxy`
iptables -t nat -A OUTPUT -p tcp --dport 80 -m owner --gid-owner $gid -j ACCEPT
iptables -t nat -A OUTPUT -p tcp --dport 80 -j DNAT --to-destination SQUIDIP:3127

Complete details at ... http://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxLocalhost

Hope that helps other people.


Oh man, this'll definitely come in handy!

Thanks Lee. Or... "C"?
Err... Thanks, C. Lee Talyor! :)


The excellent and free SquidMan makes this very easy to set up on a Mac. Make sure you select a cache size on the SquidMan Preferences/General pane.

Note that OS X doesn't come with wget so proxy settings go in ~/.curlrc - e.g.:

proxy localhost:3128


Can you format the article a bit? It is practically illegible.


whoops, site upgrade hosed the geshi library. thanks, fixed now.


Anyone had any luck installing squid on OSX using Homebrew?


Hmmm... in the squid3 access.log, I seem to be getting TCP_CLIENT_REFRESH_MISS for all modules when I use the alias:
drush="http_proxy=http://localhost:3128 drush"

Funny, cause when we specify a direct git repo in the make file, then we get a TCP_HIT, but it misses for everything else...

but hey, drush dl works fine, it's just drush make that's giving issue :S

Just posting in case someone else gets the same problem!


bc781ec37a760a6ffc97bac3f0a2a4b3 Hi Guys, I am newbie in the internet stuff and I dont know if I am writing on correct board on this website. I
have got problem with activating my account. I received email but when I click on the link it was not working, is this link is correct? http://reluctanthacker.rollett.org/?9859eacd0245,


Just a friendly heads up about drush 5.x caching:
http://drupal.org/node/1473006#comment-5881814


Hello, dear Admin.

I want to place banner on your forum.
Price $100 dollars is acceptable?

PM me please.


Add new comment

The content of this field is kept private and will not be shown publicly.