Re: [squid-users] Architecture for scaling delivery of large static files

2009-07-16 Thread Amos Jeffries
Jamie Tufnell wrote: On Thu, Jul 16, 2009 at 12:29 PM, Amos Jeffries wrote: On Thu, 16 Jul 2009 09:22:25 +1000, Jamie Tufnell We are talking files up-to-1GB in size here. Taking that into consideration, would you still recommend this architecture? Yes, the architecture itself is sound. OK

Re: [squid-users] Architecture for scaling delivery of large static files

2009-07-16 Thread Jamie Tufnell
On Thu, Jul 16, 2009 at 12:29 PM, Amos Jeffries wrote: > On Thu, 16 Jul 2009 09:22:25 +1000, Jamie Tufnell >> We are talking files up-to-1GB in size here.  Taking that into >> consideration, would you still recommend this architecture? > > Yes, the architecture itself is sound. OK cool. Thanks a

Re: [squid-users] Architecture for scaling delivery of large static files

2009-07-16 Thread Henrik Nordstrom
tor 2009-07-16 klockan 20:13 +1200 skrev Amos Jeffries: > Thanks Henrik. I was a bit unsure of that split. It's from the sfileno and sdirno being packed in the same 32-bit slot, both signed.. Relatively easy to change in the code if needed. Regards Henrik

Re: [squid-users] Architecture for scaling delivery of large static files

2009-07-16 Thread Amos Jeffries
Henrik Nordstrom wrote: tor 2009-07-16 klockan 14:29 +1200 skrev Amos Jeffries: For you with MB->GB files in Squid-2 that changes to faster Squid due to limiting RAM-cache to small files, with lots of large fast disks. Squid is limited to a few million (2^24) cache _objects_ per cache_dir, an

Re: [squid-users] Architecture for scaling delivery of large static files

2009-07-16 Thread Adrian Chadd
I was going to say; I'm tweaking the performance of a cache with 21 million objects in it now. Thats a bti bigger than 2^24. 2009/7/16 Henrik Nordstrom : > tor 2009-07-16 klockan 14:29 +1200 skrev Amos Jeffries: > >> For you with MB->GB files in Squid-2 that changes to faster Squid due to >> limit

Re: [squid-users] Architecture for scaling delivery of large static files

2009-07-15 Thread Henrik Nordstrom
tor 2009-07-16 klockan 14:29 +1200 skrev Amos Jeffries: > For you with MB->GB files in Squid-2 that changes to faster Squid due to > limiting RAM-cache to small files, with lots of large fast disks. Squid is > limited to a few million (2^24) cache _objects_ per cache_dir, and up to 32 (2^6) cache

Re: [squid-users] Architecture for scaling delivery of large static files

2009-07-15 Thread Amos Jeffries
On Thu, 16 Jul 2009 09:22:25 +1000, Jamie Tufnell wrote: > Thank you both for your responses, good to hear I might be on the right > track! > > Amos wrote: >> Just note that for MB or so scale files in memory Squid-2 is a snail, and >> Squid-3 does not yet provide collapsed forwarding. > > We ar

Re: [squid-users] Architecture for scaling delivery of large static files

2009-07-15 Thread Adrian Chadd
2009/7/16 Jamie Tufnell : > We are talking files up-to-1GB in size here.  Taking that into > consideration, would you still recommend this architecture? On disk? Sure. The disk buffer cache helps quite a bit. In memory ? (as in, the squid hot object cache; not the buffer cache) ? Not without inv

Re: [squid-users] Architecture for scaling delivery of large static files

2009-07-15 Thread Jamie Tufnell
Thank you both for your responses, good to hear I might be on the right track! Amos wrote: > Just note that for MB or so scale files in memory Squid-2 is a snail, and > Squid-3 does not yet provide collapsed forwarding. We are talking files up-to-1GB in size here. Taking that into consideration,

Re: [squid-users] Architecture for scaling delivery of large static files

2009-07-15 Thread Amos Jeffries
Jamie Tufnell wrote: Hi, I am wondering if Squid is the right tool to solve a scaling problem we're having. Our static content is currently served directly from Apache boxes to the end-user: User <=> Apache Originally it was just one Apache box but its Disk IO became saturated and now we have

[squid-users] Architecture for scaling delivery of large static files

2009-07-15 Thread Jamie Tufnell
Hi, I am wondering if Squid is the right tool to solve a scaling problem we're having. Our static content is currently served directly from Apache boxes to the end-user: User <=> Apache Originally it was just one Apache box but its Disk IO became saturated and now we have three Apache boxes, ea

Re: [squid-users] Architecture

2009-06-29 Thread Chudy Fernandez
o Cc: squid-users@squid-cache.org Sent: Tuesday, June 30, 2009 9:10:56 AM Subject: Re: [squid-users] Architecture On Mon, 29 Jun 2009 18:05:22 -0300, Ronan Lucio wrote: > Adrian, > > Adrian Chadd escreveu: >> Just another random datapoint - I've just deployed my Squid-2 >>

Re: [squid-users] Architecture

2009-06-29 Thread Amos Jeffries
On Mon, 29 Jun 2009 18:05:22 -0300, Ronan Lucio wrote: > Adrian, > > Adrian Chadd escreveu: >> Just another random datapoint - I've just deployed my Squid-2 >> derivative (which is at least as fast as Squid-2.HEAD) as a forward >> proxy on some current generation hardware. It's peaking at 700 >>

Re: [squid-users] Architecture

2009-06-29 Thread Adrian Chadd
2009/6/30 Ronan Lucio : > Could you tell what hardware do you use? > Reading Squid-Guide > (http://www.deckle.co.za/squid-users-guide/Installing_Squid) it says Squid > isn't CPU intensive, says a multiprocessor machines would not increase speed > dramatically. > Its a dual quad core amd of some s

Re: [squid-users] Architecture

2009-06-29 Thread Ronan Lucio
Adrian, Adrian Chadd escreveu: Just another random datapoint - I've just deployed my Squid-2 derivative (which is at least as fast as Squid-2.HEAD) as a forward proxy on some current generation hardware. It's peaking at 700 requests/sec and ~120mbit a sec with a ~ 30% byte hit rate. A reverse p

Re: [squid-users] Architecture

2009-06-26 Thread Adrian Chadd
2009/6/27 Chris Robertson : > I'm running a strictly forward proxy setup, which puts an entirely different > load on the system.  It's also a pretty low load (peaks of 160 req/sec at > 25mbit/sec). Just another random datapoint - I've just deployed my Squid-2 derivative (which is at least as fast

Re: [squid-users] Architecture

2009-06-26 Thread Chris Robertson
Amos Jeffries wrote: Ronan Lucio wrote: It really seems to be a better choice. Do you have any idea about how many page hit would handle one squid servers? Thinking about a Dual QuadCore 4Gb RAM serving only small files (less than 300 Kb each). Adrian has mapped Squid 2.7 as far as 800-850 r

Re: [squid-users] Architecture

2009-06-25 Thread Amos Jeffries
Ronan Lucio wrote: Hi Amos, Thank your for the valuable answer. I'm sorry for the delayed reply, but I need to first read more about CARP and architecture to better digest it. Now these things seems to be getting clear in my mind. Amos Jeffries escreveu: What kinkie means is that the effici

Re: [squid-users] Architecture

2009-06-25 Thread Ronan Lucio
Hi Amos, Thank your for the valuable answer. I'm sorry for the delayed reply, but I need to first read more about CARP and architecture to better digest it. Now these things seems to be getting clear in my mind. Amos Jeffries escreveu: What kinkie means is that the efficiency is determined

Re: [squid-users] Architecture

2009-06-23 Thread Amos Jeffries
On Tue, 23 Jun 2009 21:18:36 -0200, "Ronan Lucio" wrote: > Hi Kinkie, > > On Tue, 23 Jun 2009 21:51:17 +0200, Kinkie wrote >> Hi, >> I can't see the advantage of using lighthttpd instead of squid+carp >> as the frontend, > > The idea of putting a lighttpd server as a the frontend is for load >

Re: [squid-users] Architecture

2009-06-23 Thread Mark Nottingham
Last time I looked at lighty, it buffered the entire message when used in proxy mode; this isn't really workable. I'd use Squid on the FE in CARP mode; CARP is a way of using consistent hashing to spread the load out to multiple servers in a predictable way (like a DHT). If you need even

Re: [squid-users] Architecture

2009-06-23 Thread Ronan Lucio
Hi Kinkie, On Tue, 23 Jun 2009 21:51:17 +0200, Kinkie wrote > Hi, > I can't see the advantage of using lighthttpd instead of squid+carp > as the frontend, The idea of putting a lighttpd server as a the frontend is for load balance. What exactly do you mean with squid+carp? several squid server

Re: [squid-users] Architecture

2009-06-23 Thread Kinkie
Hi, I can't see the advantage of using lighthttpd instead of squid+carp as the frontend, and if using lighthttpd i can't see the advantage of not serving static content directly out of the balancer. Also watch out as nfs has locking and scaling issues of its own (assuming thet nfs is what you mea

[squid-users] Architecture

2009-06-23 Thread Ronan Lucio
Hi, We have a high-accessed website and I need to create a cache strategy for that. So I'm thinking to use the follow architecture. 1) A lighttpd webserver in front of everyone + mod_proxy balancing/pointing to some squid servers. 2) Some squid servers in the middle 3) Application servers

Re: [squid-users] Architecture advice

2008-08-08 Thread Adrian Chadd
Of course its been done. You have a few options. For example, see if your LB environment lets you load balance based on requested URL. Adiran 2008/8/9 Rob Williams <[EMAIL PROTECTED]>: > After doing some more research, I wonder if I'm approaching the > problem I want to solve the right way. >

[squid-users] Architecture advice

2008-08-08 Thread Rob Williams
After doing some more research, I wonder if I'm approaching the problem I want to solve the right way. I am creating a large-scale website with a large amount (terabytes) of static content. What I need is a way to scale access to my static content. I could do this many ways, but an efficient way

Re: [squid-users] Architecture Question

2004-05-19 Thread Matus UHLAR - fantomas
On 18.05 14:35, Steven Garrett wrote: > Currently we have the following Router -> Hardware Load Balancer -> many > real servers. We're looking into using squid for reverse proxy to alleviate > some of the load on our real servers. Is there a recommended/best practice > for where we should put pro

[squid-users] Architecture Question

2004-05-18 Thread Steven Garrett
Hi, Currently we have the following Router -> Hardware Load Balancer -> many real servers. We're looking into using squid for reverse proxy to alleviate some of the load on our real servers. Is there a recommended/best practice for where we should put proxy servers. I'm pretty new to this whole