[techtalk] Sick of surf and porn addicts

[James Sutherland]

On Fri, 25 May 2001, Liese wrote:

> This question is not really linux related but i'm getting desperate.
> There are about 300 users in the company where i work. Our website is
> running on a local system, using suse, apache and mysql. Last year we had
> to upgrade our line to the outside world, not because the incoming traffic
> (to the website) was too big but because the outgoing traffic was clogging
> the line.

How big's the line in question?? Presumably it's not a particularly busy
WWW site you host? If I understand you correctly, the problem is not the
content they are downloading, but the fact they're using a large amount of
bandwidth to do it?

> One year later (today) we have the same problem again. We use a proxy
> server (squid) and only give internet access to those who really need it
> (and only with permition from their manager) but it seems like everybody
> really really needs it (duuh!).
> And just giving access to the requested sites isnt possible either or i
> would have a fulltime job entering sites in squid.
> Once a month I check our proxy logs to close down all non-work-related
> sites but ...
> Its hopeless.. A month later the stats are filled again with different porn
> sites.

Yep - filtering's not the answer; maintaining your own filter lists would
be like bailing out the Titanic with a collander. Even the commercial
blocking packages do a truly lousy job - they are surprisingly good at
filtering their competitors' sites and anti-censorship sites, but block
many legitimate sites wrongly, and leave access to many "inappropriate"
sites wide open. Finally, any user can bypass the filter completely via
Akamai... and if you block that, you've just blocked off half the big
corporate sites, which is probably not an option for you!

So if you can't filter effectively, what should you do? I'd look into
Squid's "delay pools" facility: this will allow you to restrict user
bandwidth quite effectively, allowing fast "bursts" but throttling big
downloads to conserve bandwidth for other users, and other useful things.

In particular, restricting the bandwidth used by big downloads could make
a big difference: if you throttle all files over, say, 1 Mb, you may find
things become a lot better overall. OK, those NT service packs and Linux
ISOs will take a while to download, but they won't impact "normal" usage
any more.

> And thats just http. Include email in the picture and it will become
> apparent how desperate i really am.. The shit that is send by email is
> unbelievable. Porn, jokes, virusses, hoaxes, ...

Is mail a major part of the problem? Look at the volume of SMTP traffic,
compared to HTTP and other - is it significant? Remember, internal mail
doesn't count, and text is very compact; one person's WWW surfing on a
graphics-heavy site can generate more traffic than a hundred people
forwarding chain letters etc!

Setting your mail server to block some or all attachments could be a big
help here: if you do have a lot of mail traffic, attachments are probably
a large part of that. Check first, though: do your users NEED attachments
for anything? If so, you'll need to make some arrangement for that.
Perhaps block only .EXE files and other "suspect" material?

Beware of driving your users to using a webmail service, though; if you
stop them using your mail system, they could switch to Hotmail or similar.
At which point, even internal mail starts using external bandwidth, and
you're worse off than before...


The first thing to do is look carefully at how your traffic breaks down.
You will probably find the vast majority is HTTP? Next, gather more
information about how it's being used: trawl your Squid logs for big
downloads (ZIP files, ISOs, whatever). How much of your e-mail bandwidth
is going on attachments? Some restrictions there - no .exe files, perhaps
- could go a long way.

Once you know exactly where the problem lies, you can work out the right
measures to cure it: e-mail filtering, Squid delay pools or ACLs,
whatever.


James.