[techtalk] Benchmarking, Bandwidth, server speed

Raven, corporate courtesan damask0 at yahoo.com
Tue May 15 19:11:40 EST 2001


Heya --

Quoth Michelle:
> I'm trying to solve a mystery, and at the same time, figure out good
> metrics to determine our server speed, network speed, capacity, etc.

     Hoo boy.  Good idea, but these are three separate issues, so
you're going to have to troubleshoot and benchmark each one separately.

> One database-driven page in particular loads *extremely* slowly
> (like, minutes), from several computers in the client's office. I've 
> NEVER had the page load anywhere near that slowly from within our 
> network (where the server resides) or remotely (like at home, which 
> is on a cable modem connection -  but via a totally different network

> than the DSL that work is on). The client has a high-speed 
> connection. 

     These differences in load times are using the same OS, same
browser version?  That can make a big difference right there.

     Assuming that they are the same, you may have a routing problem. 
Have you run pings and traceroutes in both directions between your
server and the client's PCs?  Keep an eye on the latency time there. 
The number of hops is irrelevant -- far better to take three OC-48 hops
than one T1 hop.  [grin]  But keep an eye on those return times.  300
ms is about the most you should be seeing for a cross-country trip, end
to end.  It's possible that your data is going through a highly
congested link between providers, and that this is causing your
problem.  If so, this will show up on the traceroute.

> (the strange part is that it appears that it is one page in 
> particular, and that there are other pages that have just as much 
> info, but load much faster - I'm not sure what's up there.)

     Just one page on your server that's slower, and all the other
pages on your server of comparable size are quick?  If that's the case,
it's probably not a routing issue at all.  On the other hand, if your
server is slow and comparably sized pages at other sites are fast, it
may be routing or the server.  If it appears to be just that one page
on your server and all the others load much more quickly, try to
determine whether the latency is caused by the HTML or by your database
programs.  Debug your code, run it step by step with the same data the
client is entering, and see where it's hanging.

> We need, in general to understand how good our
> server/network/internet connection is working.

     There are several ways to do this.  How much bandwidth have you
got?  One line, or are you multihomed?

> We're running mrtg, and find that theoretically, our bandwidth is
> never anywhere near the theoretical capacity that we pay for. 

     MRTG will just show you actual usage.  It won't show you
theoretical maximums.  But there's a lot you can learn from MRTG.  Is
your bandwidth usage bursty (lots of peaks and valleys) or constant? 
Does it max out at one point in the day (say, during a remote database
sync) or not?  You can often match bandwidth usage peaks to network
events.

     The type of high-speed line you've got will dictate its behaviour
in allocating bandwidth.  Frame Relay and some sorts of ATM are
generally sold with a peak burst rate, and with an average rate called
a CIR (Committed Information Rate).  If you burst above your peak rate,
your packets may be dropped until you're back below it, causing
latency, TCP retransmissions, etc.  This happens even if you are
generally using far less than your CIR.  But from what you've said, it
sounds like you're not anywhere near your peak usage.

     T1s and T3s are less flexible.  You get 1.5 Mb and 44.5 Mbs
respectively.  There's no bursting, but you should always be able to
get up to your bandwidth limit on the circuit.  And if you have an OC-3
or better, you probably have your own network staff to do this kind of
stuff for you.  [grin]

     There are a few different ways to test your throughput.  Many ISPs
will have test files that you can FTP to and from them to use up all
the available bandwidth.  Or, you can try to pingflood your line 
(ping -f on many *nix systems will do this), or you can have the ISP
try to pingflood you if testing like that is in your service contract
with them.  There are many OSs that limit the bandwidth that any one
process can be using, so sometimes it's necessary to open many
simultaneous FTP connections to try to max out the line.  If you go the
pingflood route, make sure you try it with 1500 byte packets as well as
with standard pings.  Using larger packets will simulate live traffic
and catch path-MTU problems that tiny normal 64-byte pings will not. 

     It may seem obvious, but just in case -- pingflooding the line is
using up all your bandwidth.  While you are doing this, your users will
not be able to get to the Internet.  (Or if they do, it will be very
slow.)  It's best to do testing like this in 3 AM maintainence windows,
so as not to piss off your clients.

     Also, there are switches and routers that do Quality of Service
stuff, such that they'll rate-limit any one host to a fraction of the
bandwidth.  This can mislead you when you're running benchmark testing.
 To test the whole line, I prefer to plug in a *nix laptop with a
crossover cable directly into the Ethernet port of the router.  That
way, you're localizing the problem to WAN or LAN.  There is no
pingflood option in Windows that I know of, so you'd have to go the
multiple FTP route if that's all you've got at the client site.

> First, are there programs which can give you the "cieling" bandwidth
> of your network - the real speed that you've got?

     Pingflood testing with 1500 byte packets has been the most
reliable for me.

> I'm learning about benchmarking perl code (which is the back-end
> programming language that we use) and I know that there are 
> definitely things I need to do to improve speed there.

     Cool.  That will probably help unless your bottleneck is the
network.

> How can you test the capacity of a server to handle concurrent
> sessions, etc.? (BTW, in general, we run Debian Linux, Postgres, 
> apache and perl)

     There are a good number of benchmarking programs out there.  ab
may already be on your system.  It's the Apache benchmarking program --
try which ab to see if you've got it and man ab for its usage, or read
http://httpd.apache.org/docs-2.0/programs/ab.html  It can handle http
authentication, proxy authentication, cookies, extra headers and
attributes, all sorts of stuff.  Don't use that WebBench crap; it was
originally designed to only test with a 60k text file.  Its results
bear little resemblance to the genuine load on most servers.

     Hope that helps, and feel free to hit me with questions.

Cheers,
Raven

=====
"So what sort of presentation software are you using?"
"Ummm, vi."
"Oh.  Is there a conversion program for that?  I can't read 
 these files in NT.  Maybe you should use Windows."
 -- a recent conversation with our new technical management

__________________________________________________
Do You Yahoo!?
Yahoo! Auctions - buy the things you want at great prices
http://auctions.yahoo.com/




More information about the Techtalk mailing list