[Techtalk] Performance metrics for email processing

[Devdas Bhagat]

On 13/07/04 16:41 -0400, Terri Oda wrote:
> I'm going to be running performance tests on some email filters.
> 
> A simple "filter A took X minutes, filter B took Y minutes... for Z 
> messages." is all I have in mind at the moment, but what other metrics 
> are useful when it comes to email processing?  Has anyone seen 
> performance test suites they like?  Any potential pitfalls I should 
> know about?  Any good papers that detail ways to do this?

Parameters:

1> Plain text messages vs HTML messages.
2> Message size variation.
3> Attachments.
4> False positives.
5> False negatives.
6> Message modification (strip messages).
7> Quarantine ability.
8> Nested MIME.
9> Broken MIME.
10> Whitelisting ability.
11> Blacklisting ability.
12> The ability to generate a notification if quarantining.

Since you are speaking about filtering server side,

13> The ability for the user to specify preferences.
14> The ability for the filter to run as a dedicated user, which is not
the MTA.
15> The ability for a spam filter to query various DNSBLs, DCC, Razor.
16> Filter concurrency (how many in parallel).
17> CPU loads while filtering.
18> Can the filter run entirely in RAMdisk.
19> Mail loss issues (as opposed to false positives/negatives, this is
the mail getting silently discarded).
20> Can the rules be regexes, perl regexes, sender/source pairs, more
combinations?

Pitfalls:

Your test will be valid for your hardware.
Hardware changes will result in considerable changes in throughput.

Make sure you write the mail output to /dev/null. Writing to disk will
limit your throughput by what the disk can handle.

If you are querying external sources, the network speed will be an
issue.

Filter concurrency will depend if you are running on the server or the
client. On the server, you need to limit concurrency. Your resources are
limited and cannot be overwhelmed by the filter.

If you plan to test a filter running during the SMTP phase (SA-Exim), or
a Sendmail milter, then you need to have those finishing within the smtp
timeouts specified in RFC 2821.

Tools:
For mail testing, I like the smtp-source and smtp-sink tools that
ship with Postfix.

I hope this helps a bit.

Devdas Bhagat