One of the reasons I was first drawn to ColdFusion was the built-in functionality for such things as sending e-mail, making HTTP requests, FTP uploads - all the myriad subsidiary functions you inevitably find yourself using when you build and manage large sites.

Content has to be downloaded from here, uploaded to there, and e-mailed to thousands of users every day.

Using only standard ColdFusion functionality keeps things neat and easy to support. And for all the warnings about not running an SMTP server on the same machine as your Web server, I'm sure many developers have a site running somewhere that's doing exactly that and working extremely well. I also suspect many developers have pushed things a little too far and found that nothing soaks up resources quite like an overloaded mail server.

Even if you do run an SMTP server on a separate machine, large volumes of e-mail can still have a catastrophic effect on performance. CFMAIL is a great tag and, unlike other third-party tags such as CFFTP or CFPOP, it probably won't result in a "hanging thread." However, CFMAIL writes only text files to a spool, which then has to be sent to an SMTP server, and this spooling uses up resources. There comes a point when you not only want the SMTP service to be running on a separate machine, but the spooling as well.

There's a lot of discussion about the problems with CFMAIL in the support forums, but almost all the problems are not due to the CF-MAIL tag, which works very well, but to this secondary spooling process. It's this process that leaks file handles that can eventually bring the ColdFusion service to a stop (fixed in version 5), and it can also fail when you specify different SMTP servers in the same batch of mails.

As good as CFMAIL is, there are several occasions when you need functionality that it doesn't have. First, it can't prioritize mail, so if someone requests a password reminder while your system is sending out 2,000 newsletters, they'll have to wait. Second, it can send only text or HTML mail without the option of a text "failover" for HTML. Third, it can't embed images as inline MIME attachments.

This means that people who download their mail before reading offline won't see the image or might be annoyingly prompted to reconnect. Whether or not this is a serious problem is one of those difficult calls that has no correct answer. If you want everyone to see your mail perfectly, you need to attach inline images. If you're confident that 99% of your subscribers read your mail at corporate workstations, then spread your bandwidth load and possibly track e-mail reads by using remote images.

The lack of a text failover is a more serious problem. Although almost all mail clients can now handle HTML, some people choose to turn this functionality off. Including a "Can't read this?" link at the top of the page is no real substitute for a text alternative.

There are commercial alternatives to CFMAIL (COM and Java), but there's a simple and quick way to get around these issues. It's relatively straightforward to bypass CFMAIL and write your e-mails directly to your mail server's outbound queue folder. This allows you to use inline images and failovers, but far more important, also allows you to separate Web serving and mail spooling applications by writing mail first to a database and then to a mail queue.

That might sound like a daunting prospect, but a basic e-mail couldn't be much simpler. It needs only from, to, and subject fields and some body copy. Try saving a file like this to your mail server's outgoing queue folder.

From: tom <tom@tompeerconsulting.com>
To: test <test@tompeerconsulting.com>
Subject: Testing

Body text goes here

For the Microsoft SMTP service, this is a folder called "pickup" within the mail root (usually C:\inetpub\mailroot). Bad mails are moved to "badmail" in the same folder. If you use the ever popular MDaemon mail server (www.deerfield.com), save to the REMOTEQ folder or to a custom remote queue folder you've set up. Many other systems work in exactly the same way - you just have to find the location of the outbound queue.

If you're not interested in HTML and inline images, you already have your solution to bulk mailing - simply writing text files like this to an outgoing queue folder. It's easily scalable to more than one server by either writing to several servers from a central "broker" or else running code on each mail server to fetch batches of mail from a database at regular intervals. The former is easier (and probably cheaper); the latter more powerful and necessary when you're sending huge volumes of mail (see Figure 1).

Using the simpler first approach you can probably avoid keeping a permanent record in the database of the e-mails you're sending. For instance, if you have a newsletter to send to a list of subscribers in a table called USERS, you could select every row of USERS from the database, loop over it, and write a text file for each e-mail.

Suppose, though, something goes wrong mid-mailing. If a mail server or the broker machine fails, how do you pick up where you left off? You can either send the whole mailing again or leave out the people that didn't get it.

A better arrangement is some sort of permanent record; for example, a table of mailings in which the EMAIL_STATUS_ID indicates whether an e-mail has been sent (see Figure 2). Alternatively, you can omit this field and delete from the EMAIL_RECIPIENTS table when the mail has been sent. Using SQL Server you can then insert into the "queue" with a procedure like this:

CREATE PROCEDURE SEND_NEWSLETTER
(@EMAIL_ID INT)
AS INSERT INTO EMAIL_RECIPIENTS
(USERS_ID, EMAIL_ID, EMAIL_STATUS_ID)
SELECT USERS_ID, @EMAIL_ID, 1 FROM USERS

and fetch the mail in batches with another procedure (see Listing 1).

This allows you to use the database as a broker and keeps the level of functionality needed for the mail servers in the second scenario to a minimum - a quick substitution of the important database fields (e.g., e-mail, first_name, last_name) and a file write to the outbound queue.

If you want to send out more than simple text messages, either immerse yourself in the RFC-822 standard or else use one of the solutions available in the Developers' Exchange. CF_ADVANCEDEMAIL by Jochem van Dieten is a great tag; it takes care of text failovers and inline images, but you'll need to give it a couple of tweaks if you want to use it to write directly to an outgoing queue and not the ColdFusion spool (an annotated version is available from my Web site at www.tompeerconsulting.com). The syntax is straightforward (see Listing 2), and creating inline images couldn't be simpler. Just embed them with full URLs in your e-mail and the tag will download, encode, and attach them to the mail. Rather than call this tag every time you write a text file for a bulk mailing, a simple customization will allow you to call it once and then use the formatted result as a template for writing future e-mails.

If the problem of mail priorities is an issue on your site, add a priority field to your database table of e-mails, then write higher-priority mails to a dedicated server or a separate SMTP instance running on a different port. You may even decide to write high- priority mails to an MDaemon or equivalent mail server queue, where they can be monitored, and send less important e-mails directly via SMTP.

If, like me, you develop on Windows but rather envy Linux users their price-to-performance ratios, by separating mail sending from Web serving you can quite happily use different platforms for the two tasks. A well-specified Windows server might struggle to send out 3,000 mails an hour while a Linux system costing half as much running just Sendmail might send out 5,000 or 6,000 an hour.

Whatever platform or e-mail server you choose, keeping your mail spooling separate will enable you to scale your application simply by adding more servers, especially if you opt for the second scenario described earlier. On one project we were able to send out about 250,000 e-mails a day from three discarded office PCs running Red Hat Linux - saving thousands from a third-party contract - while operating under a budget freeze.

Finally some words of warning before you get started. Make sure you secure your SMTP servers against relaying. Be careful you don't use up all your bandwidth sending outgoing mail. Do test your mail on clients like Eudora, Netscape, and Lotus - there are more older e-mail clients than older browsers, and Outlook tolerates a great deal that they don't. If you're sending bulk mailings, make sure you comply with all relevant legislation, include an unsubscribe mechanism (URL or e-mail address), and make it clear how you came by the subscriber's details.

• Figure 1
• Figure 2
• Source Code