If we continuously process large files there are a few considerations we need to take into account. The first is to make sure that the server has the processing power and RAM to run the DW gateway. In my experience it takes more power to run them than a normal server. Because of the methods used to overcome the write completion and file locks, we could end up with a lot of sleeping/running threads at one time. You can't control the number of running DW threads...or can you? This is a topic for another time but suffice it to say, it is possible. (I can hear the e-mails coming now...)

I've said a few times already that error handling is very important in a DW gateway. This can't be overemphasized, and finding out what went wrong is also important. While preventing errors is straightforward, doing something with the error isn't. First, we'll focus on preventing errors then we can focus on displaying them.

For example, in Listing 2, I used a try/catch to trap an error that may occur. Now, going one step further I can put the whole run in a try/catch. Using try/catch is probably the best thing to do to prevent a gateway from failing. The other side of the coin is to figure out what to do when something does go wrong. This is really up to the specific occurrence of the gateway.

One thing that I've done is set up an onError event in my application.cfm for the gateway CFCs. This lets me e-mail any errors that go untrapped to myself or others. I also write the error to an error log specific to the gateway that was running. Normally we can do a CFDUMP in CF code to see what's happening. With a gateway this isn't possible. I solve this by setting up a debug function to get information out while I am working on the CFC. This takes an input and dumps it to a file that I can call from a Web browser. This is great for trying to find out what's happening inside the CFC.

We must also keep an eye on the file space used for a DW gateway. If there are failures, the failed file must be cleaned up, otherwise we could end up using lots of disk space. I handled this by creating a scheduled process which deletes files that are a day old. Your situation may call for some variation on this theme.

There are some other tricks we can do to add more robust functionality to the DW gateway.

We can have a file exclusion list based on the file extension. If the file's extension is on the list, we can delete it or handle it some other way. However, if we're going to delete it we have to make sure it's in a state that allows deletion. We could just add the code from Listing 4 to the end of the other code examples to do this.

We can also build a function to attempt to correct file naming issues, for example a file that starts with a period. This is common if a user on a Mac uploads files. In this situation we'd want to delete the file. We can also make sure the file has a valid extension and that the file name doesn't contain any invalid characters.

The DW gateway is a very powerful tool, if a bit unrestrained. With it you can do a multitude of things, both good and bad. We've seen how it lets us kick off processes that may not be able to handle the very files that are monitored by the gateway. Fortunately, we can create code that can tame DW's uncivilized tendencies and reliably manage transferred files coming from both FTP and network transfers. Hopefully this article gave you some tools to build a better DW gateway and make you a hero in your shop, too.

Thanks for the great article, Dave. I'm always glad to hear from folks "in the trenches." For you readers, if you have a war story you'd like to see printed in the pages of the Journal, drop me an e-mail at jeff@grokfusebox.com.