FETCHMAIL 3.3 Fetchmail is a full-featured, robust, well-documented POP2, POP3, APOP, and IMAP batch mail retrieval/forwarding utility intended to be used over on-demand TCP/IP links (such as SLIP or PPP connections). It retrieves mail from remote mail servers and forwards it to your local (client) machine's delivery system, so it can then be be read by normal mail user agents such as elm(1) or Mail(1). Installation: Unpack in /usr/local or equivalent. I strongly recommend that your first fetchmail run use the -v and -k options, in case there is something not quite right with your server, your local delivery configuration or your port 25 listener. Also, beware of aliases that direct your local mail back to the server host! This software is known to work with the qpop/popper series of freeware POP3 servers; also with the IMAP2bis and IMAP4 servers that are distributed with Pine from the University of Washington. FAQ: Frequently Asked Questions About Fetchmail Before reporting any bug, please read G1 for advice on how to include diagnostic information that will get your bug fixed as quickly as possible. If you have a question you think needs to be added to this list, mail it to esr@thyrsus.com. General questions: G1. I think I've found a bug. Will you fix it? G2. I have this idea for a neat feature. Will you add it? Build-time problems: B1. My C compiler libraries don't seem to have atexit(). Configuration questions: C1. Why do I need a .fetchmailrc when running as root on my own machine? C2. Why does my .fetchmailrc from 2.8 or earlier no longer work? C3. How can I arrange for a fetchmail daemon to get killed when I log out? C4. How do I know what interface and address to use with --interface? C5. How can I get fetchmail to work with ssh? C6. The .fetchmailrc parser won't accept my all-numeric user name. C7. How can I set up support for sendmail's anti-spam 571 response? Runtime problems (all installations): R1. I think I've set up fetchmail correctly, but I'm not getting any mail. R2. Fetchmail isn't working, and -v shows `SMTP connect failed' messages. R3. Why is fetched mail being logged with my name, not the real From address? R4. Spurious blank lines are appearing in the headers of fetched mail. R5. Fetchmail dumps core in -V mode, but operates normally otherwise. Runtime problems (multidrop only): M1. I tried to run a mailing list using multidrop, and I have a mail loop! M2. My multidrop fetchmail seems to be having DNS problems. M3. I'm seeing long DNS delays before each message is processed. ----------------------------------------------------------------------- G1. I think I've found a bug. Will you fix it? Yes I will, provided you include enough diagnostic information for me to go on. When reporting bugs, please include the following: 1. Your operating system and compiler version. 2. The release and patch level of the fetchmail you are running. You can see your patchlevel by typing `fetchmail -V'. 3. The output of fetchmail -V (this will not reveal your password). 4. Any command-line options you used. It is helpful if you include your .fetchmailrc, but not necessary unless your symptom seems to involve an error in configuration parsing. A transcript of the failed session with -v on is almost always useful. If the bug involves a core dump or hang, a gdb stack trace is good to have. (Bear in mind that you can attach gdb to a running but hung process by giving the process ID as a second argument.) Best of all is a mail file which, when fetched, will reproduce the bug. ----------------------------------------------------------------------- G2. I have this idea for a neat feature. Will you add it? Probably not. Most of the feature suggestions I get are for ways to set various kinds of administrative policy or add more spam filtering (the most common one, which I seem to get about four million times a week and am *really* tired of, is for tin-like kill files). You can do spam filtering better with procmail or mailagent on the server side and (if you're the server sysadmin) sendmail.cf domain exclusions. You can do other policy things better with the MDA option and script wrappers around fetchmail. If it's a prime-time-vs.-non-prime-time issue, ask yourself whether a wrapper script called from crontab would do the job. I'm not going to do these; fetchmail's job is transport, not policy, and I refuse to change it from doing one thing well to attempting two things badly. One of my objectives is to keep fetchmail simple so it stays reliable. All that said, if you have a feature idea that really is about a transport problem that can't be handled anywhere but fetchmail, lay it on me. I'm very accommodating about good ideas. ----------------------------------------------------------------------- B1. My C compiler libraries don't seem to have atexit(). Your compiler libraries are deficient (this has been reported from a bunch of older Solaris and SCO boxes). The atexit(3) function is part of the ANSI C standard and should be there. You may be able to find a linkable object for atexit(3) in your C++ library. If you can't, atexit(3) source is easily obtained. You need both atexit(3) and exit(3), since the latter provides necessary support for the former. Glenn E. Thobe tells us he found source code for both in directory /cdrom/usr/src/lib/libc/stdlib on his FreeBSD disc (BSD 4.4-lite). ----------------------------------------------------------------------- C1. Why do I need a .fetchmailrc when running as root on my own machine? Ian T. Zimmerman asked: On the machine where I'm the only real user, I run fetchmail as root from a cron job, like this: fetchmail -u "itz" -p POP3 -s bolero.rahul.net This used to work as is (with no .fetchmailrc file in root's home directory) with the last version I had (1.7 or 1.8, I don't remember). But with 2.0, it RECPs all mail to the local root user, unless I create a .fetchmailrc in root's home directory containing: skip bolero.rahul.net proto POP3 user itz is itz It won't work if the second line is just "user itz". This is silly. It seems fetchmail decides to RECP the `default local user' (ie. the uid running fetchmail) unless there are local aliases, and the `default' aliases (itz->itz) don't count. They should. Answer: No they shouldn't. I thought about this for a while, and I don't much like the conclusion I reached, but it's unavoidable. The problem is that fetchmail has no way to know, in general, that a local user `itz' actually exists. "Ah!" you say, "Why doesn't it check the password file to see if the remote name matches a local one?" Well, there are two reasons. One: it's not always possible. Suppose you have an SMTP host declared that's not the machine fetchmail is running on? You lose. Two: How do you know server itz and SMTP-host itz are the same person? They might not be, and fetchmail shouldn't assume they are unless local-itz can explicitly produce credentials to prove it (that is, the server-itz password in local-itz's .fetchmailrc file.). Once you start running down possible failure modes and thinking about ways to tinker with the mapping rules, you'll quickly find that all the alternatives to the present default are worse or unacceptably more complicated or both. ----------------------------------------------------------------------- C2. Why does my .fetchmailrc from 2.8 or earlier no longer work? The `interface', `monitor' and `batchlimit' options have changed. They used to be global options with `set' syntax like the batchlimit and logfile options. Now they're per-server options, like `protocol'. If you had something like set interface = "sl0/10.0.2.15" in your .fetchmailrc file, simply delete that line and insert `interface sl0/10.0.2.15' in the server options part of your `defaults' declaration. Do similarly for any `monitor' or `batchlimit' options. ----------------------------------------------------------------------- C3. How can I arrange for a fetchmail daemon to get killed when I log out? Fetchmail versions before 2.3 actually used SIGHUP as a wakeup signal. Newer versions use SIGUSR1 for wakeup and ignore SIGHUP entirely in order to avoid any potenntial confusion about logout-time behavior. The right way to dispatch fetchmail on logout is to arrange for the command `fetchmail -q' to be called on logout. Under bash, you can arrange this by putting `fetchmail -q' in the file `~/.bash_logout'. Most csh variants execute `~/.logout' on logout. For other shells, consult your shell manual page. ----------------------------------------------------------------------- C4. How do I know what interface and address to use with --interface? This depends a lot on your local networking configuration (and right now you can't use it at all except under Linux). However, here are some important rules of thumb that can help. If they don't work, ask your local sysop or your Internet provider. First, you may not need to use --interface at all. If your machine only ever does SLIP or PPP to one provider, it's almost certainly by a point to point modem connection to your provider's local subnet that's pretty secure against snooping (unless someone can tap your phone or the provider's local subnet!). Under these circumstances, specifying an interface address is fairly pointless. What the option is really for is sites that use more than one provider. Under these circumstances, typically one of your provider IP addresses is your mailserver (reachable fairly securely via the modem and provider's subnet) but the others might ship your packets (including your password) over unknown portions of the general Internet that could be vulnerable to snooping. What you'll use --interface for is to make sure your password only goes over the one secure links. To determine the device: 1. If you're using a SLIP link, the correct device is probably sl0. 2. If you're using a PPP link, the correct device is probably ppp0. 3. If you're using a direct connection over a local network such as an ethernet, use the command `netstat -r' to look at your routing table. Try to match your mailserver name to a destination entry; if you don't see it in the first column, use the `default' entry. The device name will be in the rightmost column. To determine the address and netmask: 4. If you're talking to slirp, the correct address is probably 10.0.2.15, with no netmask specified. (It's possible to configure slirp to present other addresses, but that's the default.) 5. If you have a static IP address, run `ifconfig ', where is whichever one you've determined. Use the IP address given after "inet addr:". That is the IP address for your end of the link, and is what you need. You won't need to specify a netmask. 6. If you have a dynamic IP address, your connection IP will vary randomly over some given range (that is, some number of the least significant bits change from connection to connection). You need to declare an address with the variable bits zero and a complementary netmask that sets the range. To illustrate the rule for dynamic IP addresses, let's suppose you're hooked up via SLIP and your IP provider tells you that the dynamic address pool is 255 addresses ranging from 205.164.136.1 to 205.164.136.255. Then interface "sl0/205.164.136.0/255.255.255.0" would work. To range over any value of the last two octets (65536 addresses) you would use interface "sl0/205.164.0.0/255.255.0.0" ----------------------------------------------------------------------- C5. How can I get fetchmail to work with ssh? This is a lightly edited version of a recipe from Masafumi NAKANE. 1. You must have ssh (the ssh client) on the local host and sshd (ssh server) on the remote mail server. And, you have to configure ssh so you can login to the sshd server host without a password. (Refer to ssh man page for several authentication methods.) 2. Add something like following to your .fetchmailrc file: poll localhost port 1234 with pop3: preconnect "ssh -f -L 1234:mailhost:110 mailhost sleep 20 /dev/null"; (Note that 1234 can be an arbitrary port number. Privileged ports can be specified only by root.) The effect of this ssh command is to forward connections made to localhost port 1234 (in above example) to mailhost's 110. This configuration will enable secure mail transfer. All the conversation between fetchmail and remote pop server will be encrypted. If sshd is not running on the remote mail server, you can specify intermediate host running it. If you do this, however, communication between the machine running sshd and the POP server will not be encrypted. And the preconnect line would be like this: preconnect "ssh -f -L 1234:mailhost:110 sshdhost sleep 20 /dev/null" You can work this trick with IMAP too, but the port number 110 in the above would need to become 143. ----------------------------------------------------------------------- C6. The .fetchmailrc parser won't accept my all-numeric user name. So put string quotes around it. :-) ----------------------------------------------------------------------- C7. How can I set up support for sendmail's anti-spam 571 response? Rachel Polanskis writes: Basically you need to use the "check_*" rules in sendmail. These are rules introduced since version 8.8.2 The idea is to generate a list of domains and addresses that are placed into a file - I call mine "sendmail.rej" and you place just one domain or email address on each line. During the SMTP transaction, this file is checked and if there is a match, the message is refused, with a suitable "Service not available" message sent back to the sender. With the feature enabled in fetchmail, the mail is simply deleted, with no further processing. The only drawback when blocking spam with fetchmail is that you do not get the satisfaction of sending an error back to the sender. To actually use the check_mail rules in sendmail 8.8.2 or better, you need to know how to generate a sendmail.cf file from the m4 config files distributed with sendmail. The actual rules can be found at the following URLS: http://www.informatik.uni-kiel.de/%7Eca/email/check.html By Claus Assman, who has documented more of sendmail then I can digest! The actual setup I used though was by David Begley, who has put together a WWW page describing how to quickly implement these rules yourself. http://www.nepean.uws.edu.au/users/david/pe/blockmail.html David's pages could be moving shortly. I will post an update if it happens. Remember, when copying these rulesets off the web, that there are tabs embedded in them, that may not be preserved. You *must* reintroduce these tabs into the rules to make them work properly. Once you have your ruleset in place, and have generated a nice sendmail.cf file, and the list of blocked sites, try telneting to your SMTP port to test it, and send a message with a blocked address in it. You should see a message similar to: "571 unsolicited email is refused" Next, if you have access to a host that you can send mail from, that is *not* your mail host, add that host to your spamlist and restart sendmail. Send a message to your mailing address from that host and then pop off the message with fetchmail, using the -v argument. You can monitor the SMTP transaction, and when the FROM address is parsed, if sendmail sees that it is an address in spamlist, fetchmail will flush and delete it. Under no circumstances put your *mailhost* or any host you accept mail from using fetchmail into your reject file. You *will* lose mail!!! The check_ rules work, and they work well. Coupled with fetchmail's ability to respond to the appropriate error messages, you can be assured of never seeing a spam from any address you put in the reject list. The only thing that is missing, as mentioned previously, is the ability to allow sendmail to process the message further and generate an error message to the sender. ----------------------------------------------------------------------- R1. I think I've set up fetchmail correctly, but I'm not getting any mail. Maybe you have a .forward set up that you've forgotten about. You should probably remove it. Or maybe you're trying to run fetchmail in multidrop mode as root without a .fetchmailrc file. This doesn't do what you think it should; see question C1. Or you may not be connecting to the SMTP listener. Run fetchmail -v and see the next question. ----------------------------------------------------------------------- R2. Fetchmail isn't working, and -v shows `SMTP connect failed' messages. Fetchmail is working, but your SMTP port 25 listener is down or inaccessible. The first thing to check is if you can telnet to port 25 and get a greeting line from the listener. If the listener is down, bring it back up. If the listener seems to be up when you test with telnet, it could have had a momentary problem due to resource exhaustion (process table full or some other problem that stopped the listener process from forking). If your SMTP host is not `localhost' or something else in /etc/hosts, the glitch could also have been caused by transient nameserver failure, or the SMTP host's actually being down. If the listener tests up, you can usually ignore the glitch (except as a symptom of other problems) because a future fetchmail run will get the mail through. If this is a recurring or constant failure mode, OTOH, you may have more serious problems in your network layer which I can't diagnose in this FAQ. One way to work around chronic SMTP connect problems is to use --mda. But this only attacks the symptom. You should really try to figure out what's going on underneath before it bites some other way. We have one report from a Linux user of 2.1 who solved his SMTP connection problem by removing the reference to -lresolv from his link line and relinking. Apparently in some recent Linux distributions the libc bind library version works better. As of 2.2, the configure script has been hacked so the bind library is linked only if it is actually needed. So under Linux it won't be, and this particular cause should go away. ----------------------------------------------------------------------- R3. Why is fetched mail being logged with my name, not the real From address? Because logging is done based on the address indicated by SMTP MAIL FROM, and some listeners are picky about that address. Some SMTP listeners get upset if you try to hand them a MAIL FROM address naming a different host than the originating site for your connection. This is a feature, not a bug -- it's supposed to help prevent people from forging mail with a bogus origin site. Since the originating site of a fetchmail delivery connection is localhost, this effectively means these picky listeners will barf on any MAIL FROM address fetchmail hands them with an @ in it! In versions up to 1.9.9 this led to pesky errors at some sites. Because of this, I hacked 2.0 to just use the calling user ID as the MAIL FROM address. Versions 2.1 and up try the header From address first and fall back to the calling-user ID. So if your server isn't picky, the log will look right. ----------------------------------------------------------------------- R4. Spurious blank lines are appearing in the headers of fetched mail. What's probably happening is that the POP/IMAP daemon on your mailserver is inserting a non-RFC822 header (like X-POP3-Rcpt:) and something in your delivery path (most likely an old version of the `deliver' program, which sendmail often calls to do local delivery) is failing to recognize it as a header. This is not fetchmail's problem. The first thing to try is installing a current version of deliver. If this doesn't work, try to figure out which other program in your mail path is inserting the blank line and replace that. If you can't do either of these things, pick a different MDA (such as procmail) and declare it with the `mda' option. ----------------------------------------------------------------------- R5. Fetchmail dumps core in -V mode, but operates normally otherwise. We've had this reported to us under Linux using libc-5.4.17 and gcc-2.7.2. It does not occur with libc-5.3.12 or earlier versions. Workaround: link with GNU malloc rather than the stock C library malloc. We're told there is some problem with the malloc() code in that version which makes it fragile in the presence of multiple free() calls on the same pointer (the malloc arena gets corrupted). Unfortunately it appears from doing gdb traces that whatever free() calls producing the problem are being made by the C library itself, not the fetchmail code (they're all from within fclose, an not an fclose called by fetchmail, either). ----------------------------------------------------------------------- M1. I tried to run a mailing list using multidrop, and I have a mail loop! This isn't fetchmail's fault. Check your mailing list. If the list expansion includes yourself or anybody else at your mailserver (that is, not on the client side) you've created a mail loop. Just chop the host part off any local addresses in the list. If you use sendmail, you can check the list expansion with sendmail -bv. ----------------------------------------------------------------------- M2. My multidrop fetchmail seems to be having DNS problems. We have one report from a Linux user (not the same one as in R2!) who solved this problem by removing the reference to -lresolv from his link line and relinking. Apparently in some recent Linux distributions the libc bind library version works better. As of 2.2, the configure script has been hacked so the bind library is linked only if it is actually needed. So under Linux it won't be, and this problem should go away. ----------------------------------------------------------------------- M3. I'm seeing long DNS delays before each message is processed. Use the aka option to pre-declare as many of your mailserver's DNS names as you can. When an address's host part matches an aka name, no DNS lookup needs to be done to check it. Sometimes delays are unavoidable. Some SMTP listeners try to call DNS on the From-address hostname as a way of checking that the address is valid. ----------------------------------------------------------------------- Jason R. Mastaler