NYCkayaker Mail policy changes here (long, geeky and boring)

[Rich Kulawiec]

[ Please direct any comments, questions, concerns, brickbats, bouquets,
or non-consecutive unmarked twenty-dollar bills to our "postmaster"
address and whoever here is least psychotic at the time will answer it. ]



We're going to implement some mail policy changes during March 2007.

I'll try to make this explanation as brief and as non-technical as
possible.  But this is still going to be lengthy.  Sorry.  But all
of us here think it's important to tell you what we're doing because
we may well need your help detecting and correcting mistakes.

Summary
-------

We're going to enforce a number of mail system requirements that
have been "on the books", so to speak, for 20+ years, or which are
widely considered best practices.  Most of these aren't anti-spam
measures, per se, but "sanity checks".   So while they'll help us
block a lot of spam, they may also end up blocking mail from
incorrectly configured (but non-spamming) mail systems...and
that'll need be dealt with on a case-by-case basis.


Rationale
---------

Spam levels went up 300% in the last quarter of 2006.  This is
not an isolated occurence: everyone else who's paying attention
is seeing the same thing.  Our incoming mail stream is now about 1-2%
real traffic, 98-99% junk (including spam, viruses, worms, etc.).

This problem is being exacerbated -- badly -- by lots of people who
don't really know how to run Internet-connected mail systems and/or who
are busy deploying ill-conceived "anti-spam" measures that actually
make the problem worse.

	[ By the way, here is the one-question certification test
	you can use to find out if the person(s) running YOUR mail
	system has/have any clue at all.  Ask them:

	"What mailbox address is required by RFC 821?"

	If they ask "what's RFC 821?" then they should be relieved
	of mail-related duties on the spot.  If they answer "postmaster"
	then they have at least some clue.  If they point out that
	RFC 821 has been supplanted by RFC 2821, then they have way
	more clue than most.  And if they also tell you that they've
	taken pains to make sure that "postmaster" doesn't use
	spam/virus-filtering because they wish to avoid a listing at
	rfc-ignorant.org, then I'd like to see a copy of their resume. ;=) ]

Still worse, many of those people have refused to fix their problems.
They've adopted a "it works for me" attitude, completely contrary to
the spirit of cooperation on which the Internet is architected.
This is the network equivalent of saying "it works for me...to throw
my trash in your river, because you can always pick it up for me".

We're not going to go along with this -- that is, we're not going
to allow them to make *their* problem *our* problem.  We run our
systems and networks properly, and expect the same of others.  We're
not required to continue extending mail privileges to those who can't
or won't.

To put it another way, we're not required to subsidize the costs
of others' incompetence.  (We have our hands full dealing with
our own occasional lapses!)

A result of this may be that some of you might have trouble sending
mail here.  If that happens, we'll do what we can to address the
problem, but please recognize that (barring a mistake by us) ultimate
responsibility for that rests solely with YOUR mail system's
administrators, and so it's there that your unhappiness/complaints/etc.
should be directed.  If they can't or won't do their jobs properly,
then perhaps you should consider (as appropriate) replacing them,
getting your mail service elsewhere, or other alternate arrangements.

	[ It might be tempting to try "the easy way out", proceeding
	on the assumption that we're just being exceptionally picky.
	We're not.  In fact, we've stuck with the long-standing Internet
	credo "be conservative in what you send, be liberal about what
	you accept" a lot longer than many.  We're just being more
	forthcoming than most about what we're doing and why.

	So if you have trouble sending mail here today, you will most
	likely have it sending to another site tomorrow and more sites
	the day after that, until the actual underlying problem(s)
	is/are fixed.  Evasion is not a viable long-term approach;
	if the problem is with the operation of YOUR mail server, then
	that's what needs to be fixed. ]


Changes we're going to make
---------------------------

1. Any host which attempts to deliver mail here and doesn't have
functioning reverse DNS will be refused.  Explanation: all Internet
hosts have (at least) one name, e.g. "mail22.example.com",
and (at least) one IP address, e.g. "10.16.0.49".  Any host that
has an IP address that doesn't map back to a name (which is what
reverse DNS does) doesn't comply with the basic requirements for
mail service, and won't be given any.

2. Any host which attempts to deliver mail here and doesn't have
reverse DNS that maps back to a valid hostname will be refused.
Explanation: if we look up 10.16.0.49 and find that it maps to
mail22.yahoo.com, then that's okay, because ".com" is a valid
top-level domain, "yahoo.com" exists, and so on.  But if it
maps back to "hn.kd.dhcp", that's not okay because there is no
such top-level domain as "dhcp".  Or if it maps back to
"frobblegazzer.net" and "frobblegazzer.net" does not exist --
that's wrong.

3. Any host which attempts to deliver mail here and has reverse
DNS that maps back to a "generic" hostname will LIKELY to be refused.
I say "likely" because we're using a database of about 13,500 such
patterns but it doesn't cover everything.  Explanation: if 10.16.0.49
maps back to mail22.example.com, that's non-generic: "mail22" looks
like a hostname that belongs to a mail server.  But if it maps
back to pool-141-153-179-27.mad.east.verizon.net, that IS generic,
and it's part of a pattern of (so far) 20,294 hosts that have tried
to deliver spam here that all have names pool-SOMETHING-verizon.net,
and we're not going to accept mail from it.

4. Any host which attempts to deliver mail here and does not say HELO
with a proper hostname will be refused.  Explanation: when mail servers
connect to each other, the sending one is required to identify itself
by saying something like "HELO mail22.example.com".  The hostname
provided must be valid and must exist in DNS -- so if it says
"HELO frobblegazzer.net" and there is no such domain: no soup for you.

5. Any host which attempts to deliver mail here and doesn't wait for
our side to send the SMTP greeting will be LIKELY be refused.  I say
"likely" because some rather well-known hosts are broken (e.g. Gmail)
and we'll need to make exceptions for them.  Explanation:  when
mail servers connect to each other, the sending one is required to
wait for the receiving one to identify itself before proceeding.
(In the same way that it's customary for a phone caller to wait for
the callee to say "hello" before they start jabbering.)  Spam senders
frequently don't bother because it'd slow them down.

6. Any sender address whose domain part resolves to a bad MX record
will be rejected.  In English: suppose we're handed a mail message
that claims to be from fred at flintstone.com.  We're going to use DNS
to look up the mail servers for flintstone.com.  If we find that
the network (IP) addresses of those servers are completely bogus,
then we're going to refuse the incoming mail message, because this
means either (a) it's spam or (b) whoever is running flintstone.com
has broken it so badly that neither we nor anyone else on the entire
Internet can send mail to it -- so we're not accepting any mail
that claims to be from it.

To put it another way: flintstone.com is telling us that their mail
servers cannot connect to ours.  We're agreeing with them.

7. Any host which issues a challenge in response to mail sent from
here is likely not only to be locally blacklisted -- permanently --
but to be submitted to numerous public and private blacklists used
by others.  Explanation: challenge/response systems are abusive,
selfish, stupid, and generate spam.  See:

	Challenge-Response Anti-Spam Systems Considered Harmful
	http://kmself.home.netcom.com/Rants/challenge-response.html

	Why Challenge-Response is a Bad Idea
	http://tardigrade.net/challengeresponse.html

8. Any host which attempts to use callbacks (also known as "sender
address verification") is likely to be not only locally blacklisted
-- permanently -- but to be submitted to numerous public and private
blacklists used by others.  Explanation: callbacks are misguided attempts
to ascertain whether sender addresses are valid.  They attempt to bypass
site security, they're abusive, they provide a spam support service,
and they are readily abused to conduct denial-of-service attacks.

	[ Both 7 & 8 represent unethical, and possibly illegal, attempts
	to force the costs of controlling one's inbound spam onto others.
	So this might be a good time to go find whoever-it-is that
	runs your mail system and make sure they're not doing 7 or 8.
	Take your paddle.  Some mail system admins have proven to be
	remarkably clue-resistant on these points and so it may be
	necessary to induce a state of enlightenment. ;-) ]

9. By the end of 2006, I'd (semi-)manually blacklisted 88,100 domains
in the .info top-level domain, with about 9,200 in the queue waiting
to be checked.  I finally got tired of this game, and blacklisted all
of .info.  I'll make exceptions on a case-by-case basis.  I'm not alone:
others -- MANY others -- are moving in the same direction or are already
there.  It seems likely that soon a .info domain will be unusable and
completely worthless -- not just here, but Internet-wide.  (This is a
pattern we've seen before: .biz domains are so utterly worthless that
not even spammers are registering them any more.)

10. Mail from any domain which has obfuscated/anonymized contact
information is likely to be refused.  I say "likely" because we
note such domains as they come to our attention, and that takes time.
In addition, once such domains are noticed, they'll be submitted to
several public and private blacklists such as rfc-ignorant.org
and open-whois.org.  Explanation: Anonymous *speech* on the Internet
is fine.  Anonymous *operation* of the Internet is intolerable, and
anyone who registers a domain is in fact trying to operate a portion
of the Internet.

(Some people naively believe that registering domains in this fashion
will help them avoid spam.  They are encouraged in this belief by
the unethical registrars who offer such registration, because, of
course, doing so makes more money for them.  It won't.  Spammers
have a wide variety of means for acquiring addresses, including
"bribe someone at the registrar", so any so-called "anti-spam"
strategy that relies on concealing addresses is doomed.)


Summary
-------

Note that points 1, 2, 4, and 5 aren't really anti-spam measures
per se; they're just enforcing compliance with an Internet standard
that's now over 20 years old...so there's really no excuse for anyone
failing to meet it.  Point 3 is just a reflection of reality: more than
95% of spam comes from such hosts.  Point 7 and 8 are becoming necessary
to deal with ill-conceived "spam blocking" measures that are really "spam
redirection" meaures.  Point 9 is a consequence of a staggering level
of registrar greed.  Point 10 simply reflects the reality that all
responsible domain owners know that operation of the Internet requires
accurate contact information.


How you'll know if this affects you
-----------------------------------

You'll find out when you try to send a mail message here and get back
a message that reads something like:

	Mail refused - host [85.108.209.109] at 85.108.209.109 listed by local dnsbl, forward this message to dec2006 at firemountain.net if in error

or

	Mail refused - ruleset-dynamic2/adsl.alicedsl.de: forward this message to dec2006 at firemountain.net if in error

or

	Mail refused - 199-2-58-30.oxfordnetworks.net listed by Spamhaus (http://www.spamhaus.org), forward this message to dec2006 at firemountain.net if in error

Those are just three examples of the sort of error message you're
likely to see -- all the rest (and there are quite a few more) follow
the same general pattern.

Please note: some amazingly-broken mail systems, in violation of not
only the relevant Internet standards, but of all common sense, attempt
to rewrite error messages.  The usual result of this is that they either
(a) mangle them into gibberish or (b) replace them with useless drivel.
We can't really do anything about this, because it's not our systems
screwing up.

( Oh, and if you get any of these:

	Invalid MX record for recipient host XXXXXX.XXX
	bogus HELO name used: XXXXXXX
	Fix reverse DNS for XXX.XXX.XXX.XXX
	Possibly forged hostname for XXX.XXX.XXX.XXX

then your own ISP has done something truly horrible to its mail servers
and we probably can't help you.   That's why these error messages,
unlike the ones above, don't tell you to forward it to us.)


What to do if this affects you
------------------------------

Do exactly what the error message says: forward the message you are
staring at, including the error message and everything, to the address
provided in the error message.  (That address changes from time to time,
so don't stash this one and try to use it later.)

This will put the message into a priority queue that gets dealt with as
quickly as we can get to it -- usually by me.  But whoever grabs it will
try to figure out what the issue is.  If it's our mistake (and sometimes
it is) we'll fix it, apologize, and figure out how to keep from making
that particular mistake again.  If it's a problem on your end (e.g.,
your mail system, your ISP, your web host) then we'll tell you as much
as we can about it and trust that you'll take it up with them -- since
we can't fix their problem.   If we can, we'll try to band-aid the
problem so that you can at least send mail here.  (But please understand
that this doesn't actually "solve" anything other than in a very limited
and superficial way: if the problem is with your mail server, then someone
on your end needs to fix it.)


Comments, questions, etc.
-------------------------

We have a functioning "postmaster" address, and that's where everything
and anything relating to the operation of the mail systems here should go.