mailfront on IPv6

[John R. Levine]

I've been using it for a month or so, and it seems to work fine.

There's a buglet in plugin-add-received. The SMTP spec says that you put
IPV6: in front of the IP address, e.g.

Received: from foo (bar [IPV6:2001:2002::123])
by zip (zap [IPV6:201a::345]) ; date

The code to add to the routine str_catfromby() is obvious, I haven't done
it since I use a different plugin that does mysql logging at the same
time.

Unrelated question: has anyone done STARTTLS? There's patches for
ofmipd or qmail-smtpd that show what you have to do, but it looks to me
like the current plugin setup isn't adequate to stick the necessary shims
to do the TLS on the way in and out. I suppose you could do it by making
a separate process and pipes, but ugh.

Regards,
John Levine, ***@iecc.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

[Scott Gifford]

On Wed, Jun 22, 2011 at 9:58 PM, John R. Levine <***@iecc.com> wrote:
[ ... ]

Unrelated question: has anyone done STARTTLS? There's patches for ofmipd
or qmail-smtpd that show what you have to do, but it looks to me like the
current plugin setup isn't adequate to stick the necessary shims to do the
TLS on the way in and out. I suppose you could do it by making a separate
process and pipes, but ugh.

I have a version of my ucspitls patch for mailfront 0.96, and will probably
update it for the newer version sometime soon. It doesn't fit neatly into
the plugin architecture, but is still fairly clean. See:

http://www.suspectclass.com/sgifford/ucspi-tls/


It would definitely be interesting to extend the plugin architecture to
allow this as a pure plugin.

-----Scott.

[John R. Levine]

Post by Scott Gifford
I have a version of my ucspitls patch for mailfront 0.96, and will probably
update it for the newer version sometime soon. It doesn't fit neatly into
http://www.suspectclass.com/sgifford/ucspi-tls/

Thanks, will take a look.

Post by Scott Gifford
It would definitely be interesting to extend the plugin architecture to
allow this as a pure plugin.

It'd need both ways to add new verbs, and some way to push filter routines
onto the I/O channels. Neither of those is particularly hard, but making
the interfaces clean and flexible can be tricky.

Regards,
John Levine, ***@iecc.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

[Scott Gifford]

Post by Scott Gifford
I have a version of my ucspitls patch for mailfront 0.96, and will

Post by Scott Gifford
probably
update it for the newer version sometime soon.

[ ... ]
It would definitely be interesting to extend the plugin architecture to

Post by Scott Gifford
allow this as a pure plugin.

It'd need both ways to add new verbs, and some way to push filter routines
onto the I/O channels. Neither of those is particularly hard, but making
the interfaces clean and flexible can be tricky.

The UCSPI-TLS approach lets sslserver do the work, and it just signals back
to switch to SSL mode, so it doesn't really require I/O filters.

-----Scott.

[Bruce Guenter]

Post by Scott Gifford
I have a version of my ucspitls patch for mailfront 0.96, and will probably
update it for the newer version sometime soon. It doesn't fit neatly into
http://www.suspectclass.com/sgifford/ucspi-tls/
It would definitely be interesting to extend the plugin architecture to
allow this as a pure plugin.

I would be interested too, with two large caveats:

1. While it would be technically possible to graft the functionality
into the existing system, I don't see why any plugin would have both an
front-end I/O filter and an internal data handler.

2. Is there any call for more front-end protocol filtering other than
TLS? Would this plugin interface be used by anything other than
STARTTLS?

[Scott Gifford]

Post by Scott Gifford

Post by Scott Gifford
I have a version of my ucspitls patch for mailfront 0.96, and will

probably

Post by Scott Gifford
update it for the newer version sometime soon. It doesn't fit neatly

into

Post by Scott Gifford
http://www.suspectclass.com/sgifford/ucspi-tls/
It would definitely be interesting to extend the plugin architecture to
allow this as a pure plugin.

1. While it would be technically possible to graft the functionality
into the existing system, I don't see why any plugin would have both an
front-end I/O filter and an internal data handler.

2. Is there any call for more front-end protocol filtering other than

Post by Scott Gifford
TLS? Would this plugin interface be used by anything other than
STARTTLS?

So the functionality required to support the ucspi-tls patch is adding new
verbs (STARTTLS/STLS) and capabilities (CAPABILITIES/EHLO/CAPA responses) to
the IMAP, POP, and SMTP. All of the I/O filtering is done by manipulating
file descriptors, which is reasonably portable since these programs run
under the UCSPI interface, which defines which FDs are used for
communication over the network.

I think new verbs and capabilities might be of general interest to plugin
developers.

-----Scott.

[Bruce Guenter]

Post by Scott Gifford
I think new verbs and capabilities might be of general interest to plugin
developers.

Only for SMTP, since QMQP/QMTP don't have "verbs" or other capabilities,
which admittedly is probably the only protocol that counts for accepting
mail.

I don't think this answers the original question -- would this
capability be used by anything other than STARTTLS?

[John R. Levine]

Post by Bruce Guenter
I don't think this answers the original question -- would this
capability be used by anything other than STARTTLS?

Possibly a filterette that did the \n -> \r\n band-aid. I'm also thinking
that some of the timing based anti-spam tricks might be easier if the
plugin could manage the I/O.

Also remember there's two models of TLS, there's STARTTLS in mid-session
on port 25, or TLS at open time on port 465.

Regards,
John Levine, ***@iecc.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

[Scott Gifford]

Post by Bruce Guenter
I don't think this answers the original question -- would this

Post by Bruce Guenter
capability be used by anything other than STARTTLS?

[ ... ]
Also remember there's two models of TLS, there's STARTTLS in mid-session on
port 25, or TLS at open time on port 465.

Certainly true, but the port 465 TLS at open-time model is easy, sslserver
and mailfront can already do that with no changes or patches. If only
STARTTLS wasn't so widely used, our lives would be very slightly easier.
:-)

------Scott.

[Bruce Guenter]

Post by John R. Levine
Possibly a filterette that did the \n -> \r\n band-aid.

All of the protocol code that handles \r\n line endings already also
accepts \n.

[Scott Gifford]

Post by Bruce Guenter

Post by Scott Gifford
I think new verbs and capabilities might be of general interest to plugin
developers.

Only for SMTP, since QMQP/QMTP don't have "verbs" or other capabilities,
which admittedly is probably the only protocol that counts for accepting
mail.

My original patch also adds this support to pop3front and imapfront. I
assumed they provided a similar plugin mechanism to smtpfront, but maybe
not?

I don't think this answers the original question -- would this

Post by Bruce Guenter
capability be used by anything other than STARTTLS?

The ability to add EHLO capabilities and new verbs? I personally probably
wouldn't use it for anything else, but to speculate on some possible uses:

- Support for more obscure SMTP features like ETRN
- Custom mail communication through custom X-verbs, for example to
provide some specialized extension between mail servers run by the same
company, or between a specialized client and server
- Some existing mailfront features could be implemented as plugins which
add a new capability and verb, in particular SMTP authentication
- There could also be alternate implementations of STARTTLS that used the
same plugin interface, for example if somebody preferred a different TLS
implementation, or wanted different tradeoffs than the ones in patched
sslserver (e.g. wanted to start up the TLS processing later in case it
wasn't used, at the cost of giving up chroot support).

------Scott.

[Bruce Guenter]

Post by Scott Gifford
My original patch also adds this support to pop3front and imapfront. I
assumed they provided a similar plugin mechanism to smtpfront, but maybe
not?

No, there are no plugins for pop3front or imapfront, and they don't make
sense for that neither since the existing plugin architecture only
handles accepting messages.

Incidentally, handling TLS in-process for the latter two is particularly
problematic, as they end up exec'ing a child process after
authentication. Either the parent would need to hang around handling
the SSL/TLS translation (thus negating the value of in-process
translation), or all the information about the translation would need to
be handed off to the child so it could continue uninterrupted.

Post by Scott Gifford
The ability to add EHLO capabilities and new verbs? I personally probably
- Some existing mailfront features could be implemented as plugins which
add a new capability and verb, in particular SMTP authentication

Ok, you've convinced me. Now the question is whether to add it to the
existing plugin framework, or add a new one just for protocol
manipulation. Is it of any use to add this facility to the pop3front or
imapfront?

[Scott Gifford]

Post by Bruce Guenter

Post by Scott Gifford
My original patch also adds this support to pop3front and imapfront. I
assumed they provided a similar plugin mechanism to smtpfront, but maybe
not?

No, there are no plugins for pop3front or imapfront, and they don't make
sense for that neither since the existing plugin architecture only
handles accepting messages.
Incidentally, handling TLS in-process for the latter two is particularly
problematic, as they end up exec'ing a child process after
authentication. Either the parent would need to hang around handling
the SSL/TLS translation (thus negating the value of in-process
translation), or all the information about the translation would need to
be handed off to the child so it could continue uninterrupted.

I would have to review my patches to see how this is handled now, but in
practice it doesn't matter: all clients in the wild start TLS before they
authenticate.

Post by Bruce Guenter

Post by Scott Gifford
The ability to add EHLO capabilities and new verbs? I personally

probably

Post by Scott Gifford
wouldn't use it for anything else, but to speculate on some possible
- Some existing mailfront features could be implemented as plugins

which

Post by Scott Gifford
add a new capability and verb, in particular SMTP authentication

Ok, you've convinced me. Now the question is whether to add it to the
existing plugin framework, or add a new one just for protocol
manipulation. Is it of any use to add this facility to the pop3front or
imapfront?

Yes, it's very valuable to add this facility to pop3front and imapfront IMO,
my existing patch does this and I use it nearly everywhere I use smtpfront.
So a plugin interface (or 3 similar plugin interfaces) that could support
all protocols would be very useful.

------Scott.

[John R. Levine]

Post by Scott Gifford
I would have to review my patches to see how this is handled now, but in
practice it doesn't matter: all clients in the wild start TLS before they
authenticate.

RFC 4954 suggests pretty strongly that you wouldn't want to do AUTH and
then STARTTLS. On the other hand, an entirely valid and occasionally
useful way to use STARTTLS is for the client to present its own
certificate in the TLS negotiation which the server uses to authenticate
the client. The STARTTLS patches for qmail-smtpd do that, and set
RELAYCLIENT appropriately.

Post by Scott Gifford

Post by Scott Gifford
The ability to add EHLO capabilities and new verbs?

That will be very handy when EAI comes along. While you're at it, you'll
want to handle extra parameters on MAIL FROM and (if you support them at
all) EXPN and VRFY.

R's,
John

[Scott Gifford]

Post by Scott Gifford
I would have to review my patches to see how this is handled now, but in

Post by Scott Gifford
practice it doesn't matter: all clients in the wild start TLS before they
authenticate.

RFC 4954 suggests pretty strongly that you wouldn't want to do AUTH and
then STARTTLS. On the other hand, an entirely valid and occasionally useful
way to use STARTTLS is for the client to present its own certificate in the
TLS negotiation which the server uses to authenticate the client. The
STARTTLS patches for qmail-smtpd do that, and set RELAYCLIENT appropriately.

My patches send the pre-parsed SSL information over a socket in an
easy-to-use format designed to be imported into the environment, so an
appropriate plugin could make use of this information to automatically
authenticate based on certificate credentials by checking these variables.
I haven't had a need for this, so I haven't implemented it, but the
architecture allows for it.

The ability to add EHLO capabilities and new verbs?

Post by Scott Gifford
That will be very handy when EAI comes along. While you're at it, you'll
want to handle extra parameters on MAIL FROM and (if you support them at
all) EXPN and VRFY.

Support for extra FROM parameters would make the SIZE extension possible as
a plugin too, I think, although it may be too simple to bother with.

-----Scott.

[Bruce Guenter]

Post by Scott Gifford
I have a version of my ucspitls patch for mailfront 0.96, and will probably
update it for the newer version sometime soon. It doesn't fit neatly into
the plugin architecture, but is still fairly clean.

On the subject of the patch itself, does there exist any helper programs
that could be forked from mailfront only when TLS is requested? While I
agree on the idea of doing the TLS interpretation out-of-process, I
don't particularly like the idea of each instance having to carry around
an additional process no matter if TLS is requested or not.

[Scott Gifford]

Post by Scott Gifford

Post by Scott Gifford
I have a version of my ucspitls patch for mailfront 0.96, and will

probably

Post by Scott Gifford
update it for the newer version sometime soon. It doesn't fit neatly

into

Post by Scott Gifford
the plugin architecture, but is still fairly clean.

On the subject of the patch itself, does there exist any helper programs
that could be forked from mailfront only when TLS is requested? While I
agree on the idea of doing the TLS interpretation out-of-process, I
don't particularly like the idea of each instance having to carry around
an additional process no matter if TLS is requested or not.

That is probably possible, but one of the security mechanisms I use in
sslserver is chroot and switching to an otherwise unused UID, both of which
are only possible to root. Providing the same security starting the process
afterwards would require a setuid root program, or for some kind of root
credentials to be maintained in order to start up this separate process. I
guess it could be done with a separate SSL server and file-descriptor
passing, but I'm not sure that could be easily shoehorned into sslserver.

Because sslserver implements the functionality of tcpserver plus SSL, it
doesn't really carry around an extra process, it just carries around a copy
of tcpserver that has some extra libraries loaded to provide SSL. I haven't
measured it, but I suspect the per-process overhead of this is pretty small.

-----Scott.

[Bruce Guenter]

Post by Scott Gifford
That is probably possible, but one of the security mechanisms I use in
sslserver is chroot and switching to an otherwise unused UID, both of which
are only possible to root. Providing the same security starting the process
afterwards would require a setuid root program, or for some kind of root
credentials to be maintained in order to start up this separate process.

Right

Post by Scott Gifford
I guess it could be done with a separate SSL server and
file-descriptor passing,

That's an interesting idea, though setting up an extra server could be a
nuisance. It could run as a parent of tcpserver (or equivalent), which
wouldn't then require an extra server.

Post by Scott Gifford
but I'm not sure that could be easily shoehorned into sslserver.

No, it wouldn't need to be sslserver, since the requirements don't
require accepting connections and setting up the environment, just
handling the SSL protocol.

Post by Scott Gifford
Because sslserver implements the functionality of tcpserver plus SSL, it
doesn't really carry around an extra process, it just carries around a copy
of tcpserver that has some extra libraries loaded to provide SSL.

But it is still another PID, right? So it has its own address space and
all. Granted, most will be mapped from the parent process if I
understand right.

[John R. Levine]

Honestly, all this extra process junk strikes me as overkill. The openssl
library is complex, but it is well debugged and I don't see that it's
notably more likely to do evil stuff to your program than anything else.

I've been using an openssl patched version of ofmipd for years without any
trouble.

Regards,
John Levine, ***@iecc.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

[Scott Gifford]

Post by John R. Levine
Honestly, all this extra process junk strikes me as overkill. The openssl
library is complex, but it is well debugged and I don't see that it's
notably more likely to do evil stuff to your program than anything else.

Maybe. The CVE vulnerability database records 10 bugs in OpenSSL over the
last 10 years that could allow arbitrary code execution, see:

http://web.nvd.nist.gov/view/vuln/search-results?query=openssl+arbitrary+code


On average, about 1 a year. Personally, I'm willing to use an extra few KB
of memory per connection to mitigate that risk, but certainly other
tradeoffs are reasonable.

-----Scott.

[Scott Gifford]

[ ... ]

Post by Bruce Guenter

Post by Scott Gifford
I guess it could be done with a separate SSL server and
file-descriptor passing,

That's an interesting idea, though setting up an extra server could be a
nuisance. It could run as a parent of tcpserver (or equivalent), which
wouldn't then require an extra server.

Post by Scott Gifford
but I'm not sure that could be easily shoehorned into sslserver.

No, it wouldn't need to be sslserver, since the requirements don't
require accepting connections and setting up the environment, just
handling the SSL protocol.

True, but it would require somebody to write such a server, which is
definitely a nontrivial undertaking.

Post by Bruce Guenter

Post by Scott Gifford
Because sslserver implements the functionality of tcpserver plus SSL, it
doesn't really carry around an extra process, it just carries around a

copy

Post by Scott Gifford
of tcpserver that has some extra libraries loaded to provide SSL.

But it is still another PID, right? So it has its own address space and
all. Granted, most will be mapped from the parent process if I
understand right.

You are right. I thought that tcpserver stuck around, and so this required
no more work than that, but on further investigation I see that tcpserver
exits after setting up the connection.

So support for STARTTLS requires that the parent process hangs around in
case TLS is activated later. It doesn't do anything unless STARTTLS is
activated, and on my system it consumes 216KB of data+stack; another 3.36MB
is consumed by the executable and libraries, but that should be shared. The
footprint is pretty small by modern standards, but certainly nonzero.

-----Scott.

[Jorge Valdes]

Post by John R. Levine
I've been using it for a month or so, and it seems to work fine.
There's a buglet in plugin-add-received. The SMTP spec says that you
put IPV6: in front of the IP address, e.g.
Received: from foo (bar [IPV6:2001:2002::123])
by zip (zap [IPV6:201a::345]) ; date
The code to add to the routine str_catfromby() is obvious, I haven't
done it since I use a different plugin that does mysql logging at the
same time.
Unrelated question: has anyone done STARTTLS? There's patches for
ofmipd or qmail-smtpd that show what you have to do, but it looks to
me like the current plugin setup isn't adequate to stick the necessary
shims to do the TLS on the way in and out. I suppose you could do it
by making a separate process and pipes, but ugh.
Regards,
Please consider the environment before reading this e-mail. http://jl.ly
---------------------------------------------------------------------

I have STARTTLS working on mailfront 1.12 by doing the following:
* I have modified ucspi-ssl in order for it to work correctly by adding
another flag to the sslserver executable. I use sslserver to handle the
SSL stuff and pass the information via environment variables to mailfront.
* I have modified the protocol-smtp.c in order to add the STARTTLS verb
to the protocol as well as other responses necessary to allow users to
know the extension is allowed. As well as modify the protocol string as
appropiate in order to generate the correct received headers.
* I have modified plugin-add-received.c in order to show the SSL
information (SSL_PROTOCOL/SSL_CIPHER_USEKEYSIZE/SSL_CIPHER) on the
received header as shown here:

Received: from jvaldes.corporate.intercom (jvaldes.corporate.intercom [172.16.2.252])
by sun21a.intercom.com.sv ([168.243.235.221])
with ESMTPA via SSL (TLSv1:256 bits [AES256-SHA]); 23 Jun 2011 17:23:00 -0000

With regards to its funcionality, I have had no user complaints and it
works great. Tested it with Outlook/Outlook Express/Thunderbird as well
as other mail clients. The only issue I still have is that sometimes the
sslserver session gets stuck @ 100% cpu usage and the process must be
killed manually, and have not been able to reproduce on a staging
server, so I need to clear these "stuck" processes. My server gets ~100K
connections/day and get between 1 and 3 stuck processes per day, so I
have not done much debugging, since I really do keep an eye on the
server throughout the day...

--
Jorge Valdes

[John R. Levine]

Post by Jorge Valdes
* I have modified ucspi-ssl in order for it to work correctly by adding
another flag to the sslserver executable. I use sslserver to handle the
SSL stuff and pass the information via environment variables to mailfront.

Does it handle IPv6?

R's,
John

[Jorge Valdes]

Post by John R. Levine

Post by Jorge Valdes
* I have modified ucspi-ssl in order for it to work correctly by adding
another flag to the sslserver executable. I use sslserver to handle the
SSL stuff and pass the information via environment variables to mailfront.

Does it handle IPv6?
R's,
John
---------------------------------------------------------------------

No. But mainly because I am not running IPV6.
I guess that an IPV6 patch for sslserver would be all I need (apart from
an IPV6 capable network) to get it to work. Here is one:
http://www.brandonturner.net/blog/2009/08/qmail-ipv6-sslserver/

Once applied, the logic for STARTTLS is driven by the SMTP protocol and
not the IP Address ipv4/ipv6.
--
Ing. Jorge Enrique Valdes
Gerente IT
IBW El Salvador
***@ibw.com
Centro Industrial Santa Elena
Calle Siemens N°3, Lote 1
Antiguo Cuscatlán, El Salvador
Oficina: (503) 2278-5068
Fax: [phone]

[John R. Levine]

Post by Jorge Valdes
I guess that an IPV6 patch for sslserver would be all I need (apart from
http://www.brandonturner.net/blog/2009/08/qmail-ipv6-sslserver/

The FreeBSD port of ucspi-tcp says that the ssl and ipv6 patches are
incompatble. Oh, well.

This week I smooshed the ipv6 qmail patches on top of other patches
including outgoing ssl and AUTH. I suppose I could do it for ucspi, but
I'd rather make it work in mailfront rather than a separate process.

Regards,
John Levine, ***@iecc.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

[Scott Gifford]

Post by Jorge Valdes
I guess that an IPV6 patch for sslserver would be all I need (apart from

http://www.brandonturner.net/**blog/2009/08/qmail-ipv6-**sslserver/<http://www.brandonturner.net/blog/2009/08/qmail-ipv6-sslserver/>

The FreeBSD port of ucspi-tcp says that the ssl and ipv6 patches are
incompatble. Oh, well.
This week I smooshed the ipv6 qmail patches on top of other patches
including outgoing ssl and AUTH. I suppose I could do it for ucspi, but I'd
rather make it work in mailfront rather than a separate process.

IMO there are substantial security advantages to running the SSL processing
in a separate process. Primarily, it allows SSL processing to be isolated
inside a low-privilege chroot jail, so that any security flaws in
OpenSSL<http://web.nvd.nist.gov/view/vuln/search-results?query=openssl&search_type=all&cves=on>
are
contained. OpenSSH uses a similar privilege separation
strategy<http://www.citi.umich.edu/u/provos/ssh/privsep.html> to
mitigate security risks. My ucspi-tls
patch<http://www.suspectclass.com/sgifford/ucspi-tls/> to
sslserver implements this privilege separation through chroot and setuid.
Also, SSL has a tricky programming model, and better to re-use the work
already done in ucspi-ssl to get this all right then to try and re-implement
it under mailfront.

The blog post above doesn't talk about adding support for both ssl and ipv6
to tcpserver, but about adding ipv6 support to sslserver. Assuming that
works, it shouldn't be too hard to throw my ucspi-tls patch into the mix.
I'd be happy to help if you run into trouble.

Of course, I may be a bit biased in this whole thing. :-)

------Scott.

[Bruce Guenter]

Post by John R. Levine
I've been using it for a month or so, and it seems to work fine.

Great.

Post by John R. Levine
There's a buglet in plugin-add-received. The SMTP spec says that you put
IPV6: in front of the IP address, e.g.
Received: from foo (bar [IPV6:2001:2002::123])
by zip (zap [IPV6:201a::345]) ; date

So, if $PROTO=TCP6 then add this prefix, right?

Post by John R. Levine
The code to add to the routine str_catfromby() is obvious,

Well, not entirely obvious, particularly since the IP is also used in
place of the host name if it's not known, and also when adding a "fixup"
header. I've attached the patch I'm testing now.

Post by John R. Levine
Unrelated question: has anyone done STARTTLS? There's patches for
ofmipd or qmail-smtpd that show what you have to do, but it looks to me
like the current plugin setup isn't adequate to stick the necessary shims
to do the TLS on the way in and out. I suppose you could do it by making
a separate process and pipes, but ugh.

I'd like to do it, but it's not going to be handled by the normal plugin
setup. Mailfront plugins operate within a somewhat generic protocol,
while STARTTLS needs to front the real wire protocol.

I actually think the separate process and pipes is probably the best way
to go about it, preferably if the parent process can get completely out
of the way, just leaving the TLS translator and a secondary mailfront
instance.