Re: Proposal to upgrade django.core.mail to Python's modern email API

[Arthur Pemberton]

>  The background workers proposal will implement a new background SMTP
EmailBackend (in django.core.mail.backends).

I had missed that fact. Thanks for the explanation.

I for one think that this is a good proposal -- this would modern
transactional email sending.

- Arthur


On Mon, Jun 24, 2024 at 3:14 PM Mike Edmunds  wrote:

> > Would this be designed to be compatible with "Proposal 14: Background
> Workers"?
>
> I wouldn't expect this to impact background workers one way or the other.
> The same goes for the async email proposal(s) floating around.
>
> Virtually all of this work would occur in django.core.mail.message, where
> Python's `email` APIs are used. A goal is to avoid changes that would break
> existing email backends (Django or third-party).
>
> The background workers proposal will implement a new background SMTP
> EmailBackend (in django.core.mail.backends). The existing SMTP EmailBackend
> doesn't directly use Python's `email` APIs, and there should be no reason
> for background SMTP to be any different. (It might be helpful to know that
> Python's `email` library isn't involved in *sending* email; that's
> handled by Python's `smtplib`, which Django uses only in the SMTP
> EmailBackend.)
>
> The existing SMTP EmailBackend *does* use one function I expect will
> become deprecated: django.core.mail.message.sanitize_address(). I haven't
> yet investigated whether that use is still necessary, or whether it's there
> to get around past bugs/limitations in Python's smtplib. If any of it is
> still needed for SMTP, I'd probably want to move that code into the SMTP
> EmailBackend(s).
>
> - Mike
> On Sunday, June 23, 2024 at 10:22:03 PM UTC-7 Arthur Pemberton wrote:
>
>> Would this be designed to be compatible with "Proposal 14: Background
>> Workers"?
>>
>>
>>
>> On Sun, Jun 23, 2024 at 10:46 PM Mike Edmunds  wrote:
>>
>>> I want to propose updating django.core.mail to replace use of Python's
>>> legacy email.message.Message (and other legacy email APIs) with
>>> email.message.EmailMessage (and other modern APIs).
>>>
>>> If there's interest, I can put together a more detailed proposal and/or
>>> ticket, but was hoping to get some initial feedback first. (I searched for
>>> relevant discussions in django-developers and the issue tracker, and didn't
>>> find any. Apologies if this has come up before.)
>>>
>>> [Note: I maintain django-anymail
>>> , which implements Django
>>> integration with several transactional email service providers.]
>>>
>>>
>>> *Background*
>>>
>>> Since Python ~3.6, Python's email package has included two largely
>>> separate implementations:
>>>
>>>- a "modern (unicode friendly) API" based on
>>>email.message.EmailMessage and email.policy.default
>>>- a "legacy API" providing compatibility with older Python versions,
>>>based on email.message.Message and email.policy.compat32
>>>
>>> (See https://docs.python.org/3/library/email.html, especially toward
>>> the end.)
>>>
>>> django.core.mail currently uses the legacy API.
>>>
>>>
>>> *Why switch?*
>>>
>>> There are no plans to deprecate Python's legacy email API, and it's
>>> working, so why change Django?
>>>
>>>-
>>>
>>>*Fewer bugs:* The modern API fixes a *lot* of bugs in the legacy
>>>API. My understanding is legacy bugs will generally not be fixed. (And in
>>>fact, there are some cases where the legacy API deliberately replicates
>>>earlier buggy behavior.)
>>>
>>>Django #35497  is an
>>>example of a legacy bug (with a problematic proposed workaround) which is
>>>just fixed in the modern API.
>>>-
>>>
>>>*Simpler, safer code:* A substantial portion
>>>
>>> 
>>>  of
>>>django.core.mail's internals implements workarounds and security fixes 
>>> for
>>>problems in the legacy API. This would be greatly simplified—maybe
>>>eliminated completely—using the modern API.
>>>
>>>Examples: the modern API prevents CR/NL injections in message
>>>headers. It serializes and folds address headers properly—even ones with
>>>unicode names. It enforces single-instance headers where appropriate.
>>>-
>>>
>>>*Easier to work with:* The modern API adds some nice conveniences
>>>for developers working in django.core.mail, third-party library 
>>> developers,
>>>and (depending on what we choose to expose) users of Django's mail APIs.
>>>
>>>Examples: populating the "Date" header with a datetime or an address
>>>header with Address
>>>
>>> 
>>>  objects—without
>>>needing intricate knowledge of email header formats. Using
>>>email.policy to generate a 7-bit clean serialization (without having
>>>to muck about with the MIME parts
>>>
>>> 

Re: Proposal to upgrade django.core.mail to Python's modern email API

['Adam Johnson' via Django developers (Contributions to Django itself)]

After my comment in the steering council vote and in-person conversation with 
Jake, I believe the SMTP backend will not be implemented for DEP 14 : 
https://forum.djangoproject.com/t/steering-council-vote-on-background-tasks-dep-14/31131/20

On Tue, 25 Jun 2024, at 10:27, Arthur Pemberton wrote:
> >  The background workers proposal will implement a new background SMTP 
> > EmailBackend (in django.core.mail.backends).
> 
> I had missed that fact. Thanks for the explanation.
> 
> I for one think that this is a good proposal -- this would modern 
> transactional email sending.
> 
> - Arthur
> 
> 
> On Mon, Jun 24, 2024 at 3:14 PM Mike Edmunds  wrote:
>> > Would this be designed to be compatible with "Proposal 14: Background 
>> > Workers"?
>> 
>> I wouldn't expect this to impact background workers one way or the other. 
>> The same goes for the async email proposal(s) floating around.
>> 
>> Virtually all of this work would occur in django.core.mail.message, where 
>> Python's `email` APIs are used. A goal is to avoid changes that would break 
>> existing email backends (Django or third-party).
>> 
>> The background workers proposal will implement a new background SMTP 
>> EmailBackend (in django.core.mail.backends). The existing SMTP EmailBackend 
>> doesn't directly use Python's `email` APIs, and there should be no reason 
>> for background SMTP to be any different. (It might be helpful to know that 
>> Python's `email` library isn't involved in *sending* email; that's handled 
>> by Python's `smtplib`, which Django uses only in the SMTP EmailBackend.)
>> 
>> The existing SMTP EmailBackend *does* use one function I expect will become 
>> deprecated: django.core.mail.message.sanitize_address(). I haven't yet 
>> investigated whether that use is still necessary, or whether it's there to 
>> get around past bugs/limitations in Python's smtplib. If any of it is still 
>> needed for SMTP, I'd probably want to move that code into the SMTP 
>> EmailBackend(s).
>> 
>> - Mike
>> On Sunday, June 23, 2024 at 10:22:03 PM UTC-7 Arthur Pemberton wrote:
>>> Would this be designed to be compatible with "Proposal 14: Background 
>>> Workers"?
>>> 
>>> 
>>> 
>>> 
>>> On Sun, Jun 23, 2024 at 10:46 PM Mike Edmunds  wrote:
 I want to propose updating django.core.mail to replace use of Python's 
 legacy email.message.Message (and other legacy email APIs) with 
 email.message.EmailMessage (and other modern APIs).
 
 If there's interest, I can put together a more detailed proposal and/or 
 ticket, but was hoping to get some initial feedback first. (I searched for 
 relevant discussions in django-developers and the issue tracker, and 
 didn't find any. Apologies if this has come up before.)
 
 [Note: I maintain django-anymail 
 , which implements Django 
 integration with several transactional email service providers.]
 
 
 
 *Background*
 
 Since Python ~3.6, Python's email package has included two largely 
 separate implementations:
 
  • a "modern (unicode friendly) API" based on email.message.EmailMessage 
 and email.policy.default
  • a "legacy API" providing compatibility with older Python versions, 
 based on email.message.Message and email.policy.compat32
 (See https://docs.python.org/3/library/email.html, especially toward the 
 end.)
 
 django.core.mail currently uses the legacy API.
 
 
 
 *Why switch?*
 
 There are no plans to deprecate Python's legacy email API, and it's 
 working, so why change Django?
 
  • *Fewer bugs:* The modern API fixes a *lot* of bugs in the legacy API. 
 My understanding is legacy bugs will generally not be fixed. (And in fact, 
 there are some cases where the legacy API deliberately replicates earlier 
 buggy behavior.)
 
 Django #35497  is an example 
 of a legacy bug (with a problematic proposed workaround) which is just 
 fixed in the modern API.
 
  • *Simpler, safer code:* A substantial portion 
 
  of django.core.mail's internals implements workarounds and security fixes 
 for problems in the legacy API. This would be greatly simplified—maybe 
 eliminated completely—using the modern API.
 
 Examples: the modern API prevents CR/NL injections in message headers. It 
 serializes and folds address headers properly—even ones with unicode 
 names. It enforces single-instance headers where appropriate.
 
  • *Easier to work with:* The modern API adds some nice conveniences for 
 developers working in django.core.mail, third-party library developers, 
 and (depending on what we choose to expose) users of Django's mail APIs.
 
 Examples: populating the "Date" header with a datetime or 

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Ronny V.]

Hi all!

Jakob Rief pointed this discussion out to me. I've been going around lately 
to make some advertisement for my idea of class-based emails.

I've implemented a package called "django-pony-express" which in a nutshell 
provides to things:

* A class-based way of creating new emails (very similar to class-based 
views)
* A test suite for easy unit-testing of emails (we are currently working on 
bringing this to Django core)

Here's a documentation link: 
https://django-pony-express.readthedocs.io/en/latest/features/introduction.html#create-a-single-email
 


The idea is to create emails like you create views. All important stuff is 
encapsulated but you can overwrite everything you need. There are a bunch 
of examples in the docs.

Apart from not having to deal with low-level email API stuff (which is a 
pain IMHO), it provides lots of neat improvements I had to (re-)invent in 
every project I was working on.

What do you people think about this? I got quite good feedback in Vigo and 
in my opinion, the solution is very Django-esque.

I'd be happy to go into more detail if required but wanted to keep this 
first commend brief.

Best from Cologne
Ronny
Mike Edmunds schrieb am Montag, 24. Juni 2024 um 21:14:47 UTC+2:

> > Would this be designed to be compatible with "Proposal 14: Background 
> Workers"?
>
> I wouldn't expect this to impact background workers one way or the other. 
> The same goes for the async email proposal(s) floating around.
>
> Virtually all of this work would occur in django.core.mail.message, where 
> Python's `email` APIs are used. A goal is to avoid changes that would break 
> existing email backends (Django or third-party).
>
> The background workers proposal will implement a new background SMTP 
> EmailBackend (in django.core.mail.backends). The existing SMTP EmailBackend 
> doesn't directly use Python's `email` APIs, and there should be no reason 
> for background SMTP to be any different. (It might be helpful to know that 
> Python's `email` library isn't involved in *sending* email; that's 
> handled by Python's `smtplib`, which Django uses only in the SMTP 
> EmailBackend.)
>
> The existing SMTP EmailBackend *does* use one function I expect will 
> become deprecated: django.core.mail.message.sanitize_address(). I haven't 
> yet investigated whether that use is still necessary, or whether it's there 
> to get around past bugs/limitations in Python's smtplib. If any of it is 
> still needed for SMTP, I'd probably want to move that code into the SMTP 
> EmailBackend(s).
>
> - Mike
> On Sunday, June 23, 2024 at 10:22:03 PM UTC-7 Arthur Pemberton wrote:
>
>> Would this be designed to be compatible with "Proposal 14: Background 
>> Workers"?
>>
>>
>>
>> On Sun, Jun 23, 2024 at 10:46 PM Mike Edmunds  wrote:
>>
>>> I want to propose updating django.core.mail to replace use of Python's 
>>> legacy email.message.Message (and other legacy email APIs) with 
>>> email.message.EmailMessage (and other modern APIs).
>>>
>>> If there's interest, I can put together a more detailed proposal and/or 
>>> ticket, but was hoping to get some initial feedback first. (I searched for 
>>> relevant discussions in django-developers and the issue tracker, and didn't 
>>> find any. Apologies if this has come up before.)
>>>
>>> [Note: I maintain django-anymail 
>>> , which implements Django 
>>> integration with several transactional email service providers.]
>>>
>>>
>>> *Background*
>>>
>>> Since Python ~3.6, Python's email package has included two largely 
>>> separate implementations:
>>>
>>>- a "modern (unicode friendly) API" based on 
>>>email.message.EmailMessage and email.policy.default
>>>- a "legacy API" providing compatibility with older Python versions, 
>>>based on email.message.Message and email.policy.compat32
>>>
>>> (See https://docs.python.org/3/library/email.html, especially toward 
>>> the end.)
>>>
>>> django.core.mail currently uses the legacy API.
>>>
>>>
>>> *Why switch?*
>>>
>>> There are no plans to deprecate Python's legacy email API, and it's 
>>> working, so why change Django?
>>>
>>>- 
>>>
>>>*Fewer bugs:* The modern API fixes a *lot* of bugs in the legacy 
>>>API. My understanding is legacy bugs will generally not be fixed. (And 
>>> in 
>>>fact, there are some cases where the legacy API deliberately replicates 
>>>earlier buggy behavior.)
>>>
>>>Django #35497  is an 
>>>example of a legacy bug (with a problematic proposed workaround) which 
>>> is 
>>>just fixed in the modern API.
>>>- 
>>>
>>>*Simpler, safer code:* A substantial portion 
>>>
>>> 
>>>  of 
>>>django.core.mail's internals implements workarounds and security fixes 
>>> for 
>>>problems in the legacy API. This would be greatly simplified—maybe 
>>>elim

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Mike Edmunds]

Hi Ronny,

django-pony-express looks really interesting, thanks. (I'm going to add a 
link in django-anymail's "you probably don't need proprietary ESP templates 
" docs.)

I think django-pony-express is very complementary to this proposal, both as 
a third-party library and if some version of class-based emails finds its 
way into Django core. 

Here's my mental model of Django's email sending functionality:

   - A django.core.mail.message.EmailMessage (or EmailMultiAlternatives*) 
   is a list of ingredients for building an email message—it's mostly just a 
   "bucket of properties" that specifies what should end up in the message.**
   - An EmailBackend is responsible for transmitting that EmailMessage to a 
   server that will actually send an email—an SMTP server, an email service 
   provider's HTTP API, etc. Each backend implements its own recipe to bake 
   the Django EmailMessage ingredients into the form expected by its server—an 
   RFC 2822+ message for SMTP, a JSON API payload for an ESP, etc.
   - Because a Django EmailMessage can be "a pain" (you're not wrong!), 
   there are convenience APIs that simplify constructing and sending it. Some 
   come with Django: send_mail(), send_mass_mail(), mail_admins(). Some come 
   from third party libraries: django-pony-express, django-templated-mail, 
   etc. *Many* get built (and repeatedly re-built) by developers in 
   individual Django projects.

To stretch your class-based view analogy, Django's EmailMessage is roughly 
akin to Django's HttpResponse. You can write a view function that builds up 
an HttpResponse from scratch. For a lot of common cases, though, it's much 
easier to work one of Django's helper functions or class-based View 
subclasses. But however you go about it, every view eventually has to 
return an HttpResponse that Django can hand off to its "http backend" 
(wsgi/asgi).

Similarly, there are several ways to build a Django EmailMessage. At the 
end of the day, though, all of these need to return a 
django.core.mail.EmailMessage that Django can hand off to an email backend 
for sending.

- Mike

_

* I suspect the EmailMultiAlternatives/EmailMessage distinction is no 
longer helpful. I'm fairly certain it's confusing to users, and I *know* it 
adds complexity for third-party email backends. Given that modern Python 
email convenience 

 
methods 

 handle 
all the multipart restructuring, I'll propose collapsing 
EmailMultiAlternatives/EmailMessage into a single class as part of this 
work.

** Along with the "bucket of properties," Django's EmailMessage also 
implements serialization to RFC 2822+ format in as_message(). This is used 
by many—but certainly not all—email backends. And (just to come full circle 
here in the footnotes), the bulk of work in this proposal is actually about 
updating as_message() and the private helpers it calls.

On Tuesday, June 25, 2024 at 7:23:42 AM UTC-7 Ronny V. wrote:

> Hi all!
>
> Jakob Rief pointed this discussion out to me. I've been going around 
> lately to make some advertisement for my idea of class-based emails.
>
> I've implemented a package called "django-pony-express" which in a 
> nutshell provides to things:
>
> * A class-based way of creating new emails (very similar to class-based 
> views)
> * A test suite for easy unit-testing of emails (we are currently working 
> on bringing this to Django core)
>
> Here's a documentation link: 
> https://django-pony-express.readthedocs.io/en/latest/features/introduction.html#create-a-single-email
>  
>
> The idea is to create emails like you create views. All important stuff is 
> encapsulated but you can overwrite everything you need. There are a bunch 
> of examples in the docs.
>
> Apart from not having to deal with low-level email API stuff (which is a 
> pain IMHO), it provides lots of neat improvements I had to (re-)invent in 
> every project I was working on.
>
> What do you people think about this? I got quite good feedback in Vigo and 
> in my opinion, the solution is very Django-esque.
>
> I'd be happy to go into more detail if required but wanted to keep this 
> first commend brief.
>
> Best from Cologne
> Ronny
> Mike Edmunds schrieb am Montag, 24. Juni 2024 um 21:14:47 UTC+2:
>
>> > Would this be designed to be compatible with "Proposal 14: Background 
>> Workers"?
>>
>> I wouldn't expect this to impact background workers one way or the other. 
>> The same goes for the async email proposal(s) floating around.
>>
>> Virtually all of this work would occur in django.core.mail.message, where 
>> Python's `email` APIs are used. A goal is to avoid changes that would break 
>> existing email backends (Django or third-party).
>>
>> The background workers proposal will implement a new back

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Mike Edmunds]

> After my comment in the steering council vote and in-person conversation 
with Jake, I believe the SMTP backend will not be implemented for DEP 14

Then I can improve my earlier response: I *am confident* this proposal will 
have *no impact* on background workers. :-)

- Mike

On Tuesday, June 25, 2024 at 4:38:11 AM UTC-7 Adam Johnson wrote:

> After my comment in the steering council vote and in-person conversation 
> with Jake, I believe the SMTP backend will not be implemented for DEP 14 : 
> https://forum.djangoproject.com/t/steering-council-vote-on-background-tasks-dep-14/31131/20
>
> On Tue, 25 Jun 2024, at 10:27, Arthur Pemberton wrote:
>
> >  The background workers proposal will implement a new background SMTP 
> EmailBackend (in django.core.mail.backends).
>
> I had missed that fact. Thanks for the explanation.
>
> I for one think that this is a good proposal -- this would modern 
> transactional email sending.
>
> - Arthur
>
>
> On Mon, Jun 24, 2024 at 3:14 PM Mike Edmunds  wrote:
>
> > Would this be designed to be compatible with "Proposal 14: Background 
> Workers"?
>
> I wouldn't expect this to impact background workers one way or the other. 
> The same goes for the async email proposal(s) floating around.
>
> Virtually all of this work would occur in django.core.mail.message, where 
> Python's `email` APIs are used. A goal is to avoid changes that would break 
> existing email backends (Django or third-party).
>
> The background workers proposal will implement a new background SMTP 
> EmailBackend (in django.core.mail.backends). The existing SMTP EmailBackend 
> doesn't directly use Python's `email` APIs, and there should be no reason 
> for background SMTP to be any different. (It might be helpful to know that 
> Python's `email` library isn't involved in *sending* email; that's 
> handled by Python's `smtplib`, which Django uses only in the SMTP 
> EmailBackend.)
>
> The existing SMTP EmailBackend *does* use one function I expect will 
> become deprecated: django.core.mail.message.sanitize_address(). I haven't 
> yet investigated whether that use is still necessary, or whether it's there 
> to get around past bugs/limitations in Python's smtplib. If any of it is 
> still needed for SMTP, I'd probably want to move that code into the SMTP 
> EmailBackend(s).
>
> - Mike
> On Sunday, June 23, 2024 at 10:22:03 PM UTC-7 Arthur Pemberton wrote:
>
> Would this be designed to be compatible with "Proposal 14: Background 
> Workers"?
>
>
>
>
> On Sun, Jun 23, 2024 at 10:46 PM Mike Edmunds  wrote:
>
> I want to propose updating django.core.mail to replace use of Python's 
> legacy email.message.Message (and other legacy email APIs) with 
> email.message.EmailMessage (and other modern APIs).
>
> If there's interest, I can put together a more detailed proposal and/or 
> ticket, but was hoping to get some initial feedback first. (I searched for 
> relevant discussions in django-developers and the issue tracker, and didn't 
> find any. Apologies if this has come up before.)
>
> [Note: I maintain django-anymail 
> , which implements Django 
> integration with several transactional email service providers.]
>
>
> *Background*
>
> Since Python ~3.6, Python's email package has included two largely 
> separate implementations:
>
>- a "modern (unicode friendly) API" based on 
>email.message.EmailMessage and email.policy.default
>- a "legacy API" providing compatibility with older Python versions, 
>based on email.message.Message and email.policy.compat32
>
> (See https://docs.python.org/3/library/email.html, especially toward the 
> end.)
>
> django.core.mail currently uses the legacy API.
>
>
> *Why switch?*
>
> There are no plans to deprecate Python's legacy email API, and it's 
> working, so why change Django?
>
>- 
>
>*Fewer bugs:* The modern API fixes a *lot* of bugs in the legacy API. 
>My understanding is legacy bugs will generally not be fixed. (And in fact, 
>there are some cases where the legacy API deliberately replicates earlier 
>buggy behavior.)
>
>Django #35497  is an 
>example of a legacy bug (with a problematic proposed workaround) which is 
>just fixed in the modern API.
>- 
>
>*Simpler, safer code:* A substantial portion 
>
> 
>  of 
>django.core.mail's internals implements workarounds and security fixes for 
>problems in the legacy API. This would be greatly simplified—maybe 
>eliminated completely—using the modern API.
>
>Examples: the modern API prevents CR/NL injections in message headers. 
>It serializes and folds address headers properly—even ones with unicode 
>names. It enforces single-instance headers where appropriate.
>- 
>
>*Easier to work with:* The modern API adds some nice conveniences for