Re: Proposal to upgrade django.core.mail to Python's modern email API

[Tom Carrick]

I'm in favour of this change, and nice that you're thinking about the
future, but if you're going to write a ticket for this I would focus it on
purely the strictly necessary parts to update to the new API, and when that
is done, make more proposals to simplify the API as you suggest.

I say this just with the goal of getting something that is easy to agree on
merged, before dealing with potentially contentious things
like EmailMultiAlternatives, MIMEBase, etc.

Ignore me if this is irrelevant and those things are indeed necessary, my
knowledge on the topic is a bit lacking.

Tom

On Thu, 27 Jun 2024 at 03:28, Mike Edmunds  wrote:

> Since the early feedback seems positive (though we're still waiting for
> more votes), here's some additional detail on the changes I think would be
> involved to update django.core.mail to use Python's modern email API.
>
> (See my earlier message
>  
> for
> background on Python's legacy vs. modern email APIs and why updating is
> useful.)
>
> Note: Django and Python both have classes named EmailMessage. I'm using
> "Django EmailMessage" to refer to django.core.mail.message.EmailMessage,
> and "Python EmailMessage" (or sometimes just "modern API") to refer to
> Python's modern email.message.EmailMessage.
> Necessary work
> 
>
>1.
>
>Update tests.mail.tests to use modern email APIs
>
>Where a test relies on legacy implementation details, try to rewrite
>it to be implementation agnostic if possible; otherwise try to retain the
>spirit of the test using modern APIs.
>
>Retain all security related tests, with updates as appropriate. (Even
>where we know Python's modern API handles the security for us, it doesn't
>hurt to double check.)
>
>Probably need to add some cases for existing behavior not currently
>covered by tests, particularly around attachments and alternatives.
>
>Remove a test case *only* if it's truly no longer relevant and can't
>be usefully updated. (And perhaps leave behind a brief comment explaining
>why.)
>
>(Legacy email APIs used in tests.mail.tests: email.charset,
>email.header.Header, MIMEText, parseaddr. Also message_from_… currently
>defaults to legacy policy.compat32.)
>2.
>
>Update django.core.mail.message.EmailMessage to use modern email APIs
>
>Change Django's EmailMessage.message() to construct a modern
>email.message.EmailMessage rather than a SafeMIME object (which is based on
>legacy email.message.Message with policy=compat32). Add a
>message(policy=default) param forwarded to Python's EmailMessage
>constructor.
>
>Hoist alternative part handling from Django's EmailMultiAlternatives
>into Django's base EmailMessage to simplify the code. (More on this below.)
>
>In _create_alternatives(), use modern add_alternative()
>
> 
>  and
>friends to replace legacy SafeMIME objects.
>
>In _create_attachments(), use modern add_attachment()
>
> .
>Handle (legacy Python) MIMEBase objects as deprecated, and convert to
>modern equivalent. This is a relatively complex topic; I'll post a separate
>message about it later. (I also have some questions about how best to
>handle content-disposition "inline" and whether we want to somehow support
>multipart/related.)
>
>Remove the private Django EmailMessage methods
>_create_mime_attachment() and _create_attachment() (without deprecation).
>
>(Legacy APIs used in django.core.mail.message: email.message.Message,
>email.charset, email.encoders, email.header, email.mime, email.utils)
>3.
>
>Deprecate unused internal APIs from django.core.mail.message
>
>Django will no longer need these (Python's modern email API covers
>their functionality), but they may be in use by third-party libraries:
>- utf8_charset
>   - utf8_charset_qp
>   - RFC5322_EMAIL_LINE_LENGTH_LIMIT
>   - BadHeaderError^ (more details below)
>   - ADDRESS_HEADERS
>   - forbid_multi_line_headers()^
>   - sanitize_address() (more details below)
>   - MIMEMixin
>   - SafeMIMEMessage
>   - SafeMIMEText^
>   - SafeMIMEMultipart^
>
>(Items marked ^ are exposed in django.core.mail via __all__. I haven't
>looked into the reason for that.)
>4.
>
>Update django.core.mail.__init__ to avoid EmailMultiAlternatives, and
>django.core.mail.backend.smtp to replace sanitize_address. (More details
>below.)
>5.
>
>Update docs
>- deprecation of legacy MIMEBase in attachments list, what to do
>   instead
>   - eliminate EmailMessage/EmailMultiAlternatives distinction
>   - dep

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Paolo Melchiorre]

I agree with the approach suggested by Tom.

And thanks for proposing this enhancement.

Ciao,
Paolo

On Thu, Jun 27, 2024, 11:14 Tom Carrick  wrote:

> I'm in favour of this change, and nice that you're thinking about the
> future, but if you're going to write a ticket for this I would focus it on
> purely the strictly necessary parts to update to the new API, and when that
> is done, make more proposals to simplify the API as you suggest.
>
> I say this just with the goal of getting something that is easy to agree
> on merged, before dealing with potentially contentious things
> like EmailMultiAlternatives, MIMEBase, etc.
>
> Ignore me if this is irrelevant and those things are indeed necessary, my
> knowledge on the topic is a bit lacking.
>
> Tom
>
> On Thu, 27 Jun 2024 at 03:28, Mike Edmunds  wrote:
>
>> Since the early feedback seems positive (though we're still waiting for
>> more votes), here's some additional detail on the changes I think would be
>> involved to update django.core.mail to use Python's modern email API.
>>
>> (See my earlier message
>>  
>> for
>> background on Python's legacy vs. modern email APIs and why updating is
>> useful.)
>>
>> Note: Django and Python both have classes named EmailMessage. I'm using
>> "Django EmailMessage" to refer to django.core.mail.message.EmailMessage,
>> and "Python EmailMessage" (or sometimes just "modern API") to refer to
>> Python's modern email.message.EmailMessage.
>> Necessary work
>> 
>>
>>1.
>>
>>Update tests.mail.tests to use modern email APIs
>>
>>Where a test relies on legacy implementation details, try to rewrite
>>it to be implementation agnostic if possible; otherwise try to retain the
>>spirit of the test using modern APIs.
>>
>>Retain all security related tests, with updates as appropriate. (Even
>>where we know Python's modern API handles the security for us, it doesn't
>>hurt to double check.)
>>
>>Probably need to add some cases for existing behavior not currently
>>covered by tests, particularly around attachments and alternatives.
>>
>>Remove a test case *only* if it's truly no longer relevant and can't
>>be usefully updated. (And perhaps leave behind a brief comment explaining
>>why.)
>>
>>(Legacy email APIs used in tests.mail.tests: email.charset,
>>email.header.Header, MIMEText, parseaddr. Also message_from_… currently
>>defaults to legacy policy.compat32.)
>>2.
>>
>>Update django.core.mail.message.EmailMessage to use modern email APIs
>>
>>Change Django's EmailMessage.message() to construct a modern
>>email.message.EmailMessage rather than a SafeMIME object (which is based 
>> on
>>legacy email.message.Message with policy=compat32). Add a
>>message(policy=default) param forwarded to Python's EmailMessage
>>constructor.
>>
>>Hoist alternative part handling from Django's EmailMultiAlternatives
>>into Django's base EmailMessage to simplify the code. (More on this 
>> below.)
>>
>>In _create_alternatives(), use modern add_alternative()
>>
>> 
>>  and
>>friends to replace legacy SafeMIME objects.
>>
>>In _create_attachments(), use modern add_attachment()
>>
>> .
>>Handle (legacy Python) MIMEBase objects as deprecated, and convert to
>>modern equivalent. This is a relatively complex topic; I'll post a 
>> separate
>>message about it later. (I also have some questions about how best to
>>handle content-disposition "inline" and whether we want to somehow support
>>multipart/related.)
>>
>>Remove the private Django EmailMessage methods
>>_create_mime_attachment() and _create_attachment() (without deprecation).
>>
>>(Legacy APIs used in django.core.mail.message: email.message.Message,
>>email.charset, email.encoders, email.header, email.mime, email.utils)
>>3.
>>
>>Deprecate unused internal APIs from django.core.mail.message
>>
>>Django will no longer need these (Python's modern email API covers
>>their functionality), but they may be in use by third-party libraries:
>>- utf8_charset
>>   - utf8_charset_qp
>>   - RFC5322_EMAIL_LINE_LENGTH_LIMIT
>>   - BadHeaderError^ (more details below)
>>   - ADDRESS_HEADERS
>>   - forbid_multi_line_headers()^
>>   - sanitize_address() (more details below)
>>   - MIMEMixin
>>   - SafeMIMEMessage
>>   - SafeMIMEText^
>>   - SafeMIMEMultipart^
>>
>>(Items marked ^ are exposed in django.core.mail via __all__. I
>>haven't looked into the reason for that.)
>>4.
>>
>>Update django.core.mail.__init__ to avoid EmailMultiAlternatives, and

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Mike Edmunds]

> focus it on purely the strictly necessary parts to update to the new API 
… before dealing with potentially contentious things 
like EmailMultiAlternatives, MIMEBase, etc.

Appreciate the advice. I'll try to keep EmailMultiAlternatives and see 
whether that makes the new code hard to follow.

Sadly, MIMEBase attachments are a documented part of Django's EmailMessage 
API (and have been necessary for certain kinds of attachments), so they'll 
have to be dealt with one way or another.

- Mike

On Thursday, June 27, 2024 at 2:14:12 AM UTC-7 Tom Carrick wrote:

> I'm in favour of this change, and nice that you're thinking about the 
> future, but if you're going to write a ticket for this I would focus it on 
> purely the strictly necessary parts to update to the new API, and when that 
> is done, make more proposals to simplify the API as you suggest.
>
> I say this just with the goal of getting something that is easy to agree 
> on merged, before dealing with potentially contentious things 
> like EmailMultiAlternatives, MIMEBase, etc.
>
> Ignore me if this is irrelevant and those things are indeed necessary, my 
> knowledge on the topic is a bit lacking.
>
> Tom
>
> On Thu, 27 Jun 2024 at 03:28, Mike Edmunds  wrote:
>
>> Since the early feedback seems positive (though we're still waiting for 
>> more votes), here's some additional detail on the changes I think would be 
>> involved to update django.core.mail to use Python's modern email API.
>>
>> (See my earlier message 
>>  
>> for 
>> background on Python's legacy vs. modern email APIs and why updating is 
>> useful.)
>>
>> Note: Django and Python both have classes named EmailMessage. I'm using 
>> "Django EmailMessage" to refer to django.core.mail.message.EmailMessage, 
>> and "Python EmailMessage" (or sometimes just "modern API") to refer to 
>> Python's modern email.message.EmailMessage.
>> Necessary work 
>> 
>>
>>1. 
>>
>>Update tests.mail.tests to use modern email APIs
>>
>>Where a test relies on legacy implementation details, try to rewrite 
>>it to be implementation agnostic if possible; otherwise try to retain the 
>>spirit of the test using modern APIs.
>>
>>Retain all security related tests, with updates as appropriate. (Even 
>>where we know Python's modern API handles the security for us, it doesn't 
>>hurt to double check.)
>>
>>Probably need to add some cases for existing behavior not currently 
>>covered by tests, particularly around attachments and alternatives.
>>
>>Remove a test case *only* if it's truly no longer relevant and can't 
>>be usefully updated. (And perhaps leave behind a brief comment explaining 
>>why.)
>>
>>(Legacy email APIs used in tests.mail.tests: email.charset, 
>>email.header.Header, MIMEText, parseaddr. Also message_from_… currently 
>>defaults to legacy policy.compat32.)
>>2. 
>>
>>Update django.core.mail.message.EmailMessage to use modern email APIs
>>
>>Change Django's EmailMessage.message() to construct a modern 
>>email.message.EmailMessage rather than a SafeMIME object (which is based 
>> on 
>>legacy email.message.Message with policy=compat32). Add a 
>>message(policy=default) param forwarded to Python's EmailMessage 
>>constructor.
>>
>>Hoist alternative part handling from Django's EmailMultiAlternatives 
>>into Django's base EmailMessage to simplify the code. (More on this 
>> below.)
>>
>>In _create_alternatives(), use modern add_alternative() 
>>
>> 
>>  and 
>>friends to replace legacy SafeMIME objects.
>>
>>In _create_attachments(), use modern add_attachment() 
>>
>> .
>>  
>>Handle (legacy Python) MIMEBase objects as deprecated, and convert to 
>>modern equivalent. This is a relatively complex topic; I'll post a 
>> separate 
>>message about it later. (I also have some questions about how best to 
>>handle content-disposition "inline" and whether we want to somehow 
>> support 
>>multipart/related.)
>>
>>Remove the private Django EmailMessage methods 
>>_create_mime_attachment() and _create_attachment() (without deprecation).
>>
>>(Legacy APIs used in django.core.mail.message: email.message.Message, 
>>email.charset, email.encoders, email.header, email.mime, email.utils)
>>3. 
>>
>>Deprecate unused internal APIs from django.core.mail.message
>>
>>Django will no longer need these (Python's modern email API covers 
>>their functionality), but they may be in use by third-party libraries:
>>- utf8_charset
>>   - utf8_chars