[Mailman-Users] Non-ascii characters missing from Pipermail archive txt and gz downloads

2021-04-09 Thread Mark Dale via Mailman-Users 
Apologies for double posting. I sent this to the MM3 list by mistake earlier.

Mailman 2.1.34
Debian 10
Postfix

Hi

I'm hoping someone can shine a light on character encoding issue I've 
encountered.

A plain-text email with non-ascii characters in the body gets posted to the 
list.

As per Mark Sapiro's guide I've captured the incoming message to file.

 - https://www.mail-archive.com/mailman-users@python.org/msg73469.html

The message is received by Mailman with the non-ascii characters displaying 
correctly. 

The header of that message has:

User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.9.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Language: en-AU
Content-Transfer-Encoding: 8bit

In the list's mbox file and archive webpage, the message displays the non-ascii 
characters correctly.

In the archive's downloaded .txt (and also .gz) file, the non-ascii characters 
are missing and displayed as "?".

I've copied the message text in below, from both the correct one from the email 
and the erroneous .txt file. Hopefully they won't get scrambled up when I send 
this.

Any advice on getting the non-ascii characters written into the archive .txt 
file would be gratefully received.

Thanks,
Mark


=== Message text as okay in mbox and as shown on the archive webpage ===

If one goes by the definition of veḷippaṭai as given in the Tamil Lexicon that 
the meaning of an ambiguous word should be disambiguated by a qualifying word, 
then aruvi āmpal does not conform to that definition since in the case of aruvi 
āmpal in Patiṟṟuppattu 63, aruvi is really made up of aru+vi, a compound. 
Moreover, the expression aṭai aṭuppu aṟiyā is already there to clarify that 
āmpal is a number and not a flower. Thus, aruvi simply provides information in 
addition to aṭai aṭuppu aṟiyā that āmpal is not a flower. The modern 
commentator Aruḷampalavaṉār also does not call it veḷippaṭai.

=== 

=== Message text with missing characters in te archive's txt and gz downloads ==

If one goes by the definition of ve?ippa?ai as given in the Tamil Lexicon that 
the meaning of an ambiguous word should be disambiguated by a qualifying word, 
then aruvi ?mpal does not conform to that definition since in the case of aruvi 
?mpal in Pati??uppattu 63, aruvi is really made up of aru+vi, a compound. 
Moreover, the expression a?ai a?uppu a?iy? is already there to clarify that 
?mpal is a number and not a flower. Thus, aruvi simply provides information in 
addition to a?ai a?uppu a?iy? that ?mpal is not a flower. The modern 
commentator Aru?ampalava??r also does not call it ve?ippa?ai.

===
--
Mailman-Users mailing list -- mailman-users@python.org
To unsubscribe send an email to mailman-users-le...@python.org
https://mail.python.org/mailman3/lists/mailman-users.python.org/
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: https://www.mail-archive.com/mailman-users@python.org/
https://mail.python.org/archives/list/mailman-users@python.org/

[Mailman-Users] Re: Non-ascii characters missing from Pipermail archive txt and gz downloads

2021-04-09 Thread Mark Sapiro 
On 4/9/21 5:55 AM, Mark Dale via Mailman-Users wrote:
> 
> In the archive's downloaded .txt (and also .gz) file, the non-ascii 
> characters are missing and displayed as "?".
...
> Any advice on getting the non-ascii characters written into the archive .txt 
> file would be gratefully received.


The message is prepared for the .txt file by the Article.as_text()
method in HyperArch.py
.
In order to do the email address obfuscation in the message body,
whether or not ARCHIVER_OBSCURES_EMAILADDRS is True, the method first
converts the body to unicode using the charset of the list's language
and then after possible obfuscation, converts it back, again using the
charset of the list's language. Both these conversions use
`errors=replace` which replaces any characters not in the charset with,
in the case of ascii, `?`.

One way to avoid this replacement would be to change the charset for
English from ascii to utf-8. See .

This isn't a complete solution in the case where the non-ascii
characters are encoded other than `utf-8`, e.g., `iso-8859-1`, in the
original message, but will probably handle most cases


-- 
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
--
Mailman-Users mailing list -- mailman-users@python.org
To unsubscribe send an email to mailman-users-le...@python.org
https://mail.python.org/mailman3/lists/mailman-users.python.org/
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: https://www.mail-archive.com/mailman-users@python.org/
https://mail.python.org/archives/list/mailman-users@python.org/

[Mailman-Users] Need to add two addresses to a bunch of lists as allowed senders.

2021-04-09 Thread Bruce Johnson 
l am mildly familiar with withlist and python but my google-fu today is weak, I 
haven’t found any example of this. The setting is ‘accept_these_nonmembers’ 
which I’m assuming is stored as a list in the database; do I need to retrieve 
it and append the new addresses to it? 

Can anyone point me to an example of doing something like this (or better, 
point me to one someone else has already made!  )


-- 
Bruce Johnson
University of Arizona
College of Pharmacy
Information Technology Group

Institutions do not have opinions, merely customs

--
Mailman-Users mailing list -- mailman-users@python.org
To unsubscribe send an email to mailman-users-le...@python.org
https://mail.python.org/mailman3/lists/mailman-users.python.org/
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: https://www.mail-archive.com/mailman-users@python.org/
https://mail.python.org/archives/list/mailman-users@python.org/

[Mailman-Users] Re: Need to add two addresses to a bunch of lists as allowed senders.

2021-04-09 Thread Christian Buser via Mailman-Users 
Hi Bruce

I never really understood the option ‘accept_these_nonmembers’ - but I
would anyway not use it.

If these are users who should be able to post messages but not want to
receive any messages, I would add them to the subscriber list and set
their mail delivery to "no message". In a cPanel installation, this is a
tick mark next to the address which needs to be set.

Christian


Bruce Johnson schrieb am 09.04.21 um 22:07:
> l am mildly familiar with withlist and python but my google-fu today is weak, 
> I haven’t found any example of this. The setting is ‘accept_these_nonmembers’ 
> which I’m assuming is stored as a list in the database; do I need to retrieve 
> it and append the new addresses to it? 
>
> Can anyone point me to an example of doing something like this (or better, 
> point me to one someone else has already made!  )
>
>

--
Mailman-Users mailing list -- mailman-users@python.org
To unsubscribe send an email to mailman-users-le...@python.org
https://mail.python.org/mailman3/lists/mailman-users.python.org/
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: https://www.mail-archive.com/mailman-users@python.org/
https://mail.python.org/archives/list/mailman-users@python.org/

[Mailman-Users] Re: Need to add two addresses to a bunch of lists as allowed senders.

2021-04-09 Thread Bruce Johnson 


On Apr 9, 2021, at 1:57 PM, Christian Buser 
mailto:lusche...@yahoo.de>> wrote:

Hi Bruce

I never really understood the option ‘accept_these_nonmembers’ - but I
would anyway not use it.

The issue here is for moderated class lists and this is pretty much exactly the 
use case for that option, afaik.


If these are users who should be able to post messages but not want to
receive any messages, I would add them to the subscriber list and set
their mail delivery to "no message". In a cPanel installation, this is a
tick mark next to the address which needs to be set.

I am looking for a way to NOT have to do this for each list individually; also 
these lists are updated via an entirely separate process nightly from our 
student database tables, so adding people as members-but-not-recievers would 
require an extra step of processing for all these lists to be implemented in 
our nightly scripts.



Bruce Johnson schrieb am 09.04.21 um 22:07:
l am mildly familiar with withlist and python but my google-fu today is weak, I 
haven’t found any example of this. The setting is ‘accept_these_nonmembers’ 
which I’m assuming is stored as a list in the database; do I need to retrieve 
it and append the new addresses to it?

Can anyone point me to an example of doing something like this (or better, 
point me to one someone else has already made!  )




--
Bruce Johnson
University of Arizona
College of Pharmacy
Information Technology Group

Institutions do not have opinions, merely customs

--
Mailman-Users mailing list -- mailman-users@python.org
To unsubscribe send an email to mailman-users-le...@python.org
https://mail.python.org/mailman3/lists/mailman-users.python.org/
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: https://www.mail-archive.com/mailman-users@python.org/
https://mail.python.org/archives/list/mailman-users@python.org/

[Mailman-Users] Re: Need to add two addresses to a bunch of lists as allowed senders.

2021-04-09 Thread Mark Sapiro 
On 4/9/21 1:07 PM, Bruce Johnson wrote:
> l am mildly familiar with withlist and python but my google-fu today is weak, 
> I haven’t found any example of this. The setting is ‘accept_these_nonmembers’ 
> which I’m assuming is stored as a list in the database; do I need to retrieve 
> it and append the new addresses to it? 
> 
> Can anyone point me to an example of doing something like this (or better, 
> point me to one someone else has already made!  )


See the script at 

-- 
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
--
Mailman-Users mailing list -- mailman-users@python.org
To unsubscribe send an email to mailman-users-le...@python.org
https://mail.python.org/mailman3/lists/mailman-users.python.org/
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: https://www.mail-archive.com/mailman-users@python.org/
https://mail.python.org/archives/list/mailman-users@python.org/

[Mailman-Users] Re: Need to add two addresses to a bunch of lists as allowed senders.

2021-04-09 Thread Bruce Johnson 
Exactly what I was looking for. Thanks, Mark!

> On Apr 9, 2021, at 2:36 PM, Mark Sapiro  wrote:
> 
> On 4/9/21 1:07 PM, Bruce Johnson wrote:
>> l am mildly familiar with withlist and python but my google-fu today is 
>> weak, I haven’t found any example of this. The setting is 
>> ‘accept_these_nonmembers’ which I’m assuming is stored as a list in the 
>> database; do I need to retrieve it and append the new addresses to it? 
>> 
>> Can anyone point me to an example of doing something like this (or better, 
>> point me to one someone else has already made!  )
> 
> 
> See the script at 
> 
> -- 
> Mark Sapiro The highway is for gamblers,
> San Francisco Bay Area, Californiabetter use your sense - B. Dylan
> --
> Mailman-Users mailing list -- mailman-users@python.org
> To unsubscribe send an email to mailman-users-le...@python.org
> https://mail.python.org/mailman3/lists/mailman-users.python.org/
> Mailman FAQ: http://wiki.list.org/x/AgA3
> Security Policy: http://wiki.list.org/x/QIA9
> Searchable Archives: https://www.mail-archive.com/mailman-users@python.org/
>https://mail.python.org/archives/list/mailman-users@python.org/

-- 
Bruce Johnson
University of Arizona
College of Pharmacy
Information Technology Group

Institutions do not have opinions, merely customs


--
Mailman-Users mailing list -- mailman-users@python.org
To unsubscribe send an email to mailman-users-le...@python.org
https://mail.python.org/mailman3/lists/mailman-users.python.org/
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: https://www.mail-archive.com/mailman-users@python.org/
https://mail.python.org/archives/list/mailman-users@python.org/