1. I expect your using IMAP based on the issues your mentioning
   IMAP works differently when it concerns expunging when using gmail. Now i don't know what mail server you are using so please let me know
   Have you seen this? http://www.mantisbt.org/wiki/doku.php/mantisbt:plugins:emailreporting#mailbox_type

2. MantisBT never updated the column type of old installations so it depends on that. I'm unsure from which version 1 mil applies but i'm pretty sure its at least MantisBT 1.3.x
   If you want me to investigate this please provide a raw_msg file create through the EmailReporting debug mode
   The truncated version can be a note or in the description
   For proper functioning everything should have been properly converted to UTF8 and the database needs to be in UTF8 mode as well

3. & 4.
   It might be that there is an error somewhere which EmailReporting encounters and stops processing further
   Can you find out if that is the case by running the bug_report_mail.php script manually when the truncation should occur?

Hi SL-Gundam, thanks for your prompt reply.

1 - we are using IMAP yes, and reaping mails from an Outlook/Exchange email account, which works fine for the most part except for the points mentioned: large email content not being entered into the description field, and the emails themselves not being deleted from the specially designated folder in my outlook, after they are reaped by the plugin.

2 - We are using the almost very latest MantisBT version 2.22.1. I will look into the other points and get back to you asap.

Best,

Andrei

Exchange is a bitch when it comes to IMAP because it seems it is using an incomplete implementation of IMAP4REV1.
We've had tons of fixes just so that it runs. It seems we have a new issue to deal with concerning Exchange

As earlier mentioned, please provide the raw_msg so that i can look into the truncation issue
As for the deletion issue i will debug it myself. I've got Office365 at my office so i will debug it there

Please find attached the raw message body of the kind of email we're dealing with, and trying to get truncated. I can't imagine you'd need the email header raw body, because it's purely a question of the size, but I can send that also if you think it would help.

I say it's all about the body size because I've done some tests ... steadily decreasing the number of rows in the ridiculously huge, repetitive table, and eventually the plugin does accept the mail body in the Description field.

Hope this helps, thanks.

Please provide the raw_msg. I want to pass it through EmailReporting to see what happens. That requires the email to be complete in its headers and structure

If you need be you can change any private information in the message but make sure the headers are structurely intact

Hi SL-Gundam, sorry for the delay. The raw header is here: https://pastebin.com/78LxHKQd . Hope this helps. Thanks, A.

I'm not sure how you created this dump but i need to raw_msg file that EmailReporting creates when its run in debug mode and processes an email.
It's need to be complete. This dump only has the headers, not the body

Please provide the complete dump

The raw message body was attached to this ticket earlier here: https://www.mantisbt.org/bugs/view.php?id=26442#c63309 ...

In fact I got both raw message body and header using Outlook! ... not EmailReporting in debug mode as you mention. Can you not put the 2 things together that I've now sent, or do we need to approach this differently?

It would take an awful lot of time to do that because of the way a raw email is structured

But I've done some testing and found an issue in case max length was encountered on old MantisBT installations.
The Email table would add extra content around the description which would put us over the 65535 characters causing a query error. So the default has now been lowered to 60000 characters.

Can you test the latest DEV version and see whether it works better
Also with your content you are at around 700.000 characters which is lower then 1.000.000. So would not be truncated. Please test with a maximum of 60.000. If it works you can adjust it later to a value of your choice

I've done a comprehensive test with an upgraded installation, now using mantisbt-2.24.0-dev, and it's no cigar. Any significantly large number of lines simply does NOT get pulled into the Description field. It doesn't even have to be that many lines. I get the same fail for instance with 1000 lines, as I do with several thousand, let alone 60,000.

There is another interesting aspect however: the failure only happens when the body of the mail contains a TABLE with a very large number of rows.

IF, however, I remove that same text from each row of the table, and make up a whole new email with the raw text alone, so it's a 1000 rows of text, but NO TABLE, that text DOES get reaped into the Description field!

Unfortunately we need to do an automated harvesting of the original mails that contain tables, but maybe this difference in behaviour will help debug the issue.

I've never tested EmailReporting HTML conversion on a large table. But i suspect it would take quite a lot of processing time with how Markdownify parses the content.
Make sure you are testing using the command line. This would allow EmailReporting to work as long as it needs. If you test through a browser there are often time limits as safety precautions in place which will stop processing after a certain time

Please provide a raw_msg (created by debug mode in EmailReporting) so that we can debug those issues further

Before providing a raw_msg created with debug mode, I have to mention the size of the mail body causing this issue is well over 60,000 bytes, in fact many times over. - 1,701,141 bytes to be exact as we can see in the attachment feature for the raw body I sent earlier at https://www.mantisbt.org/bugs/view.php?id=26442#c63309 !

The question is now: can the config option mail_max_email_body handle an email as big as 2M? I just specified 2000000 and still no cigar when testing, but maybe I need to ensure the relevant column type in the DB can handle that size also.

I refer now to the doc, which is not exactly clear:

=================================================================================================
mail_max_email_body
Name: Maximum size of the description or note being added to the issue

Description: This is the maximum size allowed for the description or a note. The choice was made to make the default 60000 bytes because 65535 used to be the default column length for old MantisBT versions prior to mid 2007 using MySQL. Since we need some overhead we lowered 65535 to 60000. New MantisBT installations after mid 2007 will be able to store upwards of 1 million bytes. Even more depending on which database backend you use. As updating the schema does not update the column size in question this would be a good default that would not cause problematic issues

Of course you can increase this if you want. Just make sure that your MantisBT database is using a column type in which this will fit. This is usually the case if you MantisBT installation was created using MantisBT 1.1.x or later

==============================================================================

[not sure how that text came out so big ..]

EmailReporting can handle anything as long as you have enough memory and the database can accept the amount of data. Because of the different databases that might be used with MantisBT, the maximum is not set in stone

I expect this to be an issue with the amount of processing time required for an email that large... either that or a memory issue...