Prevent errors when loading emails

[pmosconi]

Hi,
I had a few errors when loading emails into Mantis through the plugin and I traced them down to 2 issues:
1. the email contains UTF-8 characters that are not accepted by the database
2. the email body lenght is longer than mysql text field

To prevent these I made a small change to the [...]\plugins\EmailReporting\core\mail.api file: I hope this info can be useful to who has similar issues and ultimately it could find its way into the code.

private function add_bug( &$p_email, $p_overwrite_project_id = FALSE )

replace:
$t_bug_data->description = $t_description;
with:
// PM 20140613: change encoding from UTF-8 to avoid errors due to strange characters and check for max message lenght
define('CHARSET', 'HTML-ENTITIES');
define ('MAX_TEXT_LEN', 65535); // mysql text datatype
$t_bug_data->description = mb_convert_encoding($t_description, CHARSET);
if (strlen($t_bug_data->description) > MAX_TEXT_LEN)
{
$t_bug_data->description = substr($t_bug_data->description, 0, MAX_TEXT_LEN);
}
// $t_bug_data->description = $t_description;

Hope this helps
paolo

[SL-Gundam]

Which version of EmailReporting are you running?
Which version of MantisBT are you running?

[pmosconi]

Mantis 1.2.17
Schema 183
EmailReporting-0.9.0-DEV

[SL-Gundam]

I would like to have a debug dump of said email so that i can reproduce this myself as I've been testing and i'm unable to reproduce number 2 at the moment

What database type are you using?
If MySQL is it running in strict mode or not?

What UTF-8 characters are we talking about? There should be full support for UTF-8 since conversions already take place in EmailReporting

What errors were you getting?

[pmosconi]

Re point #1: Mysql instance is shared by several databases and was not created with Mantis, Mantis db was recently moved during the upgrade from an older version. The original db is fairly old (over 5 yrs) and was upgraded a few times. Our language encoding is ISO-8859-1, but I am no mysql dba and I have no idea how to check the db / instance setting (nor I can mess with them since it is a production instance). When an email message contains other characters, the plugin fails with a db error in the INSERT statement.

Re point #2: please, send me your email via private message so that I can forward you an example. The bug data description field is defined as text, which in mysql has a 64k max lenght: any longer message will cause a db error in the INSERT statement.

[SL-Gundam]

The Mantis DB should always be UTF-8 since at least 1.1.0 i believe. Yours would be a MantisBT bug from upgrades in the past. I suggest you fix this in the database.
Easiest would probably be to
1. extract all the data from the database (not the schema)
2. drop all MantisBT tables
3. install MantisBT
4. truncate all MantisBT tables
5. reimport all the data

In some rare configurations above might fail so backup before trying this.

email address has been sent. Point 2 might fail because of strict mode in mysql. I believe strict mode is disabled in my development environment so in my situation it automatically truncates. But i will test this more

[pmosconi]

Thanks for your help and suggestions.
I sent you an example of email that is too big for my installation of Email Reporting.

[SL-Gundam]

Alright i checked MySQL strict mode for point 2. This is not a bug in EmailReporting but rather in MantisBT. Because of that this will be added to the todolist with a low priority

See here for some more info: http://stackoverflow.com/questions/1407 ... -inserting

[pmosconi]

Once again, many thanks for your help!

[SL-Gundam]

Ah i see . In your situation the email got bloated by an error. Thats nasty. But i do have an idea with which we can fix this.

I will post here after thats done

[pmosconi]

Actually, I instructed Outlook to ignore that particular kind of message via rules. But that's just an example: it's not uncommon to have an email trail with lot of useless disclaimer texts become so big to max the field out.

[SL-Gundam]

Some changes have been pushed to fix the field length issues.

https://github.com/mantisbt-plugins/Ema ... 75b24925e7

It fixes subject and description/note issues. I still need to test it in a live environment though.

It will try moving big descriptions to the attachments if thats allowed otherwise the description will be truncated

[SL-Gundam]

While i was busy creating an issue about this in the MantisBT bugtracker i did some more checking.

The MantisBT database is supposed to be using the MYISAM database engine and using the utf8 charsets. If both of these are correct then you should not have the problems you mentioned in the first post as they are applicable to the InnoDB database engine functioning in strict mode and the database not using the utf8 charsets.

I'm going to revert some of the changes now as these are problems that should not exist in a proper MantisBT database schema. Please follow the steps mentioned here http://www.mantisbt.org/forums/viewtopi ... 294#p55294 and make sure the new database is of the MyISAM type

[pmosconi]

I cannot make the changes because the instance is shared, but now I know why and how!
Thanks

[SL-Gundam]

I don't see how that should be a problem. In a single database you can both have MYISAM tables and innodb tables (same goes for charsets) so the only issue should be a small amount of downtime while you do the necessary work.

But you know your set-up better then i do.

Anyways this issue is closed now