Prevent errors when loading emails
Moderators: Developer, Contributor
Prevent errors when loading emails
Hi,
I had a few errors when loading emails into Mantis through the plugin and I traced them down to 2 issues:
1. the email contains UTF-8 characters that are not accepted by the database
2. the email body lenght is longer than mysql text field
To prevent these I made a small change to the [...]\plugins\EmailReporting\core\mail.api file: I hope this info can be useful to who has similar issues and ultimately it could find its way into the code.
private function add_bug( &$p_email, $p_overwrite_project_id = FALSE )
replace:
$t_bug_data->description = $t_description;
with:
// PM 20140613: change encoding from UTF-8 to avoid errors due to strange characters and check for max message lenght
define('CHARSET', 'HTML-ENTITIES');
define ('MAX_TEXT_LEN', 65535); // mysql text datatype
$t_bug_data->description = mb_convert_encoding($t_description, CHARSET);
if (strlen($t_bug_data->description) > MAX_TEXT_LEN)
{
$t_bug_data->description = substr($t_bug_data->description, 0, MAX_TEXT_LEN);
}
// $t_bug_data->description = $t_description;
Hope this helps
paolo
I had a few errors when loading emails into Mantis through the plugin and I traced them down to 2 issues:
1. the email contains UTF-8 characters that are not accepted by the database
2. the email body lenght is longer than mysql text field
To prevent these I made a small change to the [...]\plugins\EmailReporting\core\mail.api file: I hope this info can be useful to who has similar issues and ultimately it could find its way into the code.
private function add_bug( &$p_email, $p_overwrite_project_id = FALSE )
replace:
$t_bug_data->description = $t_description;
with:
// PM 20140613: change encoding from UTF-8 to avoid errors due to strange characters and check for max message lenght
define('CHARSET', 'HTML-ENTITIES');
define ('MAX_TEXT_LEN', 65535); // mysql text datatype
$t_bug_data->description = mb_convert_encoding($t_description, CHARSET);
if (strlen($t_bug_data->description) > MAX_TEXT_LEN)
{
$t_bug_data->description = substr($t_bug_data->description, 0, MAX_TEXT_LEN);
}
// $t_bug_data->description = $t_description;
Hope this helps
paolo
Re: Prevent errors when loading emails
Which version of EmailReporting are you running?
Which version of MantisBT are you running?
Which version of MantisBT are you running?
Re: Prevent errors when loading emails
Mantis 1.2.17
Schema 183
EmailReporting-0.9.0-DEV
Schema 183
EmailReporting-0.9.0-DEV
Re: Prevent errors when loading emails
I would like to have a debug dump of said email so that i can reproduce this myself as I've been testing and i'm unable to reproduce number 2 at the moment
What database type are you using?
If MySQL is it running in strict mode or not?
What UTF-8 characters are we talking about? There should be full support for UTF-8 since conversions already take place in EmailReporting
What errors were you getting?
What database type are you using?
If MySQL is it running in strict mode or not?
What UTF-8 characters are we talking about? There should be full support for UTF-8 since conversions already take place in EmailReporting
What errors were you getting?
Re: Prevent errors when loading emails
Re point #1: Mysql instance is shared by several databases and was not created with Mantis, Mantis db was recently moved during the upgrade from an older version. The original db is fairly old (over 5 yrs) and was upgraded a few times. Our language encoding is ISO-8859-1, but I am no mysql dba and I have no idea how to check the db / instance setting (nor I can mess with them since it is a production instance). When an email message contains other characters, the plugin fails with a db error in the INSERT statement.
Re point #2: please, send me your email via private message so that I can forward you an example. The bug data description field is defined as text, which in mysql has a 64k max lenght: any longer message will cause a db error in the INSERT statement.
Re point #2: please, send me your email via private message so that I can forward you an example. The bug data description field is defined as text, which in mysql has a 64k max lenght: any longer message will cause a db error in the INSERT statement.
Re: Prevent errors when loading emails
The Mantis DB should always be UTF-8 since at least 1.1.0 i believe. Yours would be a MantisBT bug from upgrades in the past. I suggest you fix this in the database.
Easiest would probably be to
1. extract all the data from the database (not the schema)
2. drop all MantisBT tables
3. install MantisBT
4. truncate all MantisBT tables
5. reimport all the data
In some rare configurations above might fail so backup before trying this.
email address has been sent. Point 2 might fail because of strict mode in mysql. I believe strict mode is disabled in my development environment so in my situation it automatically truncates. But i will test this more
Easiest would probably be to
1. extract all the data from the database (not the schema)
2. drop all MantisBT tables
3. install MantisBT
4. truncate all MantisBT tables
5. reimport all the data
In some rare configurations above might fail so backup before trying this.
email address has been sent. Point 2 might fail because of strict mode in mysql. I believe strict mode is disabled in my development environment so in my situation it automatically truncates. But i will test this more
Re: Prevent errors when loading emails
Thanks for your help and suggestions.
I sent you an example of email that is too big for my installation of Email Reporting.
I sent you an example of email that is too big for my installation of Email Reporting.
Re: Prevent errors when loading emails
Alright i checked MySQL strict mode for point 2. This is not a bug in EmailReporting but rather in MantisBT. Because of that this will be added to the todolist with a low priority
See here for some more info: http://stackoverflow.com/questions/1407 ... -inserting
See here for some more info: http://stackoverflow.com/questions/1407 ... -inserting
Re: Prevent errors when loading emails
Once again, many thanks for your help!
Re: Prevent errors when loading emails
Ah i see . In your situation the email got bloated by an error. Thats nasty. But i do have an idea with which we can fix this.
I will post here after thats done
I will post here after thats done
Re: Prevent errors when loading emails
Actually, I instructed Outlook to ignore that particular kind of message via rules. But that's just an example: it's not uncommon to have an email trail with lot of useless disclaimer texts become so big to max the field out.
Re: Prevent errors when loading emails
Some changes have been pushed to fix the field length issues.
https://github.com/mantisbt-plugins/Ema ... 75b24925e7
It fixes subject and description/note issues. I still need to test it in a live environment though.
It will try moving big descriptions to the attachments if thats allowed otherwise the description will be truncated
https://github.com/mantisbt-plugins/Ema ... 75b24925e7
It fixes subject and description/note issues. I still need to test it in a live environment though.
It will try moving big descriptions to the attachments if thats allowed otherwise the description will be truncated
Re: Prevent errors when loading emails
While i was busy creating an issue about this in the MantisBT bugtracker i did some more checking.
The MantisBT database is supposed to be using the MYISAM database engine and using the utf8 charsets. If both of these are correct then you should not have the problems you mentioned in the first post as they are applicable to the InnoDB database engine functioning in strict mode and the database not using the utf8 charsets.
I'm going to revert some of the changes now as these are problems that should not exist in a proper MantisBT database schema. Please follow the steps mentioned here http://www.mantisbt.org/forums/viewtopi ... 294#p55294 and make sure the new database is of the MyISAM type
The MantisBT database is supposed to be using the MYISAM database engine and using the utf8 charsets. If both of these are correct then you should not have the problems you mentioned in the first post as they are applicable to the InnoDB database engine functioning in strict mode and the database not using the utf8 charsets.
I'm going to revert some of the changes now as these are problems that should not exist in a proper MantisBT database schema. Please follow the steps mentioned here http://www.mantisbt.org/forums/viewtopi ... 294#p55294 and make sure the new database is of the MyISAM type
Last edited by SL-Gundam on 18 Jun 2014, 11:35, edited 1 time in total.
Re: Prevent errors when loading emails
I cannot make the changes because the instance is shared, but now I know why and how!
Thanks
Thanks
Re: Prevent errors when loading emails
I don't see how that should be a problem. In a single database you can both have MYISAM tables and innodb tables (same goes for charsets) so the only issue should be a small amount of downtime while you do the necessary work.
But you know your set-up better then i do.
Anyways this issue is closed now
But you know your set-up better then i do.
Anyways this issue is closed now