View Issue Details

IDProjectCategoryView StatusLast Update
0023549mantisbtdb mysqlpublic2018-03-31 19:58
Reporterc_schmitz Assigned Toatrol  
PrioritynormalSeveritycrashReproducibilityalways
Status closedResolutionfixed 
Product Version2.7.0 
Target Version2.13.0Fixed in Version2.13.0 
Summary0023549: Entering Emojis in comments with a user mention crashes with an error
Description

When entering an emoji in a comment field along with a mention of a username Mantis crashes with error message

APPLICATION ERROR 0000401

Database query failed. Error received from database was 0001366: Incorrect string value: '\xF0\x9F\x98\xB1' for column 'body' at row 1 for the query: INSERT INTO mantis_email_table
( email, subject, body, submitted, metadata)
VALUES
(?,?,?,?,?).
Please use the "Back" button in your web browser to return to the previous page. There you can correct whatever problems were identified in this error or select another action. You can also click an option from the menu bar to go directly to a new section.

Steps To Reproduce

Added a comment to a existing bug like:

"hey @someuser , that is so bad <some_emoji>"

After posting the above crash message appears.

TagsNo tags attached.

Relationships

related to 0020431 assigneddregad Use utf8mb4 charset for new MySQL installations 
related to 0021101 closeddregad Issues with emoji's are truncated before getting saved 

Activities

c_schmitz

c_schmitz

2017-10-26 06:30

reporter   ~0058061

Related to 0021101

atrol

atrol

2017-10-26 07:07

developer   ~0058062

Last edited: 2017-10-26 07:08

I am too old to understand the importance of this emoji stuff https://www.mantisbt.org/forums/viewtopic.php?f=2&amp;t=24787

atrol

atrol

2017-10-26 07:38

developer   ~0058064

c_schmitz, I was not able to reproduce your problem with a fresh install of 2.7.0

Please provide detailed step-by-step instructions to reproduce the issue; the following additional information may also be useful:

  • Do you get any errors/or warnings (especially related to utf-8) when runing admin/check/index.php ?
  • Exact version of PHP, Database, Web server and Operating System
  • Relevant customizations (e.g. changes in config_inc.php, etc)
  • Installed plugins or custom functions ?
  • Was the MantisBT source code modified in any way ?
c_schmitz

c_schmitz

2017-10-26 08:38

reporter   ~0058065

@atrol Let me try here: �

atrol

atrol

2017-10-26 08:42

developer   ~0058066

Setting again to waiting for more feedback

atrol

atrol

2017-10-26 10:38

developer   ~0058069

I am wondering that your config_inc.php does not contain any setting for $g_db_type
It must not be set in config_inc.php as there is a default for it in config_defaults_inc.php, but it's quite typical to set it.
Can I be sure that you did not change config_defaults_inc.php, so that $g_db_type is set to 'mysqli' ?

Can you confirm that column body of table mantis_email_table is of type longtext, character set is utf8 and collation is utf8_general_ci ?

atrol

atrol

2017-11-05 07:54

developer   ~0058135

c_schmitz,

You did not provide feedback; I am therefore resolving this issue as "no change required".

Feel free to reopen the issue at a later time and provide the requested information.

c_schmitz

c_schmitz

2017-11-16 11:35

reporter   ~0058212

Can I be sure that you did not change config_defaults_inc.php, so that $g_db_type is set to 'mysqli' ?

config_defaults is unmodified and the factory one.

Can you confirm that column body of table mantis_email_table is of type longtext, character set is utf8 and collation is utf8_general_ci ?

I can confirm that.

atrol

atrol

2017-12-01 20:29

developer   ~0058298

I am still not able to reproduce the issue using the given information.
I tried on different test systems, of course they are not exactly the same than yours.
I use other PHP versions (newer ones and older ones) and Apache as web server.
Maybe you can try newer PHP versions.

Did you check for errors or warnings in Web server, PHP or Database log ?

atrol

atrol

2017-12-17 06:02

developer   ~0058404

c_schmitz,

You did not provide new feedback; I am therefore resolving this issue as "no change required".

Feel free to reopen the issue at a later time and provide the requested information.

c_schmitz

c_schmitz

2018-02-20 06:49

reporter   ~0058976

Did some more tests today.

Interestingly when I add

$g_display_errors = array(
E_USER_WARNING => DISPLAY_ERROR_HALT,
E_WARNING => DISPLAY_ERROR_HALT,
E_ALL => DISPLAY_ERROR_INLINE,
);

to config_inc.php the error is gone. Pretty weird, heh?
Actually I was hoping for some stack dump.

c_schmitz

c_schmitz

2018-02-20 07:16

reporter   ~0058977

OK, I think I have an understanding of the issue.
I think we can all agree that inserting a 4-byte UTF-8 character into MySQL table having only the collation utf8_general_ci is a problem.
Anyway, our database cranks out on that (no matter the reason why yours doesn't do that).

There has been a patch to mitigate this problem here at https://github.com/mantisbt/mantisbt/pull/797/files
Indeed this patch does sanitize the bug note, but too late - the 'mentions' are processed earlier in the code before the sanitizing by the patch occurs.

Fix:

Insert the line

   $f_bugnote_text = db_mysql_fix_utf8( $f_bugnote_text );

after the line

# Process the mentions in the added note

in bugnote_add.php

I am not sure if this is the correct way to fix it but for me it did the trick.

atrol

atrol

2018-02-20 07:50

developer   ~0058978

Thanks @c_schmitz that makes sense now.

in bugnote_add.php

Seems you are still talking about version 2.7.0.
The code has changed quite a lot since that time, so you will not able to apply the same trick to latest version.

@dregad I am not complete sure, but I think the issue happens when trying to store the note in email queue.
So we might need an additional db_mysql_fix_utf8 in email_queue_add()

c_schmitz

c_schmitz

2018-02-20 08:14

reporter   ~0058979

Just updated. You are right - I cannot reproduce this using the latest version so matter seems to be resolved now.

atrol

atrol

2018-02-20 08:24

developer   ~0058980

Last edited: 2018-03-08 05:06

I cannot reproduce this using the latest version so matter seems to be resolved now.

I wouldn't have expected this, as we still try to store 4-byte UTF-8 to 3-byte UTF-8 in MySQL.

But it was weird that I was not able to reproduce before, it was weird that you have not been able to reproduce when changing $g_display_errors and now it's weird that it's fixed for you.

We might talk about something like writing 1 Byte too much on stack, so depending on the call stack it will crash or not.

c_schmitz

c_schmitz

2018-02-20 08:45

reporter   ~0058981

I agree. So it still would be great if you can sanitize the call accordingly for next version.

atrol

atrol

2018-03-08 05:31

developer   ~0059116

Just updated. You are right - I cannot reproduce this using the latest version so matter seems to be resolved now.

Again weird, as I was able to reproduce.

This PR should finally fix it: https://github.com/mantisbt/mantisbt/pull/1313

Related Changesets

MantisBT: master 42e57d4a

2018-03-08 00:22

atrol


Details Diff
Don't store 4-byte UTF-8 characters in MySQL 3-byte utf8 encoding

Fixes 0023549
Affected Issues
0023549
mod - core/email_queue_api.php Diff File