View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0011499 | mantisbt | performance | public | 2010-02-13 00:29 | 2014-06-11 14:51 |
| Reporter | dylanc | Assigned To | grangeway | ||
| Priority | urgent | Severity | major | Reproducibility | always |
| Status | closed | Resolution | no change required | ||
| Product Version | 1.1.8 | ||||
| Summary | 0011499: Fix for huge slowdown in e-mail queueing with MySql | ||||
| Description | Change the field types for metadata and body in the e-mail database table to not be longtext, and switch them to mediumtext (still allows for 16 million bytes of data!) This is a problem in the schema preparation - the email table is defined as XL and this translates to longtext under mysql - looking around on the net shows that people never use longtext in a regular table (it allows for up to 4 terabytes of data and is inefficient if used like a normal field). The slowdowns aren't visible if using the InnoDB backend for MySql which is why reports of this slowdown are scattered, but even then I'd suggest changing the longtext to mediumtext for efficiency. This field change should be added to the database upgrade code too. | ||||
| Steps To Reproduce | update any bug when using MySql 4.x (haven't tried 5.x) and with email notifications on that sends 2-3 notifications. On a modern fast server this can take 30 seconds to a minute for the database update to complete. (the bug is updated immediately - it takes all that time solely to write out the e-mail table) | ||||
| Additional Information | I was seeing a huge slowdown in the e-mailing notifications performance, and looking on the forums I saw scattered reports of other people hitting the same (30 seconds or more on an update) slowdown. The common response was to switch on the cronjob batching but this actually made no difference because the problem is in the database storage for the batching system itself. So hopefully this will fix this obscure problem for a lot of people. | ||||
| Tags | No tags attached. | ||||
|
Thanks for this hint. We also had this problem and never thought that the e-mail table caused it. Now everything runs fine! |
|
|
Thanks, I already changed mantis_email_table->metadata/body from longtext to mediumtext.
|
|
|
We're also having huge slowdowns when updating/assigning a bug to another user.
[UPDATE] |
|
|
Notes from Dylan: Perhaps on 64-bit machines a LONGTEST field is fast? All I know is that on our servers which were old P4s at the time using the field type absolutely destroyed performance, and as I mention in the notes that particular data type isn’t used in normal databases (at least that was the sentiment at the time I recall) because well, it’s overkill. I tried all the other solutions suggested for speeding up the mailing (batching etc) and none of them were relevant to the problem I was having, although they do fix a lot of smaller speed problems. Basically the “huge speed down” I was seeing was Really slow, as in minutes to send a single mail. Perhaps there was disk-swapping going on or simply a bug in mySQL, I really don’t know, all I know is that switching the type sped it all back up again to sane speeds. Also, I seem to recall that looking at the types of fields the XL type maps to for other database solutions showed a discrepancy. XL -> LONGTEXT simply seemed to be an incorrect over-sized mapping to me. |
|
|
Marking as no change required - my email to dev list with current thoughts for if this comes up again, or we can reproduce below: There are indeed reports on the internet of similar behaviour - e.g. Looking at http://nicj.net/mysql-text-vs-varchar-performance/ and http://dev.mysql.com/doc/refman/5.7/en/internal-temporary-tables.html If I were to hazard a guess, I'd probably be inclined to blame the use of 'order by' in some version of mysql. Personally, I think I'm inclined to leave as is for now, and will close off this bug report. I'm inclined to modify email_queue_get_ids to always return ID's in the order of creation (we want emails to leave the mysql server in the order the notifications were generated - I'll submit a PR for this, so there's no need to ever do a reverse sort). If we were to take a stab at avoiding a potential issue here, I'd be inclined to remove the ORDER BY clause completely, on the basis that the default expected state of the email table is to be empty at all times - when it's non-empty i.e. emails waiting to be processed, we always process all emails in one go assuming at working mail server. However, given the probability of a mail server breaking, and the fact that we want the ID's in order, and without the order by clause the order is undefined, I'm not going to remove the clause |
|