View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0025701 | Plugin - EmailReporting | General | public | 2019-04-17 14:55 | 2019-04-25 17:50 |
Reporter | Daveid | Assigned To | SL-Gundam | ||
Priority | normal | Severity | minor | Reproducibility | have not tried |
Status | resolved | Resolution | fixed | ||
Summary | 0025701: Slow IMAP parsing | ||||
Description | The plugin has become very slow, taking 0000060:0000120 seconds to run. I have narrowed it down to the parsing the response of the FETCH imap calls I see it makes 739 FETCH calls, the first of which seems to be for a message for March 13th, it starts as follows
the last is from an email from today. 90% of the run time seems to be parsing these 739 responses, i have observed as such with logging also. I get the feeling that it's not supposed to get getting 739 messages every time and in fact these are supposed to be marked somewhere as done? Not sure when this started happening but any help is appreciated :) mantis is version 2.19.0, EmailReporting is github master, email server is office365 have attached some trace / profiling info | ||||
Tags | No tags attached. | ||||
Attached Files | |||||
0000060:0000120 was supposed to be approx 120 seconds ( |
|
Well the issue is that for every message in a folder EmailReprting needs to retrieve the status flags and verify to see if the message needs to be processed (this includes messages marked as deleted) So if you've turned off http://www.mantisbt.org/wiki/doku.php/mantisbt:plugins:emailreporting#mail_delete then this can add up quite a bit Now it happens that recently with had an issue: https://mantisbt.org/forums/viewtopic.php?f=13&t=26100 This fix resulted in a significant performance improvement concerning IMAP communication So my advise is update to 0.11.0-DEV (https://github.com/mantisbt-plugins/EmailReporting) and see whether thats also true for your situation |
|
If possible please also provide the same graphs for the new version so that we can see how much performance has improved and if there are other areas that might need looking at |
|
thanks for the speedy reply! sorry to disappoint but I updated to master after initially finding it was slow with imap, i notice that caching was enabled so hope that it might help. I added when running i see the many responses fly past quickly with the response immediately followed by done. when the 700 odd FETCH messages fly passed the |
|
I'm sorry to say i don't have any test situation where it takes that long. The function in question is part of a PHP PEAR package called Net_IMAP EmailReporting uses. Its hasn't had any official updates since 2014 I've only applied changes to that package when compatibility required it. I still have plans to start using https://www.php.net/manual/en/book.imap.php Is it possible you provide me the content of the folder so that i might be able to reproduce the issue? Otherwise you will have to dig in to this yourself and find out what must be done to improve performance The function that is at the source of this is EmailReporting's getListing function |
|
Maybe if this fixes it we will need to perform the sorting of the emails after processing the flags... |
|
Actually already found another method to fix this if you confirm the solution Please let me know if this works for you 0001-Optimized-IMAP-email-sorting.patch (2,997 bytes)
From b6638cf499e41b6979b86f78975c9a84d66964e8 Mon Sep 17 00:00:00 2001 From: SL-Gundam <slgundam@gmail.com> Date: Thu, 18 Apr 2019 01:22:55 +0200 Subject: [PATCH] Optimized IMAP email sorting Fix email processing order --- core/mail_api.php | 37 +++++++++++-------------------------- doc/CHANGELOG.txt | 2 ++ 2 files changed, 13 insertions(+), 26 deletions(-) diff --git a/core/mail_api.php b/core/mail_api.php index fc531d5..02d2ff0 100644 --- a/core/mail_api.php +++ b/core/mail_api.php @@ -346,7 +346,7 @@ class ERP_mailbox_api if ( !$this->pear_error( 'Retrieve list of messages', $t_ListMsgs ) ) { - while ( $t_Msg = array_pop( $t_ListMsgs ) ) + while ( $t_Msg = array_shift( $t_ListMsgs ) ) { $t_emailresult = $this->process_single_email( $t_Msg[ 'msg_id' ] ); @@ -445,7 +445,7 @@ class ERP_mailbox_api { $t_flags = $this->_mailserver->getFlags(); - while ( $t_Msg = array_pop( $t_ListMsgs ) ) + while ( $t_Msg = array_shift( $t_ListMsgs ) ) { $t_isDeleted = $this->isDeleted( $t_Msg[ 'msg_id' ], $t_flags ); @@ -536,35 +536,20 @@ class ERP_mailbox_api # Needed a workaround to sort IMAP emails in a certain order private function getListing() { - $t_ListMsgs = NULL; + $t_ListMsgs = $this->_mailserver->getListing(); - if ( $this->_mailbox[ 'mailbox_type' ] === 'IMAP' ) + if ( !PEAR::isError( $t_ListMsgs ) ) { - $t_getSummary = $this->_mailserver->getSummary(); - - if ( !$this->pear_error( 'IMAP Get folder info', $t_getSummary ) ) + if ( $this->_mailbox[ 'mailbox_type' ] === 'IMAP' ) { - $t_nummsg = count( $t_getSummary ); - $t_ListMsgs = array(); - for ( $i = 0; $i < $t_nummsg; $i++ ) - { - $t_getSummary[ $i ][ 'DATE' ] = strtotime( $t_getSummary[ $i ][ 'DATE' ] ); - // If strtotime fails we default back to the current time. - if ( $t_getSummary[ $i ][ 'DATE' ] === FALSE ) - { - $t_getSummary[ $i ][ 'DATE' ] = time(); - } - - $t_ListMsgs[ $t_getSummary[ $i ][ 'DATE' ] ] = array( 'msg_id' => (int) $t_getSummary[ $i ][ 'MSG_NUM' ] ); - } - - krsort( $t_ListMsgs ); + $t_ListMsgs = array_column( $t_ListMsgs, NULL, 'uidl' ); + } + else + { + $t_ListMsgs = array_column( $t_ListMsgs, NULL, 'msg_id' ); } - } - if ( $t_ListMsgs === NULL ) - { - $t_ListMsgs = $this->_mailserver->getListing(); + ksort( $t_ListMsgs ); } return( $t_ListMsgs ); diff --git a/doc/CHANGELOG.txt b/doc/CHANGELOG.txt index b136245..1f38eec 100644 --- a/doc/CHANGELOG.txt +++ b/doc/CHANGELOG.txt @@ -12,6 +12,8 @@ Sep 2018 - EmailReporting-0.11.0 - Limited max length of message id (#25433) - Add support for Windows-1257 charset - Enable the caching of IMAP flags + - Optimized IMAP email sorting + - Fix email processing order May 2018 - EmailReporting-0.10.1 - Added error if, on installation, 'api/soap/mc_file_api.php' is missing -- 2.21.0.windows.1 |
|
With that patch it's down to 19 seconds total run time, undoing the patch takes it back up to 120ish |
|
since you asked IMAP1 (alone) takes the total runtime to approx 19 seconds also |
|
Comparison of the profiles (incl column is milliseconds, and profiling makes it slower) |
|
wrong file |
|
sorry both the new runs are IMAP1 :) my bad. will get the results with the patch |
|
Changes have been pushed to master This commit is a little bit different from the patch so update your installation as well I will be examining the provided trace and profiling info later tonight Thank you |
|
I did some poking around Obviously BODY is probably needed for the plugin to work however creating |
|
i have a patch that makes this section more efficient fewer_substr.patch (1,655 bytes)
diff --git a/core_pear/Net/IMAPProtocol.php b/core_pear/Net/IMAPProtocol.php index ced0c75..7791a82 100644 --- a/core_pear/Net/IMAPProtocol.php +++ b/core_pear/Net/IMAPProtocol.php @@ -2794,23 +2794,25 @@ class Net_IMAPProtocol default: for ($pos = 0; $pos < $len; $pos++) { - if ($this->_getSubstr($str, 0, 5) == 'BODY[' - || $this->_getSubstr($str, 0, 5) == 'BODY.') { + $c = $str[$pos]; + if ($c =='B' && + ($this->_getSubstr($str, 0, 5) == 'BODY[' + || $this->_getSubstr($str, 0, 5) == 'BODY.')) { if ($str[$pos] == ']') { $pos++; break; } - } elseif ($str[$pos] == ' ' - || $str[$pos] == "\r" - || $str[$pos] == ')' - || $str[$pos] == '(' - || $str[$pos] == "\n" ) { + } else if ($c == ' ' + || $c == "\r" + || $c == ')' + || $c == '(' + || $c == "\n" ) { break; } - if ($str[$pos] == "\\" && $str[$pos + 1 ] == ' ') { + if ($str[$pos] == '\\' && $str[$pos + 1 ] == ' ') { $pos++; } - if ($str[$pos] == "\\" && $str[$pos + 1 ] == "\\") { + if ($str[$pos] == '\\' && $str[$pos + 1 ] == '\\') { $pos++; } } |
|
nope, the patch is wrong :) but close |
|
fixed the patch, i though that it was checking whether BODY was at $pos, but in fact it was just if $str started with BODY, so just need to do it once fewer_substr-2.patch (1,433 bytes)
diff --git a/core_pear/Net/IMAPProtocol.php b/core_pear/Net/IMAPProtocol.php index ced0c75..bd42a27 100644 --- a/core_pear/Net/IMAPProtocol.php +++ b/core_pear/Net/IMAPProtocol.php @@ -2793,18 +2793,21 @@ class Net_IMAPProtocol break; default: + $startsWithBody = $str[0] == 'B' + && ($this->_getSubstr($str, 0, 5) == 'BODY[' + || $this->_getSubstr($str, 0, 5) == 'BODY.'); for ($pos = 0; $pos < $len; $pos++) { - if ($this->_getSubstr($str, 0, 5) == 'BODY[' - || $this->_getSubstr($str, 0, 5) == 'BODY.') { - if ($str[$pos] == ']') { + $c = $str[$pos]; + if ($startsWithBody) { + if ($c == ']') { $pos++; break; } - } elseif ($str[$pos] == ' ' - || $str[$pos] == "\r" - || $str[$pos] == ')' - || $str[$pos] == '(' - || $str[$pos] == "\n" ) { + } elseif ($c == ' ' + || $c == "\r" + || $c == ')' + || $c == '(' + || $c == "\n" ) { break; } if ($str[$pos] == "\\" && $str[$pos + 1 ] == ' ') { |
|
If you use things like this $str[0] you will also need to add isset( $str[0] ) or !empty( $str ) to avoid a notice level error |
|
Can't see it at the moment but I think the string is guaranteed to have some length at that point and since 50% of the run time is in those two substr calls it's worth it. |
|
Sorry it took me some time but was finally able to verify the code Thank you. If you are willing to do some more work for IMAP, i could use your help with adding support for the PHP IMAP extension |
|