View Issue Details

IDProjectCategoryView StatusLast Update
0006627mantisbtperformancepublic2006-09-12 00:52
Reportermlovell Assigned Tovboctor  
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionfixed 
Product Version1.0.0rc5 
Fixed in Version1.1.0a1 
Summary0006627: Very slow performance when filing a new issue or adding a note
Description

With Mantis 1.0.0rc5 installed on our RHEL4 box, performance of all the "view" pages is currently fine. Reporting a new issue or adding a note to an existing issue, however, is taking several seconds (>10)!

I was wondering if anyone has seen anything like this. I have turned on the SQL query listing, but it doesn't seem like a consequence of a slow query at all. No query is taking longer than ~0.02 seconds, and there are only ~40 of those to start with.

Additional Information

Red Hat Enterprise Linux 4
Mantis 1.0.0rc5
PHP 4.3.9
MySQL 4.1.12

TagsNo tags attached.

Activities

mlovell

mlovell

2006-01-23 16:58

reporter   ~0011999

Last edited: 2006-01-23 17:00

The delay I'm seeing in adding a bug is more like 20 seconds. Here are some delays as measured by Benchmark/Timer.php:

[client 10.1.24.3] Hello from bugnote_add!, referer: http://wally/mantis_exp/view.php?id=7
[client 10.1.24.3] Start: -, referer: http://wally/mantis_exp/view.php?id=7
[client 10.1.24.3] Marker 1: 0.000207, referer: http://wally/mantis_exp/view.php?id=7
[client 10.1.24.3] Marker 2: 0.004432, referer: http://wally/mantis_exp/view.php?id=7
[client 10.1.24.3] Marker 3: 0.001573, referer: http://wally/mantis_exp/view.php?id=7
[client 10.1.24.3] Marker 4: 0.000922, referer: http://wally/mantis_exp/view.php?id=7
[client 10.1.24.3] Marker 5: 0.000345, referer: http://wally/mantis_exp/view.php?id=7
[client 10.1.24.3] Marker 6: 0.002532, referer: http://wally/mantis_exp/view.php?id=7
[client 10.1.24.3] Marker 7: 20.296918, referer: http://wally/mantis_exp/view.php?id=7
[client 10.1.24.3] Marker 8: 0.197588, referer: http://wally/mantis_exp/view.php?id=7
[client 10.1.24.3] Stop: 0.000024, referer: http://wally/mantis_exp/view.php?id=7
[client 10.1.24.3] Total: 20.504541, referer: http://wally/mantis_exp/view.php?id=7

Correction!
The large time (~20.3 seconds) is entirely the call to email_bugnote_add(). I'll attempt to instrument that routine and see where the delay is originating.

mlovell

mlovell

2006-01-23 17:20

reporter   ~0012000

The delay appears to be entirely with the

$mail->Send()

invocation in email_send(). Any information on what could cause phpMailer such delays?

vboctor

vboctor

2006-01-23 17:45

manager   ~0012001

How many users are sent a notification when you add the note? How long does the $email->send operation take?

mlovell

mlovell

2006-01-23 18:39

reporter   ~0012002

I traced the long delay down to PHP waiting on sendmail to exit. Experimenting a little further, I modified email_api.php's email_send() routine such that,
when phpMailer_method is sendmail, one further tweak is made:


switch ( config_get( 'phpMailer_method' ) ) {
case 0: $mail->IsMail();
break;

  case 1: $mail->IsSendmail();
          $mail->Sendmail = 'sendmail -O DeliveryMode=defer';  #NEW
          break;

This allows the mail to be enqueued and the PHP script to continue very quickly.

Is this a viable solution? Should I have any concerns running with this option to sendmail?

Thanks.

mlovell

mlovell

2006-01-23 19:06

reporter   ~0012003

Responding to vboctor's questions...

One user is receiving the notification when the note is added.

The send() operation is taking about 20 seconds. It appears to be a consequence of phpMailer running sendmail synchronously. The box on which Mantis is running cannot actually send mail to anything but our mail server (sendmail has been configured as a smart relay). I suspect that setup causes synchronous operation to be slow. Running sendmail on the command line is just as slow as what PHP is showing.

Thus, I'm wondering about the DeliveryMode option in my other note. Any insights would be appreciated.

mlovell

mlovell

2006-01-23 19:20

reporter   ~0012004

Hmmmm, I think I'm finding the DeliveryMode=defer is not really an option. Sendmail says such email has been queued, but is not actually delivering it; some other postings on the net seem to suggest deferred mode has been closed due to spam...grrr...

So, that still leaves me wondering how I can get phpMailer to just queue mail for delivery, rather than waiting the mail to actually be accepted by our mail server.

mlovell

mlovell

2006-01-23 19:50

reporter   ~0012005

Hmmmm...the problem may have been entirely local.

Turns out one of our kerberos servers was down for most of the day. Although I don't understand the linkage, this caused the mail server (and thus our sendmail) to be extremely slow.

Once that server was brought back up, all mail (independent of phpMailer setting) went back to being quite fast, returning Mantis responsiveness back to normal. All of the mail which had been deferred was also very quickly delivered.

So, there probably is not any major issue with phpMailer and sendmail. My apologies for the noise. It was educational, albeit confusing, to the performance linkage to kerberos server availability. I doubt it does me much good, but educational.

vboctor

vboctor

2006-01-23 19:50

manager   ~0012006

We will probably need to do the following:

  • Create a Mantis queue for outgoing emails, this will be a table which has fields like email_id, to, cc, bcc, subject, body.

  • Have a script that gets run regularly that sends the queued entries or some of them. This script will be executed from the cronjob.

  • Failures to send emails should probably be flagged. Hence, if emails can't be delivered, the admin can go in and check the error message returned.

We should still keep the current synchronous behaviour as an option for users who don't want to use a scheduler or a cronjob. However, we may have an underlying common implementation for both options.

With this new approach, actions that trigger notifications will just create records in this table and then it will be sent later, and hence will improve the performance and usability of Mantis.

mlovell

mlovell

2006-01-23 20:27

reporter   ~0012007

The table-based queue sounds like a nice feature! I would definitely keep the synchronous one available as an option.

On a semi-related note, have any "nag scripts" been developed for Mantis? In particular, I'm interested in email users (every Monday) with a list of issues they need to resolve or to close. The manager of the project should also get a list of News issues with no current owners.

Thanks.

ghansen

ghansen

2006-04-17 07:48

reporter   ~0012517

We had the same issue, modifing an entry (new/change/close) took up to 60 seconds.
I tracked it down also to the emails sending routine. As we have appr. 60 users using mantis and our emails server is hosted on the other side of the planet, we needed to change something to speed things up.

In our case the issue was that for every user that gets notified a new emails is sent.
This means 60 users equals 60 emails and at a rate of one email per second, we had our 60 seconds delay.

I changed the default sending behaviour to a batched email sending mode using CC or BCC.

Therefore only one email is sent per change, depending on of you have a closed community using mantis or not, you can use CC or BCC, so not every other user can see who also did get notified.

This lowered the delay down to 1 second, constantly, even if we increase the amount of users in the future.

Thought this might be interresting to others.
Patch available upon request.

vboctor

vboctor

2006-08-14 04:34

manager   ~0013261

The email queuing and the support for cronjobs are now implemented and will be part of Mantis 1.1.

To use the cronjob option set the option below to ON and setup a cronjob to call core/send_emails.php.

$g_email_send_using_cronjob = OFF;