View Issue Details

IDProjectCategoryView StatusLast Update
0012544mantisbtldappublic2015-09-06 17:37
ReporterdregadAssigned Todregad 
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionreopened 
Product Version1.2.1 
Target Version1.3.0-beta.3Fixed in Version1.3.0-beta.3 
Summary0012544: LDAP unavailability causes severe performance problems
Description

Last night, we had a problem on our LDAP servers, which was unavailable. This caused severe performance issues on Mantis, which still responded, but after each and any type of request, users had to wait 2-3 minutes for the server to respond.

Note: I am connecting to LDAP (AD) with the following URI: ldap://domain.com:3268/ where ldap.domain.com is actually resolved to multiple IP addresses, which are tried in sequence, one after the other. It turned out that the crashed LDAP, was the first IP returned.

The ldap_bind is apparently falling over to the next IP in line after some predefined timeout (looks like 3 minutes). After I changed my config_inc.php setting to point to the 2nd LDAP host and restarted Apache, performance was back to normal.

Would it be possible to implement a parameter for this timeout value ?

Additional Information

Mantis LDAP log file - look at the times between the "Attempting to bind" and "bind successful" - 3 minutes delay

2010-11-18 18:39 CET ldap Binding to LDAP server
2010-11-18 18:39 CET ldap Attempting connection to LDAP server 'ldap://domain.com:3268/' port '999999'.
2010-11-18 18:39 CET ldap Connection accepted to LDAP server
2010-11-18 18:39 CET ldap Setting LDAP protocol to to ldap server to 3
2010-11-18 18:39 CET ldap Attempting bind to ldap server with username and password
2010-11-18 18:42 CET ldap bind to ldap server successful
2010-11-18 18:42 CET ldap Searching for (&(sAMAccountName=XXXXXX))
2010-11-18 18:42 CET ldap Binding to LDAP server
2010-11-18 18:42 CET ldap Attempting connection to LDAP server 'ldap://domain.com:3268/' port '999999'.
2010-11-18 18:42 CET ldap Connection accepted to LDAP server
2010-11-18 18:42 CET ldap Setting LDAP protocol to to ldap server to 3
2010-11-18 18:42 CET ldap Attempting bind to ldap server with username and password
2010-11-18 18:45 CET ldap bind to ldap server successful
2010-11-18 18:45 CET ldap Searching for (&(sAMAccountName=XXXXXX))

TagsNo tags attached.

Relationships

Activities

dregad

dregad

2010-11-19 05:29

developer   ~0027415

I checked on PHP docs http://www.php.net/manual/en/function.ldap-set-option.php , and there is an existing option that may do the trick (LDAP_OPT_NETWORK_TIMEOUT) but it's only available in php 5.3 and Mantis requirements is only >= 5.1.x / 5.2.x recommended.

Is it a reasonable option to change the minimum required PHP version for Mantis ?

Are there any alternative options ?

atrol

atrol

2010-11-19 05:44

developer   ~0027416

I don't think this is a reason to change the minimun required version.
What about something like
if (version_compare(PHP_VERSION, '5.3.0') >= 0) {
do_my_ldap_stuff ;
}

dregad

dregad

2010-11-19 10:51

developer   ~0027432

Well, the "do_my_ldap_stuff" here would probably involve checking whether the LDAP server is alive, before executing the bind (since PHP does not allow to specify the timeout).

Other ideas ?

atrol

atrol

2010-11-19 11:18

developer   ~0027433

Maybe you did not get my idea.
If you are right that setting LDAP_OPT_NETWORK_TIMEOUT fixes your problem
the idea is to add some code to function ldap_connect_bind (ldap_api.php)

if ( version_compare(PHP_VERSION, '5.3.0') >= 0 ) {
ldap_set_option( $t_ds, LDAP_OPT_NETWORK_TIMEOUT, config_get( 'ldap_network_timeout' ));
}

Be aware that I have no experience dealing with LDAP

atrol

atrol

2010-11-19 11:22

developer   ~0027434

Last edited: 2010-11-19 11:25

View 2 revisions

I found this example http://simplesamlphp.googlecode.com/svn/trunk/lib/SimpleSAML/Auth/LDAP.php
instead of version_compare(PHP_VERSION, '5.3.0') >= 0
the example uses
if (defined('LDAP_OPT_NETWORK_TIMEOUT'))

I don't know how PHP interpretes the code, so maybe this is the right solution because using constant LDAP_OPT_NETWORK_TIMEOUT with PHP < 5.3 might produce a syntax error (depends on lazy evaluation)

Be aware that I have also no experience with PHP ;-)

dregad

dregad

2010-11-19 11:40

developer   ~0027435

Be aware that I have also no experience with PHP ;-)
LOL... what are you doing, being a Mantis developer then ;-)

Thanks for your feedback, and the link. To be honest, I am not sure whether LDAP_OPT_NETWORK_TIMEOUT would resolve my problem or not, at this time I have only looked for options and not actually written any code to test.

Also, my previous note was triggered by my misreading your if statement, I thought you intended to act in case the version was < 5.3 when in fact you wrote the contrary.

In any case, I wanted to gather some ideas first, before starting to write code (for a change).

atrol

atrol

2010-11-21 16:10

developer   ~0027438

developer just means, that I am able to delete your notes and issues.

I don't think that setting LDAP_OPT_NETWORK_TIMEOUT will solve the problem because ldap_set_option is called after ldap_connect.

SL-Gundam

SL-Gundam

2010-11-21 20:21

reporter   ~0027440

Last edited: 2010-11-21 20:22

View 2 revisions

I would not be surprised if the default_socket_timeout ini setting affects the ldap_connect function. I'm not able to test it though.

dregad

dregad

2010-11-22 05:59

developer   ~0027445

Last edited: 2010-11-22 06:00

View 2 revisions

Thanks for your feedback.

ldap_set_option is called after ldap_connect, thus
This is fine, since the problem is with ldap_bind, so I can still set the option before that call.

I tested with the following code:

<?php
$host = "1.2.3.4";
$ds = ldap_connect($host) or die ("can't connect to $host");

ldap_get_option( $ds, LDAP_OPT_NETWORK_TIMEOUT, $o );
echo "Test connect $host timeout " . $o . "\n";
$t = time();
$r = ldap_bind($ds);
echo "status $r - elapsed " . (time() - $t) . "\n";

$s = ldap_set_option( $ds, LDAP_OPT_NETWORK_TIMEOUT, 2 );
ldap_get_option( $ds, LDAP_OPT_NETWORK_TIMEOUT, $o );
echo "Test connect $host timeout " . $o . "\n";
$t = time();
$r = ldap_bind($ds);
echo "status $r - elapsed " . (time() - $t) . "\n";

$s = ldap_close($ds);
?>

Resulting in the following output on my desktop:

$ php /tmp/ldap.php
Test connect 1.2.3.4 timeout
PHP Warning: ldap_bind(): Unable to bind to server: Can't contact LDAP server in /tmp/ldap.php on line 8
status - elapsed 21
Test connect 1.2.3.4 timeout 2
PHP Warning: ldap_bind(): Unable to bind to server: Can't contact LDAP server in /tmp/ldap.php on line 15
status - elapsed 2

Then I changed the value of default_socket_timeout in php.ini, from default 60 to 5 and ran the same script, but got the exact same results. Nice try, but no cigar.

Running the same script on my server (although without using the LDAP_OPT_NETWORK_TIMEOUT option as PHP is only on 5.2.5 there), (and default_socket_timeout = 60 in php.ini) gave me

Test connect 1.2.3.4 timeout
PHP Warning: ldap_bind(): Unable to bind to server: Can't contact LDAP server in /tmp/ldap.php on line 8
status - elapsed 189

There are my 3 minutes...

Bottom line:

  • default_socket_timeout apparently does not come into play here
  • LDAP_OPT_NETWORK_TIMEOUT fixes my problem, but can't be used as a general fix unless running PHP 5.3.x

Any other ideas ?

grangeway

grangeway

2011-09-12 17:03

reporter   ~0029694

Setting $g_ldap_contexts to multiple server entries, and g_ldap_options to include LDAP_OPT_NETWORK_TIMEOUT should do what you are trying to achieve. [available in git only]

dregad

dregad

2011-09-13 03:10

developer   ~0029695

grangeway, re: "available in git only"

I found no trace of either $g_ldap_contexts or $g_ldap_options, in any of the mantisbt github branches (master and next). Is this part of your long-awaited revamp of ldap, in some local branch that you likely have not yet merged with master ?

If so, please update this issue whenever the new functionality is ready for testing, I'll be happy to test-drive it. In the meanwhile, I don't think this can be marked as resolved.

Having a single config option for all LDAP settings sounds really good (using an array ?). However, I'm not really sure what you mean by ldap contexts and how it would apply in my case (I have a single DNS entry, which resolves to multiple IP's for failover - they are essentially the same LDAP).

JeromyK

JeromyK

2014-01-21 04:56

reporter   ~0039110

Hello everybody

I only want to say, that we have the same problem with LDAP and multiple IPs (aka multiple DCs) in our configuration. If there is some testing I can do for you in our environment, I would be happy to help.

For the first time, we will update PHP to version 5.3.0 and try if that really fix the issue. I will send Feedback here if we done.

atrol

atrol

2014-01-21 07:27

developer   ~0039111

we will update PHP to version 5.3.0
You should use a version >= 5.3.2 as you will get other issues when using 5.3.0

JeromyK

JeromyK

2014-01-21 09:02

reporter   ~0039112

Thx for that. I think we will update directly to PHP 5.3.27

Related Changesets

MantisBT: master ca813d8a

2011-09-12 12:35:29

Damien Regad


Committer: dregad Details Diff
Add new ldap_network_timeout config option

Introduction of PHP 5.3 support with 1.3.x allows setting of LDAP
network timeout (LDAP_OPT_NETWORK_TIMEOUT) before binding the server.

When the LDAP server is not available with this option set to its
default value of 0 (infinite), the system will seem to hang for several
minutes, depending on the TCP stack settings, until control is finally
returned to Mantis.

Setting this option to a low value makes the system more responsive,
and also allows automatic and rapid failover to the next available LDAP
server when the hostname defined in $g_ldap_server resolves to multiple
IP addresses, which are tried in sequence by ldap_bind().

Fixes 0012544
mod - config_defaults_inc.php Diff File
mod - core/ldap_api.php Diff File
mod - docbook/Admin_Guide/en-US/config/auth.xml Diff File

Issue History

Date Modified Username Field Change
2010-11-19 05:28 dregad New Issue
2010-11-19 05:29 dregad Note Added: 0027415
2010-11-19 05:44 atrol Note Added: 0027416
2010-11-19 10:51 dregad Note Added: 0027432
2010-11-19 11:18 atrol Note Added: 0027433
2010-11-19 11:22 atrol Note Added: 0027434
2010-11-19 11:25 atrol Note Edited: 0027434 View Revisions
2010-11-19 11:40 dregad Note Added: 0027435
2010-11-21 16:10 atrol Note Added: 0027438
2010-11-21 20:21 SL-Gundam Note Added: 0027440
2010-11-21 20:22 SL-Gundam Note Edited: 0027440 View Revisions
2010-11-22 05:59 dregad Note Added: 0027445
2010-11-22 06:00 dregad Note Edited: 0027445 View Revisions
2011-09-12 12:13 dregad Assigned To => dregad
2011-09-12 12:13 dregad Status new => assigned
2011-09-12 12:13 dregad Product Version => 1.2.1
2011-09-12 12:13 dregad Target Version => 1.3.0-beta.1
2011-09-12 17:03 grangeway Note Added: 0029694
2011-09-12 17:03 grangeway Status assigned => resolved
2011-09-12 17:03 grangeway Resolution open => no change required
2011-09-13 03:10 dregad Note Added: 0029695
2011-09-13 03:10 dregad Status resolved => feedback
2011-09-13 03:10 dregad Resolution no change required => reopened
2012-09-13 09:50 dregad Assigned To dregad =>
2012-09-13 09:50 dregad Status feedback => confirmed
2014-01-21 04:56 JeromyK Note Added: 0039110
2014-01-21 07:27 atrol Note Added: 0039111
2014-01-21 09:02 JeromyK Note Added: 0039112
2014-01-21 18:42 atrol Target Version 1.3.0-beta.1 =>
2015-04-16 19:25 dregad Changeset attached => MantisBT master ca813d8a
2015-04-16 19:25 dregad Assigned To => dregad
2015-04-16 19:25 dregad Status confirmed => resolved
2015-04-16 19:25 dregad Fixed in Version => 1.3.0-beta.3
2015-04-16 19:25 dregad Target Version => 1.3.0-beta.3
2015-09-06 17:37 vboctoradmin Status resolved => closed