0013680: Configuration page takes a very long time to load

(0030623)
JanHegewald (reporter)
2011-12-15 05:25
edited on: 2011-12-15 05:26

Additional information: this happens on 1.2.1.

And my php log tells me, that most of the time the problem is because of this line in session_api.php (line 154): $_SESSION[ $this->key ][ $p_name ] = serialize( $p_value );

Edit: removed typo

(0030636)
rombert (developer)
2011-12-16 08:56

Can you share the contents of your config table? Alternatively, please attach the result of loading the page with xdebug profiling enabled ( http://xdebug.org/docs/profiler [^] ).

(0030653)
JanHegewald (reporter)
2011-12-19 05:09

I attached an output file of xdebug profiler. Because the file has a size of around 34 MB, I had to pack it. I hope this is ok.

One additional information: every time I reload the page, it gets slower. It's slowing down approx. 2 to 5 seconds per reload. When I close the browser (Firefox 8.0), wait for some time, open the browser again and reopen the config page, it loads very much faster (around 5 secondes for the first time, I think).

(0030654)
rombert (developer)
2011-12-19 06:21

Thanks for the profiling output. It seems that printing a button ( for each modified configuration option ) triggers a lot of work behind the scenes, which creates performance problems. I do not have an immediate solution, but I'll target this towards 1.2.x .

(0030785)
JanHegewald (reporter)
2012-01-04 04:27
edited on: 2012-01-04 04:40

Thanks for your feedback!

Edit: it seems that commentig out the lines in adm_config_report.php where the button is generated fixes the issue temporary. Now the duration for page load is constant or even decreasing a little bit. I know that this is just a temporary solution, but for now it's perfect for me.

Again, thanks for investigating!

(0030786)
atrol (developer)
2012-01-04 05:52

Maybe another workaround:
If you want the buttons and can live with a possible security risc (protection against Cross-Site Request Forgery) add the following line to config_inc.php
$g_form_security_validation = OFF;

(0030791)
dregad (developer)
2012-01-04 07:33

The bottleneck is actually the serialize/unserialize calls performed while storing/retrieving the token from the PHP session.

Maybe we could shift to json encoding to improve performance ?

https://bugs.php.net/bug.php?id=51267 [^]
http://stackoverflow.com/questions/804045/preferred-method-to-store-php-arrays-json-encode-vs-serialize [^]

(0031981)
dhx (developer)
2012-06-02 07:50

Our CSRF support at present is far from being user friendly or sane from a performance perspective. There is no reason why we need multiple CSRF tokens per page load. We don't even need a unique CSRF token for each page load. This is something we need to fix in a future release.

(0034543)
dregad (developer)
2012-12-14 15:53

@dhx I suppose you were referring to https://github.com/mantisbt/mantisbt/commit/a45d0ef551e9359d3a469287f5de8ef08096e12f [^] ?

(0034638)
dregad (developer)
2012-12-28 19:06

I have implemented basic filtering on the manage config page which I believe would be an effective workaround for the performance issue reported here, thanks to the default filter criteria.

Please have a look at https://github.com/dregad/mantisbt/tree/manage-config [^] - testing and feedback would be appreciated.

(0034664)
dregad (developer)
2013-01-02 09:30

Reminder sent to: dhx

@dhx,

I made an attempt at fixing this issue [1] and would appreciate if you could take a look at the changes I made. Let me know what you think.

[1] https://github.com/dregad/mantisbt/tree/fix-13680 [^]

(0034690)
dhx (developer)
2013-01-03 08:31

@dregad: Thanks for the reminder and proposed patch. I've reviewed https://github.com/dregad/mantisbt/commit/db0e3d3b103750197cbc656323ff2984f803fa47 [^] and it looks good to commit. Keep up the good work :)

(0034719)
dregad (developer)
2013-01-06 19:31

Thanks for feedback dhx.

For the record, I did some benchmarking for this patch, using a sample DB having a bit less than 700 records in mantis_config_table, configuring the filter as appropriate to display all of them, for the following branches:

baseline: current 1.2.x branch HEAD (commit 66050dc2)
case 1: filter branch https://github.com/dregad/mantisbt/tree/fix-14559 [^]
case 2: performance fix branch https://github.com/dregad/mantisbt/tree/fix-13680 [^]

Average execution times on 3 runs (as reported at page bottom by $g_show_timer) are as follows:

baseline: 13.3 seconds
1: 20.1 seconds (due to displaying 2 buttons per line instead of 1)
2: 1.4 seconds

I have a bit more testing to do on other aspects of the feature branch https://github.com/dregad/mantisbt/tree/manage-config [^] before merging.

(0036075)
grangeway (developer)
2013-04-05 17:56

Marking as 'acknowledged' not resolved/closed to track that change gets ported to master-2.0.x branch

Related Changesets
MantisBT: master-1.2.x 259f95cd Timestamp: 2012-12-14 15:10:52 Author: dregad [ Details ] [ Diff ]	Issue 0014559: improve adm_report_config.php filter The following changes were made - revised UI to make it more similar to the issues filter - make use of existing constants and language strings - filter defaults to All Users / All Projects / All options which avoids performance issues in installations having a large number of entries in the config table (workaround for issue 0013680) - filter form uses post instead of get method - comply to coding guidelines
	mod - adm_config_report.php	[ Diff ] [ File ]
	mod - lang/strings_english.txt	[ Diff ] [ File ]
	mod - lang/strings_french.txt	[ Diff ] [ File ]

MantisBT: master-1.2.x db0e3d3b Timestamp: 2013-01-02 13:13:18 Author: dregad [ Details ] [ Diff ]	Fix performance issue on adm_config_report.php In systems with large numbers of config items in mantis_config_table, the Configuration Report page can take a very long time to load. This behavior is due to each of the 'Delete' buttons being printed with its own form, each one having a security token. The performance bottleneck is actually the serialize/unserialize calls executed while storing/retrieving the token from the PHP session. To avoid this problem, the print_button() and form_security_field() functions have been modified to accept a security token as an optional parameter. This allows the calling page to generate a single token, which is shared by all buttons. Furthermore, print_button() also allows the security token parameter to be 'OFF', which prevents the function from displaying a security field. This is useful for buttons not resulting in modifications (i.e. not requiring CSRF protection). Fixes 0013680
	mod - adm_config_report.php	[ Diff ] [ File ]
	mod - core/form_api.php	[ Diff ] [ File ]
	mod - core/print_api.php	[ Diff ] [ File ]

MantisBT: master 5c3ac412 Timestamp: 2013-01-02 13:13:18 Author: dregad [ Details ] [ Diff ]	Fix performance issue on adm_config_report.php In systems with large numbers of config items in mantis_config_table, the Configuration Report page can take a very long time to load. This behavior is due to each of the 'Delete' buttons being printed with its own form, each one having a security token. The performance bottleneck is actually the serialize/unserialize calls executed while storing/retrieving the token from the PHP session. To avoid this problem, the print_button() and form_security_field() functions have been modified to accept a security token as an optional parameter. This allows the calling page to generate a single token, which is shared by all buttons. Furthermore, print_button() also allows the security token parameter to be 'OFF', which prevents the function from displaying a security field. This is useful for buttons not resulting in modifications (i.e. not requiring CSRF protection). Fixes 0013680
	mod - adm_config_report.php	[ Diff ] [ File ]
	mod - core/form_api.php	[ Diff ] [ File ]
	mod - core/print_api.php	[ Diff ] [ File ]

Issue History
Date Modified	Username	Field	Change
2011-12-15 05:22	JanHegewald	New Issue
2011-12-15 05:25	JanHegewald	Note Added: 0030623
2011-12-15 05:26	JanHegewald	Note Edited: 0030623	View Revisions
2011-12-15 06:11	atrol	Relationship added	related to 0012113
2011-12-16 08:56	rombert	Note Added: 0030636
2011-12-16 08:56	rombert	Status	new => feedback
2011-12-19 05:06	JanHegewald	File Added: cachegrind.out.rar
2011-12-19 05:09	JanHegewald	Note Added: 0030653
2011-12-19 05:09	JanHegewald	Status	feedback => new
2011-12-19 06:21	rombert	Note Added: 0030654
2011-12-19 06:21	rombert	Status	new => confirmed
2011-12-19 06:21	rombert	Target Version	=> 1.2.9
2011-12-19 06:21	rombert	Description Updated	View Revisions
2011-12-19 06:22	rombert	File Added: screenshot.png
2012-01-04 04:27	JanHegewald	Note Added: 0030785
2012-01-04 04:40	JanHegewald	Note Edited: 0030785	View Revisions
2012-01-04 05:52	atrol	Note Added: 0030786
2012-01-04 07:33	dregad	Note Added: 0030791
2012-03-04 09:23	atrol	Target Version	1.2.9 => 1.2.10
2012-04-02 02:33	atrol	Target Version	1.2.10 => 1.2.11
2012-06-02 07:50	dhx	Note Added: 0031981
2012-06-02 07:50	dhx	Assigned To	=> dhx
2012-06-02 07:50	dhx	Status	confirmed => assigned
2012-06-06 23:54	jreese	Target Version	1.2.11 => 1.2.12
2012-08-08 15:10	atrol	Relationship added	related to 0014559
2012-11-10 19:04	dregad	Target Version	1.2.12 => 1.2.13
2012-12-14 15:53	dregad	Note Added: 0034543
2012-12-28 19:06	dregad	Note Added: 0034638
2012-12-28 19:06	dregad	Status	assigned => feedback
2013-01-02 09:30	dregad	Note Added: 0034664
2013-01-03 08:31	dhx	Note Added: 0034690
2013-01-03 08:31	dhx	Assigned To	dhx => dregad
2013-01-03 08:31	dhx	Status	feedback => assigned
2013-01-06 19:31	dregad	Note Added: 0034719
2013-01-06 19:55	dregad	Relationship added	related to 0015347
2013-01-22 08:55	dregad	Changeset attached	=> MantisBT master-1.2.x 259f95cd
2013-01-22 08:55	dregad	Changeset attached	=> MantisBT master-1.2.x db0e3d3b
2013-01-22 08:55	dregad	Status	assigned => resolved
2013-01-22 08:55	dregad	Resolution	open => fixed
2013-01-22 08:55	dregad	Fixed in Version	=> 1.2.13
2013-01-22 09:47	dregad	Status	resolved => closed
2013-03-08 11:05	dregad	Changeset attached	=> MantisBT master 5c3ac412
2013-03-13 06:17	jayavel	Issue cloned: 0015625
2013-04-05 17:56	grangeway	Status	closed => acknowledged
2013-04-05 17:56	grangeway	Note Added: 0036075
2013-04-05 19:43	grangeway	Relationship added	related to 0015721
2013-04-06 03:39	dregad	Status	acknowledged => resolved
2013-04-06 07:21	grangeway	Status	resolved => acknowledged
2013-04-06 09:22	dregad	Tag Attached: 2.0.x check
2013-04-06 09:23	dregad	Status	acknowledged => closed

Notes
(0030623) JanHegewald (reporter) 2011-12-15 05:25 edited on: 2011-12-15 05:26	Additional information: this happens on 1.2.1. And my php log tells me, that most of the time the problem is because of this line in session_api.php (line 154): $_SESSION[ $this->key ][ $p_name ] = serialize( $p_value ); Edit: removed typo

(0030636) rombert (developer) 2011-12-16 08:56	Can you share the contents of your config table? Alternatively, please attach the result of loading the page with xdebug profiling enabled ( http://xdebug.org/docs/profiler [^] ).

(0030653) JanHegewald (reporter) 2011-12-19 05:09	I attached an output file of xdebug profiler. Because the file has a size of around 34 MB, I had to pack it. I hope this is ok. One additional information: every time I reload the page, it gets slower. It's slowing down approx. 2 to 5 seconds per reload. When I close the browser (Firefox 8.0), wait for some time, open the browser again and reopen the config page, it loads very much faster (around 5 secondes for the first time, I think).

(0030654) rombert (developer) 2011-12-19 06:21	Thanks for the profiling output. It seems that printing a button ( for each modified configuration option ) triggers a lot of work behind the scenes, which creates performance problems. I do not have an immediate solution, but I'll target this towards 1.2.x .

(0030785) JanHegewald (reporter) 2012-01-04 04:27 edited on: 2012-01-04 04:40	Thanks for your feedback! Edit: it seems that commentig out the lines in adm_config_report.php where the button is generated fixes the issue temporary. Now the duration for page load is constant or even decreasing a little bit. I know that this is just a temporary solution, but for now it's perfect for me. Again, thanks for investigating!

(0030786) atrol (developer) 2012-01-04 05:52	Maybe another workaround: If you want the buttons and can live with a possible security risc (protection against Cross-Site Request Forgery) add the following line to config_inc.php $g_form_security_validation = OFF;

(0030791) dregad (developer) 2012-01-04 07:33	The bottleneck is actually the serialize/unserialize calls performed while storing/retrieving the token from the PHP session. Maybe we could shift to json encoding to improve performance ? https://bugs.php.net/bug.php?id=51267 [^] http://stackoverflow.com/questions/804045/preferred-method-to-store-php-arrays-json-encode-vs-serialize [^]

(0031981) dhx (developer) 2012-06-02 07:50	Our CSRF support at present is far from being user friendly or sane from a performance perspective. There is no reason why we need multiple CSRF tokens per page load. We don't even need a unique CSRF token for each page load. This is something we need to fix in a future release.

(0034543) dregad (developer) 2012-12-14 15:53	@dhx I suppose you were referring to https://github.com/mantisbt/mantisbt/commit/a45d0ef551e9359d3a469287f5de8ef08096e12f [^] ?

(0034638) dregad (developer) 2012-12-28 19:06	I have implemented basic filtering on the manage config page which I believe would be an effective workaround for the performance issue reported here, thanks to the default filter criteria. Please have a look at https://github.com/dregad/mantisbt/tree/manage-config [^] - testing and feedback would be appreciated.

(0034664) dregad (developer) 2013-01-02 09:30	Reminder sent to: dhx @dhx, I made an attempt at fixing this issue [1] and would appreciate if you could take a look at the changes I made. Let me know what you think. [1] https://github.com/dregad/mantisbt/tree/fix-13680 [^]

(0034690) dhx (developer) 2013-01-03 08:31	@dregad: Thanks for the reminder and proposed patch. I've reviewed https://github.com/dregad/mantisbt/commit/db0e3d3b103750197cbc656323ff2984f803fa47 [^] and it looks good to commit. Keep up the good work :)

(0034719) dregad (developer) 2013-01-06 19:31	Thanks for feedback dhx. For the record, I did some benchmarking for this patch, using a sample DB having a bit less than 700 records in mantis_config_table, configuring the filter as appropriate to display all of them, for the following branches: baseline: current 1.2.x branch HEAD (commit 66050dc2) case 1: filter branch https://github.com/dregad/mantisbt/tree/fix-14559 [^] case 2: performance fix branch https://github.com/dregad/mantisbt/tree/fix-13680 [^] Average execution times on 3 runs (as reported at page bottom by $g_show_timer) are as follows: baseline: 13.3 seconds 1: 20.1 seconds (due to displaying 2 buttons per line instead of 1) 2: 1.4 seconds I have a bit more testing to do on other aspects of the feature branch https://github.com/dregad/mantisbt/tree/manage-config [^] before merging.

(0036075) grangeway (developer) 2013-04-05 17:56	Marking as 'acknowledged' not resolved/closed to track that change gets ported to master-2.0.x branch