View Issue Details

IDProjectCategoryView StatusLast Update
0017577mantisbtperformancepublic2023-10-31 16:36
Reporterdregad Assigned Todregad  
PrioritynormalSeverityminorReproducibilityalways
Status assignedResolutionopen 
Product Version1.2.17 
Target Version2.27.0 
Summary0017577: Improve print_user_option_list() performance
Description

On instances with large number of users, the function has very poor
performance (e.g. > 20 seconds to load 28K users on this tracker).

This is an issue e.g. when filtering issues by reporter or when using adm_config_report.php, especially when the current project is 'All Projects'.

TagsNo tags attached.

Relationships

has duplicate 0017737 closedatrol adm_config_report.php is slow with larger number of users 
has duplicate 0019942 closedatrol Reporter list is loading way too long 
related to 0017962 closeddregad Implement auto-complete for reporter field in bug_update_page 
related to 0017738 new return_dynamic_Filters.php performance with large number of users 
related to 0024124 new Editing the report of a bug crashes browser with large user base 
related to 0025666 confirmed Replace user selection list with an autocomplete widget progressive remote search 

Activities

dregad

dregad

2014-08-07 19:49

developer   ~0041032

PR https://github.com/mantisbt/mantisbt/pull/238

The proposed fix is for master, I plan to backport it to 1.2.x if accepted.

atrol

atrol

2015-09-09 09:50

developer   ~0051427

Reminder sent to: dregad

Should we target to 1.3.x or maybe even 2.0.x?

dregad

dregad

2015-09-09 18:51

developer   ~0051429

The reason for targeting 1.2 was that our tracker was heavily affected by this issue. Now that we have upgraded it, a 1.3.x target is probably best (assuming I ever find the time to get to it :/).

The impact is still major IMO, although not blocking for release since not so many instances are likely to be affected by this.

cproensa

cproensa

2015-10-26 20:51

developer   ~0051728

for solving half of the problem: why not rewrite the user selector in filter section to fetch all users that appear in existing bugs?

This would be a query (fine tuned to some degree by filter parameters), but surely less work on the server.
The real problem here is that for "all projects", its fetching and combining a list of all users, repeating for every project.

And, for an improved resolution, change the select into an autocomplete widget (when available by js), with progressive remote search

dregad

dregad

2015-10-27 04:20

developer   ~0051732

change the select into an autocomplete widget (when available by js), with progressive remote search

This is exactly where I want to go. I'm pretty sure I have a feature branch with initial research on this somehere, but I can't find it ATM.

atrol

atrol

2015-10-27 04:39

developer   ~0051734

I'm pretty sure I have a feature branch

Maybe this one?
https://github.com/dregad/mantisbt/commits/dynamic-reporter-select

Independet from AJAX some other optimizations that did not make their way into the product but might be worth to review them again.

https://github.com/mantisbt/mantisbt/pull/238
https://github.com/mantisbt/mantisbt/pull/271

cproensa

cproensa

2015-10-27 06:48

developer   ~0051735

^^^ regarding that:
please have a look at 0020225, its somewhat related to the affected core functions

cproensa

cproensa

2015-10-29 08:04

developer   ~0051753

i have this branch with a rewrite of the affected functions
https://github.com/cproensa/mantisbt/tree/project_user_list

this shows great improvement, as testing with 20k users, 0000054:0000100 projects
still it can get faster, but i have some questions before it can be done
should i open a PR to comment there?

cproensa

cproensa

2015-10-31 21:41

developer   ~0051780

PR: https://github.com/mantisbt/mantisbt/pull/668

vboctor

vboctor

2015-12-22 03:38

manager   ~0052164

Last edited: 2016-05-09 12:03

Changing status to minor based on @dregad's 0017577:0051429 and offline discussion with @atrol since we are trying to identify blocking issues for v1.3. Though this may make it before the release, it is not blocking.

dregad

dregad

2016-05-21 04:31

developer   ~0053204

cproensa's PR https://github.com/mantisbt/mantisbt/pull/769 (see related changeset below) greatly improves this performance issue.

atrol

atrol

2016-06-14 02:17

developer   ~0053370

Independent from the AJAX approach and other optimizations, but will also enhance performance at least for some of the user lists, e.g. "Reporter" on "View Issues" page:

Wouldn't it be better to list just the users that reported an issue for the current project (distinct reporter_id from bug table)?

This will reduce the number of listed users and solve also the issue that we are not able to search for issues that have been reported by deactivated or deleted users, see 0010141.

cproensa

cproensa

2016-06-14 03:37

developer   ~0053374

distinct xxx from bug table

yes, that would be the simplest solution. For several filter fields this should be a valid approach.

able to search for issues that have been reported by deactivated or deleted users

but, irrc, then someone can complain that a reporter gets a hint of the existence of the deactivated users (i dont think this is an issue, just saying)

Now that theres some will to change things it filter api, this is a thing that should be done

atrol

atrol

2016-06-18 13:17

developer   ~0053412

someone can complain that a reporter gets a hint of the existence of the deactivated users

We could add a new option for it, something like $g_show_deactivated_users_threshold = MANAGER;
But this will not prevent a user from using "View Issues" page to list all issues and look for users which are displayed strike out.
So adding this option would just let some people think that users with access level < $g_show_deactivated_users_threshold can't find deactivated users.

dregad

dregad

2017-01-24 08:46

developer   ~0055289

Unless someone objects, I'd like to mark this one as resolved, considering cproensa's commit MantisBT master b433e7e6 - on this tracker it takes about 3-5 seconds to load the reporters' list (about 35K users), which I think is acceptable.

Other ideas for improvement should be tracked separately.

atrol

atrol

2017-01-24 15:49

developer   ~0055298

or when using adm_config_report.php, especially when the current project is 'All Projects'

This is still quite slow.
Didn't have a deeper look, might be caused by missing indices on mantis_config_table.

dregad

dregad

2019-03-29 09:40

developer   ~0061796

@cproensa wrote in 0017577:0051728

And, for an improved resolution, change the select into an autocomplete widget (when available by js), with progressive remote search

Opened 0025666 to track this

Related Changesets

MantisBT: master b433e7e6

2016-04-29 13:13

cproensa

Committer: dregad


Details Diff
Don't cache users in project_get_all_user_rows()

Most of the callers of project_get_all_user_rows use the return array
directly, and do not need to fetch the user rows. There's no need to
always cache the calculated users. Especially when the user list is very
large, eg: print users option list with public access. By not caching
them, memory and time is saved (see issue 0017577)

Of all the callers of this function, those that actually use the
cached data are:
- email_notify_new_account(): Adds an explicit call to cache
- email_collect_recipients(): Was already calling to cache users

Signed-off-by: Damien Regad <dregad@mantisbt.org>
Affected Issues
0017577
mod - core/email_api.php Diff File
mod - core/project_api.php Diff File