View Issue Details

IDProjectCategoryView StatusLast Update
0005297mantisbtcustom fieldspublic2005-07-23 02:22
Reportermno Assigned Tothraxisp  
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionfixed 
Product Version0.19.0 
Fixed in Version1.0.0rc1 
Summary0005297: performance problem when filtering on custom fields
Description

There is a huge difference when it comes to performance when filtering on standard fields as opposed to custom fields (about 140 times)
We have about 2000 bug entries and about 30 users.

This issue might be related to bug 0004345

Additional Information

Custom Fields timing:

Time: 112.976389 seconds.
44 total queries executed.
32 unique queries executed.

Time: 113.880447 seconds.
102 total queries executed.
90 unique queries executed.

Non custom fields timing:

Time: 0.798021 seconds.
98 total queries executed.
86 unique queries executed.

TagsNo tags attached.

Relationships

has duplicate 0005458 closedthraxisp Slow queries on projects with many bugs and custom fields 

Activities

grangeway

grangeway

2005-03-04 11:51

reporter   ~0009478

hello mno,

are you able to tell what query(ies) is performing slowly?

mno

mno

2005-03-08 09:38

reporter   ~0009499

not really, how do I do that?

grangeway

grangeway

2005-03-09 10:46

reporter   ~0009503

Set:
$g_show_queries_list = ON;
$g_show_queries_count = On

I believe

Paul

mno

mno

2005-03-11 08:39

reporter   ~0009514

Time: 123.923095 seconds.
105 total queries executed.
93 unique queries executed.
1 0.0007 SELECT id FROM mantis_user_table WHERE cookie_string='5fbcc83d9a65e5ce3dcf48895406a43d153eeaf62456d54d5df2df8dc9799627'
2 0.0006 SELECT FROM mantis_user_table WHERE id='18'
3 0.0006 SELECT
FROM mantis_filters_table WHERE id='2'
4 0.0005 SELECT FROM mantis_project_table WHERE id='0'
5 0.0021 SELECT
FROM mantis_filters_table WHERE user_id='18' AND project_id='0' AND name=''
6 0.0007 SELECT id FROM mantis_custom_field_table ORDER BY name ASC
7 0.0006 SELECT id FROM mantis_custom_field_table ORDER BY name ASC
8 0.0008 SELECT id FROM mantis_custom_field_table ORDER BY name ASC
9 0.0006 SELECT FROM mantis_custom_field_table WHERE id='8'
10 60.6542 SELECT COUNT( DISTINCT mantis_bug_table.id ) as count FROM mantis_project_table, mantis_bug_table LEFT JOIN mantis_custom_field_string_table as mantis_custom_field_string_table_8 ON mantis_custom_field_string_table_8.bug_id = mantis_bug_table.id WHERE mantis_project_table.enabled = 1 AND mantis_project_table.id = mantis_bug_table.project_id AND (( mantis_custom_field_string_table_8.field_id = 8 AND mantis_custom_field_string_table_8.value = 'BO-GUI' ))
11 62.4930 SELECT DISTINCT mantis_bug_table.
FROM mantis_project_table, mantis_bug_table LEFT JOIN mantis_custom_field_string_table as mantis_custom_field_string_table_8 ON mantis_custom_field_string_table_8.bug_id = mantis_bug_table.id WHERE mantis_project_table.enabled = 1 AND mantis_project_table.id = mantis_bug_table.project_id AND (( mantis_custom_field_string_table_8.field_id = 8 AND mantis_custom_field_string_table_8.value = 'BO-GUI' )) ORDER BY last_updated DESC, date_submitted DESC
12 0.0012 SELECT FROM mantis_user_pref_table WHERE user_id='18' AND project_id='0'
13 0.0009 SELECT DISTINCT( id ), name FROM mantis_project_table WHERE enabled=1 ORDER BY name
14 0.0007 SELECT
FROM mantis_project_table WHERE id='16'
15 0.0006 SELECT FROM mantis_project_table WHERE id='15'
16 0.0006 SELECT
FROM mantis_project_table WHERE id='17'
17 0.0006 SELECT FROM mantis_project_table WHERE id='14'
18 0.0023 SELECT
FROM mantis_filters_table WHERE user_id='18' AND project_id='0' AND name=''
19 0.0008 SELECT id FROM mantis_custom_field_table ORDER BY name ASC
20 0.0022 SELECT FROM mantis_filters_table WHERE user_id='18' AND project_id='0' AND name=''
21 0.0007 SELECT id FROM mantis_custom_field_table ORDER BY name ASC
22 0.0007 SELECT id FROM mantis_custom_field_table ORDER BY name ASC
23 0.0009 SELECT id FROM mantis_custom_field_table ORDER BY name ASC
24 0.0005 SELECT type, possible_values FROM mantis_custom_field_table WHERE id='8'
25 0.0006 SELECT
FROM mantis_custom_field_table WHERE id='4'
26 0.0006 SELECT type, possible_values FROM mantis_custom_field_table WHERE id='4'
27 0.0421 SELECT mantis_custom_field_string_table.value FROM mantis_custom_field_string_table WHERE mantis_custom_field_string_table.field_id='4' GROUP BY mantis_custom_field_string_table.value
28 0.0011 SELECT FROM mantis_custom_field_table WHERE id='5'
29 0.0005 SELECT type, possible_values FROM mantis_custom_field_table WHERE id='5'
30 0.0007 SELECT
FROM mantis_custom_field_table WHERE id='6'
31 0.0005 SELECT type, possible_values FROM mantis_custom_field_table WHERE id='6'
32 0.0006 SELECT FROM mantis_custom_field_table WHERE id='3'
33 0.0005 SELECT type, possible_values FROM mantis_custom_field_table WHERE id='3'
34 0.0418 SELECT mantis_custom_field_string_table.value FROM mantis_custom_field_string_table WHERE mantis_custom_field_string_table.field_id='3' GROUP BY mantis_custom_field_string_table.value
35 0.0010 SELECT
FROM mantis_custom_field_table WHERE id='7'
36 0.0006 SELECT type, possible_values FROM mantis_custom_field_table WHERE id='7'
37 0.0032 SELECT FROM mantis_filters_table WHERE (project_id='0' OR project_id='0') AND name!='' AND filter_string!='' ORDER BY is_public DESC, name ASC
38 0.0023 SELECT
FROM mantis_filters_table WHERE user_id='18' AND project_id='0' AND name=''
39 0.0007 SELECT id FROM mantis_custom_field_table ORDER BY name ASC
40 0.0007 SELECT id FROM mantis_custom_field_table ORDER BY name ASC
41 0.0031 SELECT FROM mantis_filters_table WHERE (project_id='0' OR project_id='0') AND name!='' AND filter_string!='' ORDER BY is_public DESC, name ASC
42 0.0026 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1959' ORDER BY last_modified DESC
43 0.0006 SELECT project_id, access_level FROM mantis_project_user_list_table WHERE user_id='18'
44 0.0022 SELECT bug_id, COUNT(bug_id) AS attachments FROM mantis_bug_file_table GROUP BY bug_id
45 0.0008 SELECT
FROM mantis_user_table WHERE id='28'
46 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1971' ORDER BY last_modified DESC
47 0.0007 SELECT FROM mantis_user_table WHERE id='9'
48 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1917' ORDER BY last_modified DESC
49 0.0009 SELECT
FROM mantis_user_table WHERE id='26'
50 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1929' ORDER BY last_modified DESC
51 0.0025 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1970' ORDER BY last_modified DESC
52 0.0026 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1928' ORDER BY last_modified DESC
53 0.0008 SELECT FROM mantis_user_table WHERE id='11'
54 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1969' ORDER BY last_modified DESC
55 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1967' ORDER BY last_modified DESC
56 0.0025 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1966' ORDER BY last_modified DESC
57 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1957' ORDER BY last_modified DESC
58 0.0027 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1955' ORDER BY last_modified DESC
59 0.0025 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1920' ORDER BY last_modified DESC
60 0.0025 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='746' ORDER BY last_modified DESC
61 0.0012 SELECT
FROM mantis_user_table WHERE id='15'
62 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1831' ORDER BY last_modified DESC
63 0.0025 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1943' ORDER BY last_modified DESC
64 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1935' ORDER BY last_modified DESC
65 0.0028 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1908' ORDER BY last_modified DESC
66 0.0008 SELECT FROM mantis_user_table WHERE id='8'
67 0.0025 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1860' ORDER BY last_modified DESC
68 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1923' ORDER BY last_modified DESC
69 0.0025 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1861' ORDER BY last_modified DESC
70 0.0008 SELECT
FROM mantis_user_table WHERE id='7'
71 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1924' ORDER BY last_modified DESC
72 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1927' ORDER BY last_modified DESC
73 0.0025 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1926' ORDER BY last_modified DESC
74 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1925' ORDER BY last_modified DESC
75 0.0027 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1832' ORDER BY last_modified DESC
76 0.0025 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1809' ORDER BY last_modified DESC
77 0.0027 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1762' ORDER BY last_modified DESC
78 0.0025 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1896' ORDER BY last_modified DESC
79 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1199' ORDER BY last_modified DESC
80 0.0025 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='301' ORDER BY last_modified DESC
81 0.0007 SELECT FROM mantis_user_table WHERE id='6'
82 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='312' ORDER BY last_modified DESC
83 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='299' ORDER BY last_modified DESC
84 0.0025 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='311' ORDER BY last_modified DESC
85 0.0025 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='307' ORDER BY last_modified DESC
86 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1922' ORDER BY last_modified DESC
87 0.0026 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1921' ORDER BY last_modified DESC
88 0.0025 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1700' ORDER BY last_modified DESC
89 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1919' ORDER BY last_modified DESC
90 0.0025 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1821' ORDER BY last_modified DESC
91 0.0008 SELECT
FROM mantis_user_table WHERE id='24'
92 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1894' ORDER BY last_modified DESC
93 0.0026 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1898' ORDER BY last_modified DESC
94 0.0008 SELECT FROM mantis_user_table WHERE id='22'
95 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1907' ORDER BY last_modified DESC
96 0.0025 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1887' ORDER BY last_modified DESC
97 0.0033 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1911' ORDER BY last_modified DESC
98 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1915' ORDER BY last_modified DESC
99 0.0027 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1862' ORDER BY last_modified DESC
100 0.0008 SELECT
FROM mantis_user_table WHERE id='32'
101 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1914' ORDER BY last_modified DESC
102 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1913' ORDER BY last_modified DESC
103 0.0024 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1912' ORDER BY last_modified DESC
104 0.0026 SELECT last_modified FROM mantis_bugnote_table WHERE bug_id='1743' ORDER BY last_modified DESC
105 0.0006 UPDATE mantis_user_table SET last_visit= '2005-03-11 14:41:16' WHERE id='18'
123.4053

thraxisp

thraxisp

2005-03-11 09:20

reporter   ~0009516

Is there any possibility of retesting this with the CVS version? The filter code (and queries) have been rewritten several times since 0.19.0.

mno

mno

2005-04-13 09:49

reporter   ~0009805

sorry for not answering for such a long time.
Today I tested the new 0.19.2 version and
it is in fact twice as fast as the 0.19.0,
and I am happy about the improvement, nevertheless
a query still takes 75 seconds
(150 seconds with the 0.19.0). It takes about
1 second if I do not use the custom field as a filter.
I can't tell exactly, because somehow for the new
version I do not see the timing at the bottom of
the page anymore, although I used my previous config
config file. The same config file with the old version
does produce the timing.

thraxisp

thraxisp

2005-04-13 10:56

reporter   ~0009806

Great. The filter code has been been rewritten to improve performance between the 0.19.2 release and now. My testing shows that it is about 50% faster.

I think that there is a problem in 0.19.2 that sometimes prevents the queries list from showing. This was fixed in CVS a while ago.

Can we consider this closed?

mno

mno

2005-04-13 12:04

reporter   ~0009807

How do I do the donation?
You guys definitely earned the money, but I am ready to pay another 50$ for cutting the current processing time by half and than another 50$ if this trick could be done again :)

but if you feel there is not much that can be done in that direction, you can close the bug.

So which files from CVS do I need to update having 0.19.2 already installed?

view_all_set.php only or some other files too?

thraxisp

thraxisp

2005-04-13 14:31

reporter   ~0009808

Normally, one of the developers would contact you after the code is released to arrange payment. PayPal is usually the simplest mechanism.

As for picking up just the changes for the filters, it's hard to tell the minumum set of modules because a lot has changed. There have been several featues added to the filter code since 0.19.2. The best bet for a complete set of code would be a CVS image. These are getting pretty stable now.

mno

mno

2005-04-22 05:20

reporter   ~0009862

Hi, I have found the solution to my problem and the filters are just flying now. I have added an index to the mantis_custom_field_string_table on bug_id. Thank's for the previous work, the 30$ is yours.

thraxisp

thraxisp

2005-06-17 18:29

reporter   ~0010573

Fixed in CVS.

admin/upgrades/1_00_inc.php -> 1.5