View Issue Details

IDProjectCategoryView StatusLast Update
0004843mantisbtfilterspublic2014-01-13 20:11
Reporterpolzin Assigned Tojreese  
PrioritynormalSeverityfeatureReproducibilityN/A
Status closedResolutionfixed 
Fixed in Version1.2.0 
Summary0004843: feature request: improving fulltext search
Description

As full text search is used quite often, I think it would be nice to have a full text search with improved capabilities. As Google is the current standard for search engines, Mantis could mimic the Google syntax.

Probably this a difficult task, but I can not believe that this has not been done 100 times somewhere in open-source code that could be reused. I saw that mysql has a "FULLTEXT INDEX" functionality with some other boolean search syntax, but probably this is not sufficient (as we have search in multiple tables)...

A similar issue was discussed in 0002921.

Additional Information

probably everyone knows google´s syntax, but anyway...

A B C - search for issue with A and B and C
"A B C" - search for issue with the phrase "A B C"
A -B - search for issue with A and not with B
desc:A - search only the issue description for A

TagsNo tags attached.

Relationships

has duplicate 0006192 closedthraxisp In the free search: search for more than one keyword 
has duplicate 0007854 closed Need advanced text search options 
has duplicate 0009105 closedvboctor Enhanced free text search 
has duplicate 0001878 closedvboctor Search capability needs more flexibility. 
has duplicate 0010782 closeddhx "NOT CONTAIN" filter 
related to 0005525 new Regex-based search filter. 

Activities

vwegert

vwegert

2005-03-15 14:52

reporter   ~0009545

BTW, the documentation still insists that the bugnotes are not searched.

djcarr

djcarr

2008-09-25 04:18

reporter   ~0019452

Last edited: 2008-09-25 05:07

I am trialing the following change on my Mantis 1.1.1 install to use mysql 5.0's builtin BOOLEAN MODE text search which is very similar to Google's logic operators: http://dev.mysql.com/doc/refman/5.0/en/fulltext-boolean.html

Replace the following 2 lines in core/filter.api, approx line 1235:

$t_textsearch_where_clause = '(' . db_helper_like( 'summary', "%$c_search%" ) .
' OR ' . db_helper_like( "$t_bug_text_table.description", "%$c_search%" ) . 
' OR ' . db_helper_like( "$t_bug_text_table.steps_to_reproduce", "%$c_search%" ) .
' OR ' . db_helper_like( "$t_bug_text_table.additional_information", "%$c_search%" ) .
" OR ( $t_bug_table.id = '$c_search_int' ) )";

$t_textsearch_wherejoin_clause = '(' . db_helper_like( 'summary', "%$c_search%" ) .
' OR ' . db_helper_like( "$t_bug_text_table.description", "%$c_search%" )
' OR ' . db_helper_like( "$t_bug_text_table.steps_to_reproduce", "%$c_search%" ) .
' OR ' . db_helper_like( "$t_bug_text_table.additional_information", "%$c_search%" ) .
' OR ' . db_helper_like( "$t_bug_table.id", "%$c_search%" ) .
' OR ' . db_helper_like( "$t_bugnote_text_table.note", "%$c_search%" ) . ' )';

With:

$t_textsearch_where_clause = "( MATCH (summary, $t_bug_text_table.description, $t_bug_text_table.steps_to_reproduce, $t_bug_text_table.additional_information) ".
"AGAINST ('" . stripslashes(str_replace(""","\"", $c_search)) . "' IN BOOLEAN MODE) ".
" OR ( $t_bug_table.id = '$c_search_int' ) )";

$t_textsearch_wherejoin_clause = "( MATCH (summary, $t_bug_text_table.description, $t_bug_text_table.steps_to_reproduce, $t_bug_text_table.additional_information, $t_bugnote_text_table.note) ".
"AGAINST ('" . stripslashes(str_replace(""","\"", $c_search)) . "' IN BOOLEAN MODE) ".
" OR ( $t_bug_table.id = '$c_search_int' ) )";

To resolve this issue we still need:

  • other mysql users to try this and report their experiences here
  • users of psql and other dbs to work out a variant for their platforms
  • new config option to activate this
  • checks against db_is_mysql, mysql version, db_is_psql and so on
djcarr

djcarr

2008-12-02 22:07

reporter   ~0020207

Last edited: 2008-12-02 22:08

Issues with above approach so far:

  • Can only handle words of length 4 or greater, either need to edit mysql.ini to support 3 letter words, or incorporate fallback to original search for short words.
  • Does not seem to handle punctuation well, eg. a search on "notepad.exe" does not return the existing instances. May need a means to fallback on original search in this case.
Buga

Buga

2008-12-03 03:09

reporter   ~0020208

It would also be useful to have a config option where we can add customfields to the search.

djcarr

djcarr

2009-03-15 19:12

reporter   ~0021046

update to the earlier code. This incorporates the old literal search in as well as the new google-like search. This addresses the two issues identified earlier.

        $t_tidied_search = substr( stripslashes( str_replace( "'", " ", str_replace(chr(13),' ', $c_search) ) ), 0, 60);
        $t_textsearch_where_clause = "( MATCH (summary, $t_bug_text_table.description, $t_bug_text_table.steps_to_reproduce, $t_bug_text_table.additional_information) ".
                                         "AGAINST ('" . $t_tidied_search . "' IN BOOLEAN MODE) ".
                                        ' OR ' . db_helper_like( 'summary', "%$c_search%" ) .
                                        ' OR ' . db_helper_like( "$t_bug_text_table.description", "%$c_search%" ) . 
                                        ' OR ' . db_helper_like( "$t_bug_text_table.steps_to_reproduce", "%$c_search%" ) .
                                        ' OR ' . db_helper_like( "$t_bug_text_table.additional_information", "%$c_search%" ) .
                                        " OR ( $t_bug_table.id = '$c_search_int' ) )";
        $t_textsearch_wherejoin_clause = "( MATCH (summary, $t_bug_text_table.description, $t_bug_text_table.steps_to_reproduce, $t_bug_text_table.additional_information, $t_bugnote_text_table.note) ".
                                         "AGAINST ('" . $t_tidied_search . "' IN BOOLEAN MODE) ".
                                        ' OR ' . db_helper_like( 'summary', "%$c_search%" ) .
                                        ' OR ' . db_helper_like( "$t_bug_text_table.description", "%$c_search%" ) .
                                        ' OR ' . db_helper_like( "$t_bug_text_table.steps_to_reproduce", "%$c_search%" ) .
                                        ' OR ' . db_helper_like( "$t_bug_text_table.additional_information", "%$c_search%" ) .
                                        ' OR ' . db_helper_like( "$t_bugnote_text_table.note", "%$c_search%" ) . ' )';
                                         " OR ( $t_bug_table.id = '$c_search_int' ) )";
jreese

jreese

2009-11-30 14:16

reporter   ~0023825

Implementation committed to 1.2.x and master branches.

Related Changesets

MantisBT: master-1.2.x ff7f362f

2009-11-30 12:40

jreese


Details Diff
Issue 0004843: Implemented improved text filtering

The free text filter box now searches for individual terms in the field
separately, obeys quoting patterns, and allows for negation. All terms
found are required to be found somewhere in the issue (or not found
anywhere for negated terms) in order for the issue to match the filter.

Example searches and their expected results:

bus car
Should find any issue that mentions both "bus" and "car"

"micro computer"
Should find any issue that mentions "micro computer" exactly; this
matches the existing search behaviors

apple -banana
Should find any issue that mentions "apple" but has no mention of
"banana"

circuit -"power line"
Should find any issue that mentions "circuit" but has no mention of
"power line"
Affected Issues
0004843
mod - core/filter_api.php Diff File

MantisBT: master 60a4d24a

2009-11-30 12:40

jreese


Details Diff
Issue 0004843: Implemented improved text filtering

The free text filter box now searches for individual terms in the field
separately, obeys quoting patterns, and allows for negation. All terms
found are required to be found somewhere in the issue (or not found
anywhere for negated terms) in order for the issue to match the filter.

Example searches and their expected results:

bus car
Should find any issue that mentions both "bus" and "car"

"micro computer"
Should find any issue that mentions "micro computer" exactly; this
matches the existing search behaviors

apple -banana
Should find any issue that mentions "apple" but has no mention of
"banana"

circuit -"power line"
Should find any issue that mentions "circuit" but has no mention of
"power line"
Affected Issues
0004843
mod - core/filter_api.php Diff File