MantisBT
Main | My View | View Issues | Change Log | Roadmap | Wiki | ManTweet | Repositories
  

View Issue Details [ Jump to Notes ] [ Wiki ] [ Related Changesets ] [ Issue History ] [ Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0004843mantisbtfilterspublic2004-11-10 03:152010-02-22 14:35
Reporterpolzin 
Assigned Tojreese 
PrioritynormalSeverityfeatureReproducibilityN/A
StatusclosedResolutionopen 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version1.2.0 
Summary0004843: feature request: improving fulltext search
DescriptionAs full text search is used quite often, I think it would be nice to have a full text search with improved capabilities. As Google is the current standard for search engines, Mantis could mimic the Google syntax.

Probably this a difficult task, but I can not believe that this has not been done 100 times somewhere in open-source code that could be reused. I saw that mysql has a "FULLTEXT INDEX" functionality with some other boolean search syntax, but probably this is not sufficient (as we have search in multiple tables)...

A similar issue was discussed in 0002921.
Additional Informationprobably everyone knows google´s syntax, but anyway...

A B C - search for issue with A and B and C
"A B C" - search for issue with the phrase "A B C"
A -B - search for issue with A and not with B
desc:A - search only the issue description for A
TagsNo tags attached.
Attached Files

- Relationships
has duplicate 0006192closedthraxisp In the free search: search for more than one keyword 
has duplicate 0007854closed Need advanced text search options 
has duplicate 0009105closedvboctor Enhanced free text search 
has duplicate 0001878closedvboctor Search capability needs more flexibility. 
has duplicate 0010782closeddhx "NOT CONTAIN" filter 
related to 0005525new Regex-based search filter. 

-  Notes
User avatar (0009545)
vwegert (developer)
2005-03-15 14:52

 BTW, the documentation still insists that the bugnotes are not searched.
User avatar (0019452)
djcarr (reporter)
2008-09-25 04:18
edited on: 2008-09-25 05:07

 I am trialing the following change on my Mantis 1.1.1 install to use mysql 5.0's builtin BOOLEAN MODE text search which is very similar to Google's logic operators: http://dev.mysql.com/doc/refman/5.0/en/fulltext-boolean.html [^]

Replace the following 2 lines in core/filter.api, approx line 1235:

    $t_textsearch_where_clause = '(' . db_helper_like( 'summary', "%$c_search%" ) .
    ' OR ' . db_helper_like( "$t_bug_text_table.description", "%$c_search%" ) .
    ' OR ' . db_helper_like( "$t_bug_text_table.steps_to_reproduce", "%$c_search%" ) .
    ' OR ' . db_helper_like( "$t_bug_text_table.additional_information", "%$c_search%" ) .
    " OR ( $t_bug_table.id = '$c_search_int' ) )";

    $t_textsearch_wherejoin_clause = '(' . db_helper_like( 'summary', "%$c_search%" ) .
    ' OR ' . db_helper_like( "$t_bug_text_table.description", "%$c_search%" )
    ' OR ' . db_helper_like( "$t_bug_text_table.steps_to_reproduce", "%$c_search%" ) .
    ' OR ' . db_helper_like( "$t_bug_text_table.additional_information", "%$c_search%" ) .
    ' OR ' . db_helper_like( "$t_bug_table.id", "%$c_search%" ) .
    ' OR ' . db_helper_like( "$t_bugnote_text_table.note", "%$c_search%" ) . ' )';

With:

    $t_textsearch_where_clause = "( MATCH (summary, $t_bug_text_table.description, $t_bug_text_table.steps_to_reproduce, $t_bug_text_table.additional_information) ".
    "AGAINST ('" . stripslashes(str_replace(""","\"", $c_search)) . "' IN BOOLEAN MODE) ".
    " OR ( $t_bug_table.id = '$c_search_int' ) )";

    $t_textsearch_wherejoin_clause = "( MATCH (summary, $t_bug_text_table.description, $t_bug_text_table.steps_to_reproduce, $t_bug_text_table.additional_information, $t_bugnote_text_table.note) ".
    "AGAINST ('" . stripslashes(str_replace(""","\"", $c_search)) . "' IN BOOLEAN MODE) ".
    " OR ( $t_bug_table.id = '$c_search_int' ) )";

To resolve this issue we still need:

- other mysql users to try this and report their experiences here
- users of psql and other dbs to work out a variant for their platforms
- new config option to activate this
- checks against db_is_mysql, mysql version, db_is_psql and so on
User avatar (0020207)
djcarr (reporter)
2008-12-02 22:07
edited on: 2008-12-02 22:08

 Issues with above approach so far:

- Can only handle words of length 4 or greater, either need to edit mysql.ini to support 3 letter words, or incorporate fallback to original search for short words.
- Does not seem to handle punctuation well, eg. a search on "notepad.exe" does not return the existing instances. May need a means to fallback on original search in this case.
User avatar (0020208)
Buga (reporter)
2008-12-03 03:09

 It would also be useful to have a config option where we can add customfields to the search.
User avatar (0021046)
djcarr (reporter)
2009-03-15 19:12

 update to the earlier code. This incorporates the old literal search in as well as the new google-like search. This addresses the two issues identified earlier.

            $t_tidied_search = substr( stripslashes( str_replace( "'", " ", str_replace(chr(13),' ', $c_search) ) ), 0, 60);
            $t_textsearch_where_clause = "( MATCH (summary, $t_bug_text_table.description, $t_bug_text_table.steps_to_reproduce, $t_bug_text_table.additional_information) ".
                                             "AGAINST ('" . $t_tidied_search . "' IN BOOLEAN MODE) ".
                                            ' OR ' . db_helper_like( 'summary', "%$c_search%" ) .
                                            ' OR ' . db_helper_like( "$t_bug_text_table.description", "%$c_search%" ) .
                                            ' OR ' . db_helper_like( "$t_bug_text_table.steps_to_reproduce", "%$c_search%" ) .
                                            ' OR ' . db_helper_like( "$t_bug_text_table.additional_information", "%$c_search%" ) .
                                            " OR ( $t_bug_table.id = '$c_search_int' ) )";
            $t_textsearch_wherejoin_clause = "( MATCH (summary, $t_bug_text_table.description, $t_bug_text_table.steps_to_reproduce, $t_bug_text_table.additional_information, $t_bugnote_text_table.note) ".
                                             "AGAINST ('" . $t_tidied_search . "' IN BOOLEAN MODE) ".
                                            ' OR ' . db_helper_like( 'summary', "%$c_search%" ) .
                                            ' OR ' . db_helper_like( "$t_bug_text_table.description", "%$c_search%" ) .
                                            ' OR ' . db_helper_like( "$t_bug_text_table.steps_to_reproduce", "%$c_search%" ) .
                                            ' OR ' . db_helper_like( "$t_bug_text_table.additional_information", "%$c_search%" ) .
                                            ' OR ' . db_helper_like( "$t_bugnote_text_table.note", "%$c_search%" ) . ' )';
                                             " OR ( $t_bug_table.id = '$c_search_int' ) )";
User avatar (0023825)
jreese (administrator)
2009-11-30 14:16

 Implementation committed to 1.2.x and master branches.

- Related Changesets
MantisBT: master-1.2.x ff7f362f
Timestamp: 2009-11-30 17:40:50
Author: jreese
[ Details ] [ Diff ] 		Issue 0004843: Implemented improved text filtering

The free text filter box now searches for individual terms in the field
separately, obeys quoting patterns, and allows for negation. All terms
found are required to be found somewhere in the issue (or not found
anywhere for negated terms) in order for the issue to match the filter.

Example searches and their expected results:

  bus car
    Should find any issue that mentions both "bus" and "car"

  "micro computer"
    Should find any issue that mentions "micro computer" exactly; this
    matches the existing search behaviors

  apple -banana
    Should find any issue that mentions "apple" but has no mention of
    "banana"

  circuit -"power line"
    Should find any issue that mentions "circuit" but has no mention of
    "power line"
mod - core/filter_api.php [ Diff ] [ File ]
MantisBT: master 60a4d24a
Timestamp: 2009-11-30 17:40:50
Author: jreese
[ Details ] [ Diff ] 		Issue 0004843: Implemented improved text filtering

The free text filter box now searches for individual terms in the field
separately, obeys quoting patterns, and allows for negation. All terms
found are required to be found somewhere in the issue (or not found
anywhere for negated terms) in order for the issue to match the filter.

Example searches and their expected results:

  bus car
    Should find any issue that mentions both "bus" and "car"

  "micro computer"
    Should find any issue that mentions "micro computer" exactly; this
    matches the existing search behaviors

  apple -banana
    Should find any issue that mentions "apple" but has no mention of
    "banana"

  circuit -"power line"
    Should find any issue that mentions "circuit" but has no mention of
    "power line"
mod - core/filter_api.php [ Diff ] [ File ]

- Issue History
Date Modified Username Field Change
2004-11-10 03:15 polzin New Issue
2005-03-02 05:27 mno Sponsorship Added mno: US$ 20
2005-03-02 05:27 mno Sponsorship Total 0 => 20
2005-03-11 08:44 mno Sponsorship Updated mno: US$ 30
2005-03-11 08:44 mno Sponsorship Total 20 => 30
2005-03-15 14:04 vwegert Note Added: 0009542
2005-03-15 14:52 vwegert Note Added: 0009545
2005-03-15 14:53 vwegert Note Deleted: 0009542
2005-04-13 03:23 mno Sponsorship Updated mno: US$ 50
2005-04-13 03:23 mno Sponsorship Total 30 => 50
2005-04-27 16:35 thraxisp Relationship added related to 0005525
2005-08-30 17:11 thraxisp Relationship added related to 0006192
2005-08-30 17:12 thraxisp Relationship deleted related to 0006192
2005-08-30 17:14 thraxisp Relationship added has duplicate 0006192
2008-06-06 17:51 daryn Relationship added has duplicate 0007854
2008-07-24 13:26 vboctor Relationship added has duplicate 0009105
2008-07-24 13:27 vboctor Relationship added has duplicate 0001878
2008-07-24 13:28 vboctor Status new => acknowledged
2008-08-18 10:44 jreese Status acknowledged => assigned
2008-08-18 10:44 jreese Assigned To => daryn
2008-09-25 04:18 djcarr Note Added: 0019452
2008-09-25 04:19 djcarr Note Edited: 0019452
2008-09-25 04:19 djcarr Note Edited: 0019452
2008-09-25 04:20 djcarr Note Edited: 0019452
2008-09-25 04:22 djcarr Note Edited: 0019452
2008-09-25 04:23 djcarr Note Edited: 0019452
2008-09-25 04:25 djcarr Note Edited: 0019452
2008-09-25 04:26 djcarr Note Edited: 0019452
2008-09-25 04:26 djcarr Note Edited: 0019452
2008-09-25 05:07 djcarr Note Edited: 0019452
2008-12-02 22:07 djcarr Note Added: 0020207
2008-12-02 22:08 djcarr Note Edited: 0020207
2008-12-03 03:09 Buga Note Added: 0020208
2009-03-15 19:12 djcarr Note Added: 0021046
2009-11-30 14:10 jreese Changeset attached master-1.2.x ff7f362f =>
2009-11-30 14:10 jreese Changeset attached master 60a4d24a =>
2009-11-30 14:16 jreese Note Added: 0023825
2009-11-30 14:16 jreese Assigned To daryn => jreese
2009-11-30 14:16 jreese Status assigned => resolved
2009-11-30 14:16 jreese Fixed in Version => 1.2.2
2010-02-22 14:35 jreese Status resolved => closed
2010-02-24 22:37 dhx Relationship added has duplicate 0010782

MantisBT 1.2.16dev master-1.2.x-8c2bd07 [^]
Copyright © 2000 - 2013 MantisBT Team
Time: 0.1472 seconds.
memory usage: 2,927 KB
Powered by Mantis Bugtracker