[Issue #29] (Team 2) Dynamic Programming Algorithm for Query Rewriter by ss3n · Pull Request #120 · apache/texera

ss3n · 2016-05-24T03:38:49Z

Added English dictionary file, with frequencies.
Modified FuzzyParser to implement the dynamic programming algorithm, implemented by @JavierJia which is referenced in the comment thread, under (Team 2) Design: Query Rewriter #29
Modified the QueryRewriterTest to reflect the new expected results from the QueryRewriter

@chenlica : Can you give this a review?

chenlica · 2016-05-24T03:46:43Z

 */
-public class FuzzyTokenizer
-{
+public class FuzzyTokenizer {


I think the name FuzzyTokenizer is not very accurate. Can we rename it to QueryRewriter?

I just noticed that it's a helper class for QueryRewriter. Still I prefer a different name. Any suggestions?

How about QuerySegmenter? http://www2003.org/cdrom/papers/poster/p052/xhtml/querysegmentation.html

Understood... Refactoring to QuerySegmenter
Just asking, after the refactoring, is it ok to have the method named getFuzzyTokens ?

We want to avoid overloading the word "fuzzy". So I prefer to use getTokens().

Understood.

ss3n · 2016-05-24T03:49:01Z

Actually QueryRewriter is the main class... It in turn calls the FuzzyTokenizer class to detect missing white spaces. The reason we have separated the two classes is because later, QueryRewriter would also perform "synonym" matching as well.

chenlica · 2016-05-24T03:50:12Z

+    /**
+     * When called returns a single-member list containing most-likely tokenization of phrase
+     * Splits phrase into terms by whitespace delimiter
+     * For each term, rewrites(tokenizes) term to detect likely missing spaces


Add an example to the comments.

ss3n · 2016-05-24T04:03:22Z

@chenlica : Added an example in the comments as suggested... Kindly take a look

chenlica · 2016-05-24T04:07:11Z

In the QueryRewriter class, is it better to make fuzzyTokenizer.getFuzzyTokens() a static function if it only needs a query string?

chenlica · 2016-05-24T04:08:53Z

        this.phrase = phrase;
    }

+    /**


For line 42 public FuzzyTokenizer(String phrase, rename phrase to queryString since phrase has a special meaning in other modules.

It does make sense for fuzzyTokenizer.getFuzzyTokens() to be a static function. We are making changes as suggested.

chenlica · 2016-05-24T04:22:37Z

High-level comments:

Make FuzzyTokenizer.getFuzzyTokens() a static function.
Possibly rename FuzzyTokenizer to QuerySegmenter. If you don't agree, that's fine.
Wait for @JavierJia to review the DP algorithm.

chenlica · 2016-05-24T04:26:17Z

+                }
+            }
+        }
+


Rename tokenization to tokenizedQuery.

ss3n · 2016-05-24T04:45:46Z

Refactored FuzzyTokenizer to QuerySegmenter
Refactored phrase to queryString
Refactored getFuzzyTokens to getTokens
Converted getTokens to static function and made corresponding changes to QueryRewriter

@chenlica : Kindly take a look.

chenlica · 2016-05-24T04:46:58Z

To unblock you, no need to wait for the review from @JavierJia . Go ahead to do the merge after making those changes. We will let @JavierJia doublecheck the logic even after the merge.

chenlica · 2016-05-24T04:49:29Z

Looks good. Please go ahead to do the merge!

ss3n · 2016-05-24T04:53:16Z

Missed refactoring a variable name...

chenlica · 2016-05-24T05:22:47Z

Looks good. Please go ahead to do the merge.

ss3n · 2016-05-24T06:43:03Z

Overloaded QueryRewriter constructor to accept flag for returning all possible tokenizations
Included corresponding test cases in QueryRewriterTest
Cleaned some noisy entries in word-frequency map

@chenlica : Can you take a look at the changes?

chenlica · 2016-05-24T06:48:02Z

+            List<String> queryStrings;
+            if(allSegmentations)
+                queryStrings = QuerySegmenter.bruteGetTokens(searchQuery);
+            else


bruteGetTokens -> getTokensBruteForce

We were thinking of refactoring bruteGetTokens -> getAllTokens and getTokens -> getLikelyTokens.
getAllTokens can internally call rewriteTermBruteForce and getLikelyTokens can call rewriteTermDP
@chenlica : Do you think that would be ok?

Sounds good. The name rewriteTermDP can be better.

shiladitya added 2 commits May 23, 2016 20:31

Implemented DP algorithm for FuzzyTokenizer; modified test case

8cf2ddb

Merge branch 'yoda-dp' of https://github.com/TextDB/textdb into yoda-dp

c9d76e0

ss3n added the 2-query-rewritter label May 24, 2016

ss3n assigned kishore-narendran May 24, 2016

kishore-narendran mentioned this pull request May 24, 2016

(Team 2) Design: Query Rewriter #29

Closed

ss3n changed the title ~~(Team 2) Dynamic Programming Algorithm for Query Rewriter~~ [Issue #29] (Team 2) Dynamic Programming Algorithm for Query Rewriter May 24, 2016

chenlica reviewed May 24, 2016
View reviewed changes

Added a working example in comment for DP processing

8253f34

chenlica reviewed May 24, 2016
View reviewed changes

Refactored FuzzyTokenizer->QuerySegmenter; Made getTokens static

3e51009

Refactored phrase to query

76c7fa8

Changed QueryRewriter to allow brute force tokenization; included tests

78fd055

chenlica reviewed May 24, 2016
View reviewed changes

Conversation

ss3n commented May 24, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ss3n commented May 24, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ss3n commented May 24, 2016

Uh oh!

chenlica commented May 24, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chenlica commented May 24, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ss3n commented May 24, 2016

Uh oh!

chenlica commented May 24, 2016

Uh oh!

chenlica commented May 24, 2016

Uh oh!

ss3n commented May 24, 2016

Uh oh!

chenlica commented May 24, 2016

Uh oh!

ss3n commented May 24, 2016

Uh oh!

chenlica May 24, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ss3n commented May 24, 2016 •

edited

Loading

chenlica May 24, 2016 •

edited

Loading