(Team 2) Design: Query Rewriter

@kishore-narendran

Team 2:

Please do the following for your task.

Put the design to https://github.com/TextDB/textdb/wiki/CS290-2016S-Task:-Character-based-Fuzzy-Search
Use this issue to keep track of the progress.

Per our discussion in the lecture today, please do the following:

Model the features as operators.
For each operator, come up with a few test cases.
Put your initial design (including the presentation) to your wiki page https://github.com/TextDB/textdb/wiki/CS290-2016S-Task:-Character-based-Fuzzy-Search.
Update this issue with your progress.

@kishore-narendran

@shiladityasen and @kishore-narendran: Please refine your design to come up with an interface for the classes. We can schedule a meeting this week when you are ready.

Here is the Chinese tokenizer code in SRCH2 codebase: https://github.com/SRCH2/srch2-ngn/blob/master/src/core/analyzer/ChineseTokenizer.cpp#L197

Thank you Professor,

We will go through the literature to find how to incorporate that in our
module.
For now we are going to define the interfaces, write some tests and build a
very naive algorithmic version in sandbox.

On Wed, Apr 20, 2016 at 8:45 PM, Jianfeng Jia notifications@github.com
wrote:

Here is the Chinese tokenizer code in SRCH2 codebase:
https://github.com/SRCH2/srch2-ngn/blob/master/src/core/analyzer/ChineseTokenizer.cpp#L197

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#29 (comment)

@chenlica

The current status of the issue is such:

The interface to the QueryRewriter has been setup
QueryRewriter internally uses FuzzyTokenizer to find the permutations of search strings needed.
A few test cases have also been written in QueryRewriterTest.java
A new type of IField was written to reflect the type of ITuple that the QueryRewriter will return i.e. a list of Strings.

The above changes were submitted in the PR #75

This is what is up next -

We will write a naive version of the FuzzyTokenizer and test it in textdb-sandbox
The naive version will then be ported into textdb-dataflow to the actual FuzzyTokenizer class
We will then optimize and improve the FuzzyTokenizer algorithm, and iterate

@chenlica @JavierJia

This is currently the status of the issue!

@chenlica

QueryRewriter will require certain dictionaries to be present (or provided by user) such as dictionary of English words. Which would be the ideal directory to store this dictionary in?
@chenlica @JavierJia @rajesh9625 @sandeepreddy602

@shiladityasen... You can create one more source folder src/test/resources and add your directories there.

@sandeepreddy602

@sandeepreddy602 : This English word dictionary is part of the operator, so it should NOT be part the test folder, right? Can each operator have its own local "resources" directory for such files?

@chenlica

@chenlica.. In that case we can create the source folder src/main/resources and create directory for each operator. We can add the operator related data files in the corresponding directory.

@sandeepreddy602

@sandeepreddy602 : Agreed. We can use this PR to create this folder structure. I hope it's easy for each operator to locate that file using a relative path.

@chenlica

@chenlica @sandeepreddy602 : Just to clarify, we are deciding to create a separate package src/main/resources for every project (such as textdb-dataflow)? Then every operator implemented in the project can access the files in src/main/resources.

Is this correct?

Merge pull request #83 from TextDB/yoda-query

@JavierJia

This is the current status of this issue:

An interface, and tests for QueryRewriter was completed in Issue #29: Query Rewriter Interface and Tests #75
A naive implementation of the QueryRewriter was completed in Issue #29: QueryRewriter Operator Naive Implementation and Tests #83

This is what is up next:

An optimized version of the implementation will be implemented in an upcoming PR, which will either utilize @JavierJia's dynamic programming approach, OR, we are alternatively looking at an algorithm that utilizes a Trie data structure to perform the tokenization.

Team 2: Any update on your task? Thanks for agreeing to present the dynamic programming algorithm today. It will be good to create a Google presentation for your work.

Hello Professor,

We are deciding the basic layout of the presentation and some examples to
illustrate the working and where it fits in our application.

@kishore-narendran

@shiladityasen and @kishore-narendran when can you raise a PR to finish it?

@chenlica

@chenlica : Professor, we intend to raise a PR later tonight or by tomorrow morning.

#120

The above PR, implements:

The Dynamic Programming algorithm for performing tokenization with highest likelihood.
Modifies test cases.

Merge pull request #120 from TextDB/yoda-dp

@chenlica

@chenlica We completed the documentation, and this can be found at https://github.com/TextDB/textdb/wiki/CS290-2016S-Task:-Query-Rewriter. Do you think we can proceed to close this issue?

I will review it first. Thanks.

I reviewed and polished your wiki page. Now you can close the issue.

Congratulations!

Thank you so much, Professor :)

On Tue, Jun 7, 2016 at 5:58 PM, Chen Li notifications@github.com wrote:

I reviewed and polished your wiki page. Now you can close the issue.

Congratulations!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#29 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/APbwhOH_GNnSKLkm61up4uyWidfbzcjfks5qJhOogaJpZM4H_v0D
.

Merge branch 'main' into fix/local-dev-fresh-db-replay

Navigation Menu

Repository navigation

(Team 2) Design: Query Rewriter #29

11 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

(Team 2) Design: Query Rewriter #29

Description

Activity

chenlica commented on Apr 12, 2016

chenlica commented on Apr 19, 2016

JavierJia commented on Apr 21, 2016

ss3n commented on Apr 21, 2016

kishore-narendran commented on Apr 26, 2016

ss3n commented on Apr 26, 2016

sandeepreddy602 commented on Apr 26, 2016

chenlica commented on Apr 26, 2016

sandeepreddy602 commented on Apr 26, 2016

chenlica commented on Apr 26, 2016

ss3n commented on Apr 27, 2016

11 remaining items

kishore-narendran commented on May 2, 2016

chenlica commented on May 9, 2016

ss3n commented on May 9, 2016

chenlica commented on May 23, 2016

ss3n commented on May 23, 2016

kishore-narendran commented on May 23, 2016

kishore-narendran commented on Jun 7, 2016

chenlica commented on Jun 7, 2016

chenlica commented on Jun 8, 2016

ss3n commented on Jun 8, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions