How to add a big dataset

Hi,

I have big file JSON object (700 articles from Pocket's API) i'd like to add, but I experience some problems.

Adding 50 elements (articles) takes me 2 sec
Adding 100 elements takes me 20 sec
Adding all 700 elements crashes my app on the device

        //Create/get model from storage
        this.model = await reactNativeStore.model('articles');
        //Add our test data to storage
        for(var element in arr) {
            await this.model.add(arr[element]);
            //console.log("Item Added")
        }

Is there a solution or workaround for this?

Read/write performance is an issue right now that is next on my to-do list when I have more time to dedicate to personal projects during the holidays. The problem is that currently the whole database is written on each create/update/delete.

What I can suggest right now as a temporary hack is to add all your data in one add call and instead of using the interface in the Model class, copy filter.js to your working directory. You can write the whole array in read call and make your queries using filter.js like so:

var filter = require('./filter.js');

//other code here...

this.model = await reactNativeStore.model('articles');
this.filter = new Filter();
await model.add({ articles: arr });
var allData = await model.find();

var desiredArticles = this.filter.apply(allData.articles, {
    where: {
        //Query parameters here, note _id wont work since these
        //were elements of rows in the database
    }
});

Updating, deleting will be a hassle. If there is some unique property to an article, like a url, name, etc. you can update/delete the element in the allData array using that property and then update the model:

this.model.update(allData, {
    where: {
        _id: allData._id
    }
});

Thank you! I am just trying to implement this.

Quick question, how would I write my filter function if I want to find all articles with "favorite" == "1" (see object screenshot)?

http://i.imgur.com/HyM1O3O.jpg

var desiredArticles = this.filter.apply(allData.articles, {
    where: {
        //Query parameters here, note _id wont work since these
        //were elements of rows in the database
    }
});

This is how you would test for equality:

var desiredArticles = this.filter.apply(allData.articles, {
    where: {
        favorite: 1
    }
});

Note that every other comparison operator other than '==' is different:

var desiredArticles = this.filter.apply(allData.articles, {
    where: {
        favorites: { neq: 1 } // != 1

    }
});

See the LoopBack Where filter documentation for better reference, which were the specs the filter's in react-native-store were based on.

@terminull

Hi @terminull and @thorbenandresen
I need to do a similar thing (adding something like 2000 rows to the DB), so tried this:

// Add data to local database
var num_rows = responseData.data.length;
for(var key in responseData.data) {
    if(key == num_rows-1) {
        await thisModel.add(responseData.data[key]);
    } else {
        thisModel.add(responseData.data[key]);
    }
}

It basically makes all iterations async except for the final iteration, which it'll make the rest of the code wait for. It seems to work almost instantly.

Do you see any issues with this?

@terminull

Scratch that, my suggestion doesn't work 👎

I instead tried creating a separate API method - https://github.com/mcnamee/react-native-store/blob/develop/src/model.js#L79

If it looks ok with you @terminull - I'll create a pull request

+1 performant bulk-add is critical

+1

I switched to SQLite which is super quick (around 1.5 seconds for 2500 record insert).

@mcnamee

@mcnamee Does SQLite have the same simple query syntax? I found the multiAdd to be pretty quick, but need some more advanced querying capabilities.

It's pretty simple - you just pass SQL queries (be it an insert, search, update, delete) into a method. It provides a way more advanced option for finding records, given the ability to use SQL.

You can use multiAdd, may be it's help for you

Navigation Menu

Repository navigation

How to add a big dataset #21

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

How to add a big dataset #21

Description

Activity

kintsugi commented on Dec 2, 2015

thorbenandresen commented on Dec 3, 2015

kintsugi commented on Dec 5, 2015

mcnamee commented on Jan 12, 2016

mcnamee commented on Jan 12, 2016

mericsson commented on Jan 17, 2016

srhise commented on Mar 8, 2016

mcnamee commented on Mar 8, 2016

srhise commented on Mar 8, 2016

mcnamee commented on Mar 11, 2016

yuzhewo commented on Nov 6, 2016

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions