Skip to content

Confusing / inconsistent operation of '$lt' and '$lte' in queries run on docs with missing props #321

Closed

Description

@VladimirTechMan

To @techfort and @obeliskos: Thought with a small extra delay, I am opening this issue to make you aware of this specific case (in case if you were not) and discuss it. I am not sure yet where we will come to, with this discussion. At the minimum, it is worth exchanging opinions on it, I believe. And possibly documenting it properly, so that people are aware.

Here is the issue: Suppose that we have populated a collection in Loki with some documents. The documents do not confirm to just a single "template" in terms of their inner fields, but to a few different ones (maybe many). Now, (1) let's assume that we want to run a query against the collection, using a record-set or a dynamic view. And (2) let's assume that we want to query all the documents where a specific field is less than (or is less than or equal to) a certain value. A string value or (even more likely in practice) a numeric value. Finally, (3) let's assume that the field in consideration is present only in a fraction of the documents in the collection (or that the field is not present at all in any document).

Here is what will happen in case of a field that is queried as a direct property of documents in the collection:

  • When evaluating the { 'fieldName': {'$lt': value} } expression, the find() method of ResultSet will get the value of property 'fieldName' for each document referenced by the filteredrows(or collection data, or an existing index).
  • For the documents that do contain that specific property in them, the LokiOps.$lt function will operate against the property's value.
  • For the documents that do not contain that specific property, the operation's function $lt will be run against the undefined value. And as the '$lt' operation relies on ltHelper which also serves as the ordering criteria (together with gtHelper), the latter case with a non-existing property will evaluate to true. While that is good for ordering purposes, it is quite unlikely that developers will want and expect to get the result-set will all the records not containing the field referenced by the query.

Activity

  1. VladimirTechMan commented on Jan 19, 2016

    @VladimirTechMan
    CollaboratorAuthor

    To continue, it is worth noting that the above situation will not be the same when applying the same type of query to nested ("dot-chained") properties. In this case, non-existent properties on documents will result in false from dotSubScan() as that method of ResultSet stops scanning the "path" upon encountering a non-existent property of a document. Thus, we have an issue with the case described in my initial comment. And an inconsistency compared with the case I am describing in this paragraph.

    But the inconsistency goes further in this case: The dotSubScan() method also aborts scanning the "path" upon finding undefined or null values of the referenced property. Which means that checking those properties with $lt or $lte on indirect document properties will also result in false from the method, thus not including the related documents into the result-set. In contrast, referencing a direct property of the document where the property's value is null or undefined (the latter is not exactly the same as a non-existent property in JavaScript) will result in the inclusion of the document in the result-set. In this case, one may argue either way of operation, I guess (?) But they need to be uniform between the direct and indirect document properties in queries.

  2. VladimirTechMan commented on Jan 19, 2016

    @VladimirTechMan
    CollaboratorAuthor

    What can be done about it? I am intentionally not submitting any pull requests here now. Instead, I'd like to discuss with you how you see this situation. And we could approach it going forward. For the start, I can outline the following approaches:

    (1) Don't change the current implementation of find(). But explicitly document it to make the developers aware. And suggest them a workaround, e.g.

    • Instead of using just the {'field': {'$lt': value} } node in the query, use a more complex '$and' sub-expression node, like {'$and': [ {'field': {'$ne': void 0 }}, {'field': {'$lt': value }} ] }. There can be other ways, but the idea is clear, I think. However, that will immediately come with a performance cost for just a "simple less-than operator". Which should not be an issue for small data sets, but will affect queries on large data sets, of course.

    (2) Introduce a special complementary set of comparison operators to the existing '$lt', '$lte', '$gt', '$gte' operations. Something like '$emlt', '$emlte', '$emgt', '$emgte' (where 'em' stands for "embedded"). Those would be implemented in LokiOps as simple native JavaScript comparison operators, no extra processing, like ltHelper and gtHelper do. The good side is that we don't need to change find() in that case. Same was as the previous approach, this one cannot benefit from the binary index range optimizations, of course. In terms of the speed, I guess this approach would be quicker in most cases compared to approach (1).

    (3) Do modify the find() method – the specific cases in it dealing with direct document properties – to check if the referenced property exists in each document. That will make the current '$lt' and '$lte' operators consistent with a typical developer's expectation. But it will obviously come at certain price, in terms of the performance. More speed tests would need to be done here to see if it is really bad or not. It is also important that special handling would be required in this case for the binary index-based range optimization – should be doable, but I did not check how much the logic would have to be altered. Possibly, you @techfort and @obeliskos can comment better on that from your knowledge of implementation details.

    I hope that helps. And look forward to your reply.

  3. VladimirTechMan commented on Jan 19, 2016

    @VladimirTechMan
    CollaboratorAuthor

    As the final note. :) Whichever of the above approaches (or maybe an approach you'll propose) we take, the inconsistency between comparing the existing undefined and null values of document properties for direct and indirect properties will need to be fixed. For which, we need to agree what the expected behavior is. And then document it in Loki's public documentation.

  4. changed the title [-]Confusing / inconsistent operation of '$lt' and '$lte' in queries run on records with missing props[/-] [+]Confusing / inconsistent operation of '$lt' and '$lte' in queries run on docs with missing props[/+] on Jan 19, 2016
  5. obeliskos commented on Jan 19, 2016

    @obeliskos
    Collaborator

    Might take me a while to read and I can tell you have thought a lot about this, so I will just offer initial impressions that are just thoughts about how I remember it and we let it evolve.

    So my initial takeaway is that your main concern is mainly about missing properties and how we handle them across operators as well as sorting (since they use same helpers).

    One approach (just like your ne void example) would be to add $exists operator and if you have dirty data, you can add an $exists filter along with your comparison filter within a chained query or dynamic view filter.

    Another approach would be to structure our responses from helpers (ltHelper, gtHelper) as well as dotSubScan to return response objects instead of Boolean true/false. A response might just be something like { result : true/false, exists: false } where result is our existing return value and exists is result of hasOwnProperty which we can test if the 'value' is undefined.

    So as for policy, just to clarify if the user does a '$lt' op on a collection, retains results, does $gte op on same collection and unions the results.... AND the property in both of those filters are the same and only exists in half the documents... that the end union would only be half the documents (the docs without the property are neither less than nor greater than nor equal to anything)... ?

    Personally I would gain value in an { $exists: true/false } filter op for controlling this myself if I needed to. As for 'default' behavior i'm ok with dropping those records if performance isn't impacted significantly.

    @techfort is probably GMT 0 so he may chime in tomorrow :)

  6. techfort commented on Jan 20, 2016

    @techfort
    Owner

    hey @VladimirTechMan ! first of all thanks a lot for taking the time to explain the issue and for spotting it in the first place.
    I don't want LokiJS to be too opinionated so I would never enforce schemas on collections, however I am satisfied that contravening document structure consistency is a developer's risk.

    The most important implication of changing the policy on $lt and $lte and similar operators is performance.

    The most important implication of not changing this policy is documentation.

    Weighing the two, I'd be happy to put additional lines of documentation explaining the policy of the operator and the behaviour on non-existent fields.

    However, @obeliskos 's suggestion about the $exists operator is attractive, as it puts the onus of additional checks due to inconsistent structures on the developers.

    Do let me know if I missed something - I appreciate the length you guys go to to make LokiJS better, especially in a period in which my spare time is next to zero.

  7. techfort commented on Jan 20, 2016

    @techfort
    Owner

    on a side note: @VladimirTechMan I really appreciate the work you've been doing on LokiJS. If you want to become an official contributor to the library I'd be more than happy to give you such rights, the God of Mischief can do with the help of one more trickster here in Midgard.

  8. VladimirTechMan commented on Jan 22, 2016

    @VladimirTechMan
    CollaboratorAuthor

    Thanks for the comments, @techfort and @obeliskos. I did consider introducing '$exists' as well. (Indeed, as @obeliskos correctly pointed, using '$exists' is along the lines of the possible "workaround" that I noted with approach (1) – though, with some subtle diffs, but still, in general, it is about the same explicit filtering out of "undefined values"). That is basically what I am doing in my code anyway, thus it would not make things worse for me. I would love to be able to make it quicker of course, but it won't be my major concern for sometime. And I still hopeful that we'll be able to optimize and speedup things just in general, which will help in this case too. (I have some more ideas that I am playing with now.)

    I would say that – between us and for the public documentation to be updated – we need to clarify on how Loki treats the undefined JavaScript value. I believe that @techfort tries to mimic MongoDB's behavior in many aspects in terms of the query language. Which is good, to me. As MongoDB does have operator '$exists' – yes, it makes perfect sense then to "replicate" it in Loki, instead of "inventing" something similar, but different in details (name or parameter values) in Loki. With that in mind, it is worth noting that MongoDB have officially deprecated the undefined values in its queries (while preserving the null value as valid). Thus, we may want to follow the same avenue, and to document that aspect explicitly to the developers. And, thus, to implement '$exists' just in terms of checking if accessing the referenced document field gives a result that is (or is not) undefined, be it a real value or a result of accessing a non-existent property in the object.

    Implementing '$exists' with hasOwnProperty() or the in JS operator would require deeper changes – for example, by splitting the "root" object reference and "property" name into two arguments of the LokiOps functions or by introducing a separate code branch for '$exists', like those for '$or' and '$and'. I guess it is less desirable to you.

    Taking the official "deprecation" of undefined as a real value of documents one step further, a recommendation to the Loki users maybe to always use null as a special "no-value / to-be-set-value" mark in the documents. And never use undefined in that way, or convert it to null before inserting a document to Loki. Then '$exists', if it is implemented as in the former option, will cause less of ambiguity, if any. Even better, in this case, we may be able to use the existing binary indices to quickly get all "non-existent" (val === undefined) or all "existent" (val !== undefined) documents as index ranges where the range optimization could be applicable. (What do you think?)

    P.S. The "side effect" of applying the '$lt' and '$lte' query operations on the documents that do not have the referenced field will need to be explicitly mentioned in Loki's public documentation.

    P.P.S. My only concern remains around the difference when dotSubScan() is involved vs. when it is not. But we may possibly postpone it – and later approach it with a different issue / pull request (maybe coming from me as well ;-) ).

  9. VladimirTechMan commented on Jan 22, 2016

    @VladimirTechMan
    CollaboratorAuthor

    Hey @techfort. Replying on your side note above: I don't mind it! The stream of the fixes and improvements from my side may vary, of course, depending on my work-load and different activities. Next months may be more challenging from that perspective. (It is also worth noting that I was on vacation for a few last weeks, thus I had some more time to dedicate to Loki.) In any case, I do like Loki as the project. My main priority is my own initiative with the log format and a browser for it (which I mentioned to you in one of the past pull requests), but as Loki is its core part and Loki's features and performance is critical to my project's success, I will (almost inevitably :-) keep improving Loki's internals where I hit limitations standing in my way.

    P.S. To mention, I am possibly not very fluent in the Norse mythology (though I do have a good general idea about it and some major characters, like "Loki") – more fluent with the Slavic mythology. :-) But I hope that is not a prerequisite for Loki's core contributors? ;-)

  10. techfort commented on Jan 22, 2016

    @techfort
    Owner

    @VladimirTechMan no requirement to be a LokiJS comtributor other than
    passion, but if you have any interesting underground facts about Perun &
    Co. I'm sure we'd all find it entertaining :-D

    On Fri 22 Jan 2016 20:07 VladimirTechMan notifications@github.com wrote:

    Hey @techfort https://github.com/techfort. Replying on your side note
    above. I don't mind it. The stream of the fixes / improvements from my side
    may vary, of course, depending on my work-load and different activities.
    Next months may be more challenging in that way. (It is also worth noting
    that I was on vacation for a few last weeks, thus I had some more time to
    dedicate to Loki.) In any case, I do like Loki as the project. My main
    priority is my own initiative (which I mentioned to you in one of the past
    pull requests), but as Loki is its core part and Loki's features and
    performance is critical to my project's success, I will (almost inevitably
    :-) keep improving Loki's internals where I hit limitations in my way.

    P.S. To mention, I am possibly not very fluent in the Norse mythology
    (though I do have a good general idea about it and some major characters,
    like "Loki") – more fluent with the Slavic mythology. :-) But I hope that
    is not a prerequisite for Loki's core contributors? ;-)

    —
    Reply to this email directly or view it on GitHub
    #321 (comment).

  11. VladimirTechMan commented on Jan 23, 2016

    @VladimirTechMan
    CollaboratorAuthor

    Good. Thanks, @techfort. I believe I comply. :-)

    BTW, did you have a chance to look at my reply above to you and Dave regarding the main topic of this issue? Any thoughts on that?

  12. techfort commented on Jan 23, 2016

    @techfort
    Owner

    @VladimirTechMan yes absolutely, I am gathering my thoughts and will follow with more detailed feedback.

  13. stale commented on Jul 13, 2018

    @stale

    This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions