Repository navigation
Mongoengine is very slow on large documents compared to native pymongo usage #1230
Description
(See also this StackOverflow question)
I have the following mongoengine model:
class MyModel(Document):
date = DateTimeField(required = True)
data_dict_1 = DictField(required = False)
data_dict_2 = DictField(required = True)In some cases the document in the DB can be very large (around 5-10MB), and the data_dict fields contain complex nested documents (dict of lists of dicts, etc...).
I have encountered two (possibly related) issues:
- When I run native
pymongo find_one()query, it returns within a second. When I runMyModel.objects.first()it takes 5-10 seconds. - When I query a single large document from the DB, and then access its field, it takes 10-20 seconds just to do the following:
m = MyModel.objects.first()
val = m.data_dict_1.get(some_key)
The data in the object does not contain any references to any other objects, so it is not an issue of objects dereferencing.
I suspect it is related to some inefficiency of the internal data representation of mongoengine, which affects the document object construction as well as fields access.
Activity
touilleMan commentedon Feb 8, 2016
Hi,
Have you profiled the execution with
Profile/cProfile? A graph of it withobjgraphshould give us a better view of where the trouble is.baruchoxman commentedon Feb 8, 2016
Hi,
Please see the following chart: http://i.stack.imgur.com/qAb0t.png (attached in the answer to my StackOverflow question) - it shows that the bottleneck is in "DictField.to_python" (being called 600,000 times).
thedrow commentedon Mar 10, 2016
Have you tried using MongoEngine with PyPy? Most of the overhead is gone because of PyPy's JIT.
But you're right. We need a better benchmarks suite.
[-]Mongoengine is very slow on large documents comapred to native pymongo usage[/-][+]Mongoengine is very slow on large documents compared to native pymongo usage[/+]amcgregor commentedon Jun 23, 2016
The entire approach of eager conversion is potentially fundamentally flawed. Lazy conversion on first access would defer all conversion overhead to the point where the structure is actually accessed, completely eliminating it in the situation where access is not made. (Beyond such a situation indicating proper use of
.only()and related helpers is warranted.)amcgregor commentedon Jun 23, 2016
Duplicate of #1137 ?
lafrech commentedon Jun 23, 2016
amcgregor commentedon Jun 23, 2016
lafrech commentedon Jun 23, 2016
apolkosnik commentedon Mar 8, 2017
I've played a bit with a snippet and with some modifications( https://gist.github.com/apolkosnik/e94327e92dd2e0642e2b263efd87d1b1 ), then I ran it against Mongoengine 0.8.8 and 0.11:
Please see the pictures...
On 0.8.8:
mongoengine with dict took 16.95s
On 0.11:
mongoengine with dict took 32.74s
apolkosnik commentedon Mar 16, 2017
It looks like some change from 0.8.8 to 0.9+ caused the get() in ComplexBaseField class to go on a dereference spree for dicts().
25 remaining items
amcgregor commentedon Apr 5, 2019
I'd love to formalize this benchmark set (akin to template engine's "bigtable" test) and add more contenders to it. The code below the file of older results demonstrates, effectively side-by-side, identical solutions.
This is a more direct comparison of querying, specifically and in isolation, with the note that
as_queryis entirely unnecessary on the MM side; just passfind_{one/many}theFilterinstance; it's a suitable mapping natively. (Oh, and ME appears to be unable to "continue" from a "compiled" query, e.g. reconstitute the rich Q object from a plain dict; at least, when I made that comparison.)olka commentedon Nov 27, 2019
Same here -
to_pythoncall produces 70% of overheadpikeas commentedon Oct 10, 2020
Is this still an issue? I'm using Pymodm and would prefer switching to MongoEngine as a more popular ODM, but poor large object performance would be a deal breaker.
amcgregor commentedon Oct 11, 2020
@pikeas Yes, with some variance for some additional optimization here, and further complication over there… the underlying mechanism remains "eager", that is, upon retrieval of a record MongoEngine recursively casts all elements of the document it can to native Python types via repeated
to_pythoninvocation.This contrasts with my own DAO's approach (if I'm going to be fixing everything, might as well start from scratch) which is purely lazy: transformers to "cast" (or just generally process) MongoDB values to native values are executed on attribute access. Bypassed by dictionary dereferencing. The Document class' equivalent
from_mongofactory class method only performs the outermost Document object lookup and wrapping. Mine was written after many years of MongoEngine use and frustration with lack of progress on numerous fronts. Parts are still enjoyably crazy, but at least I can very exactly explain the “crazy” in mine. 😉Edited to add: Note that a double underscore-prefixed (dunderscore / soft private) initializer argument is available to disable eager conversion. The underlying machinery iteratively utilizes both explicit
to_pythoninvocation, and indirect invocation via setters (L125), which doesn't make it much easier to follow. 🙁Using my silly simple benchmark, the latest deserialization numbers:
next) 0.08598494529724121sAdmittedly, other areas differ in the opposite direction. Unsaved instance construction is faster under MongoEngine:
bagerard commentedon Oct 12, 2020
Note that I'll probably start experimenting with lazy loading the attribute soon in mongoengine (i.e defer the python de-serialization until it's actually called, like it is done in pymodm)
amcgregor commentedon Oct 12, 2020
The initial lazy version I completed myself 5 years ago — with minor deficits later corrected, e.g.
from_mongoof an already cast Document. I hope you don't mind that I didn't wait.bagerard commentedon Oct 12, 2020
You do whatever you want in your own project :) I'll make sure to check how you dealt with that (compared to pymodm) when I'll work on that, out of curiosity. I understand the reasons that made you move away from mongoengine but I would appreciate if we could keep the discussions in the mongoengine project constructive.
amcgregor commentedon Oct 13, 2020
@bagerard As the tag on my comments identify, I'm a past direct code contributor.
marrow/contentment#12 — I indexed my issues for handy reference. A number date to 2013: almost 8 years with little to no progress. Progress on some, of course! #1136 (regression in
limituse in 0.10.0) was a milestone for giving me the needed kick to get going on mine.Many others documented there were simply never engineered to be problems in the first place, e.g. by removal of a "Document" v. "DynamicDocument" differentiation, clear segregation and isolation of "active collection"-like behavior, and explicit avoidance of the self-saving (change tracking) "active record" pattern, no global registry demanding unique model class names, plus virtually no form of implicit caching or automatic (edit: eager/recursive) casting/conversion. No result set middleware, or connection management middleware, and so on. The Document instances are usable anywhere a dictionary is, within plain PyMongo APIs, with zero use-time work. (And near-zero effort to encapsulate raw PyMongo result set records.) It's, unfortunately, almost exactly the opposite of MongoEngine. 😕
bagerard commentedon Oct 13, 2020
I've seen your post many times and also all the references that it created in our tickets, I didn't need you to elaborate. If you don't like MongoEngine and gave up on improving it, that's ok but if we could keep the discussions in the MongoEngine project (thus its Issues) to actually improve MongoEngine, that would be more helpful
bkntr commentedon Dec 7, 2020
Is there a plan to add lazy init to MongoEngine any time soon?
neilsh commentedon Jan 17, 2022
Any news on this, or ways others can help?
cundi commentedon Mar 1, 2023
PyMODM is better than this lib, does someone knows lib which like it.
amcgregor commentedon Nov 10, 2023
I… somewhat hate to do this, but given how stale this (and numerous others; see my previous comments) have become, I wrote a replacement for myself. I haven't really gone out of my way to advertise it, but it follows some of the suggestions given here re: lazy conversion. (The
DocumentandQueryclasses act like a plain, PyMongo-compatible dictionaries/mappings, always storing MongoDB-safe types internally thus no conversion on final use.) It is rather stable, extremely well unit tested, and in active use.It does not, however, reimplement the full, nested change-tracking active record approach, nor automatic foreign record lookup. (Those are also performance and memory utilization problems.) It uses a "layer your needs" approach. The base
Documentclass is essentially just a fancy, declaratively-defined dictionary with validation and typecasting. Mix-ins such asQueryable(a type ofCollectionofIdentifiedrecords) adds inbind(to connect a PyMongo DB or collection),find,insert_one, &c. methods. There are also data defining mix-ins beyondIdentifiedsuch asLocalizedorPublished..get()or.first()replaced by class dereferencing on bound classes:User[identifier]To allow multiple installed packages to cooperate in defining fields or traits, entry_points-based pseudo-namespaces can be imported from:
marrow.mongo.document,marrow.mongo.field, andmarrow.mongo.trait. Register an entry_point in the appropriate namespace, your custom document, field, and trait can be imported from there, too. (Making conflicts explicit.) There is a minimal beginning to proper documentation.neilsh commentedon Mar 14, 2025
In case it's handy for others, I've found the
no_dereferencecontext manager handy for speeding up performance for Documents that don't need ReferenceField or DBField dereferencing. According to our profiling, mongoengine spent a lot of time recursively checking for (nonexistent) fields that needed to be dereferenced.https://docs.mongoengine.org/apireference.html#mongoengine.context_managers.no_dereference