Repository navigation
ProgressEvent's messages and internal fields #286
Description
Activity
- changed the title
[-]`ProgressEvent` `messages` and `internal` fields[/-][+]ProgressEvent's messages and internal fields[/+]on Aug 20, 2019 I would totally recommend considering at least one other possibility, now.
With access to "custom" logging, where the user can slot in their own logger with their own type, you should be able to capture the actual frontiers, rather than string representations of them. The trade off is that the person on the other end of the logging stream would need to correctly "guess" what the type was when reading it back in. Perhaps that would work?
To be more concrete, just like how differential adds in its own logging for arrangement stuff, a subgraph (which is what creates the progress tracker) could create and register its own logging for progress events, which use the specific timestamp types. It would need to be named something like
"timely/subgraph-123415/progress"or such, which is gross but maybe ok.Having played around with the logging recently, I'm a bit inclined to figure out how to tease apart the logging streams (so, more in the direction above) where one of the implementations could be to slam each event into a common enum, but which allows a variety of different logging streams that can be picked up and processed by consumers who have specific opinions.
I'm pretty sure that would work (at least for me).
Could we surface the various subgraph loggers in a combined way so that they still end up in a single log event stream (to be consumed by
tdiag,snailtrailet al.)? I guess that would be some variation on the "slam each event into a common enum" implementation?Probably not. This is the trade-off with actually putting the type in the log event: it is hard to write one type a-priori that can wrap each of the timestamp types one might end up using. This is why the current enum has
Stringfor its timestamp stuff, because this is the best way we know how to represent timestamps without many constraints (justDebug, really).I see. Then it might make more sense for me to stick with the old
ProgressEventimplementation plus some indication on whether "real" progress was made (currently, even an endless iteration stuck at some pointstamp will happily provideProgressEvents that I can't discern from progress that advances the computation), and look into extracting the actual frontiers in a second step where I'm more interested into a per-subgraph analysis.Hrm. I'm not sure I understand "endless iteration stuck at a pointstamp". I think the logged events are meant to be only in response to actual progress traffic that occurs, so something should be going on when the events are logged.
The descriptors mean to follow the representation used by the
Progcaster, but it looks like this has evolved and the logging hasn't kept pace. The intent was to log theChangeBatch<(Location, T)>that is either sent or received, where a location is roughly a(usize, usize)with a tag on the type (either a source or target, corresponding to internal or message respectively). We should probably just shift that to be aVec<(Location, String, isize)>, I think.That's interesting; perhaps I made a mistake while inspecting an infinite iterate closure's log events. I will provide an example if that isn't the case. Thanks for your help in any case!
Should I still create a PR that includes the updated
ProgressEventfields?Sounds good! Extra good if you think you can make sense of their contents too. I haven't sanity checked that they are actually useful, but I think if you maintain a
ChangeBatch<(Location, String)>with these contents, you should at least see the current outstanding capabilities in the system (just, not be able to compare them correctly as they will be strings).
Hi Frank!
For profiling progress & visualizing dataflow frontiers, I believe I need some information in
ProgressEvents on what kind of progress actually happened. I think that themessagesandinternalfields could help me with that [1].Currently they are only mocked up in
send(...)andreceive(...). I'd be happy to create a PR and implement them, if you can guide me a little on what theVecfields are supposed to be. E.g., what are target and source descriptor in this context (I assume they are the first twousizecoordinates in theVec's tuples)?Cheers,
Malte
[1]