Ability to set highWaterMark on transform stream? #8855

Activity

1. mscdex

   added

   questionIssues asking questions about Node.js.Issues asking questions about Node.js.

   streamIssues and PRs related to Node.js streams.Issues and PRs related to Node.js streams.

   on Sep 30, 2016
2. mscdex commented on Sep 30, 2016

   

   mscdex

   on Sep 30, 2016

   Last edited by mscdex

   Contributor

   More actions

   All of the constructors can already accept options, including highWaterMark. For example:

   var Transform = require('stream').Transform;
   var inherits = require('util').inherits;
   
   function MyTransform(opts) {
     if (typeof opts !== 'object' || opts === null)
       opts = {};
     Transform.call(this, { highWaterMark: opts.highWaterMark });
     // Or just: `Transform.call(this, opts);`
   
     // Other initialization ...
   }
   inherits(MyTransform, Transform);
   
   MyTransform.prototype._transform = function(chunk, enc, cb) {
     // ...
   };

   👍React with 👍1Reacted by Páll Árpád
3. olalonde commented on Sep 30, 2016

   

   olalonde

   on Sep 30, 2016

   Last edited by olalonde

   ContributorAuthor

   More actions

   Ok, thanks. Maybe I'm misunderstanding something then. I just tested the following code, which outputs 16384 instead of the expected 100 (highWaterMark) to the console:

   var Transform = require('stream').Transform
   var RandomStream = require('common-streams').RandomStream
   
   class CustomTransform extends Transform {
     constructor() {
       super({
         highWaterMark: 100,
       })
     }
   
     _transform(chunk, enc, cb) {
       console.log(chunk.length)
       cb(null, chunk)
     }
   }
   
   const rand = new RandomStream(10 * 1024 * 1024)
   
   rand.pipe(new CustomTransform())

   Is highWaterMark a minimum which can possibly be exceeded?
4. mscdex commented on Sep 30, 2016

   

   mscdex

   on Sep 30, 2016

   Contributor

   More actions

   highWaterMark is just an indicator to communicate to upstream that no more data should be written/pushed. Obviously you can write more data and it will just continue to be buffered in memory. highWaterMark does not dictate the size of chunks passed to _transform()/_write().
5. olalonde commented on Sep 30, 2016

   

   olalonde

   on Sep 30, 2016

   ContributorAuthor

   More actions

   Ah ok, so it's more of a signal to upstream but not enforced, thanks.
6. olalonde

   closed this as completedon Sep 30, 2016
8. kennylbj commented on Sep 4, 2020

   

   kennylbj

   on Sep 4, 2020

   More actions

   How can I set the chunk size in _transform function?

   👍React with 👍2Reacted by Ishaan Taylor and Stud
9. bbrzoska commented on Jul 18, 2024

   

   bbrzoska

   on Jul 18, 2024

   Last edited by bbrzoska

   More actions

   If you're here because your @aws-sdk/lib-storage Upload code started failing after upgrading node from v18 to v22, and have this line:

   Readable.toWeb(stream)

   then try changing it to:

   Readable.from(stream)

   Which fixes the problem for me.