流沙网缘

I wrote the async library back when Node first removed Promises from core (yes, that really was the case). Back then, I preferred to do with plain callbacks what was usually done by including third-party Promise, Future or Continuable libraries instead.

I find a certain elegance in describing complex patterns using simple parts, particularly when the parts are made composable through a common interface. In Node, and JavaScript in general, you're likely to use a combination of Promises, callbacks, Streams, Event Emitters and even ES6 Generators. To me, however, these all represent values in the future. What if we could reduce entire programs to transformations over a Stream, and have just one API to rule them all?

Well, here is my proposal, for your consideration and feedback:
Highland, a high-level streams library

var _ = require('highland');
var doubled = _([1, 2, 3, 4]).map(function (x) {
 return x * 2;
});
var data = filenames.map(readFile).parallel(4);
data.errors(function (err, rethrow) {
 
});
data.pipe(output);
var output = fs.createWriteStream('output');
var docs = db.createReadStream();
_(docs).filter(isBlogpost).pipe(output);
docs.pipe(_().filter(isBlogpost)).pipe(output);
var clicks = _('click', btn).map(1);
var counter = clicks.scan(0, _.add);
counter.each(function (n) {
 $('#count').text(n);
});

This is not a new idea, but I believe it is a new combination of features, which is important. Those of you that work with Streams in Node may be aware of the excellent modules by Dominic Tarr, similarly if you're from the browser you might use FRP libraries such as RxJS. These are all great, but they hint at a deeper abstraction, one which would allow us to write entire programs using Streams. In an attempt to achieve this, Highland implements:

• Back-pressure support - Data sources are regulated so that slow consumers are not overwhelmed
• Laziness - So we can use the reading of Streams to sequence the execution of code - this means we can choose to read from files in parallel or series, or stop reading after an error, for example
• Asynchronous operations - So we're able to handle async data sources and async transformations
• Error and data channels - So we can manage error propagation from sync and async code
• Compatible with Node Streams - So we can pipe Node Streams to and from Highland Streams and play nicely with the Node ecosystem

If you find this idea intriguing and want to explore it further, then check out the Highland website. I'd love to hear your experiences.

Highland on GitHub