Re: ocaml-cohttp client

[Anil Madhavapeddy <anil@xxxxxxxxxx>]

One thing I forgot to mention is that Lwt_stream isn't quite ideal for our 
purposes, as it seems designed for combinator parsing rather than network 
traffic.  The current interface is non-blocking between the iterator and the 
iteratee. The push function type is:

('a option -> unit)

whereas we want:
('a option -> unit Lwt.t)

...so that the receiver can exert backpressure on the transmitter. Also, the 
receiver function cannot signal a shutdown to the transmitter currently. It is 
currently:

Lwt_stream.iter_s (fun v -> <do stuff>) stream

whereas we ideally want an Lwt exception from the receiver to be propagated 
back to the transmitter, so the next push function will immediately return that 
exception to signal a receiver has closed.

More concerningly, the push function seems to shut down under some 
circumstances, due to the stream itself getting GCed. This is because 
Lwt_stream uses Weak references in order to allow push/pull pairs to continue 
working even if the stream value itself is GCed (and the Weak ref prevents a 
memory leak).  This results in our TCP stack sometime mysteriously shutting 
down, annoyingly! Balraj and I tried to track this down, but we can't generate 
a small test case: it happens in the full Mirage stack, but not in toy examples.

So we came to the conclusion on Thursday that it would be best to write a 
simple Lwt_iteratee module that is modelled on Lwt_stream, but that does not 
having the stream-cloning functionality nor the weak references, and that adds 
the flow control mentioned above. It should be pretty much a drop-in 
replacement for most uses of Lwt_stream in the existing tree.

-anil