Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make fromAction and fromActionStep able to pass a value from an action to the next one #1448

Open
giulioforesto opened this issue Aug 31, 2021 · 3 comments · May be fixed by #1533
Open

Make fromAction and fromActionStep able to pass a value from an action to the next one #1448

giulioforesto opened this issue Aug 31, 2021 · 3 comments · May be fixed by #1533

Comments

@giulioforesto
Copy link

Context

On servant-server, I must serve some data that I query from a store. The persistance interface returns me some Stream of resources, so actual queries to the store are done only when the stream is consumed.

Unsatisfactory solution

Fold the stream into a list and respond with that list in a normal Servant endpoint. This is unsatisfactory because by folding the stream, all resources are read from the persistance layer and accumulated in memory before being sent, which leads me to huge memory usage.

Objective

Pipe the resources directly from the Prelude.Streaming.Stream into a Servant.Types.SourceT, so that the persistance layer is queried only when the resource must be sent in the response stream, and thus avoid to accumulate many resources in memory.

To achieve this, I can generate a SourceIO with a fromAction or fromActionStep, that reads the next element of the stream.

Issue

Since all the actions run by fromAction are actually the same, I cannot consume the stream by accessing the next values.

Solution

Tweak fromActionStep to make it pass a value from an action to the next one:

fromActionStep' :: Functor m => (c -> m (Maybe (a,c))) -> c -> StepT m a
fromActionStep' action = loop where
  loop c = Effect $ step <$> action c
  step Nothing = Stop
  step (Just (x,t)) = Yield x $ loop t

And then call it on Prelude.Streaming.uncons.

Of course, Maybe (a,c) can be generalized to any type v, provided that we pass additionally some stop :: v -> Bool, value :: v -> a and rest :: v -> v functions.

Question

Was there an easier out-of-the-box solution I did not see?

Otherwise, is it worth adding such fromActionStepWithConsumable method to the library?
@alpmestan
Copy link
Contributor

@giulioforesto Hello! Wouldn't StreamResponse from this package allow you to seamlessly plug your existing stream into servant's server machinery?

I can see there: type ServerT (StreamResponse method status contentTypes :: *) m = m (Stream (Of ByteString) (ResourceT IO) ())

which means that if your endpoint runs in some monad YourMonad, the return type of your handler would have to be YourMonad (Stream (Of ByteString) (ResourceT IO) ()).

Regarding the implementation, you can see how the plumbing is done in https://hackage.haskell.org/package/servant-streaming-server-0.3.0.0/docs/src/Servant.Streaming.Server.Internal.html (bottom of that module)

@giulioforesto
Copy link
Author

Hi! Thank you for your answer!

I did not know about that package. It could indeed work. However, I saw that it's no longer maintained (notably, because it is supposed to have been replaced by Servant.API.Stream by the way), and it's not on Stackage either. Since I'm working on professional project, I cannot rely on deprecated libraries, so in any case it would need some rework. Besides, as my workaround with fromActionStep' is not so complicated, it is maybe easier to keep this solution.

It's just that I was surprised that there was no way to pass a value to the action of fromActionStep, to be able to consume something, like in my case Stream, or possibly any other data structure.

More generally, it would be nice to have the possibility to easily create a SourceT m from something else than a List (see source), notably from structures that cannot be converted to a List (without loosing interesting properties, like in my case the Stream).

@alpmestan
Copy link
Contributor

Fair enough, it seems reasonable to focus on the more recent streaming infrastructure we got.

I think the existing functions are mostly about supporting whatever was needed at the time, and could indeed be expanded to facilitate the life of users who end up relying on this by covering more scenarios, like that unfold-ish function you've been describing. All in all, while it's possible to implement something like this in your codebase by writing this function yourself in terms of StepT/SourceT constructors directly, I personally wouldn't be against adding more generic utility functions like yours if you were to put together a patch. :-)

@giulioforesto giulioforesto linked a pull request Feb 15, 2022 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(SourceT): add fromConsumableActionStep that creates a StepT from… giulioforesto/servant
2 participants
@alpmestan @giulioforesto