Precisely. When you get down to building flows, your data starts to get transformed during the flow, and when it gets to the endpoint it’s not in any structured form, except in terms of the flow it’s just come out of. The lake is basically “I’m done with this now so I’ll just leave this here until I might need it”.

You don’t need to optimise your data in a structured way for querying, unless that optimisation makes sense. The problem is that it rarely makes sense to optimise the data in the context of a flow. Even after the flow is finished, it may not make sense to push that data into a structured form.

The larger the application gets the more you need to structure the data so that it makes sense across the application — “this is an X and an X is related to a Y by this type of connection” — which is why when we’re building an application data store, something with a structure makes the most sense e.g. an RDBMS. It’s only when multiple pieces of data come together that you need to know how they relate.

Within a flow, you should not be trying to work out how the data passed in relates to your entire application because it’s irrelevant. So you shouldn’t need that much context.

Written by

ServerlessDays CoFounder (Jeff), ex AWS Serverless Snr DA, experienced CTO/Interim, Startups, Entrepreneur, Techie, Geek and Christian

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store