Incremental window functions using AWS Glue Bookmarks

The out-of-order data landing problem Applying window functions over data is non-trivial if data arrives out-of-order (with respect to the dimension the window function is applied across). For clarity, lets take timeseries data for this example as our window dimension. If timeseries data arrives from Tuesday through Thursday of a week, then at a later time data from Monday of that week arrives, the data has arrived out-of-order. As a window function output is sensitive to its surroundings in timespace, the results of the window function would be altered by the new out-of-order data that landed....

 · 3 min · Hamish Lamotte