Proposition

It is quite safe to assume that we operate in a complex environments past mere survival. Should it be societies, sciences, religions, nations or circumventing a yearly tax declaration.

“Complexity”

Cognition and reasoning are attributed with helping both envision such complex structures and overcoming accompanying convolutions. Ever more complex environment we imagine requires exponentially more cognition capacity to effectively orient yourself in. While cognition being our most expensive trait and the body can support only this much, we can rely on it only to a limited quantity and quality. …


This article is about using (rolling) window when applying mathematical aggregative function (i.e. statistic wiki) on data sampled from a table. Sample here is a number of consecutive elements that fit into a window.

We’ll see how this window-calculation can be realized with different tools: python, pandas, SQL and Spark.

For starters, lets define the process and try to make an imperative example with python, that will serve as a basis for understanding what is going on. Then we will jump to declarative examples and tools (pandas/SQL/Spark).

Reviewing the essence of a window

Imagine you have any kind of data array. Now you’d like to estimate…

VNimec

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store