Data in Motion
The method of operating on data-in-motion in which data is operated on immediately as it arrives without the requirement that it first be stored in some kind of database is an alternative to batch processing. It is typically implemented as an in-memory technology with extremely low end-to-end latency. Depending on the operations being performed and the available processing hardware it is not unusual to have overall latency from the time a new data element arrives to when the results have been updated to be in the sub-millisecond range.
Some aspects of the solutions based on a software based analysis framework are that they need to act more “intelligent” and be more tolerant of uncertainty than traditional software based systems. These characteristics are, to some extent, present in the way that humans approach the same kinds of problems. Although the purpose of an analysis framework is not necessarily to mimic biological thought processes, there is sufficient common ground to make it a logical place to begin a design.
An obvious and important use for data-in-motion stream processing is in intensive care unit. More data is being generated today than ever and most if it is underutilized. The characteristics of stream processing and more specifically the InfoSphere Streams platform are ideally suited to consume this fire hose of data and process it into a form relevant to the clinician.