Utilising condor for data parallel analytics in an IoT context — An experience report

The current emphasis on sensor-based intelligent and ubiquitous systems, more commonly known as “cyber-physical systems”, has the potential to give rise to a new generation of systems and services encompassing several domains such as e-Governance, healthcare, transportation, waste management, energy & utilities, insurance, etc., resulting in the metamorphosis of the Internet as we see it, into the Internet of Things (IoT). One probable commonality in each of these services will be the abundance of different types of data from different sources with the success of the systems depending on real-time or near real-time analysis of data. Such analyses are normally performed via well-known algorithms with a time-constraint on the execution, thus creating a requirement for parallel execution techniques. Some of these analyses may have a higher frequency of execution on a relatively small set of data, in which case the current big-data frameworks may actually add an overhead. Further, the frameworks like Hadoop demand the algorithms to be mapped onto a particular paradigm, which may not always be a suitable option. This paper, which is a work-in-progress, provides an experience report on the use of Condor, a well known Grid framework, for data-parallel “black-box” style execution of analysis algorithms in the context of Internet of Things. We concentrate on algorithms which are already in use, and can be partitioned into data-parallel subtasks without any modification and use Condor, which has traditionally been used for high-performance or high-throughput computing, as the execution framework.