Massive Processing of Activity Logs of a Virtual Campus

Online web-based application that heavily require user interaction, either among users or among users and the application, generate huge amounts of data. Recording such user interaction data, usually in the form of log data files, could be very useful for different purposes such as user modelling, user activity analysis, data analytics, security, monitoring, etc. However, such data is not ready to be analysed due log files are to be pre-processed and cleaned up from redundant and futile information. Due to the large amounts of data generated daily, the massive processing is a foremost step in extracting useful information from log data files. In this work we study the viability of massive processing of log data files of a real Virtual Campus using different distributed infrastructures. More precisely, we study the time performance of processing daily log files of a Virtual Campus using cluster computing(under Open Grid Engine) and Planet Lab platform. The study reveals the complexity and challenges of massive processing in the big data era.