Analysis and classification of web proxy logs based on patterns of traffic rates

Logs are typically used for performing post mortem for abnormal activities. Most Internet service providers keep the history of users´ web accesses in terms of proxy logs for investigating a misuse or fraud. However, the majority of the logs represent normal behavior, and no thorough analysis of such logs is usually performed, keeping them on storage would consume very big space. This paper analyzes the characteristics of such logs and classifies them into normal, medium, high and burst rate using five main attributes: IP address, bandwidth, duration, file category, and file type. Our experimental results show different rates for each file type in five popular file categories. The results will be used in classifying web access logs and filtering out abnormal from normal logs so that only abnormal logs are kept for fast investigation.