The integration technology of sensor network based on web crawler

Along with the development of the sensor system and sensor network, the wide applications of sensor networks have arisen at the historic moment. In reality, all kinds of sensors monitor every aspect of our life, which provides various services and brings the challenge: how to effectively integrate those distributed sensor resources and then can be used to find more advanced information or implement the sharing of resources are the big problems to be solved. Based on the framework of Sensor Web Enablement(SWE) which was proposed by Open GIS Consortium (OGC)and combined with the function of web crawler, we study and find Sensor Observation Service (SOS) service which is the core components of the SWE then we design a system based on the web crawler technology and the Istituto Scienze della Terra Sensor Observation Service (Istsos) architecture. The design of sensor network technology integration architecture includes three parts. The layer of data access which is the lowest layer encapsulates the access to the database or other source of resources. The layer of business logic it provides the core operation of component which was named Request Operator, this layer is used for processing various requests from the lowest layer in order to return the classes of listening. The layer of web and the client is connected, which can provide some thin client of SOS. The published server includes the ability of new services creation, addition of new sensors and relative metadata, visualization, and manipulation of stored observations, registration of new measures and setting of system properties like observable properties and data quality codes. In order to get sensor data, web crawler technology is used in our research, which can make us get sensor data from the target website, and the standardized sensor data is gotten by filtering the original data and then the data is uploaded to the database of Istsos with the standardized format. At last, the implementation of SOS architecture has been configured. The test´s results show that the integrated architecture of services can effectively obtain the required sensor data and display them graphically.