INVESTIGATION OF THE EFFICIENCY OF DISTRIBUTED INFORMATION SYSTEMS BASED ON THE PROCESSING OF LARGE AMOUNTS OF DATA

Abstract

The features of processing large arrays of information for distributed systems are investigated. The method of singular data decomposition is applied, which can be used to reduce the amount of data processed by discarding excess data. Dependencies of computational efficiency on distributed systems were obtained using the MPI messaging protocol and the MapReduce nodal interaction model. The efficiency of the application of each technology for the processing of data sets of different sizes is analyzed. It has been determined that the MPI protocol enables more efficient computation of small amounts of information. As the data arrays increase, it is advisable to use the Map Reduce model. On shared resource systems, the device processes only some of the data that is being received for calculation. After the individual calculators complete their tasks, all parts are combined and the result is obtained. Further devices receive portions of other data, etc. This approach improves system performance because large amounts of information are processed much faster. Distributed computing systems have a number of other advantages, in particular, the scalability and reliability of data processing and storage. In such systems, the data is distributed by different devices to small parts, which significantly reduces the loss of information in the event of errors and damage to the latter Conventional retained data processing systems are inefficient for large amounts of information due to poor computing performance. It is proposed to use distributed systems that use the method of singular decomposition of data, which will reduce the amount of information processed. The study of systems using the MPI protocol and the MapReduce model obtained the dependence of the duration of the calculations on the number of processes, which testify to the expediency of using distributed computing in the processing of large data sets. It has also been found that distributed systems using the MapReduce model work much more efficiently than MPI, especially with large amounts of data.