SYSTEM ARCHITECTURE OF DATA DEDUBLATION AND DISTRIBUTION IN CLOUD STORES DURING BACKUP

Authors

  • Bohdan Pavlovych Rusyn GV Physical and Mathematical Institute Karpenko NAS of Ukraine
  • Liubomyr Volodymyrovych Pohreliuk GV Physical and Mathematical Institute Karpenko NAS of Ukraine
  • Viktoriia Anatoliivna Vysotska NU Lviv Polytechnic, Department of Information Systems and Networks
  • Mykhailo Mykhailovych Osypov NU Lviv Polytechnic, Department of Information Systems and Networks
  • Yarema Yuriiovych Varetskyi GV Physical and Mathematical Institute Karpenko NAS of Ukraine
  • Oleh Viroslavovych Kapshii GV Physical and Mathematical Institute Karpenko NAS of Ukraine

DOI:

https://doi.org/10.31649/1999-9941-2019-45-2-40-63

Keywords:

data deduplication, data sharing, cloud environment, cloud computing, Rabbin algorithm, data hashing, hybrid deduplication

Abstract

The conceptual model of the system is developed and described in detail. An intelligent system of deduplication and distribution of data in the cloud storage is developed, the description of the software is described, the stages of the user's work are considered. Testing of the projected system was carried out. Several control samples are described and results are analyzed. The purpose of the system is to deduplicate and distribute data in cloud repositories in such a way that the end result of the backup is to eliminate duplicate pieces of data using distributed computing and cloud repositories. By picking the right approach to distribute tasks and data during deduplication, you can harness the full potential of cloud-based distributed systems to increase backup speed and bandwidth. Analyzes (disadvantages and advantages of using different approaches) are analyzed and effective methods of solution are selected: hybrid block-level deduplication, splitting of data flow on the basis of Rabin's digital imprint, distribution of data based on hash values of blocks of deduplication and use of distributed index. Block-level deduplication involves two types of data flow splitting into blocks, a fixed-length, algorithm-based split. Fixed-length partitioning is rather trivial and fast with respect to the complexity of the algorithm, but the downside is that data is displaced at the beginning of the stream, since the blocks that will follow after the changes will be considered new. However, in the case of partitioning of blocks of variable length, the point of proper partitioning is determined by the algorithm. This algorithm should work with infinite data flows using the ring hash function. The algorithm absorbs each input byte of data from the stream, and as soon as the value of the annular hash function corresponds to the previously specified template, it also serves as a point of splitting the stream into blocks. Thus, if the data is changed or displaced by a couple of bytes, only the data block that covers the data will be considered new. However, in order to track changes and correctly set breakpoints, it is necessary to check the input data for a specific preset digital pattern - a hash value. It is a common practice to calculate a hash value every time an input byte is received in a data stream. The point of partition will be the moment when the resulting hash value matches the specified pattern. To do these calculations effectively, an algorithm has been devised for the ring hash. One of the most common ring hash algorithms is a digital Rabin imprint. During the analysis of the solutions, the Rust programming language for client-side writing, the Scala programming language for the server-side, the Akka distributed computing management tool, and Amazon S3 as the cloud repository were selected.

Author Biographies

Bohdan Pavlovych Rusyn, GV Physical and Mathematical Institute Karpenko NAS of Ukraine

DSc, profesor,chief of depertment of methods and systems of remote sensing Karpenko Physico-Mechanical Institute

Liubomyr Volodymyrovych Pohreliuk, GV Physical and Mathematical Institute Karpenko NAS of Ukraine

postgraduate depertment of methods and systems of remote sensing Karpenko Physico-Mechanical Institute

Viktoriia Anatoliivna Vysotska, NU Lviv Polytechnic, Department of Information Systems and Networks

PhD, Associate Professor, Associate Professor of Information Systems and Networks Department

Mykhailo Mykhailovych Osypov, NU Lviv Polytechnic, Department of Information Systems and Networks

MSc of Information Systems and Networks Department, Lviv Polytechnic National University

Yarema Yuriiovych Varetskyi, GV Physical and Mathematical Institute Karpenko NAS of Ukraine

Phd,senior research fellow depertment of methods and systems of remote sensing Karpenko Physico-Mechanical Institute Nasu

Oleh Viroslavovych Kapshii, GV Physical and Mathematical Institute Karpenko NAS of Ukraine

Phd,researcher depertment of methods and systems of remote sensing   Karpenko Physico-Mechanical Institute

References

Understanding Data Deduplication [Електронний ресурс]. – Режим доступу: https://www.druva.com/understanding-data-deduplication

Explaining deduplication rates and single-instance storage to clients [Електронний ресурс]. – Режим доступу: https://searchitchannel.techtarget.com/tip/Explaining-deduplication-rates-and-single-instance-storage-to-clientshttp://zakon.rada.gov.ua/laws/show/2469-19

Inline vs. post-processing deduplication appliances [Електронний ресурс]. – Режим доступу: https://searchdatabackup.techtarget.com/tip/Inline-vs-post-processing-deduplication-appliances

Introduction to Data Deduplication [Електронний ресурс]. – Режим доступу: https://www.petri.com/data-deduplication-introduction

Rabin M. O. Fingerprinting by random polynomials / M. O. Rabin // Center for Research in Computing Technology Harvard University Report – Harvard, 1981.

Tanenbaum A.S. Distributed Systems / A.S. Tanenbaum, M. van Steen. – Upper Saddle River : Pearson Prentice Hall, 2017. – 15 c.

Amdahl G. The validity of the single processor approach to achieving large-scale computing capabilities. / G. Amdahl. – Atlantic City : Proceedings of AFIPS, 1967.

Using StorReduce for cloud-based data deduplication [Електронний ресурс]. – Режим доступу: https://cloud.google.com/solutions/partners/storreduce-cloud-deduplication

OpenDedup Overview [Електронний ресурс]. – Режим доступу: https://opendedup.org/odd/overview/

10. Rumbaugh J. The unified modeling language reference manual / J. Rumbaugh, I. Jacobson, G. Booch // Addison Wesley Longman Inc. – 1999.

Rolling hash, Rabin Karp, palindromes, rsync and others [Електронний ресурс]. – Режим доступу: https://www.infoarena.ro/blog/rolling-hash

Vysotska V. Methods based on ontologies for information resources processing / V. Vysotska, L. Chyrun, V. Lytvyn. - LAP Lambert Academic Publishing, 2016.

Vysotska V. Information technologies of gamification for training and recruitment / V. Vysotska, N. Shakhovska. - Saarbrücken, Germany: LAP LAMBERT Academic Publishing, 2018.

Висоцька, В.А. Особливості проектування та впровадження систем електронної комерції. / В.А. Висоцька // Комп’ютерні науки та інформаційні технології, Вісник Національного університету “Львівська політехніка”. – Львів 2008. – № 629. – С. 34-45.

Vysotska V. Web resources processing based on ontologies / V. Vysotska, V. Lytvyn. - Saarbrücken, Germany: LAP LAMBERT Academic Publishing, 2018.

Vysotska V. Tekhnolohiyi elektronnoyi komertsiyi ta Internet-marketynhu / V. Vysotska. - Saarbrücken, Germany: LAP LAMBERT Academic Publishing, 2018.

Vysotska V. Internet systems design and development based on Web Mining and NLP / V. Vysotska. - Saarbrücken, Germany: LAP LAMBERT Academic Publishing, 2018.

Vysotska V. Computer linguistics for online marketing in information technology: Monograph / V. Vysotska. - Saarbrücken, Germany: LAP LAMBERT Academic Publishing, 2018.

Lytvyn V. A Method of Construction of Automated Basic Ontology / V. Lytvyn, V. Vysotska, W. Wojcik, D. Dosyn // 1st International Conference Computational Linguistics and Intelligent Systems, COLINS’2017. – 21 April 2017, Kharkiv. – P. 75-83.

Lytvyn V. Intelligent System Structure for Web Resources Processing and Analysis / V. Lytvyn, V. Vysotska, L. Chyrun, A. Smolarz, O. Naum // 1st International Conference Computational Linguistics and Intelligent Systems, COLINS’2017. – 21 April 2017, Kharkiv. – P. 56-74.

Berko A. Features of information resources processing in electronic content commerce / Andriy Berko, Victoria Vysotska, Lyubomyr Chyrun // Applied Computer Science. ACS journal. – Volume 10, Number 2. – Poland, 2014. – ISSN 2353-6977 (Online), ISSN 1895-3735 (Print). – PP. 5-19.

Берко, А.Ю. Методи та засоби оцінювання ризиків безпеки інформації в системах електронної комерції / А.Ю. Берко, В.А. Висоцька, І.В. Рішняк // Інформаційні системи та мережі. Вісник Національного університету “Львівська політехніка”. – Львів 2008. – № 610. – С.20-33.

Vysotska Victoria. Web Content Processing Method for Electronic Business Systems / Victoria Vysotska, Lyubomyr Chyrun // International Journal of Computers & Technology. – Vol 12, No 2. – December 2013. – PP. 3211-3220.

Висоцька В.А. Моделювання етапів життєвого циклу комерційного web-контенту / В.А. Висоцька, Л.Б Чирун, Л.В. Чирун // Інформаційні системи та мережі. Вісник Національного університету “Львівська політехніка”. – Львів 2011. – № 715. – С. 69-87.

Берко, А.Ю. Алгоритми опрацювання інформаційних ресурсів в системах електронної комерції / А.Ю. Берко, В.А. Висоцька, Л.В. Чирун // Комп’ютерні науки та інформаційні технології. Вісник Національного університету “Львівська політехніка”. – Львів 2008. – № 616. – Стор.128-136.

Vysotska Victoria. Commercial Web Content Lifecycle Model: conference proceedings, November 16-19, 2011 / Victoria Vysotska, Lubomyr Chyrun, Lilya Chyrun // The 6th International Scientific and Technical Conference “Computer Sciences and Information Technologies” (CSIT’2011) which will be held November 16-19, 2011 at Lviv Polytechnic National University (Lviv, Ukraine) – Lviv 2011. – P. 160-163.

Берко А.Ю. Проектування навігаційного графу Web-сторінок бази даних систем електронної комерції. / А.Ю. Берко, В.А. Висоцька // Комп’ютерні науки та інформаційні технології, Вісник Національного університету “Львівська політехніка”. – Львів 2009. – № 638. – С. 3-14.

Берко А.Ю. Семантична інтеграція неповних та неточних даних / А.Ю. Берко, В.А. Висоцька, В.В. // Збірник наукових праць «Системи обробки інформації. Безпека та захист інформації в інформаційних системах», Випуск 7 (79). – Харків 2009. – С. 93-98.

Берко, А.Ю. Моделі та методи проектування інформаційних систем електронної комерції / А.Ю. Берко, В.А. Висоцька // Автоматизовані системи управління та прилади автоматики. Науково-технічний журнал. – Харків 2007. – № 138. – С.55-66.

Алєксєєва К.А. Управління Web-ресурсами за умов невизначеності / К.А. Алєксєєва, А.Ю. Берко, В.А. Висоцька // Журнал «Технологический аудит и резервы производства». – Vol 2, No 2(22) (2015). – Харків, 2015. – ISSN (print) 2226-3780, ISSN (on-line) 2312-8372. – С. 4-7.

Vysotska V. Designing features of architecture for e-commerce systems / Victoria Vysotska, Lyubomyr Chyrun // MEST Journal (Management Education Science & Society Technologie). – Vol.2 No.1. – P. 57-70.

Vysotska V. Set-theoretic models and unified methods of information resources processing in e-business systems / Victoria Vysotska, Lyubomyr Chyrun // Applied Computer Science. ACS journal. – Volume 10, Number 3. – Poland, 2014. – ISSN 2353-6977 (Online), ISSN 1895-3735 (Print). – P. 5-2.

Downloads

Abstract views: 441

Published

2019-10-15

How to Cite

[1]
B. P. Rusyn, L. V. Pohreliuk, V. A. Vysotska, M. M. Osypov, Y. Y. Varetskyi, and O. V. Kapshii, “SYSTEM ARCHITECTURE OF DATA DEDUBLATION AND DISTRIBUTION IN CLOUD STORES DURING BACKUP”, ІТКІ, vol. 45, no. 2, pp. 40–63, Oct. 2019.

Issue

Vol. 45 No. 2 (2019): Information technology and computer engineering

Section

Computer systems and components

Metrics

Downloads

Download data is not yet available.