Distributed Fundamentals based Conducting the Web Crawling Approaches and Types (Focused, Incremental, Distributed, Parallel, Hidden Web, Form Focused and Breadth First) Crawlers

[1] M. M. Sadeeq, N. M. Abdulkareem, S. R. Zeebaree, D. M. Ahmed, A. S. Sami, and R. R. Zebari, “IoT and Cloud computing issues, challenges and opportunities: A review,” Qubahan Academic Journal, vol. 1, no. 2, pp. 1-7, 2021. Search in Google Scholar

[2] S. R. Zeebaree, H. M. Shukur, L. M. Haji, R. R. Zebari, K. Jacksi, and S. M. Abas, “Characteristics and analysis of hadoop distributed systems,” Technology Reports of Kansai University, vol. 62, no. 4, pp. 1555-1564, 2020. Search in Google Scholar

[3] P. Y. Abdullah, S. Zeebaree, K. Jacksi, and R. R. Zeabri, “An hrm system for small and medium enterprises (sme) s based on cloud computing technology,” International Journal of Research-GRANTHAALAYAH, vol. 8, no. 8, pp. 56-64, 2020. Search in Google Scholar

[4] J. Saeed and S. Zeebaree, “Skin lesion classification based on deep convolutional neural networks architectures,” Journal of Applied Science and Technology Trends, vol. 2, no. 01, pp. 41-51, 2021. Search in Google Scholar

[5] P. Y. Abdullah, S. Zeebaree, H. M. Shukur, and K. Jacksi, “HRM system using cloud computing for Small and Medium Enterprises (SMEs),” Technology Reports of Kansai University, vol. 62, no. 04, p. 04, 2020. Search in Google Scholar

[6] S. Zeebaree, R. R. Zebari, K. Jacksi, and D. A. Hasan, “Security approaches for integrated enterprise systems performance: A Review,” Int. J. Sci. Technol. Res, vol. 8, no. 12, pp. 2485-2489, 2019. Search in Google Scholar

[7] N. O. Salim, S. R. Zeebaree, M. A. Sadeeq, A. Radie, H. M. Shukur, and Z. N. Rashid, “Study for food recognition system using deep learning,” in Journal of Physics: Conference Series, 2021, vol. 1963, no. 1: IOP Publishing, p. 012014. Search in Google Scholar

[8] K. Jacksi, R. K. Ibrahim, S. R. Zeebaree, R. R. Zebari, and M. A. Sadeeq, “Clustering documents based on semantic similarity using HAC and K-mean algorithms,” in 2020 International Conference on Advanced Science and Engineering (ICOASE), 2020: IEEE, pp. 205-210. Search in Google Scholar

[9] R. R. Zebari, S. R. Zeebaree, Z. N. Rashid, H. M. Shukur, A. Alkhayyat, and M. A. Sadeeq, “A Review on Automation Artificial Neural Networks based on Evolutionary Algorithms,” in 2021 14th International Conference on Developments in eSystems Engineering (DeSE), 2021: IEEE, pp. 235-240. Search in Google Scholar

[10] S. A. Mostafa et al., “Applying Trajectory Tracking and Positioning Techniques for Real-time Autonomous Flight Performance Assessment of UAV Systems,” Journal of Southwest Jiaotong University, vol. 54, no. 3, 2019. Search in Google Scholar

[11] Z. S. Hammed, S. Y. Ameen, and S. R. Zeebaree, “Investigation of 5G wireless communication with dust and sand storms,” Journal of Communications, vol. 18, no. 1, 2023. Search in Google Scholar

[12] L. M. Abdulrahman, S. R. Zeebaree, and N. Omar, “State of Art Survey for Designing and Implementing Regional Tourism Web based Systems,” Academic Journal of Nawroz University, vol. 11, no. 3, pp. 100-112, 2022. Search in Google Scholar

[13] R. K. Ibrahim et al., “Clustering Document based on Semantic Similarity Using Graph Base Spectral Algorithm,” in 2022 5th International Conference on Engineering Technology and its Applications (IICETA), 2022: IEEE, pp. 254-259. Search in Google Scholar

[14] N. M. ABDULKAREEM and S. R. ZEEBAREE, “OPTIMIZATION OF LOAD BALANCING ALGORITHMS TO DEAL WITH DDOS ATTACKS USING WHALE OPTIMIZATION ALGORITHM,” Journal of Duhok University, vol. 25, no. 2, pp. 65-85, 2022. Search in Google Scholar

[15] R. K. Ibrahim, S. R. Zeebaree, K. Jacksi, M. A. Sadeeq, H. M. Shukur, and A. Alkhayyat, “Clustering document based semantic similarity system using TFIDF and k-mean,” in 2021 International Conference on Advanced Computer Applications (ACA), 2021: IEEE, pp. 28-33. Search in Google Scholar

[16] B. W. Salim and S. R. Zeebaree, “Kurdish Sign Language Recognition Based on Transfer Learning,” International Journal of Intelligent Systems and Applications in Engineering, vol. 11, no. 6s, pp. 232-245, 2023. Search in Google Scholar

[17] H. M. Zangana and S. R. Zeebaree, “Distributed Systems for Artificial Intelligence in Cloud Computing: A Review of AI-Powered Applications and Services,” International Journal of Informatics, Information System and Computer Engineering (INJIISCOM), vol. 5, no. 1, pp. 1-20, 2024. Search in Google Scholar

[18] S. Mohsin, B. W. Salim, A. K. Mohamedsaeed, B. F. Ibrahim, and S. R. Zeebaree, “American Sign Language Recognition Based on Transfer Learning Algorithms,” International Journal of Intelligent Systems and Applications in Engineering, vol. 12, no. 5s, pp. 390-399, 2024. Search in Google Scholar

[19] L. M. Abdulrahman, S. H. Ahmed, Z. N. Rashid, Y. S. Jghef, T. M. Ghazi, and U. H. Jader, “Web Phishing Detection Using Web Crawling, Cloud Infrastructure and Deep Learning Framework,” Journal of Applied Science and Technology Trends, vol. 4, no. 01, pp. 54-71, 2023. Search in Google Scholar

[20] A. A. Yazdeen, R. Qashi, H. S. Malallah, L. M. Abdulrahman, and M. A. Omer, “Internet of Things Impact on Web Technology and Enterprise Systems,” Journal of Applied Science and Technology Trends, vol. 4, no. 01, pp. 19-33, 2023. Search in Google Scholar

[21] H. S. Malallah, R. Qashi, L. M. Abdulrahman, M. A. Omer, and A. A. Yazdeen, “Performance Analysis of Enterprise Cloud Computing: A Review,” Journal of Applied Science and Technology Trends, vol. 4, no. 01, pp. 01-12, 2023. Search in Google Scholar

[22] M. A. Omer, A. A. Yazdeen, H. S. Malallah, and L. M. Abdulrahman, “A Survey on Cloud Security: Concepts, Types, Limitations, and Challenges,” Journal of Applied Science and Technology Trends, vol. 3, no. 02, pp. 47-57, 2022. Search in Google Scholar

[23] V. Shrivastava, “A methodical study of web crawler,” Journal of Engineering Research and Application, vol. 8, no. 11, pp. 1-8, 2018. Search in Google Scholar

[24] F. Albertos-Marco, J. A. Gallud, and V. M. Penichet, “A study on how users perceive distributed interactions on web applications,” IEEE access, vol. 6, pp. 57389-57399, 2018. Search in Google Scholar

[25] G. Baldassarre, P. L. Giudice, L. Musarella, and D. Ursino, “The MIoT paradigm: Main features and an “ad-hoc” crawler,” Future Generation Computer Systems, vol. 92, pp. 29-42, 2019. Search in Google Scholar

[26] A. Lagopoulos, G. Tsoumakas, and G. Papadopoulos, “Web robot detection: A semantic approach,” in 2018 IEEE 30th International Conference on Tools with Artificial Intelligence (ICTAI), 2018: IEEE, pp. 968-974. Search in Google Scholar

[27] F. Navarro, S. Conjeti, F. Tombari, and N. Navab, “Webly supervised learning for skin lesion classification,” in Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11, 2018: Springer, pp. 398-406. Search in Google Scholar

[28] M. Klein, L. Balakireva, and H. Van de Sompel, “Focused crawl of web archives to build event collections,” in Proceedings of the 10th ACM Conference on Web Science, 2018, pp. 333-342. Search in Google Scholar

[29] M. Kumar, A. Bindal, R. Gautam, and R. Bhatia, “Keyword query based focused Web crawler,” Procedia Computer Science, vol. 125, pp. 584-590, 2018. Search in Google Scholar

[30] S. M. Nakashe and K. R. Kolhe, “Smart approach to crawl web interfaces using a two stage framework of crawler,” in 2018 Fourth International Conference on Computing Communication Control and Automation (ICCUBEA), 2018: IEEE, pp. 1-6. Search in Google Scholar

[31] S. Raj, R. Krishna, and A. Nayak, “Distributed component-based crawler for AJAX applications,” in 2018 Second International Conference on Advances in Electronics, Computers and Communications (ICAECC), 2018: IEEE, pp. 1-6. Search in Google Scholar

[32] K. Aggarwal, “An efficient focused web crawling approach,” in Software Engineering: Proceedings of CSI 2015, 2019: Springer, pp. 131-138. Search in Google Scholar

[33] M. Biagiola, A. Stocco, F. Ricca, and P. Tonella, “Diversity-based web test generation,” in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2019, pp. 142-153. Search in Google Scholar

[34] P. Koloveas, T. Chantzios, C. Tryfonopoulos, and S. Skiadopoulos, “A crawler architecture for harvesting the clear, social, and dark web for IoT-related cyber-threat intelligence,” in 2019 IEEE World Congress on Services (SERVICES), 2019, vol. 2642: IEEE, pp. 3-8. Search in Google Scholar

[35] S. Mehak, R. Zafar, S. Aslam, and S. M. Bhatti, “Exploiting filtering approach with web scrapping for smart online shopping: Penny wise: A wise tool for online shopping,” in 2019 2nd International Conference on Computing, Mathematics and Engineering Technologies (iCoMET), 2019: IEEE, pp. 1-5. Search in Google Scholar

[36] M. Nasr, H. Fahmy, and M. Thabet, “Efficiency Improvement Approach of Deep Web Data Extraction,” in 2019 14th International Conference on Computer Engineering and Systems (ICCES), 2019: IEEE, pp. 214-221. Search in Google Scholar

[37] T. Nathezhtha, D. Sangeetha, and V. Vaidehi, “WC-PAD: web crawling based phishing attack detection,” in 2019 International Carnahan Conference on Security Technology (ICCST), 2019: IEEE, pp. 1-6. Search in Google Scholar

[38] T. Pflanzner, Z. Feher, and A. Kertesz, “A crawling approach to facilitate open iot data archiving and reuse,” in 2019 Sixth International Conference on Internet of Things: Systems, Management and Security (IOTSMS), 2019: IEEE, pp. 235-242. Search in Google Scholar

[39] W. Zhu, Y. Li, Y. Xu, and X. Cui, “Optimal bandwidth allocation for web crawler systems,” in 2019 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), 2019: IEEE, pp. 1146-1153. Search in Google Scholar

[40] A. Prusty, O. Mejia, A. Shah, P. Kancherlapalli, A. Suresh, and R. Schiebel, “Horizontally Scalable Web Crawler using Containerization and a Graphical user Interface.” Search in Google Scholar

[41] C. Pushpa, G. Deepak, A. Kumar, J. Thriveni, and K. Venugopal, “OntoDisco: improving web service discovery by hybridization of ontology focused concept clustering and interface semantics,” in 2020 IEEE international conference on electronics, computing and communication technologies (CONECCT), 2020: IEEE, pp. 1-5. Search in Google Scholar

[42] M. Biagiola, A. Stocco, F. Ricca, and P. Tonella, “Dependency-aware web test generation,” in 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), 2020: IEEE, pp. 175-185. Search in Google Scholar

[43] U. Upadhyay, R. Busa-Fekete, W. Kotlowski, D. Pal, and B. Szorenyi, “Learning to crawl,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2020, vol. 34, no. 04, pp. 6046-6053. Search in Google Scholar

[44] D. Arillotta et al., “Novel opioids: systematic web crawling within the e-psychonauts’ scenario,” Frontiers in neuroscience, vol. 14, p. 149, 2020. Search in Google Scholar

[45] K. Avrachenkov, K. Patil, and G. Thoppe, “Online algorithms for estimating change rates of web pages,” Performance Evaluation, vol. 153, p. 102261, 2022. Search in Google Scholar

[46] M. Jakubíček, V. Kovář, P. Rychlý, and V. Suchomel, “Current challenges in web corpus building,” in Proceedings of the 12th Web as Corpus Workshop, 2020, pp. 1-4. Search in Google Scholar

[47] K. Jacksi and S. M. Abass, “Development history of the world wide web,” Int. J. Sci. Technol. Res, vol. 8, no. 9, pp. 75-79, 2019. Search in Google Scholar

[48] P. Hegade, R. Shilpa, P. Aigal, S. Pai, and P. Shejekar, “Crawler by inference,” in 2020 Indo– Taiwan 2nd International Conference on Computing, Analytics and Networks (Indo-Taiwan ICAN), 2020: IEEE, pp. 108-112. Search in Google Scholar

[49] E. Uzun, “A novel web scraping approach using the additional information obtained from web pages,” IEEE Access, vol. 8, pp. 61726-61740, 2020. Search in Google Scholar

[50] X. Wang, Q. Sun, and J. Liang, “Json-ld based web api semantic annotation considering distributed knowledge,” IEEE access, vol. 8, pp. 197203-197221, 2020. Search in Google Scholar

[51] J. Xu, L. Du, C. Duan, and M. Li, “The application of web crawler in city image research,” in 2020 IEEE 5th International Conference on Cloud Computing and Big Data Analytics (ICCCBDA), 2020: IEEE, pp. 270-274. Search in Google Scholar