Ιδρυματικό Αποθετήριο [SANDBOX]
Πολυτεχνείο Κρήτης
EN  |  EL

Αναζήτηση

Πλοήγηση

Ο Χώρος μου

Scaling text processing pipelines using Apache Spark

Katsani Merieme

Απλή Εγγραφή


URIhttp://purl.tuc.gr/dl/dias/6D2F87E6-508B-4687-97B1-9A5CA530ACB6-
Αναγνωριστικόhttps://doi.org/10.26233/heallink.tuc.70651-
Γλώσσαen-
Μέγεθος48 pagesen
ΤίτλοςScaling text processing pipelines using Apache Sparken
ΔημιουργόςKatsani Meriemeen
ΔημιουργόςΚατσανι Μεριεμεel
Συντελεστής [Επιβλέπων Καθηγητής]Deligiannakis Antoniosen
Συντελεστής [Επιβλέπων Καθηγητής]Δεληγιαννακης Αντωνιοςel
Συντελεστής [Μέλος Εξεταστικής Επιτροπής]Garofalakis Minosen
Συντελεστής [Μέλος Εξεταστικής Επιτροπής]Γαροφαλακης Μινωςel
Συντελεστής [Μέλος Εξεταστικής Επιτροπής]Lagoudakis Michailen
Συντελεστής [Μέλος Εξεταστικής Επιτροπής]Λαγουδακης Μιχαηλel
ΕκδότηςΠολυτεχνείο Κρήτηςel
ΕκδότηςTechnical University of Creteen
Ακαδημαϊκή ΜονάδαTechnical University of Crete::School of Electrical and Computer Engineeringen
Ακαδημαϊκή ΜονάδαΠολυτεχνείο Κρήτης::Σχολή Ηλεκτρολόγων Μηχανικών και Μηχανικών Υπολογιστώνel
ΠερίληψηBig data, which is derived from humans or machines, starting with social media and extending to smartphones or sensors, in forms of texts, images or transactions, is a continuously evolving field. Thus, the ongoing increase of data generated creates a need for knowledge extraction from it, through data analysis. Several areas are engaged in data mining, and in particular the area of machine learning which has been well established over the past years. Various techniques and methods of machine learning are trying to solve big data problems and these two areas consist now an integral part. This particular combination is the main subject of this study, which aims to implement a large-scale text processing architecture. More specifically, this architecture focuses on processing streaming texts derived from Reddit in real-time and the classification thereof as sarcastic or non-sarcastic through a machine learning model. The architecture uses the latest technologies in the field of information processing through distributed platforms such as Apache Kafka and Spark as well as state-of-the-art but also simple and powerful ML algorithms, i.e Random Forests, Naive Bayes and Logistic Regression. After comparing the methodology and design of each individual piece forming the final layout, a selection of the most appropriate model is made followed by the implementation of the framework. Success rates exported were quite close to the relevant literature and sometimes higher, depending on each technique examined. Finally, results are indexed in the distributed search engine Elasticsearch and are evaluated through the Kibana plugin.en
ΤύποςΔιπλωματική Εργασίαel
ΤύποςDiploma Worken
Άδεια Χρήσηςhttp://creativecommons.org/licenses/by/4.0/en
Ημερομηνία2018-01-02-
Ημερομηνία Δημοσίευσης2017-
Θεματική ΚατηγορίαBig dataen
Θεματική ΚατηγορίαMachine learningen
Θεματική ΚατηγορίαNLPen
Θεματική ΚατηγορίαNatural Language Processingen
Βιβλιογραφική ΑναφοράMerieme Katsani, "Scaling text processing pipelines using Apache Spark", Diploma Work, School of Electrical and Computer Engineering, Technical University of Crete, Chania, Greece, 2017en

Διαθέσιμα αρχεία

Υπηρεσίες

Στατιστικά