A scalable heterogeneous big data framework for e-learning systems

—The adoption of e-learning systems in higher education is a remarkable phenomenon that has redefined teaching and learning. Initially, it was proposed to allow people to learn for personal accomplishment without physically attending any traditional university or academic settings. While these systems continue to provide an efficient and flexible approach for teaching and learning, the rapid integration of ICTs and the expansion of data from these systems remain much concern to the education community. In this study, we propose a smart and secure data flow architectural framework for e-learning that uses a rich set of big data tools within a distributed and parallel analysis platform. Reference Model of Open Distributed Processing (RM-ODP) reference model guided the development of this Big Data framework for e-learning analytics. Using the RM-ODP model as a benchmark, we classify the educational institution’s architecture in terms of existing elements, functions and processes to understand stakeholders’ views for the development of the framework. Our approach uses an existing distributed computing environment and of fers an adaptable standard framework to improve the data acquisition, storage, processing, analysis, security and virtualization for e-learning systems. We implement a scalable and adaptable big data framework for e-learning (BiDeL) and point out how big data concept could integrate into online learning systems to improve teaching and learning in higher education using Apache Spark as a case. The proposed framework was applied to both batch and streaming dataset of students online activities on moodle LMSs. BiDel framework performance shows improved data integration and data governance. This big data framework and the general view of the current state of the art in “big data” technologies will serve as a guide for the creation of e-learning systems, which create new value from existing but underused data. The acquired value will provide decision-makers in higher education with new insights from data to enhance productivity in teaching and learning. These insights can lead to innovation, competitiveness in the academic space and even create entirely new teaching and learning models.