Chemistry Massive Open Online Course: Validity, reliability and undergraduate students’ perception

This is a design and development research (DDR) implementing ADDIE instructional design model. An xMOOC typed of Chemistry MOOC had been developed in the OpenLearning platform following the phases in the ADDIE model. A needs assessment was carried out during the Analysis phase to get the information from the lecturers on the chapters which are suitable to be delivered to the students in the form of MOOC. Types of content and assessment materials were collected and designed during the design phase. All the materials were then uploaded in the OpenLearning platform during the Development phase. Pilot study was done during the implementation phase to obtain the validity and reliability index of the modules in Chemistry MOOC, and finally perception questionnaires were distributed to the respondents to study their perception of the MOOC during the evaluation phase.

The main learning outcome to be achieved through this Chemistry MOOC is to enable the learners to apply fundamental chemistry concepts to solve chemistry problems. The content of this Chemistry MOOC is based on four chemistry topics, namely (1) Stoichiometry of Formulas and Equations, (2) Quantum Theory and Periodic Table, (3) Chemical Bonding and (4) Nuclear Chemistry. The e-content module compiles all the notes for the four topics in the form of PowerPoint slides, videos and animations. On the other hand, the e-assessment module provides exercises in multiple-choice questions, short answer questions, drag and drop activities, crossword puzzles, mix and match questions and true or false questions. The exercises were presented in two categories. The remedial exercise provides simple questions with difficulty level 1 and 2. While enrichment exercise consists of more challenging questions with difficulty level 3 and 4.

Sample of Research

Sample of research was chosen based on the objectives of the study: validity, reliability and perception. Four content experts were assigned to evaluate the content validity of the e-content module (three experts) and the e-assessment module (three experts). Since the Chemistry MOOC was developed for the undergraduate chemistry topics, a group of 23 undergraduate students was involved in the pilot study to answer the reliability questionnaire. A total of 129 undergraduate students from the Faculty of Science and Mathematics enrolled in Chemistry courses were randomly chosen as the respondents for perception study. The demographic information of the respondents is shown in Table I.

Table I. Demographic Information of Respondents

No.	Objectives	Demographic information	Categories	Frequency (n)	Percentage (%)
1.	Validity	Expertise	Physical chemistry	1	25.00
			Organic chemistry	1	25.00
			Inorganic chemistry	1	25.00
			Analytical chemistry	1	25.00
2.	Reliability	Gender	Male	9	39.13
		Gender	Female	14	60.87
		Program	Chemistry	19	82.61
			Biology	2	8.70
			Mathematics	2	8.70
3.	Perception	Gender	Male	28	21.70
		Gender	Female	101	78.30
		Program	Chemistry	112	86.80
			Biology	3	2.30
			Physics	6	4.70
			Mathematics	8	6.20

Instruments and Procedure

This study's three different instruments are the content validity evaluation form, reliability questionnaire, and perception questionnaire. These instruments were distributed to different groups of respondents in different phases based on the study's objectives.

Content Validity Evaluation Form

Content validity evaluation form was developed based on the content presented in both e-content and e-assessment modules in the Chemistry MOOC. For instance, the evaluation form for the e-content module listed all the videos and notes delivered in the module (Hamid, Lee, Taha, Rahim, & Sharif, 2021). On the other hand, the evaluation form for the e-assessment module listed all questions and exercises asked in the module (Kamarudin, Tien, Sharif, Taha & Rahim, 2020). The evaluation form was given to the four content experts during the Implementation phase of the ADDIE model. Experts were given the link of the Chemistry MOOC in the OpenLearning platform (Figure 1), and they were briefed on how to use both the e-content and e-assessment module in the MOOC. They need to evaluate the relevance of content in the e-content module and relevance of questions in the e-assessment module based on a 4-point scale: 1 = totally irrelevant, 2 = irrelevant, 3 = relevant, 4 = highly relevant.

Figure 1. Promotional Page of Chemistry MOOC

Reliability Questionnaire

The reliability questionnaires used in the study were adapted from Noah and Ahmad’s study (2005), which suggested two ways of identifying the reliability of a module. First is the ability of learners to achieve learning outcomes of the module, and second is the ability of learners to follow all the activities presented in the module. This study followed the second option. A group of undergraduate students were randomly chosen to answer the reliability questionnaires. The purpose of giving the reliability questionnaire to the respondents was to obtain the reliability index of the e-content and e-assessment module in the Chemistry MOOC. There are two sets of reliability questionnaire that focus on the e-content and e-assessment module activities. The items in the questionnaire are meant to check whether students can follow the content in the e-content module and answer the questions in the e-assessment module. Example of items in the reliability questionnaire is listed in Table II.

Table II. Sample Items in Reliability Questionnaire (Hamid et al., 2021; Kamarudin et al., 2020)

No.	e-Content Module	e-Assessment Module
1.	I understand the notes titled “The Mole”.	I understand all the instructions of exercise in the module.
2.	I can follow the explanation in the video titled “Stoichiometry Made Easy: Stoichiometry Tutorial”.	I can choose the correct answer in the multiple choice questions.
3.	I can follow all the content delivered in “Stoichiometry” topic.	I can solve the cross word puzzle.

Perception Questionnaire

Undergraduate students were gathered in a lecture hall, and they were introduced to the Chemistry MOOC. Students were given the QR code or website address to join the Chemistry MOOC. After they signed up or signed in, the course's homepage will be shown to the respondents. The respondents were briefed about how to use the e-content module and the e-assessment module in the MOOC. All respondents were given one to two weeks to explore the Chemistry MOOC before answering the perception questionnaire.

The perception questionnaire consists of three constructs, namely instructional design elements (20 items), acceptance (12 items) and usage barriers (8 items). The sub-constructs and the relative number of items are shown in Table III. This questionnaire was adapted from Fesol, Salam and Shaarani (2017) for the instructional design elements items and Daud, Zulkifli, Rahman and Khalid (2017) for the acceptance and usage barrier items. This study uses a 5-point Likert scale questionnaire, with 1 indicating ‘strongly disagree’, 2 indicates ‘disagree’, 3 indicates ‘neutral’, 4 indicates ‘agree’, and 5 indicates ‘strongly agree’. Three experts were assigned to check the content validity of the instrument. The experts gave comments to improve the items. After revision, the instrument obtained a content validity index (I-CVI) of 1.00 for all the items in the questionnaire. The reliability of the questionnaire was analyzed using internal consistency, and the Cronbach’s Alpha value obtained is 0.93.

Table III. Distribution of Items based on Constructs and Sub Constructs

No.	Construct	Sub-Construct	Distribution of items	Total item
1.	Acceptance	Performance Expectancy (PE)	A1 – A3	3
		Effort Expectancy (EE)	A4 – A6	3
		Social Influence (SI)	A7 – A9	3
		Facilitating Conditions (FC)	A10 – A12	3
		Total		12
2.	Usage barrier		A13 - A20	8
3.	Instructional design elements	Course Information (CI)	B1 – B4	4
		Course Resources (CR)	B5 – B8	4
		Active Learning (AL)	B9 – B12	4
		Monitoring Learning (ML)	B13 – B16	4
		Interaction (IR)	B17 – B20	4
		Total		20
4.	Total			40

Research Design