Validity and Reliability of Instrument Competencies Framework of Agriculture Educators: The Rasch Measurement Model

: This study aims to produce empirical evidence of the validity and reliability of instrument items for the competency framework of agricultural teaching staff in Malaysian agricultural vocational colleges. The validity and reliability of the framework were analyzed using Rasch Model Measurement assisted by Winsteps 3.72 software. This research instrument contained 116 items, which was distributed to 30 instructors at the Teluk Intan Agricultural Vocational College, Malaysia. The selection of respondents was made by strata random where the researcher makes the strata of the population according to the percentage and then selects randomly based on the desired percentage. Validity analysis of the instrument was done through four functional testings. For reliability and separation of respondents, it was found that the individual reliability value was very good and acceptable. The results of the item polarity analysis detected no negative value (-) in the Point Measure Correlation value. Item matching analysis found that 11 items had to be dropped as they failed to meet the required conditions. From the analysis on local dependence that determines dependent items based on the standardized residual correlation value, it was discovered that the correlation value for the items used was detected; 13 items need attention. The results of the data analysis checking the functionality of the items suggested that some items should be dropped. The omission of these items has provided evidence that the instrument of competence of agricultural instructors is crucial to have a high level of validity and reliability for use in actual studies.


Introduction
The competence of Technical and Vocational Education Training (TVET) educators is an issue discussed at the national level, as stated in the Eleventh Malaysia Plan (2016)(2017)(2018)(2019)(2020) and Malaysia Education Blueprint 2013-2025. The process of strengthening the field of TVET requires instructors' confidence and competence in performing tasks (Ramlan et al., 2017). Competent employees should have the skills, abilities, and competencies to effectively perform their duties, and the teaching staff's efficiency influences student learning achievement. Ahmad (2017) defines competence in general as the ability to do something brilliantly in terms of knowledge, attitude, and skills. This opinion is in line with that by Kulshrestha and Pandey (2013) as well as Nasir (2016) stating that teachers should have high competence in teaching pedagogy, teaching techniques, and, most importantly, the ability to master the content of practical knowledge. As an agricultural educator at KV, it is necessary to master the elements of knowledge competence (Cognitive), skills (Psychomotor), and values (Affective) in conducting practical's in the classroom. KV of the agricultural stream in Malaysia is a KV that makes agriculture a field that can generate income for graduates. Therefore, KV agriculture needs to supply competent agricultural instructors so that graduates in the agricultural industry market can perform their duties competently.
In recent years, technology in agriculture has been developing rapidly. Among the latest technologies include the operation of "drones" for fertilization and pesticide spraying activities in agriculture. Technology in agriculture requires skilled and competent workforce in the practice of agriculture. Technology and digital skills in professional practice are seen as necessary in the era of widespread digitalization in agriculture (Ramjattan et al., 2020). The more advanced the agricultural technology, the higher the perceptions and expectations given to agricultural educators. The field of agricultural education needs to be given attention to increase national production. The issue of agricultural education that is seen to interfere with this process comes from the lack of qualified and competent instructors to fill agricultural education programs (Camp et al., 2002;Connors, 1998;Myers et al., 2005). This opinion is in line with the study of Raof et al., (2020) who stated that the biggest challenge to agricultural educators is the issue of competence. Therefore, there is a need to review the competencies that need to be possessed by agricultural educators, especially in vocational colleges.
The construction of a questionnaire instrument is part of the main thing in a quantitative study. Prior to the actual study, a pilot study was conducted to ensure that the instruments used have a high level of validity and reliability. Questionnaire instruments answered by the respondents in the pilot study were diagnosed through purification and removal. Through this approach item, function testing was done in-depth through several analyzes using Winsteps software to analyze quantitative data with Rasch Model Measurement (RMM) method. Therefore, this study was conducted to analyze the pilot data to measure the validity and reliability of the questionnaire used for the actual purpose.

Purpose of study
The purpose of this study was to explore the psychometric nature and study the validity and reliability of the newly produced agricultural instructor competency instruments.

Objectives
A pilot study is a small-scale study carried out before the actual study is done to observe the feasibility or reasonableness of a study (Chua, 2014). The pilot study also reviews instructions, test items in questionnaires and estimates response time (Bond & Fox, 2015;Lim et al., 2009).
The objective of conducting the current pilot study is to examine the functionality of items from the aspects of (i) reliability and separation of item-respondent; (ii) item polarity; (iii) item fit; and (iv) standardized residual correlation value in determining the dependent items.

Research Design
Survey techniques were used in data collection using the instrument of competency framework of agricultural instructors. The questionnaire was distributed to 30 agricultural instructors at the agricultural vocational college in Perak. In this study, the researcher self-developed the research based on Borich's (1980) needs assessment model to assess the capability and importance of each competency. This list of agricultural education field competencies was developed from previous research (Boone & Boone, 2009;Garton & Chung, 1996;Layfield & Dobbins, 2003;Mundt & Connors, 1999;Sorensen et al., 2014) and modified to meet the needs of agricultural teaching staff in Malaysian agricultural vocational colleges. The instrument of this study involved the use of a set of questionnaires using a quantitative approach.

Sampling
In the administration of the pilot study, the researcher conducted a pilot study on n = 30 people taken from the population of vocational stream teachers in the agricultural stream Vocational College in Malaysia. Respondents were selected through the strata random method, where the researcher made the strata of the population according to percentage and then randomly selected the respondents based on the desired percentage. This type of sample is very suitable for this study as the respondents were selected based on the program taught.

Analyzing of Data
This questionnaire has 116 questions focusing on the elements of competence and constraints faced by teachers in achieving competence. Polytomous (Likert) data were used in this questionnaire and analyzed using the Rasch Model with the assistance of Winsteps Version 3.72 software to obtain accurate instrument formation analysis information. The statistical analysis conducted for the pilot stage was aimed at selecting the appropriate and best items (Cohen & Swerdlik, 2002). Rasch Model Measurement can identify the level of difficulty of the questionnaire items used in a study. In addition, this model considers the ability and capability of the respondents answering the instrument. In this study, Rasch Model Measurement was used in analyzing the pilot study data to check the functionality of items for each study construct through four aspects explained in Table 1 below. Accepted range of 0.5-1.5 Determines the dependent item based on the standardized residual correlation value

Local dependence
The correlation value of the two items did not exceed 0.7

Findings
This section reports the findings of the pilot study analysis related to the validity and reliability of the instruments used in this study.

Item and person reliability
The level of reliability of the study can be determined using the interpretation of Cronbach's alpha values ranging between 0.00 to 1.0. If the range value approaches 1.0, it means that the level of reliability is at a good, high, and practical level. On the other hand, the range close to 0.00 reflects a low level of reliability (Yusof et al., 2012). Table 2 shows the rating scale instrument quality criteria (Fisher, 2007)  To determine the reliability of items in the instrument, statistical analysis with the Rasch measurement model approach was used for reliability values and item isolation. The alpha value (Table 2) indicated a good reliability value with the alpha drink value of 0.90 (Hopkins, 1998). Table 3 and Table 4 show the Cronbach's Alpha values of reliability and isolation index of respondents as well as items for the competency construct. Overall, the reliability values of the respondents and items were excellent and acceptable. Meanwhile, the isolation index of respondents and items obtained showed the strata of abilities identified in the sample group.   Table 3 and Table 4 show the findings of reliability and separation of respondent and item results. The analysis shows all the findings qualify in determining the reliability.

Polarity of items measuring the constructs
Point Measure correlation (PTMEA CORR) value examination detects the polarity of moving items towards the measured construct (Fox et al., 2009;Linacre, 2008). If the PTMEA CORR value indicates a positive value, it means that all the items are functioning in parallel to measure the construct formed. However, if there is a negative value, it indicates that the item needs to be re-analyzed with the correct or dropped method. This indication clarifies that the dropped item did not lead to a question or was difficult for the respondents to answer. Examination of the PTMEA CORR value in this instrument found no items with negative values. These results indicate all constructs are moving in parallel with other items and attempting to measure what should be measured.

Item Fit
Item fit used to measure developed constructs can determine item fit in measuring constructs as observations on study data. In the Rasch analysis report, the statistic used was known as (chi-square), which includes infit and outfit mean square (MNSQ) (Raof et al., 2021). The MNSQ index of infit and outfit is always referenced for item matching checks (Mariani, 2016). Azrilah et al. (2017) define MNSQ infit as sensitivity or matching that conforms to response patterns to targeted items and respondents. To determine the suitability of the constructed study items, the MNSQ outfit value should be in the range between 0.5-1.5 (Azrilah et al., 2017). If the value obtained exceeds 1.5, it means that the item is confusing. On the other hand, if the value is less than 0.5, it indicates that the item is too simple to be expected by the respondents. Based on Bond and Fox (2015), the value of the ZSTD outfit should be in the range of -2 to +2. However, if the MNSQ outfit's value is accepted, the ZSTD index can be ignored (Linacre, 2002). Therefore, if this condition is not met, then the item can be considered for elimination. Several related references and opinions can be used to determine the MNSQ value for the fit value (Table 5). Table 6 indicates the measurement implications of the results from the MNSQ value.  Distorting or weakening the measurement system probably due to only one or two observations. 1.5-20 Less successful for measuring construct but not debilitating. 0.5-1.5 Successful enough for measurement < 0.5 Less successful for measurement but not debilitating. Likely to result in confusing reliability and separation coefficients. Source: What do Infit and Outfit, MNSQ mean? Linacre JM...Rasch Measurement Transactions, 2002, 16:2 p. 878 The following discussion is to refer to item matching for the constructs involved. Table 7 shows the MNSQ values of outfit and infit and the Zstd values for the entire construct.  Table 7 shows 11 items detected had MNSQ outfit values out of the referenced range of 0.5-1.5. A value higher than 1.5 indicates that the item is not homogeneous with other items on a measurement scale. Through this diagnosis, these items were dropped; skill constructs (3), knowledge constructs (4), social role constructs (2), and trait constructs (2).

Standardized Residual Correlations
The measurement of standardized residual correlation values aims to determine overlapping and non-singular items. A high residual correlation of more than 0.7 for the two items indicates that the items are dependent and not singular since the items have similar characteristics or because the two combine several other dimensions that are shared. According to Linacre (2008), if the correlation value of two (2) items exceeds 0.7, it indicates a high correlation value.
Thus, only one item should be retained, whereas the others should be dropped. Item selection also refers to the MNSQ value where a value close to 1.00 will be maintained (Linacre, 2002). Unidimensional measurement constructs can be produced by independent or singular items (Wright & Masters, 2002). Table 8 shows the analysis report of the pilot study to detect the existence of dependent items on other items according to the construct. In Table 8, 14 items failed to meet the requirements of this analysis. Therefore, those 14 items will be dropped in this diagnosis.

Discussion
This study aims to confirm the construct of the competency instrument for agricultural vocational stream instructors in Malaysian agricultural vocational colleges. This item analysis used a Rasch measurement model to examine the functionality of the items from the aspect of reliability and isolation of items and respondents. This is to ensure whether or not the instrument used is consistently used on other respondents. The implications of this analysis can help researchers develop instruments that are truly capable of measuring the competence of agricultural educators. The use of RMM in analyzing this instrument has been seen to obtain a set of questionnaires that can measure the objectives in the actual study. Hence, a test to perform an inspection of item functionality from the aspects of (i) reliability and item-respondent isolation; (ii) item polarity; (iii) the fit of the item and (iv) the correlation value of the standardized residue in determining the dependent item is essential before conducting a study. This situation allows the researcher to determine whether or not the response to the item is consistent with the item and respondent isolation index.
The results of the item and person reliability in this study show an excellent reliability index and item and respondent separator. These results indicate that indicating that the measures are well separated in comparison to the errors. Stages of instrument testing were conducted to improve the scale, and the results are propitious. (Suhairom, 2016). Person reliability interpretation is equivalent with Alpha Cronbach (KR-20), which is 0.99. These results indicate that this study has a relevant sample to distinguish between high and low performers. However, some constructs have item separators, and acquisition respondents of 0.74, 0.71 and 0.76, respectively for item reliability during the third instrument testing signifies that the measurement instrument has been working well to perform further analysis (Md Yunos et al., 2017). Investigating item polarity, which is indicated by the PTMEA CORR value, is essential. Linacre (2008) and Bond and Fox (2015) suggested items with a negative value to be dropped from the lists to ensure the quality of the developed instrument. In this study, it was found that no item with a PTMEA CORR value less than 0.4. Thus the study findings for polarity items meet the proposed criteria (Fisher, 2007). Appropriate fit items indicate the quality of an instrument and can be used to represent the accuracy of the instrument (Suhairom, 2016). Generally, the diagnoses of Item Fit statistics follow the requirement of Infit/ Outfit MNSQ between 0.6 logits and 1.40 logit (for polytomous scale) as well as the value of Outfit ZSTD of -2 <ZSTD value <+2 (Bond & Fox, 2015).
This study found that all the constructs analyzed were in line with the recommended conditions. From the findings, the threat regarding construct irrelevant-variance was minimum based on the dimensionality test as well as the withinrange fit indices. The instrument in this study found that most items were moving in a parallel direction; however, some items have failed to measure constructs. This pilot study found that 22 items should be dropped and 94 items retained. The omission of items in this questionnaire is necessary to ensure that the instruments used are of high validity and reliability so that the study on the competence of vocational stream instructors in the field of agriculture of vocational colleges can be used as an instrument that can answer research questions accurately. Overall, the objective of this study is to produce a set of questionnaires that have consistency to be applied. The data obtained from this actual study can also be used to form a competency framework for agricultural teaching staff.

Conclusion
After the data was analyzed, an item review was performed for each item based on the index standards and the conditions that need to be followed to achieve the standard of validity and instrument reliability based on the Rasch Measurement Model. Removal and item refinement is done by referring to and taking into account the views and evaluations of experts. Based on this pilot study examining the validity and reliability of the instrument, the Competence of Agricultural Instructors has a quality that can be used. The implications of this analysis can help researchers in developing instruments according to the desired criteria. This pilot study is an initial step in assisting researchers in identifying the competencies of agricultural teaching staff in implementing teaching in vocational colleges. This is to ensure that agricultural teachers have the next element of competence to deal with the issue of incompetent TVET teachers.

Recommendations
The results of this study have produced an instrument that has high reliability to measure the competence of agricultural educators in vocational colleges. However, this study only focused on a few constructs in competence. Future studies are also expected to explore other elements and problems that exist. Therefore, it is necessary to conduct other studies, either with different variables or with the same research subject at different educational levels or with different samples and populations. It is hoped that future studies can produce instruments in different constructs and elements. Therefore, the technical and vocational education system in Malaysia can be improved.

Limitations
There are two limitations to this study. First, the number of study samples is limited as the overall population of the teaching staff is limited. Second, only one vocational college was involved in this pilot study, given the constraints caused by the COVID-19 pandemic.