Development of Performance Assessment Instruments for Measuring Drawing Skills in Vocational Students Competence in Mechanical Engineering Expertise

ABSTRACT


INTRODUCTION
The 21st century is referred to as the century of knowledge, the century of information technology, globalization, the Industrial Revolution 4.0, and so on.In this century, there have been rapid and unpredictable changes in all aspects of life, including education.This rapid change can provide opportunities if appropriately utilized, but it can also be disastrous if not anticipated systematically, structured, and measurable (Fakhruddin et al., 2022).This change triggers changes in the skills needed in the world of work.To predict the skills needed will be very difficult because it depends on the field and subwork that is the focus of these skills, so students are required to be more creative and able to adapt to face the conditions of the business world and the industrial world in the future (Dwivedi et al., 2021).
The Industrial Revolution 4.0 no longer requires a workforce only skilled in operating machines.However, it is certainly necessary to understand AI (Artificial Intelligence) better, which has been included in the latest machines (Javaid et al., 2022).In addition, (1) The production process no longer uses pure mechanisms; (2) Manual production machines have been abandoned and are not in production again; (3) All manufacturing technologies began to use numerical control; (4) Numerical control This was adopted from the drawings, closely related to this, especially the Department of Mechanical Engineering (manufacturing), which is the student's drawing skills because Drawing is the initial stage of the manufacturing process before production (Setyono et al., 2023).So, this drawing skill must be possessed by everyone in the business or industrial world who concentrates on production/manufacturing.
Drawing skills from students can be obtained from student learning at school, skills in the narrow sense, namely ease, speed, and accuracy in motor behavior, also called average skills (Miranda, 2021).In a broad sense, skills include aspects of average skills, intellectual skills, and social skills.Skills are purposeful patterns of activity that require manipulation and coordination of learned information.Amirullah and Budiyono say, "Skill or skill is an ability to translate knowledge into practice to achieve the desired goal."From some of the opinions above, skill is the ability to do something quickly and precisely (van Laar et al., 2020).To assess the skills of student learning outcomes, an assessment.The assessment or evaluation process is one of the tasks of the teacher, who will determine the direction of the following learning process (Goss, 2022).Evaluation can be expressed as a systematic process in determining the achievement of instructional objectives (Al-Alawi & Alexander, 2020).Every educational organization implements a program from the planning stage to the evaluation (Karmila & Suchyadi, 2020).
Another opinion states that evaluation is a systematic process to determine values based on data collected through measurement (Reed et al., 2021).Grades must be taken objectively; subjective elements are not considered and assessed (Linda & Yusup, 2020).In other words, evaluation includes both steps ahead, namely measuring and judging.Assessment is a systematic and continuous process or activity to collect information about students' learning processes and outcomes to make decisions based on specific criteria and considerations (Nasution, 2022).The assessment aims to determine how far the teacher has succeeded in carrying out the learning process, which is used for feedback for the teacher in planning the following learning process (de Vries et al., 2022).Job satisfaction is one factor that supports teachers in performing at their best (Heyder, 2019).If teachers are satisfied, they will work passionately and responsibly (Li et al., 2023).Often, in the teaching and learning process, aspects of evaluating learning outcomes are ignored (Mafarja et al., 2023).Because teachers focus too much on what will be taught to their students, the teaching and learning process runs nicely and neatly, but the assessment tools used no longer see the targets to be assessed.Improving the quality of education must be distinct from applying assessments that can precisely measure the final results of a learning process, meaning that quality measuring instruments are needed to assess the final results of learning (Abdulrahman et al., 2020).The teacher's ability to compile test instruments certainly affects student learning outcomes.With assessment test instruments that meet the criteria, student learning outcomes will be detected well and can be used as evaluation material for the next learning program.A test is said to be good if it has credibility, among others: (1) validity, (2) reliability, and (3) practical value.
Student assessment activities are an essential and integral component of school teaching and learning activities.An assessment of learning outcomes is needed to obtain information about the achievement of the student's learning process results by the objectives set.An essential function for educators in evaluating student learning is providing feedback to students in considering the effectiveness and efficiency of the learning process (Firmansyah et al., 2021;Maros et al., 2023;Munna & Kalam, 2021).
Miller defines student learning assessment as various procedures for obtaining student learning information and determining decisions about student performance or learning outcomes.Assessment of student learning outcomes is a teacher activity related to making decisions about achieving competencies or student learning outcomes during the learning process.The assessment of learning outcomes must meet the principles as expressed by Anderson (2003), namely (1) meaningfulness, (2) transparency or explicitness, (3) fairness, and student assessment data collected by teachers through assessment procedures and tools that are by the competencies that must be achieved by students or indicators that have been determined to be assessed.
Permendiknas Number 66 of 2013 also mentions the assessment of student learning outcomes, including cognitive competencies (knowledge), affective, and psychomotor (skills), which are carried out in a balanced manner.The ideal assessment in learning is an assessment that covers all three areas.The assessment commonly used to measure student skills is called performance assessment (Pramana & Putra, 2019).Performance assessment is a test of deeds.In this assessment, students are expected to practice and carry out several activities, and then an assessment of students is carried out based on assessment guidelines (Sudirman et al., 2023).Performance assessment is an assessment carried out by observing student activities in doing something (Sani, 2022).According to Hidayat (2015), performance assessment is carried out by observing student activities in doing something.This assessment is suitable for assessing the achievement of competencies that require students to perform specific tasks, such as practice in the laboratory or practice, presentations, discussions, role-playing, playing musical instruments, singing, reading poetry/declamation, etc.This assessment is more authentic than the written test because the assessed attitudes better reflect the student's ability.Performance assessment needs to consider the following: 1) Performance measures expected by students to demonstrate the performance of a competency; 2) Completeness and accuracy of aspects to be assessed in the performance; 3) Specific abilities needed to complete the task; 4) Strive for the abilities to be assessed not too much so that all can be observed; 5) The abilities to be assessed are sorted in the order in which they will observed.Performance appraisal is an assessment that focuses on aspects of skills related to the psychomotor realm that can be demonstrated by students (Munandar & Junita, 2022).
Performance assessment to measure students' drawing skills is an assessment that asks students to perform several performances in a practicum related to manufacturing design that has been done.This performance assessment reviews several aspects of the assessment, namely aspects of the process, results, and work of student drawing.An instrument performance assessment of the three cognitive, affective, and psychomotor aspects must be conducted.Even more so When measuring students' drawing skills.Drawing Skills have very complex parameters, so developing instruments suitable for evaluating student skills is necessary.Therefore, it is necessary to conduct research titled "Development of Performance Assessment Instruments to Measure Drawing Skills in Vocational Students Mechanical Engineering Expertise Competencies," which aims to make it easier for teachers to measure students' design skills.

RESEARCH METHOD
The research and development method is a research method used to produce specific products and test the effectiveness of these products.The research design used in this study is a research design for developing 4-D models.This includes four stages: defining, design, development, and dissemination (Thiagarajan, 1974).The flow of research activities is shown in Figure 1.The research subjects in this instrument development research are 31 students of Mechanical Engineering at State Vocational School 2 Surabaya class XI.The defining stage helps determine and define needs in the learning process and collects various information related to the product to be developed.This stage is divided into several steps, namely: 1) front-end analysis, 2) learner analysis, 3) task analysis, and 4) concept analysis (Febrianti & Rambe, 2022;Rahmatsyah & Dwiningsih, 2021;Ramdani et al., 2021).Design Stage After getting problems from the defining stage, the design stage is carried out.This design stage aims to design instruments used to measure drawing skills.The design stage here is to design an instrument to measure drawing skills.Instruments designed to measure cognitive, psychomotor, and affective.
The development phase aims to produce instruments to measure drawing skills that have been revised based on expert input and trials to students.There are two steps in this stage, which are as follows: 1) Expert appraisal: This expert validation serves to validate the content of the instrument to measure drawing skills before testing, and the validation results are used to revise the initial product.Material experts and linguists will then assess the instrument to determine whether the instrument to measure drawing skills is feasible.The results of this validation are used as refinement material for the perfection of the instrument, measuring the developed drawing skills.After draft I is validated and revised, draft II is produced.Draft II will then be tested on students in a limited field trial phase; 2) Product Testing (development testing).After expert validation, limited field trials were carried out on vocational school students in Surabaya to find out the results of the application of instruments measuring drawing skills in vocational school Competence of mechanical engineering expertise.The results obtained from this stage are in the form of instruments to measure revised drawing skills.
Validation of instrument products was carried out by five experts with expertise in Vocational Education and tested on vocational students.Instrument validation is carried out in three stages: validation of instrument content with the content validity ratio (CVR) method, item-total correlation and reliability, and fit-item analysis.An Define Design Develop Disseminate instrument is said to be suitable for use if it meets the following requirements: (1) CVR value ≥ 0.30; (2) item-total correlation value ≥ 0.20 and reliability value ≥ 0.60; and (3) infit mean square (IMS) and outfit mean square (OMS) values of 0.50-1.50.The dissemination stage is next after limited trials, and the instrument has been revised.However, this research is only limited to the product trial stage.

Result
The defining stage is concept analysis, which aims to determine the content of the instrument to measure drawing skills, which are part of the technical drawing competencies that must be possessed by vocational students, especially the Mechanical Engineering expertise program, and only focuses on drawing skills.The design stage is the development of instruments, including test preparation (criterion-test construction).The preparation of instrument tests is based on learning objectives that become a benchmark for students' abilities in the form of products, processes, and psychomotor during and after technical drawing learning activities.This development phase aims to produce an instrument performance assessment to measure revised drawing skills based on expert input and trials with students.There are two steps in this stage, which are as follows: 1) Content validation of the instrument is carried out by five experts with areas of expertise shown in Table 3. Validation instruments are arranged based on the Likert scale 1-4.Scale 1 means very bad, scale 2 means not good, scale 3 is good, and scale 4 is perfect.The validator can provide suggestions for necessary improvements to each item.

5
The developed instrument is tested for feasibility with validity and product trials to determine the drawing skills of Vocational School students of the mechanical engineering expertise program.Products in the form of observation instruments tested on Vocational School students in Surabaya, for the results of instrument trials, can be seen in Table 4.

Discussion
Based on content validity tests, item-total correlation and reliability, and item fit analysis, they will be compared to make it easier to conclude which items will be retained and revised/discarded.Based on the results presented in Table 6, several items must meet the CVR, total correlation, OMS, and IMS criteria, so these items must be revised or discarded.In developing the drawing skills instrument, there were two items, namely item number 15 and number 19.Meanwhile, no other items did not meet the CVR, total correlation, OMS, and IMS criteria.Thus, all items can be maintained and used as research instruments.An instrument is said to be suitable for use if it meets the following requirements: (1) CVR value ≥ 0.30; (2) item-total correlation value ≥ 0.20 and reliability value ≥ 0.60; and (3) IMS and OMS values of 0.50-1.50.
CVR is a way to evaluate the level of consistency between the assessment of experts and the items presented in the proposed Instrument (Handoyo et al., 2023).CVR is the ratio between the standard deviation and the average value of all expert assessments for each Instrument item.CVR results can be interpreted as follows: 1) Consistency of Assessment: A high CVR value indicates a high degree of agreement among experts in assessing the items in the instrument.This indicates that the information presented in the CV is considered relevant and consistent by the assessors; 2) Item Quality: CVR can also indicate overall item quality.The higher the CVR value, the greater the confidence that the item presents relevant and reliable information; 3) Reliability of Expert Assessment: CVR testing also helps evaluate the reliability of the experts' judgment in the CV appraisal process.A low CVR value can indicate a significant difference in assessment among experts, which may indicate an improvement in the assessment process or may need increased consistency between raters; 4) Item Repair: CVR test results can also guide instrument makers to improve the developed instrument (Almanasreh et al., 2019).Those items with low CVR values can be identified as areas needing improvement.In the CVR testing carried out in this study, all items obtained a value of 0.30, so all instrument items developed are unnecessarily revised.
Interpreting this total correlation value can provide a deeper understanding of the items being tested: 1) Relationships Between Variables: A high total correlation value indicates a strong relationship between the items in the test.This may indicate that there is a significant dependence between the various aspects observed; 2) Data Complexity: The higher the total correlation value, the more complex the relationship between variables in the test result data; 3) Model Fit: In some cases, low Total Correlation may indicate that the model used may not be complex enough to capture the relationships present in the test data; 4) Item selection: Items that have a low contribution to total correlation can be considered as candidates for deletion or revision in further analysis (Dash & Paul, 2021).The total correlation test results obtained two items with a value of less than 0.20, namely items 15 and 19, which need attention.
OMS and IMS are two ways to evaluate item responses in the tests.The interpretation follows: 1) Item Response Consistency: OMS/IMS values close to 1 indicate item responses consistent with the test's overall response pattern (Lim & Huh, 2019).This shows that the item has a level of difficulty that matches the respondent's ability; 2) Non-Compliant Items: A high OMS/STI may indicate an item that is too difficult or too easy for the respondent's ability level, while a low OMS may indicate a pattern of unexpected responses or ambiguous items; 3) Item Selection and Update: Items with low OMS/IMS may be revised or deleted, while items with high OMS/IMS may need to be reformulated to improve consistency with the RSCH model; 4) Test Validity: The validity of the test may be questioned if there are many items with inappropriate OMS/IMS values.Therefore, OMS/IMS analysis is essential in validating the instruments created (Rahim & Haryanto, 2021).From the results of trials on the developed instrument, eight items were obtained that did not meet the criteria, namely items 1, 10, 11, 12, 15, 17, 19, 27.Based on the CVR, total correlation, OMS, and IMS criteria, some items need to be met, so they must be revised or discarded.In the developed instrument, there are two items, namely item no. 15 and 19.While other items are declared to meet the criteria of CVR, total correlation, OMS, and IMS.Thus, 28 items can maintained and used as research instruments.

CONCLUSION Fundamental Findings:
The developed performance instrument can measure students' drawing and machining engineering skills competency.Two types of instruments were developed regarding Performance Assessment for Measuring Drawing Skills, namely assessment sheets and checklist sheets.Implications: Teachers can use this instrument to measure vocational school students' drawing skills more accurately because it refers to technical drawing learning outcomes, especially machining engineering skills programs.Limitations: Drawing Competence is very broad, so the development of this instrument only focuses on basic drawings (technical drawings); it cannot be used to measure other drawing competencies, such as CAD or CAM.Further Research: The results of the development of this instrument will be used to conduct further research regarding essential intelligence that influences the drawing skills of vocational school students in machining engineering expertise programs.

Table 2 .
Developed variables and indicators.
Type A (thin letter) with a thickness of 1/14 h with upright and oblique shapes.The angle of inclination for italics is 75 0 concerning the horizontal line.13.Type B (regular letter) has a thickness of 1/10 h with upright 12-13

Table 3 .
Areas of expertise and origin of validator institutions.

Table 4 .
Results of CVR, item test correlation, OMS, and IMS analysis.

Table 5 .
Recapitulation of validity and reliability.

Table 6 .
Recapitulation of items that do not meet the criteria.