Introduction

Instrumentation and data sources are foundational components of scholarship of teaching and learning (SoTL) research. These elements provide the means through which researchers can gather empirical evidence to support their inquiries into educational practices and outcomes. Instruments refer to the tools or methods used to collect primary data directly from participants, such as surveys, tests, and observational protocols. Data sources, on the other hand, often refer to secondary data, which include existing records or datasets that have been previously collected for other purposes, such as grades on assignments, online discussion threads, and end-of-course surveys.

For SoTL research, the selection of feasible instruments and data sources is crucial. The data must be accessible within the teaching environment and compliant with institutional review board (IRB) requirements. Researchers must consider the practicality of data collection methods, the availability of existing data, and the ethical implications of using such data. This section will explore the different types of instruments and data sources available to SoTL researchers, offering guidance on how to choose appropriate tools that align with the research objectives and comply with ethical standards.

Background

In SoTL research, instruments and data sources are utilized to measure various educational variables and outcomes. Instruments can range from traditional tools like surveys and tests to more innovative methods such as digital analytics and virtual observations. Secondary data sources, including archival records and institutional databases, provide a rich repository of information that can be used to complement primary data collection.

Instrumentation in SoTL research often involves the use of standardized surveys and tests to assess student learning outcomes, engagement, and satisfaction. For example, student engagement can be measured using the National Survey of Student Engagement (NSSE), which provides a comprehensive assessment of student participation in educational practices (Kuh, 2009). Similarly, course evaluations and feedback surveys are widely used to gather students' perceptions of teaching effectiveness and course quality (Spooren et al., 2013).

Instrument Validity and Reliability

Validity and reliability are critical considerations when selecting instruments for SoTL research. Validity refers to the extent to which an instrument measures what it is intended to measure. There are several types of validity, including content validity, construct validity, criterion validity, and face validity. Ensuring that an instrument has strong validity means that the data collected will accurately reflect the construct being studied. For example, a survey intended to measure student engagement should include items that comprehensively cover various aspects of engagement, such as participation in class discussions, completion of assignments, and interaction with peers and instructors.

Reliability, on the other hand, refers to the consistency and stability of the measurements obtained using an instrument. A reliable instrument produces consistent results when administered under similar conditions. For instance, if a student engagement survey is administered to the same group of students at different times, it should yield similar results if the students' engagement levels have not changed.

In SoTL research, it is essential that the instruments used are both valid and reliable. This ensures that the data collected are trustworthy and can be used to make meaningful inferences about the research problem. Additionally, there must be a strong alignment between the construct being tested and the variable's measurement. For instance, if the research examines the impact of active learning strategies on student engagement, the instruments used should accurately capture the aspects of engagement that are influenced by active learning practices. Misalignment can lead to erroneous conclusions and reduce the study's overall validity.

Secondary data sources, such as grades and attendance records, offer valuable insights into student performance and behaviors. These data are particularly useful for retrospective, where researchers can examine behaviors, and longitudinal studies, where researchers can track changes and trends over time. For instance, archived discussion threads from online courses can be analyzed to understand patterns of student interaction and engagement (Dennen, 2008). However, using archival data comes with limitations, such as the potential for incomplete or outdated information and the lack of control over how the data were originally collected.

Limitations in using instruments in the classroom include potential disruptions to the learning environment and the need for time-consuming administration and analysis. Additionally, students may experience survey fatigue if asked to complete multiple assessments, potentially affecting the reliability of their responses. In online classrooms, technical issues and variations in student access to technology can pose challenges to data collection.

Archival data also have their limitations. In face-to-face classrooms, records may be inconsistently maintained or vary in detail, while in online settings, data may be subject to privacy concerns and restrictions on access. Ensuring the reliability and validity of secondary data is crucial, as is the need to obtain appropriate permissions to use such data.

IRB requirements play a critical role in the ethical collection and retrieval of data. Researchers must ensure that their data collection methods comply with institutional and federal regulations designed to protect the rights and privacy of participants. This includes obtaining informed consent from participants, ensuring data confidentiality, and minimizing any potential risks associated with the research. The IRB review process ensures that these ethical standards are upheld, providing oversight and guidance to researchers throughout their studies (Office for Human Research Protections, 2016).