Faculty ProjectsTablets for Socratic Test Expansion
Faculty
Ilya Mikhelson, ECE
Amount Requested
$4,450
Summary
The modern grading system is fatally flawed, with the main reason being a lack of proper assessment tools. This is true now more than ever before. Students can easily get answers from AI for anything that is take-home. In-class written exams can only test shallow understanding, such as memorization and basic problem solving. Much better is the oral exam, but it is not scalable.
In response to this, I have created the Socratic Test. This is a conversational examination administered through a chat interface by AI. During the exam, students converse with the AI examiner, which asks them unique questions guided by their problem-solving. It is effectively an oral exam with trivial scaling. Students bring their laptops and take the exam synchronously in the classroom to ensure that the demonstration of knowledge is truly their own. I ran a pilot of the Socratic Test in three classes during Spring quarter, and it produced remarkable results. The students were not intimidated by it; they even preferred it to written exams! And faculty stated that the conversations were able to map out their students' understanding accurately.
There remain a few challenges with implementation. First, some students may not have laptops, although the chance of that is exceedingly low. However, some students may be experiencing laptop issues regardless. Therefore, I want to have a small number of loaner laptops on hand. Second, some classes require drawing diagrams or writing equations. For this, I give out USB graphical tablets. I currently have 70 on hand, but I want to purchase many more for a wider rollout.
My overall goal is to bring the Socratic Test to as many classes as possible. The benefit to students is difficult to overstate. This will necessitate deep understanding of course topics, which will naturally deter students' reflexive use of AI to offload thinking. They will learn to use AI as a thought partner, not a replacement, which will have numerous ripple effects throughout their education and careers.
Planned Activities/Investments
To achieve my goals, I have researched and compiled a list of items, as detailed in the Budget section. I have also been recruiting participants to use the Socratic Test during Fall quarter. In the coming year, I plan to study the benefits of the Socratic Test to properly validate it. Pending the study results, I hope to expand it even further.
Impact
This project will impact all students in the participating classes, and hopefully many more in coming years if everything is successful. The literature affirms that oral examinations elicit much deeper thought and measure deeper understanding of topics than written exams. However, they have some drawbacks in the form of anxiety and grader bias. With the Socratic Test, the anxiety is reduced as the student is talking to AI, not performing in front of a professor. And the grading bias is largely eliminated, as the AI grades consistently and doesn't succumb to favoritism or inter-grader variability.
Students will have to shift their study habits from surface-level learning and memorization to deep synthesis and understanding. This will hopefully have the ripple effect of improving student performance and reducing reliance on AI for all aspects of thinking.
For evaluation, I will administer surveys throughout each quarter. Furthermore, as part of the overarching study, I will have professors, who are unaffiliated with the particular course, interview a sample of students to measure the correlation of the professors' assessment of the students' deep knowledge to that measured by the Socratic Test.
Deliverables
- The ability to administer the Socratic Test in much larger classes, and in many more classes.
- Validation that the Socratic Test measures deep learning while maintaining reasonable levels of student anxiety.
Sustainability
After obtaining a fleet of devices, they can be reused for further classes. Every so often, devices will break or get lost, and they can be replaced by department or school funds, as individual costs are not high.
Budget Overview
- 230 graphical tablets: $6,900
- 10 Chromebooks: $2,000
Total Budget Amount: $8,900
Matching Funds
The Department of Electrical and Computer Engineering will match the requested funds by 100%, thus bringing the total to $8,900. The total covers all of the expenses listed in the Budget section.