Faculty Projects
Embedded Foundation and Language Model Labs for Hands-On Edge AI Education

Stephen Xia headshot

Faculty

Stephen Xia, ECE

Amount Requested

$22,000

Summary

This project will create a new set of weekly hands-on laboratory modules for Northwestern's CE 395: Embedded Artificial Intelligence course [1]. This upper-level undergraduate course introduces students to emerging techniques for deploying modern AI, such as large language models and foundation models, onto pervasive compute-constrained devices that are everywhere throughout our physical world, like our smartphones, smartwatches, and smart glasses. Topics cover a wide range of interdisciplinary topics including energy efficiency, model compression, on-device training and adaptation, dynamic inference, memory and caching, and heterogeneous sensor processing. This course covers a relatively new area of research that has only begun to see heavy interest beginning in the late 2010s and early 2020s. As such, there are relatively few courses covering the topic and even fewer that provide students with hands-on experience.

Murphy funding will support the purchase of hardware components (NVIDIA Jetson-class devices, sensors, and peripherals) that can support multiple years and iterations of the course, as well as a senior undergraduate or graduate student to help develop lab materials. The project will produce a reusable lab sequence focused on deploying and creating pipelines for modern AI, such as language and foundation models, onto compute-constrained devices. Unlike other courses and labs offered at other institutions and online courseware [2-8], which focus primarily on TinyML and microcontroller-class devices that are typically constrained to deploying machine learning architectures from prior generations with less compute requirements (e.g., convolutional neural networks), this course and the developed labs will explicitly focus on deploying state-of-art and modern architectures (e.g., language and foundation models, transformer architectures) that are orders of magnitude more computationally expensive.

The expected outcome is a stronger undergraduate learning experience that pushes students to think about how modern AI can be efficiently and sustainably deployed into computer systems with compute, energy, and latency constraints. Students will leave the course and be better prepared for the increasingly large language model (LLM) and AI-centric workforce with practical skills in embedded generative AI, foundation-model deployment, and hardware-aware AI system design.

[1] Northwestern University McCormick School of Engineering. “COMP_ENG 395, 495: Embedded Artificial Intelligence.” Northwestern University. https://www.mccormick.northwestern.edu/electrical-computer/academics/courses/descriptions/395-495-embedded-ai.html.

[2] University of Pennsylvania. “ESE 3600: Tiny Machine Learning.” TinyML @ Penn. https://tinyml.seas.upenn.edu/.

[3] University of Washington. “EE P 564: Tiny Machine Learning for Ultra Low-Power Edge Computing.” University of Washington Course Catalog. https://www.washington.edu/students/crscat/eep.html.

[4] Harvard Online. “Professional Certificate in Tiny Machine Learning (TinyML).” Harvard University. https://harvardonline.harvard.edu/program/professional-certificate-in-tiny-machine-learning-tinyml.

[5] HarvardX / edX. “Deploying TinyML.” edX. https://www.edx.org/learn/tinyml/harvard-university-deploying-tinyml.

[6] Edge Impulse. “Courseware: Embedded Machine Learning.” GitHub. https://github.com/edgeimpulse/courseware-embedded-machine-learning.

[7] MIT HAN Lab. “MIT 6.5940: TinyML and Efficient Deep Learning Computing.” MIT HAN Lab. https://hanlab.mit.edu/courses/2024-fall-65940.

[8] Arm Education / edX. “Machine Learning at the Edge on Arm: A Practical Introduction.” edX. https://www.edx.org/learn/machine-learning/arm-education-machine-learning-at-the-edge-on-arm-a-practical-introduction.

Planned Activities/Investments

The project will develop five guided hands-on labs for the Embedded Artificial Intelligence course. The labs will center around deployment considerations for large language models and foundation models using the NVIDIA Jetson Orin Nano compute platform [1], which has similar hardware capabilities and specifications as modern smartphones. The planned activities for the project are as follows:

  • Fall 2026: Stephen, along with the hired senior student, will develop a sequence of 5 weekly laboratories.
  • Winter 2027: The developed labs will be tested by Stephen's Ph.D. students, who conduct research in the area, and iteratively refined. The lab materials, such as documentation, starter code, and instructions, will be finalized.
  • Spring 2027: The labs will be deployed for the first time with students in Stephen's Embedded AI course.
  • Summer 2027: Feedback from the Spring 2027 deployment will be used to further refine the labs for future years.

The sequence of labs we create will roughly be as follows:

  • Lab 1: Jetson Setup and Local Foundation Model Deployment. Students will configure the Jetson Orin Nano, install the required software stack, deploy a small language model and a small foundation model, and collect baseline latency, memory, throughput, and power-related measurements.
  • Lab 2: Fine-tuning and On-Device Adaptation. Students will adapt a small language or foundation model using techniques such as LoRA, adapters, prompt tuning, or last-layer adaptation. They will compare the base model and adapted model in terms of task performance and deployment cost.
  • Lab 3: Model Compression for Embedded LLMs. Students will apply quantization and pruning to a small LLM or foundation model and evaluate how compression affects accuracy, memory footprint, latency, and throughput on the Jetson platform.
  • Lab 4: Distillation and Transfer Learning for Small Edge Models. Students will use a larger teacher model to generate labels, explanations, or embeddings, then train or adapt a smaller student model for embedded deployment. This lab will help students understand when distillation preserves useful behavior and when it fails.
  • Lab 5: Dynamic Inference for Embedded Generative AI. Students will implement or evaluate early exit, conditional computation, or speculative decoding using small draft and target models. Students will study when adaptive inference improves latency and when its overhead outweighs the benefit.
  • After the guided labs (first 5-6 weeks), students will spend the last half of the course (3-4 weeks) deep-diving into any specific topic on embedded artificial intelligence through an open-ended final project and present their findings at the end.

[1] SparkFun Electronics. “NVIDIA Jetson Orin™ Nano Super Developer Kit | Gen AI Computer.” SparkFun Electronics. https://www.sparkfun.com/nvidia-jetson-orin-nano-developer-kit.html.

Impact

This project will directly impact students enrolled in Embedded Artificial Intelligence (typically senior undergraduate and graduate students in electrical and computer engineering and computer science) by giving them practical experience with the systems challenges of modern AI deployment. Students will learn how these models behave under real embedded constraints such as limited memory, limited power, constrained compute, and real-time interaction requirements, which is necessary and becoming more prevalent for computer systems that interact with our physical world.

The project will also create durable course infrastructure that can be reused and updated in future offerings. Beyond AI and embedded computer systems, the labs will help students build skills that are relevant in a wide range of important fields impacted by physical AI deployments, such as robotics, mobile computing, wearables, healthcare sensing, and smart environments. This will help prepare Northwestern's student population for an increasingly AI-centric workforce across almost all areas of computer science and electrical engineering.

The impact and effectiveness of this project will be evaluated through objective and subjective measures. On the objective side, we will measure lab completion rates and student sentiment captured by student surveys and pre/post lab and course assessments. On the subjective side, we will seek actionable feedback from students throughout and after the Spring 2027 course offering, as well as evaluate the quality of the student final projects that the labs are used to build foundational experience and knowledge for.

Deliverables

The project will produce:

  • Five weekly reusable and guided foundational labs for embedded AI.
  • Reusable hardware and sensing kits based on the Jetson Orin Nano that students will use to complete the labs.
  • Starter code, reference implementations, instructions, and documentation for executing and deploying the labs.
  • Instructor and TA-facing notes, troubleshooting guidelines, and expected output examples (e.g., the "correct" answer) for evaluating lab completion.

Sustainability

The project is designed to be sustainable beyond the initial funding period. The Jetson hardware, sensors, and peripherals will be reusable across future offerings of the course. Lab materials will be maintained in a version-controlled repository so that models, libraries, and assignments can be updated as the field evolves. After the initial development year, Stephen will maintain the material through regular course preparation and teaching assistant support when available.

Moreover, Stephen's NSF CAREER proposal (pending) proposes an integrated education plan that includes additional funding (15,000 USD) to develop and deploy hardware and software toolkits into courses and for outreach activities focusing directly on deploying multimodal AI pipelines onto a wide range of edge devices. The funds from this project will be used to maintain the hardware component of the lab (e.g., support more platform types beyond Jetson Orin Nano or replacing hardware) and expand lab content in future years as the field evolves.

Budget Overview

  • Jetson Orin Nano Hardware Kits [1]: $15,000 — This supports 60 kits at $250 each, allowing each student to work through the labs independently in a 60-student course (typical enrollment) or up to 120 students if they work in pairs.
  • Sensors and Peripherals: $3,000 — This supports sensors and other peripherals (cables, power, etc.) and replacement parts to make the kits reliable for repeated use (around $50 per kit).
  • Student lab developer: $4,000 — This supports 250 hours of student work (part-time for 1 quarter or three months) at $20/hour to develop, test, document, and package the lab modules. The hourly rate is based on Northwestern's student wage ranges. Because developing labs for a new and emerging field requires "advanced duties, experience, and complexity", the requested salary is based on Group C students [2].

[1] SparkFun Electronics. “NVIDIA Jetson Orin™ Nano Super Developer Kit | Gen AI Computer.” SparkFun Electronics. https://www.sparkfun.com/nvidia-jetson-orin-nano-developer-kit.html.

[2] Northwestern University. “Wages: Student Employment.” Northwestern University. https://www.northwestern.edu/student-employment/resources-and-policies/wages.html.

Total Budget Amount: $22,000

Matching Funds

Stephen's NSF CAREER proposal (pending) has additional funding (15,000 USD) for additional purchasing, development, and deployment of educational hardware toolkits and labs for edge AI and multimodal processing. The funds from the CAREER proposal will be used for maintenance of the artifacts and labs developed for this project, as well as expanding the topics for future years as the field develops.