It looks like AI is here to stay. Not only that, its impact on society will rival or exceed that of the world wide web. However, we are likely just scratching the surface and AI is still in its infancy. The expected growth is immense and there is an active arms race in both hardware and algorithms. Nvidia dominates hardware today, largely because we are still in the training era, where time-to-accuracy at any cost rules. In this era, algorithmic innovations offer the biggest benefits and require a robust programmable platform (i.e., Nvidia GPUs) upon which to innovate. However, there are signs the inference era is imminent, where the rules are different. The projected scales of AI inference deployments will demand highly efficient hardware. With AI touching all layers of the computing stack today, we can leverage algorithm-hardware co-design to uncover interesting opportunities and trends for efficiency via customization and acceleration. This talk presents an overview of a NSF-funded project we had proposed to develop a cloudless universal translator, which we thought eventually possible given the state of ML technology at the time. Throughout

the project, we designed three custom AI SoCs--targeting automatic speech recognition (FlexASR), natural language processing (EdgeBERT), and on-device learning/fine-tuning (CAMEL). These designs highlight ways to maximize energy efficiency given the probabilistic nature of machine learning algorithms. Finally, the talk closes with some thoughts on

chiplets and carbon.