There is growing momentum--from industry, government, and academia--to use AI for automating cybersecurity tasks. Yet practitioners remain skeptical: while 87% of security leaders expect AI to enhance their roles, only 9% believe it will replace significant parts of them. This gap stems from two fundamental barriers: limited capability and lack of trust. In this talk, I present my research on addressing these barriers through explainable AI. I first introduce StateMask, a method that automatically identifies critical decision steps in AI agent trajectories, enabling security professionals to understand why an AI-generated patch succeeded or failed. A user study with 41 experienced developers shows that 89% find our explanations aligned with their reasoning. I then present GPO, which leverages these explanations to synthesize high-quality training data without expensive expert annotation, thereby improving model capability. GPO-trained open-source models achieve performance competitive with leading commercial models on vulnerability patching, and its extension, EntroPO, ranks 1st on SWE-Bench Lite among all open-weight models. I conclude by discussing future directions toward building AI systems that are robust to imperfect data, trusted by security professionals, and capable of tackling real-world cybersecurity challenges.