This dissertation investigates the use of large language models (LLMs) as variation operators for evolving programs. Whereas a traditional variation operator modifies program syntax without regard to program behavior, an LLM can propose changes informed by the semantics of the code. This capability has enabled new forms of program evolution and produced promising results, yet the behavior of LLM-based evolutionary operators remains comparatively understudied. This dissertation examines three forces that shape the resulting evolutionary process: the representation of candidate programs, the method of selection that determines which candidates persist across generations, and the dynamics of the variation operator itself. Each of these factors influences the capacity for sustained, open-ended exploration.

To study representation, we evolved agent rules for multi-agent simulations implemented in NetLogo and introduced three benchmark environments for evaluating LLM-generated code in this setting. Three ways of representing the same rule were compared: executable code, code with comments, and pseudocode that a separate model call converts into executable code. Across all three environments, evolving commented code produced the highest-performing agents, while evolving pseudocode produced the lowest-performing agents, indicating that natural language aids the variation operator when it augments executable code rather than replacing it.

To study selection, we varied the balance between objective performance and behavioral novelty while evolving agent rules. Final fitness was largely unaffected by this balance, but greater emphasis on novelty consistently yielded a higher proportion of qualitatively superior code structures. These findings suggest that diversity-driven search can improve the quality of evolved code, revealing a hidden cost of objective-centric optimization.

To study the operator itself, we asked whether a language model that repeatedly mutates a program continues to explore new forms or instead returns to recurring structures. Mutation chains were analyzed in the absence of selection pressure within a domain-specific language, varying prompt design, model family, and stochastic replication. The chains converged on restricted regions of program space, with most variation confined to substitutions within recurring structural templates, while a classical genetic programming mutation operator showed no comparable convergence under matched conditions.

These three forces do not act independently. The representation determines the extent to which the capabilities of the variation operator can be exploited, selection determines which resulting programs persist and become available for further variation, and the dynamics of the operator shape the structures that repeated mutation continues to produce. LLM-driven program evolution can effectively evolve agent rules for multi-agent systems, but its outcomes are sensitive to design choices that are easily left implicit. The selection regime changes the quality of the program structures that survive, while the same semantic capabilities that allow an LLM to transform programs can also introduce a systematic bias toward structural homogeneity. Whether these systems can sustain open-ended exploration depends on the interaction among representation, selection, and variation.