Seeding Genetic Programming: From Memorization to Generalization

TL;DR

Traditionally, Genetic Programming (GP) starts from a "blank slate" of random code. This paper flips the script: start with a "perfect" (but bulky) program that has already memorized the training data and use evolution to "prune" it into a sleek, generalizable solution. Utilizing Pareto multi-objective optimization, the authors demonstrate that GP can effectively perform variable selection and logic simplification, turning complex "if-else" forests into elegant mathematical expressions.

Problem & Motivation: The Memorization Bottleneck

In standard Machine Learning, we distinguish between memorizing (learning facts) and learning (finding patterns). Most GP runs spend the majority of their early generations trying to simply "hit" the training cases—a task computers are already good at via deterministic algorithms.

The authors argue: Why waste evolution on memorization? If we can programmatically generate a solution that fits the data perfectly (even if it's massive and ugly), we can jump straight to the hard part: Generalization.

Methodology: The Pareto "Boil Down"

The core of this approach is a two-step process:

1. Deterministic Initialization: Use decision trees or nested logic to create a "Seed" that scores 100% on training data.
2. Parsimony Pressure via Pareto Selection: Instead of a single fitness score, the authors use a Pareto tournament. An individual stays in the population if it is either more accurate or smaller than its peers.

Architecture: Seed Structure

The seeds are often constructed as long chains of IF-THEN-ELSE statements. For every training case, a clause is added to return the correct class. While 100% accurate on the training set, these seeds are naturally terrible at predicting unseen data (Generalization).

Note: The GP acts as a refiner, using crossover and mutation to break down these rigid structures into flexible patterns.

Experiments: Pima Diabetes & Breast Cancer

The authors tested this on several UCI benchmarks. A standout result occurred in the Pima Indians Diabetes dataset.

• The Discovery: Starting with a seed based on only 20 training records was enough to guide the GP toward high-performance models.
• Data Selection: The GP naturally discarded irrelevant variables. In some runs, it evolved a program that only used a single input variable while maintaining competitive accuracy—a remarkable feat of automated feature selection.

Table: Comparison of Random vs. Seeded populations. Seeded runs (20, 288, 576 nodes) consistently achieved higher verification hits compared to random Pareto runs.

The "Bloat" Warning

The researchers found a catch: Bigger is not always better. When seeds were created from the entire large training set, the resulting programs were so massive that they slowed down the GP significantly (O(Program Size × Dataset Size)). These massive programs often suffered from "bloat" and were harder for evolution to simplify effectively.

Fig: The evolution of the Pareto front shows the population rapidly spreading as shorter, more general programs are discovered.

Critical Insight & Conclusion

This work highlights a unique strength of Genetic Programming: Symbolic Transparency. Because GP works with symbolic structures, we can "pre-load" it with knowledge from other algorithms (like C4.5 decision trees).

Takeaways:

• Seed Sparingly: You don't need a perfect seed of the whole dataset; a seed based on a small sample often provides the right "genetic compass."
• Pareto is Essential: Without a strong pressure to reduce size, seeded populations will simply overfit and bloat.
• Hybrid Potential: GP should be viewed as a powerful post-processor for other machine learning techniques, capable of refining rigid decision boundaries into smooth, symbolic functions.

Future Outlook: This methodology paves the way for "Knowledge-Informed GP," where domain expertise or existing heuristics are encoded into seeds, allowing evolution to fine-tune human logic rather than starting from absolute chaos.