Do large language models have metacognitive knowledge for mathematical problem solving?

Metacognition here means naming the skill a task needs

The paper borrows a term from human pedagogy. Metacognitive knowledge is a learner's accumulated knowledge about their own thinking and reasoning processes. The authors ask whether large language models hold something analogous, and they test it through one concrete behavior: given a math question, can a model say which skill or procedure the question calls for?

That framing matters for reading every result that follows. The paper does not claim to find a metacognitive module inside the network, and it does not read internal activations. It treats a model's ability to produce a sensible, reusable skill name for a question as evidence of metacognitive knowledge, then checks whether that naming is useful. The authors describe the skill names as a representation of the model's metacognitive knowledge.

From thousands of skill names to a usable repository

The procedure has three stages, all driven by prompts to a strong model. First, GPT-4-0613 labels each training question with a single skill name and a reason. This first pass is extremely fine-grained: about 500 unique skill names for the roughly 7,000 GSM8K training instances, and about 5,000 skill names for the 7,500 MATH training examples. The authors read that difference as a sign of MATH's greater difficulty and diversity.

Such specificity is impractical, because an identically worded question is unlikely to recur. So the model is then asked to group similar skills into broader categories and name each group. This semantic clustering reduces the set to 22 skills for GSM8K and 117 for MATH. The model then relabels the training questions with the coarser names.

The result is a skill exemplar repository: a list of skill names, each paired with training questions and answers. At test time a solver is shown the skill list, picks the skill it believes fits the question, and receives randomly selected solved examples carrying that label as in-context prompts. The authors report that the coarse labels look interpretable to humans.

Skill-matched examples beat topic-matched and random ones

The main test is whether label-based example selection actually helps a model solve problems. On the MATH dataset with GPT-4-0613, the skill-based Chain-of-Thought setup reaches 53.88 overall accuracy, against 42.2 for plain Chain-of-Thought, 50.30 for Complex CoT, and 50.31 for a topic-based approach that selects examples by dataset topic. The paper describes the gain over standard CoT as 11.6%.

The comparison with Complex CoT is the informative one. Complex CoT selects harder examples on the assumption that difficulty proxies for usefulness; skill-based selection wins by about 3.5 points, which the authors take to mean that relevance to the specific problem matters more than example complexity.

The gain is not uniform across topics. In the same MATH table, skill-based prompting scores 74.28 on Pre-Algebra, 73.12 on Algebra, 58.01 on Probability, 51.10 on Number Theory, 41.75 on Geometry, 33.70 on Pre-Calculus, and 27.02 on Intermediate Algebra. Geometry and Intermediate Algebra remain the weakest areas even after the improvement.

The labels also help code-based solving and weaker models

Program-aided prompting asks a model to write Python that computes the answer, which offloads arithmetic and logic to a compiler. The authors modify the PAL prompt to use three skill-matched text examples plus one fixed code example, four in-context examples in total. On a 500-example subset of the MATH test set with GPT-4-0613, this reaches 59.32 overall against 52.0 for four-shot PAL, a 7.52% improvement, and a seven-example variant reaches 62.00.

Transfer is tested separately. Using the GPT-4-created MATH skill labels and repository, Mixtral 8x7B improves from 26.14 with topic-based examples to 28.44 with skill-based examples, and from 30.75 to 32.44 when each is combined with self-consistency over four sampled chains. The authors report that skill-based prompting without self-consistency matches CoT with self-consistency, and that adding self-consistency gives a 4.0% gain over CoT with self-consistency.

Cross-dataset transfer is weaker but still positive. Applying the GSM8K-derived repository to six other math word-problem datasets, the skill-based approach beats base CoT on all six and beats a CoT plus PAL hybrid on four of six. Here the authors use pre-clustering skill labels, because those datasets contain finer-grained problems for which the coarser clusters were less effective.

Label quality depends on which model does the labeling

Not every model can serve as the annotator. The authors compared GPT-4, GPT-3.5, and Mixtral-8x7B on skill labeling and clustering, and tried Llama-2 70B, which failed to produce a sensible skill name for any example and was discarded. Mixtral could label questions but collapsed all skills into a single cluster, so it could not support clustering.

GPT-4 and GPT-3.5 both completed labeling and clustering, so the authors built separate repositories from each and compared them on MATH. The GPT-4 repository gives 53.8 overall against 51.9 for the GPT-3.5 repository, with the largest gaps on Algebra (73.12 versus 69.41) and Number Theory (51.10 versus 46.29). The paper reads this as GPT-4 producing higher-quality skill annotations, and its worked examples show GPT-4 labels that are more descriptive and sometimes more accurate than GPT-3.5's.

Skill-matched examples do not fix conceptual errors

The paper is explicit that relevant examples are not a cure. It presents cases where the model still answers incorrectly despite receiving exemplars with the matching skill, and classifies these as skill errors, secondary skill errors, and calculation errors. In one example, a convex hexagon diagonal-counting problem is labeled with counting and number theory, and the model's skill-based solution counts four diagonals per vertex instead of three, producing 12 rather than 9.

The authors' expectation is that skill-based prompting should mainly reduce skill errors, and they report that the skill success rate metric moves in line with that hypothesis. Per-skill accuracy on GSM8K favors the skill-based approach for 11 of 18 skills, so the aggregate gain hides skills where it does not help.

A further boundary is the evidence type. All of this is behavioral: prompt-guided labeling, clustering, and accuracy on benchmark test sets. The paper's claim about metacognitive knowledge is an interpretation of that behavior, and the method is presented as domain-agnostic even though the experiments cover math only. The results also come from a preprint under review, so the reported numbers have not been through peer review in this evidence.

Skill relevance beats topic or difficulty as a selection rule

The paper's most decision-relevant finding is a comparison among ways to choose in-context examples. Selecting by fine-grained skill label outperformed selecting by dataset topic and outperformed selecting by example complexity on MATH, and it also improved code-based solving and transferred from GPT-4's labels to Mixtral and from GSM8K to other word-problem datasets.

The practical priority that follows is to invest in the quality of the annotator and the granularity of the label set, not merely in adding more examples. GPT-4 labels beat GPT-3.5 labels on the same MATH test set, and the finer pre-clustering labels were the better choice for the smaller, more granular transfer datasets. Where the labels are coarse or the annotator cannot cluster sensibly, the advantage narrows.

The ceiling is also visible in the same evidence. Geometry and Intermediate Algebra stay below 42 accuracy on MATH even with skill-matched examples, and the paper documents failures where the model applies the right skill incorrectly. Skill labels improve which examples a solver sees; they do not guarantee the reasoning that follows.

Sources used in this answer