Abstract
Large Language Models (LLMs) exhibit strong potential in mathematical reasoning, yet their effectiveness is often limited by a shortage of high-quality queries.This limitation necessitates scaling up computational responses through self-generated data, yet current methods struggle due to spurious correlated data caused by ineffective exploration across all reasoning stages.To address such challenge, we introduceMARGE: ImprovingMathReasoning withGuidedExploration, a novel method that enhances mathematical reasoning through hit-guided exploration.MARGE systematically explores intermediate reasoning states derived from self-generated solutions, enabling adequate exploration and improved credit assignment throughout the reasoning process.Notably, MARGE improves both single-shot accuracy and exploration diversity, mitigating a common trade-off in alignment methods.These results demonstrate MARGE's effectiveness in enhancing mathematical reasoning capabilities and unlocking the potential of scaling self-generated training data.