Enhancing Machine Learning Through Algorithmic Stability

 

Machine learning often applies optimisation algorithms to train models that perform well on training datasets. However, for complex models, this strong performance may not carry over to new or untested datasets, leading to a phenomenon known as overfitting. To address this, our researcher has developed a framework to study how well models generalise—that is, how well they perform on new, unseen data— by using an important concept in statistics called algorithmic stability.

Through this framework, we have uncovered a new connection between optimisation and generalisation: effective optimisation can significantly improve how well a model performs on new data. 

 

This connection offers a new perspective to understand the remarkable generalisation behaviour of deep learning models, as optimisation algorithms in practice often achieve near perfect performance on training data. Our findings also offer a clear method to determine when to stop optimisation to achieve the best predictive performance.

 

機器學習通常使用一種稱為「優化算法」的方法來訓練模型，使其能在訓練數據上表現良好。然而，當模型過於複雜時，即使在訓練數據上表現出色，也可能無法在新數據或未見過的數據上表現同樣出色，這種現象被稱為「過擬合」。我們的研究人員利用統計學中的一個重要概念——「算法穩定性」，來研究模型如何學會「泛化」，也就是如何在新數據上維持良好的表現。我們的研究揭示了「優化」（提升訓練表現的過程）與「泛化」（在新數據上表現良好的能力）之間的密切關係。