Exploring Stability and Performance of hybrid Gradient Boosting Classification and Regression Models in Sectors Stock Trend Prediction: A Tale of Preliminary Success and Final Challenge

In the dynamic field of financial analytics, the ability to predict stock market trends is crucial for effective trading strategies, which is the task for FedCSIS 2024 Data Science Challenge: Predicting Stock Trends. This paper presents a comprehensive study on the use of hybrid gradient boosting models, incorporating both classification and regression approaches, to forecast stock trends across different sectors of the S&P 500. Utilizing a rich dataset comprising key financial indicators for 300 companies over a decade, our research aims to unravel the complexities of sector-specific trend predictions. The model leverages 58 financial indicators per company, along with their annual change metrics, to predict the future stock movements. In the preliminary phase of the competition, our hybrid model demonstrated promising results, achieving the lowest weighted error of 0.5941 among competitors. However, despite the initial success, the final phase of the model evaluation revealed a significant performance decline with the error rising above 0.84. This discrepancy highlights potential issues in model stability and preliminary performance when transitioning from a controlled to a truly unseen testing environment. This work not only underscores the complexities of predictive modeling in finance but also sets the stage for future research into creating more resilient AI-driven trading systems.