Are markets so different? forecasting markets' return with LSTM model and optimizations
|
ABSTRACT 3
LIST OF ABBREVIATIONS 6
LIST OF FIGURES 7
1. INTRODUCTION 10
1.1 Background 10
1.2 Research Goal 12
1.3 Research Questions 13
2. LITERATURE REVIEW 18
2.1 Efficient Market Hypothesis (EMH) 19
2.2 Random Walk Hypothesis (RWH) 21
2.3 Traditional Models vs LSTM Model 25
2.4 Parameter Optimization Techniques for LSTM Model 30
3. RESEARCH METHODOLOGY 32
3.1 Data Collection and Preprocessing 32
3.2 Data Analysis Techniques 34
3.3 Training of the Model 38
3.4 Optimization Technique and Mean Square Error 39
3.5 Research Methodology Conceptual Framework 41
4. RESULTS 42
4.1 Data Description 42
4.2 Empirical Analysis 43
4.3 Results Interpretation 49
4.4 Comparative Analysis Across Markets 60
5. DISCUSSION 63
5.1 Possible Implications 63
5.2 Directions for Future Research 64
6. CONCLUSION 65
7. REFERENCES 68
LIST OF ABBREVIATIONS 6
LIST OF FIGURES 7
1. INTRODUCTION 10
1.1 Background 10
1.2 Research Goal 12
1.3 Research Questions 13
2. LITERATURE REVIEW 18
2.1 Efficient Market Hypothesis (EMH) 19
2.2 Random Walk Hypothesis (RWH) 21
2.3 Traditional Models vs LSTM Model 25
2.4 Parameter Optimization Techniques for LSTM Model 30
3. RESEARCH METHODOLOGY 32
3.1 Data Collection and Preprocessing 32
3.2 Data Analysis Techniques 34
3.3 Training of the Model 38
3.4 Optimization Technique and Mean Square Error 39
3.5 Research Methodology Conceptual Framework 41
4. RESULTS 42
4.1 Data Description 42
4.2 Empirical Analysis 43
4.3 Results Interpretation 49
4.4 Comparative Analysis Across Markets 60
5. DISCUSSION 63
5.1 Possible Implications 63
5.2 Directions for Future Research 64
6. CONCLUSION 65
7. REFERENCES 68
Recent years have seen financial markets becoming more intricate and unstable, induced by com-plex multi-faceted factors of geopolitics, economics, and technology. The better a market movement can be forecasted, the more informed decisions can be made, and the more competitive each stakeholder stays in the fast-evolving ecosystem of global finance. Market forecasting has traditionally been done using econometric and statistical methods generally based on the Efficient Market Hypothesis. Market returns are movements in the cost of a financial market or an investment portfolio over a given period. They are commonly used to gauge how well a particular market is performing. Market returns have implications for the market efficiency hypothesis. Financial markets are efficient according to the EMH implying that security prices reflect all available information, thus making it difficult for investors to consistently obtain higher than average returns through trading strategies. This means that the average return should only have a risk measure and be proportionate to the return predicted by investors. On this basis, it is necessary to examine how market returns relate with investor behavior under an efficient market environment. Investors who think that markets are efficient and that prices reflect all the available information mightchoose passive investments like funds indexed on broad equity portfolios that try to replicate aggregate market performance net of costs, transaction fees and taxes. Such investors would expect to earn the average market return over long periods.
Depending on the specific market or investment being considered, market returns can fluc-tuate significantly. Stock market returns, for instance, generally denote the general performance of a stock market index, namely S&P 500 and Dow Jones Industrial Average. Conversely, bond market returns depict how bonds and fixed-income securities have fared during a certain period. It is important to remember that there are inherent doubts and risks associated with market returns. Various factors such as economic conditions, interest rates, geopolitical events, corporate earnings, and investor sentiment influence financial markets. Therefore, market return might be volatile at times making it hard to predict. The presence of huge volumes of data, the rate at which artificial intelligence and machine learning techniques are advancing and the improved computational ability of this machine, enable generation of complex algorithms for predicting stock prices. Meanwhile, today’s investment markets in stocks are more opaque than ever before due to the proliferation of investing options.Oneapproach that recently became quite popular and shows promising results is applying deep learning methods, particularly Long Short-Term Memory models, for financial forecasting. LSTMs are a type of Recurrent Neural Networks that have proven to be very efficient in capturing temporal dependencies and nonlinear relationships within sequential data. Given these properties, LSTMs are widely used for time series predictions tasks. However, there are still numerous issues in optimizing their work for financial forecasting and ensuring robustness across different types of a market. Firstly, markets can vary significantly by efficiency, volatility, and liquidity, which makes forecasting an interesting but quite challenging task. Secondly, data preprocessing, feature selection and hyperparameters impact the performance and efficiency of LSTMs.By presenting this thesis, the challenges and the success of LSTM model and parameter optimization methodologies are disclosed; as well as the computational and technical processes to uncover the force majeure of LSTM in predicting market returns when trained with actual market data.
Also, it is projected that such research reveals valuable insights in addressing predictability of the market and ensure the most suitable ML model for time-series trading in all market condi-tions. The LSTM model employs hard and soft optimization strategies to forecast market returns, thereby addressing these challenges. It is against this backdrop that the research seeks to examine the feasibility of using Long Short-Term Memory (LSTM) models as well as optimization strategies in formulating an answer to some of the problems encountered or faced in forecasting market re-turns. In this regard, a comparative study will be carried out across 6 different indices such as NASDAQ 100 (NDX), Standard’s & Poor 500 (SPX), Dow Jones Industrial Average (DJI), Finan-cial Times Stock Exchange - FTSE 100 (UKX), Stoxx Europe 600 (SXXP) and Nikkei 225 (NKY); and empirical analysis will be conducted to present deeper insight into what triggers market returns and how LSTM models can capture and predict such patterns. This knowledge would aid investors and policy makers in utilizing advanced predicting analytics through describing strengths and limi-tations associated with LSTM based approaches in financial forecasting.The most essential feature that stems from an LSTM model is its memory cell, capable of storing and accessing information for a long period. In addition, gates in the form of input gate, forget gate, and output gate were at-tached to it.These gates control how information is stored, forgotten, or output from memory cells of LSTM allowing it to selectively remember or forget available information depending on its re-levance. Internally, during the training process internal parameters including weights and biases are adjusted by the LSTM model to reduce discrepancies between predicted outputs and actual res-ponses. For application of LSTM as a tool for forecasting market return historical market data sup-plemented with relevant features provides inputs into the model such that it learns patterns as well as relationships among data hence can predict future market returns.
...
Depending on the specific market or investment being considered, market returns can fluc-tuate significantly. Stock market returns, for instance, generally denote the general performance of a stock market index, namely S&P 500 and Dow Jones Industrial Average. Conversely, bond market returns depict how bonds and fixed-income securities have fared during a certain period. It is important to remember that there are inherent doubts and risks associated with market returns. Various factors such as economic conditions, interest rates, geopolitical events, corporate earnings, and investor sentiment influence financial markets. Therefore, market return might be volatile at times making it hard to predict. The presence of huge volumes of data, the rate at which artificial intelligence and machine learning techniques are advancing and the improved computational ability of this machine, enable generation of complex algorithms for predicting stock prices. Meanwhile, today’s investment markets in stocks are more opaque than ever before due to the proliferation of investing options.Oneapproach that recently became quite popular and shows promising results is applying deep learning methods, particularly Long Short-Term Memory models, for financial forecasting. LSTMs are a type of Recurrent Neural Networks that have proven to be very efficient in capturing temporal dependencies and nonlinear relationships within sequential data. Given these properties, LSTMs are widely used for time series predictions tasks. However, there are still numerous issues in optimizing their work for financial forecasting and ensuring robustness across different types of a market. Firstly, markets can vary significantly by efficiency, volatility, and liquidity, which makes forecasting an interesting but quite challenging task. Secondly, data preprocessing, feature selection and hyperparameters impact the performance and efficiency of LSTMs.By presenting this thesis, the challenges and the success of LSTM model and parameter optimization methodologies are disclosed; as well as the computational and technical processes to uncover the force majeure of LSTM in predicting market returns when trained with actual market data.
Also, it is projected that such research reveals valuable insights in addressing predictability of the market and ensure the most suitable ML model for time-series trading in all market condi-tions. The LSTM model employs hard and soft optimization strategies to forecast market returns, thereby addressing these challenges. It is against this backdrop that the research seeks to examine the feasibility of using Long Short-Term Memory (LSTM) models as well as optimization strategies in formulating an answer to some of the problems encountered or faced in forecasting market re-turns. In this regard, a comparative study will be carried out across 6 different indices such as NASDAQ 100 (NDX), Standard’s & Poor 500 (SPX), Dow Jones Industrial Average (DJI), Finan-cial Times Stock Exchange - FTSE 100 (UKX), Stoxx Europe 600 (SXXP) and Nikkei 225 (NKY); and empirical analysis will be conducted to present deeper insight into what triggers market returns and how LSTM models can capture and predict such patterns. This knowledge would aid investors and policy makers in utilizing advanced predicting analytics through describing strengths and limi-tations associated with LSTM based approaches in financial forecasting.The most essential feature that stems from an LSTM model is its memory cell, capable of storing and accessing information for a long period. In addition, gates in the form of input gate, forget gate, and output gate were at-tached to it.These gates control how information is stored, forgotten, or output from memory cells of LSTM allowing it to selectively remember or forget available information depending on its re-levance. Internally, during the training process internal parameters including weights and biases are adjusted by the LSTM model to reduce discrepancies between predicted outputs and actual res-ponses. For application of LSTM as a tool for forecasting market return historical market data sup-plemented with relevant features provides inputs into the model such that it learns patterns as well as relationships among data hence can predict future market returns.
...
The process of researching included building and implementing an LSTM model to predict market prices for the indices. During the process of data preprocessing, modelling building, training, and evaluation, these steps were iterated to achieve the best forecast accuracy possible. The findings derived from the LSTM model's forecasts for different market indices are not only interesting but have also provided answers to the questions of the research in a meaningful way. These findings are based on the synthesis and testing of the model's operations, accuracy, and forecasting impact.
In regard to the 1st research question, LSTM performance demonstrates different ac-curacy levels in different markets, based on their trends, which is normal. The results of the model on several indices including NDX, SPX, DJI, UKX, SXXP, and NKY shows that indeed the model is able to capture the movements and direction of the trend for each index, however, these occurrences may slightly differ and are negligible. For example, the LSTM model precisely tracked the NDX and SPX indices with small over/under estimations indicating that it was highly reliable and would deliver the results with high precision. In contrast, the model for the UKX and SXXP indices tends to give underestimating more consistently, suggesting that the model's parameters could be further adjusted to consider this specific part. Again, as stated above these underestima-tions are of really small values between predictions and real data.
In response to the 2nd research questions, the p-values for all indices are significantly greater than 0,05 which is evidence that we cannot reject the null hypothesis that the series are characterized as unit roots. Hence the data series are non-stationary which shows the random walk pattern. The successful market returns forecasting with LSTM can prove results against the Random Walk Theory. In the Random Walk theory, stock prices change in a random manner and hence, they become impossible to predict based on their prior prices. However, the LSTM model’s ability to predict future market returns with a high level of accuracy implies that there are discoverable patterns in the historical data the model can learn and exploit to create forecasts.
The ADF test outputs indicate a mixed picture on the stationarity of the indices. While some indices such as the UKX and SXXP show a certain stationarity (ADF statistic closer to the critical value), other indices like the NDX and SPX do not. In spite of this, the LSTM model proved with its predictions that it could identify non-random trend and patterns that drive the process. The conformity of the LSTM model's forecasts to the actual values of all considered indices in this research work demonstrates that there is indeed some predictive power in the past movements of price. The credible forecasts, in which the model explains the general trend and the price changes slightly around it, speak against the belief in completely random price variations. LSTMs are capable of identifying complex, non-linear relationships within data which are otherwise not apparent through traditional methods of analysis. Whereas the ADF test results confirm the random walk expectation (i.e., non-stationarity and unpredictability), the accuracy with which the LSTM models forecast market returns is an indication against the random walk hypothesis. This means that the LSTM models have the capability to learn and use those underlying structures or patterns in market data to be used in prediction.
Answering the 3rd research question, the adjustment of the learning rate, early stopping with patience, and the addition of dropout layers were used to boost the model performance. These me-thods assist in the elimination of overfitting and good generalization, and the result is the optimiza-tion of model parameters. Consequently, we end up with more precise forecasts.Through the ap-plication of the parameter optimization methods mentioned above and the choice of the optimal hyperparameters such as the number of time steps, epochs, batch size, number of units, number of layers, dropout rate, learning rate, optimizer, loss function, regularization, early stopping; the re-searchers can determine the most powerful strategies in order to enhance forecasting accuracy of LSTM model for stock market returns. The current research adds new information to the finance field by proving with the example of LSTM models that the neural networks can be successfully applied to the task of market returns forecasting for different indices. It emphasizes the role played by parameter optimization processes for the purpose of improving forecast precision. Furthermore, the research gives rise to improved simulation methodologies in finance and supplies essential in-sights for both the researchers and the investors who are looking to use machine learning for prediction.
Concluding this thesis, the research involved developing and deploying an LSTM model to anticipate market indices through a series of steps including data processing, model building, train-ing, and validation with the aim of attaining market indices accuracy parameters. It is supported by the fact that the LSTM model is able to faithfully reflect the tendencies of indices of different mar-kets, and only minor but normal differences in accuracy are observed. The LSTM output showed reliability for all indices without a doubt. The model, however, is highly accurate despite non-stationary data series and patterns in historical data that contradict the Random Walk Theory. Para-meters optimizations proved to be effective in improving the model’s response and forecast accura-cy by highlighting the usefulness of neural networks in market forecast as well as parameter tun-ing.Generally, this paperdelivers critical knowledge to the world of finance and forecasting models, and it improves our understanding of market prediction with those advanced ML approaches.
In regard to the 1st research question, LSTM performance demonstrates different ac-curacy levels in different markets, based on their trends, which is normal. The results of the model on several indices including NDX, SPX, DJI, UKX, SXXP, and NKY shows that indeed the model is able to capture the movements and direction of the trend for each index, however, these occurrences may slightly differ and are negligible. For example, the LSTM model precisely tracked the NDX and SPX indices with small over/under estimations indicating that it was highly reliable and would deliver the results with high precision. In contrast, the model for the UKX and SXXP indices tends to give underestimating more consistently, suggesting that the model's parameters could be further adjusted to consider this specific part. Again, as stated above these underestima-tions are of really small values between predictions and real data.
In response to the 2nd research questions, the p-values for all indices are significantly greater than 0,05 which is evidence that we cannot reject the null hypothesis that the series are characterized as unit roots. Hence the data series are non-stationary which shows the random walk pattern. The successful market returns forecasting with LSTM can prove results against the Random Walk Theory. In the Random Walk theory, stock prices change in a random manner and hence, they become impossible to predict based on their prior prices. However, the LSTM model’s ability to predict future market returns with a high level of accuracy implies that there are discoverable patterns in the historical data the model can learn and exploit to create forecasts.
The ADF test outputs indicate a mixed picture on the stationarity of the indices. While some indices such as the UKX and SXXP show a certain stationarity (ADF statistic closer to the critical value), other indices like the NDX and SPX do not. In spite of this, the LSTM model proved with its predictions that it could identify non-random trend and patterns that drive the process. The conformity of the LSTM model's forecasts to the actual values of all considered indices in this research work demonstrates that there is indeed some predictive power in the past movements of price. The credible forecasts, in which the model explains the general trend and the price changes slightly around it, speak against the belief in completely random price variations. LSTMs are capable of identifying complex, non-linear relationships within data which are otherwise not apparent through traditional methods of analysis. Whereas the ADF test results confirm the random walk expectation (i.e., non-stationarity and unpredictability), the accuracy with which the LSTM models forecast market returns is an indication against the random walk hypothesis. This means that the LSTM models have the capability to learn and use those underlying structures or patterns in market data to be used in prediction.
Answering the 3rd research question, the adjustment of the learning rate, early stopping with patience, and the addition of dropout layers were used to boost the model performance. These me-thods assist in the elimination of overfitting and good generalization, and the result is the optimiza-tion of model parameters. Consequently, we end up with more precise forecasts.Through the ap-plication of the parameter optimization methods mentioned above and the choice of the optimal hyperparameters such as the number of time steps, epochs, batch size, number of units, number of layers, dropout rate, learning rate, optimizer, loss function, regularization, early stopping; the re-searchers can determine the most powerful strategies in order to enhance forecasting accuracy of LSTM model for stock market returns. The current research adds new information to the finance field by proving with the example of LSTM models that the neural networks can be successfully applied to the task of market returns forecasting for different indices. It emphasizes the role played by parameter optimization processes for the purpose of improving forecast precision. Furthermore, the research gives rise to improved simulation methodologies in finance and supplies essential in-sights for both the researchers and the investors who are looking to use machine learning for prediction.
Concluding this thesis, the research involved developing and deploying an LSTM model to anticipate market indices through a series of steps including data processing, model building, train-ing, and validation with the aim of attaining market indices accuracy parameters. It is supported by the fact that the LSTM model is able to faithfully reflect the tendencies of indices of different mar-kets, and only minor but normal differences in accuracy are observed. The LSTM output showed reliability for all indices without a doubt. The model, however, is highly accurate despite non-stationary data series and patterns in historical data that contradict the Random Walk Theory. Para-meters optimizations proved to be effective in improving the model’s response and forecast accura-cy by highlighting the usefulness of neural networks in market forecast as well as parameter tun-ing.Generally, this paperdelivers critical knowledge to the world of finance and forecasting models, and it improves our understanding of market prediction with those advanced ML approaches.
