Design and Implementation of an Automated Trading System Based on Reinforcement Learning for Financial Markets

Abedini, Erfan; Mohammadi Mohammadi, Hadi; Saadatian Farivar, Mohammad Hassan

doi:10.22075/ijnaa.2025.37619.5455

Design and Implementation of an Automated Trading System Based on Reinforcement Learning for Financial Markets

Document Type : Research Paper

Authors

Erfan Abedini ¹

Hadi Mohammadi Mohammadi ²

Mohammad Hassan Saadatian Farivar ³

¹ M.A. Student, Department of Financial Management, Faculty of Management and Economics, SR.C., Islamic Azad University, Tehran, Iran

² Associate Professor of Agricultural Economics, Department of Economics, Faculty of Economics and Accounting, CT.C., Islamic Azad University, Tehran, Iran

³ Associate professor, Department of Financial Management, Accounting and Financial Engineering, Faculty of Management and Economics, SR.C., Islamic Azad University, Tehran, Iran

10.22075/ijnaa.2025.37619.5455

Abstract

In recent years, the rapid growth of technology and data-driven financial markets has led to an increased interest in using artificial intelligence, and especially reinforcement learning, in designing automated trading systems. In this study, an automated trading system based on reinforcement learning for the stock market is designed and implemented. The main objective of this study is to investigate the efficiency of reinforcement learning models including Proximal Policy Optimization (PPO), Advantage Actor-Critic (A2C), and Deep Q-Network (DQN) compared to traditional investment methods. To train the models, real stock market data and technical indicators are used, and two different reward functions are proposed to evaluate the system performance. The performance of the models is analyzed with criteria such as total return and Sharpe ratio. Experimental results show that reinforcement learning models can provide higher returns than traditional methods in certain conditions, and choosing the appropriate reward function plays a significant role in improving trading strategies. This research, while emphasizing the importance of carefully designing the reward function and selecting appropriate features, offers new perspectives for the development of intelligent systems in financial markets.

Keywords

Reinforcement learning

artificial intelligence

financial markets

automated trading system

reward function

Subjects

Machine Learning

[1] A. Aziz, How to Day Trade for a Living, Bear Bull Traders, Vancouver, Canada, 2016.

[2] A. Briola and et al. , Deep reinforcement learning for active high frequency trading, arXiv preprint arXiv: 2101.07107 (2021).

[3] J. Carapuço, R. Neves and N. Horta, Reinforcement learning applied to Forex trading, Applied Soft Computing 73 (2018), no. 6, 783-794.

[4] Y. Deng and et al. , Deep direct reinforcement learning for financial signal representation and trading, IEEE transactions on neural networks and learning systems 28 (2016), no. 3, 653-664.

[5] C. Y. Huang, Financial trading as a game: A deep reinforcement learning approach, arXiv preprint arXiv: 1807.02787 (2018).

[6] C. F. Huang and H. C. Li, An Evolutionary Method for Financial Forecasting in Microscopic High‐Speed Trading Environment, Computational Intelligence and Neuroscience 10 (2017), no. 1, 9580815.

[7] G. Jeong and H. Y. Kim, Improving financial trading decisions using deep Q-learning: Predicting the number of shares, action strategies, and transfer learning, Expert Systems with Applications 117 (2019), no. 8, 125-138.

[8] A. Kaur, N. Gupta, Revolutionizing stock market: Role of AI and its impact on stockholders, 10 (2025), no. 2, 845-848.

[9] I. K. Kavonius and V. M. Tormalehto, Is the Financial Market Driving Income Distribution?, An analysis of the linkage between income and wealth in Europe, 2022.

[10] A. Kumanireng, Application of Fundamental Analysis in Stock Valuation in the Capital Market and Investment Decisions by Price Methods Earnings Ratio (PER), Available at SSRN, 2023.

[11] S. K. Inani and et al. , Navigating the technical analysis in stock markets: Insights from bibliometric and topic modeling approaches, Investment Management and Financial Inno. vations 21 (2024), no. 1, 275-288.

[12] R. Islam, D. Chakraborty, A. Kumar Roy, M. M. Rahman and M. M. Rahman Enam, Effectiveness of AI-based bachine learning algorithms in predicting global market movements, Journal of Engineering Research and Reports 26 (2024), no. 8, 343-354.

[13] B. Itri and et al. , Deep reinforcement learning strategy in automated trading systems, in 2023 3rd International Conference on Inno. vative Research in Applied Science, Engineering and Techno. logy (IRASET) (2023), IEEE.

[14] E. Molina, P. Cortes, I. Molina and F. Sobrino, The Impact of Artificial Intelligent Tools on Decision Making Behavioral and Neural Dynamics, Research Square, 2024, https://doi.org/10.21203/rs.3.rs-4651166/v1.

[15] S. Otabek and J. Choi, Multi-level deep Q-networks for Bitcoin trading strategies Scientific Reports 14 (2024), no. 1, 771-790.

[16] J. D. D. Roger, AI-driven financial modeling techniques: Transforming investment strategies, The Journal of Applied Business and Economics 26 (2024), no. 4, 63-74.

[17] R. G. Sloan, Fundamental analysis redux, The Accounting Review 94 (2019), no. 2, 363-377.

[18] P. E. Tsinaslanidis and A. D. Zapranis, Technical analysis for algorithmic pattern recognition, Springer publication, 2016.

[19] P. Yang and X. Hou, Research on dynamic characteristics of stock market based on big data analysis, Discrete Dynamics in Nature and Society 1 (2022), no. 2, 8758976.

[20] H. Yang and et al. , Deep reinforcement learning for automated stock trading: An ensemble strategy, in Proceedings of the first ACM international conference on AI in finance, 2020.

[21] Z. Zhang, S. Zohren and S. Roberts, Deep reinforcement learning for trading, arXiv preprint arXiv: 1911.10107 (2019).