The process of controlling railway systems is one of the important and effective topics in the process of maintaining the flow of trains’ movement and organizing the travel process, as well as providing early readings of any defect or problem that occurs in the railway network to avoid, treat accidents and ensure a safe environment for the movement of train cars across the geographical area. Therefore, a continuous follow-up of the railway condition must be provided to ensure that services continue to be provided. Intelligent railway maintenance improves safety and efficiency. This work presents the design and implementation of a real-time monitoring system for railways based on WSN. This study proposes a system consisting of a base station server and a Rail controller. The Base Station (BS) can automatically monitor and control most railway paths. For that, a classification-based deep learning model for object detection near the railway and making appropriate decisions were proposed. To improve classification-model performance, the yolov3 algorithm for object detection was proposed. On the Rail controller side, the Raspberry Pi 4 was utilized as a low-cost processing unit that can be used as a control unit to control some processes, such as streaming video from a camera, gathering information from railway sensors, and sending data to the central station server (PC) by using WiFi protocol. The model can detect and control railway issues in real time by receiving streaming data and directly detecting, classifying issues, and making the best decisions. By alarming or controlling the desired train and stopping it.