In this article, we present new research on the impact of various optical filters on the functionality of an orthogonal Optical Frequency Division Duplexing (OFDM) system for the first time. This paper focused on long-haul optical systems using four Quadrature Amplitude Modulation (4-QAM) methods for OFDM systems. The analysis demonstrates the impact of choosing a Gaussian, Rectangular, or Trapezoidal optical Filter on the effectiveness of the optical system used for long-distance high data rate transmission. Compared with other systems for long-distance transmission using various filter types, 4-QAM OFDM showed a high Bit Error Rate (BER) performance. Significant findings like transmitting Power received Power, and Error Vector Magnitude (EVM) is compared with the system's propagation length.