Nowadays, the IoT attracts a multitude of research and industrial interests. Smaller and smarter devices are being implemented daily in multiple IoT domains. However, protecting IoT devices from cyber-attacks is critical to their operation. Confidential data is leaked as a result of malicious acts. As a result, device performance becomes crucial. Security risks are frequently made in IoT-based structures that impact their standard work. Therefore, to eliminate and mitigate these issues (attacks), the Intrusion Detection System (IDS) was proposed to fulfill this purpose. This paper aims to study the state of the art of the proposed IDS. Moving on, we critically review the proposed IDS-based machine learning algorithms. Based on this evaluation criteria, the solutions covering architectures, intelligent prediction and algorithms are critically reviewed. To achieve our goals, the paper presents challenges and open research areas in IoT design.