As the penetration of distributed generation sources increases at the distribution level, the necessity of designing microgrids is highlighted. Microgrids can operate in either islanded (separated from the utility grid) or grid-connected mode. Apart from the technical and economic benefits of microgrids, they may cause some protection issues in distribution systems. The most important of these issues is the difference between the short circuit levels in islanded and grid-connected operation modes. In this paper, an intelligent overcurrent relay that can adapt to the microgrid operation mode is proposed; that is, the relay detects the microgrid operation mode by means of an adaptive algorithm. The proposed adaptive method optimizes the accuracy and speed of the algorithm for detecting the microgrid operation mode. Then, the optimal settings that have been already stored in the relay are activated according to the microgrid operation mode. All internal processes of the relay use the voltage and current sampled at the relay location. As a result, the relay can be used in all networks without communication infrastructures. Finally, the proposed method is implemented in a sample microgrid. The superiority of the intelligent relay performance over conventional overcurrent relays is shown by numerical results.