Not all mushrooms are edible because some are poisonous. The edible or poisonous mushrooms can be identified by paying attention to the morphological characteristics of mushrooms, such as shape, color, and texture. There is an issue: some poisonous mushrooms have morphological features that are very similar to edible mushrooms. It can lead to the misidentification of mushrooms. This work aims to recognize edible or poisonous mushrooms using a Deep Learning approach, typically Convolutional Neural Networks. Because the training process will take a long time, Transfer Learning was applied to accelerate the learning process. Transfer learning uses an existing model as a base model in our neural network by transferring information from the related domain. There are Four base models are used, namely MobileNets, MobileNetV2, ResNet50, and VGG19. Each base model will be subjected to several experimental scenarios, such as setting the different learning rate values for pre-training and fine-tuning. The results show that the Convolutional Neural Network with transfer learning method can recognize edible or poisonous mushrooms with more than 86% accuracy. Moreover, the best accuracy result is 92.19% obtained from the base model of MobileNetsV2 with a learning rate of 0,00001 at the pre-training stage and 0,0001 at the fine-tuning stage.