The recognition of Bangla handwritten numerals (BHNR) has recently emerged as a very interesting area for machine learning and pattern recognition research. Recently, the technology for character or object recognition has also advanced. Bangla handwritten number recognition can serve as a foundation for creating an Optical Character Recognition (OCR) in the Bangla language. However, the lack of a sizable and accurate dataset makes Bangla's handwritten numeral recognition study insufficient in contrast to that of other well-known languages. Similar to MNIST for English digits, NumtaDB is by far the largest dataset collection for handwritten digits in the Bangla language. The most used datasets for the recognition of Bangla handwritten numerals in the past were NumtaDB, CMATERdb, and ISI. The majority of approaches now in use rely on feature extraction and a few outdated machine learning algorithms. Although some approaches operate quickly enough to meet practical demands, they are not always accurate. Even while certain techniques work quite well for languages other than Bangla, they still require improvement. Convolutional Neural Networks (CNN), in particular, are demonstrating excellent achievements in this discipline with higher accuracy. In this work, we’ve used custom CNN architectures to build our model to recognize digits using all the existing datasets with a high degree of accuracy. Our CNN model shows an average of 98% accuracy recognizing Bangla numeric in respect of above datasets. We have cross verified our model with mixed datasets and the result is also promising.