Recently, multiple and heterogeneous remote sensing images have provided a new development opportunity for Earth observation research. utilizing deep learning to gain the shared representative information between different modalities is important to resolve the problem of geographical region classification. In this paper, a CNN-based multi-modal framework for low-sample-size data classification of remote sensing images is introduced. This method has three main stages. Firstly, features are extracted from high- and low-resolution remote sensing images separately using multiple convolution layers. Then, the two types of features are fused at the fusion algorithm layer. Finally, the fused features are used to train a classifier. The novelty of this method is that not only it considers the complementary relationship between the two modalities, but enhances the value of a small number of samples. Based on our experiments, the proposed model can obtain a state-of-the-art performance, being more accurate than the comparable architectures, such as single-modal LeNet, NanoNets and multi-modal H&L-LeNet that are trained with a double size of samples.