Human brain is the most complex system in the world. Within this system, multiple neurons, neural circuits or brain regions are connected to form a complex structural network. Using graph thereotical network analysis approaches, many studies have demonstrated that the structural and functional brain networks can be constructed by electrophysiological and neuroimaging data (electroencephalography, magnetoencephalography, structural MRI, diffusion MRI and functional MRI), and that the resultant brain networks show many important topological features such as the “small-world” property, modular structure and several “core” regions in the association cortex. Moreover, there is also accumulating evidence that various neurological and psychiatric diseases (e.g., Alzheimer’s disease and schizophrenia) are associated with abnormal topological architecture in their structural and functional brain networks. These studies might offer a novel avenue to elucidate pathological mechanisms underlying these brain disorders, and to uncover an imaging-based biomarker for early diagnosis and monitoring of the diseases. In this review, we first introduce some basic concepts of the human connectome and complex network theory. Then, we move on recent studies regarding the human structural and functional brain networks. Finally, we raise several technical challenges and key issues which need to be addressed in the future.