The drive for higher performance in optical fiber systems has renewed interest in coherent detection. We review detection methods, including noncoherent, differentially coherent, and coherent detection, as well as a hybrid method. We compare modulation methods encoding information in various degrees of freedom (DOF). Polarization-multiplexed quadrature-amplitude modulation maximizes spectral efficiency and power efficiency, by utilizing all four available DOF, the two field quadratures in the two polarizations. Dual-polarization homodyne or heterodyne downconversion are linear processes that can fully recover the received signal field in these four DOF. When downconverted signals are sampled at the Nyquist rate, compensation of transmission impairments can be performed using digital signal processing (DSP). Linear impairments, including chromatic dispersion and polarization-mode dispersion, can be compensated quasi-exactly using finite impulse response filters. Some nonlinear impairments, such as intra-channel four-wave mixing and nonlinear phase noise, can be compensated partially. Carrier phase recovery can be performed using feedforward methods, even when phase-locked loops may fail due to delay constraints. DSP-based compensation enables a receiver to adapt to time-varying impairments, and facilitates use of advanced forward-error-correction codes. We discuss both single- and multi-carrier system implementations. For a given modulation format, using coherent detection, they offer fundamentally the same spectral efficiency and power efficiency, but may differ in practice, because of different impairments and implementation details. With anticipated advances in analog-to-digital converters and integrated circuit technology, DSP-based coherent receivers at bit rates up to 100 Gbit/s should become practical within the next few years.