A detailed study of guided modes in a nanoscale metallic rectangular waveguide is presented by using the effective dielectric constant approach. The guided modes, including both traditional waveguide mode and surface plasmon mode, are investigated for the silver rectangular waveguide. The mode evolution in narrow waveguide is also discussed with the emphasis on the dependence of mode dispersion with waveguide height. Finally, the red-shift of the cutoff wavelength of the fundamental mode is observed when the waveguide height decreases, contrary to the behavior of regular metallic waveguide with PEC boundary. The comprehensive analysis can provide some guideline in the design of subwavelength optical devices based on the dispersion characteristics of metallic rectangular bore.