This paper presents an analytical procedure to simulate vibrations in gear transmission systems. This procedure couples the dynamics of the rotor-bearing gear system with the vibration in the gearbox structure. The modal synthesis method is used in solving the overall dynamics of the system, and a variable time-stepping integration scheme is used in evaluating the global transient vibration of the system. Locally each gear stage is modelled as a multimass rotor-bearing system using a discrete model. The modal characteristics are calculated using the matrix-transfer technique. The gearbox structure is represented by a finite element model, and modal parameters are solved by using NASTRAN. The rotor-gear stages are coupled through nonlinear compliance in the gear mesh while the gearbox structure is coupled through the bearing supports of the rotor system. Transient and steady state vibrations of the coupled system are examined in both time and frequency domains. A typical three-geared system is used as an example for demonstration of the developed procedure.