We provide a topological understanding on phonon Hall effect in dielectrics with Raman spinphonon coupling. A general expression for phonon Hall conductivity is obtained in terms of the Berry curvature of band structures. We find a nonmonotonic behavior of phonon Hall conductivity as a function of magnetic field. Moreover, we observe a phase transition in phonon Hall effect, which corresponds to the sudden change of band topology, characterized by the altering of integer Chern numbers. This can be explained by touching and splitting of phonon bands.