We provide a topological view on phonon Hall effect in dielectrics with Raman spin-phonon coupling. A general expression for phonon Hall conductivity is obtained in terms of the Berry curvature of band structures. With the increasing magnetic field, we find a nonmonotonic behavior of phonon Hall conductivity, which even becomes oscillatory at low temperatures. Moreover, we identify a phase transition in phonon Hall effect, which corresponds to band topology change, characterized by the altering of integer Chern numbers, and explained by bands touching and splitting