We prove that when suitably normalized, small enough powers of the absolute value of the characteristic polynomial of random Hermitian matrices, drawn from one-cut regular unitary invariant ensembles, converge in law to Gaussian multiplicative chaos measures. We prove this in the so-called $L^2$-phase of multiplicative chaos. Our main tools are asymptotics of Hankel determinants with Fisher-Hartwig singularities. Using Riemann-Hilbert methods, we prove a rather general Fisher-Hartwig formula for one-cut regular unitary invariant ensembles.