We show how to detect optimal Berry-Ess\'een bounds in the normal approximation of functionals of Gaussian fields. Our techniques are based on a combination of Malliavin calculus, Stein's method and the method of moments and cumulants, and provide de facto local (one term) Edgeworth expansions. The findings of the present paper represent a further refinement of the main results proved in Nourdin and Peccati (2009b). Among several examples, we discuss three crucial applications: (i) to Toeplitz quadratic functionals of continuous-time stationary processes (extending results by Ginovyan (1994) and Ginovyan and Sahakyan (2007)), (ii) to "exploding" quadratic functionals of a Brownian sheet, and (iii) to a continuous-time version of the Breuer-Major CLT for functionals of a fractional Brownian motion.