The present paper deals with weak nonlinear stability analysis of heat transfer in a nanofluid saturated porous layer. We consider a set of new boundary conditions for the nanoparticle fraction, which is physically more realistic. The new boundary condition is based on the assumption that the nanoparticle fraction adjusts itself so that the nanoparticle flux is zero on the boundaries. We use Darcy model that incorporates the effects of Brownian motion and thermophoresis. The governing equations has been reduced to Ginzburg-Landau equation and solved by homotopy analysis method (HAM). The obtained results have been compared with the numerical results obtained by Mathematica NDSolve. The results are valid for the feasible domain with high accuracy. Thermal Nusselt number and Nanoparticle Nusselt number are calculated for different values of parameters. The results have been depicted graphically.