This article examines the challenges of evaluating Mahila Samakhya (MS), a government sponsored programme for the empowerment and education of poor women. The programme has no fixed targets and aims at enabling such women to go beyond mere literacy and engage in a learning process that facilitates their collective agency and action to question and overcome deep-seated gender barriers. The article explores the issue of evaluation at several levels—negotiating with donors on evaluation parameters and methods, developing qualitative evaluation parameters and also at the conceptual domain of what or who needs to be evaluated especially in programmes that aim at challenging and transforming both gender and social inequity and inequality. The article also comments on the changing external context with its strong emphasis on measurable outcomes and quantitative measures of assessment and its implications for programmes such as MS, where the struggle is to ensure that the processes of empowerment that are often intangible are not overlooked.