A series of recent papers demonstrate that mobile phone metadata, in conjunction with machine learning algorithms, can be used to estimate the wealth of individual subscribers, and to target resources to poor segments of society. This paper uses survey data from an emergency cash transfer program in Haiti, in combination with mobile phone data from potential beneficiaries, to explore whether similar methods can be used for impact evaluation. A conventional regression discontinuity-based impact evaluation using survey data shows positive impacts of cash transfers on household food security and dietary diversity. However, machine learning predictions of food security derived from mobile phone data do not show statistically significant effects; nor do the predictions accurately differentiate beneficiaries from non-beneficiaries at baseline. Our analysis suggests that the poor performance is likely due to the homogeneity of the study population; when the same algorithms are applied to a more diverse Haitian population, performance improves markedly. We conclude with a discussion of the implications and limitations for predicting welfare outcomes using big data in poor countries.