Adeline Lo, Princeton University
Abstract: High dimensional (HD) data, where the number of covariates and/or meaningful covariate interactions might exceed the number of observations, is increasing used in prediction in the social sciences. An important question for the researcher is how to select the most predictive covariates among all the available covariates. Common covariate selection approaches use ad hoc rules to remove noise covariates, or select covariates through the criterion of statistical significance or by using machine learning techniques. These can suffer from lack of objectivity, choosing some but not all predictive covariates, and failing reasonable standards of consistency that are expected to hold in most high-dimensional social science data. The literature is scarce in statistics that can be used to directly evaluate covariate predictivity. We address these issues by proposing a variable screening step prior to traditional statistical modeling, in which we screen covariates for their predictivity. We propose the influence (I) statistic to evaluate covariates in the screening stage, showing that the statistic is directly related to predictivity and can help screen out noisy covariates and discover meaningful covariate interactions. We illustrate how our screening approach can removing noisy phrases from U.S. Congressional speeches and rank important ones to measure partisanship. We also show improvements to out-of-sample forecasting in a state failure application. Our approach is applicable via an open-source software package.
Adeline Lo is a postdoctoral research associate at the Department of Politics at Princeton University. Her research lies in the design of statistical tools for prediction and measurement for applied social sciences, with a substantive interest in conflict and post-conflict processes. She has an ongoing research agenda on high dimensional forecasting, especially in application to violent events. Her work has been published in the Proceedings of the National Academy of Sciences, Comparative Political Studies and Nature. She will be joining the Department of Political Science at the University of Wisconsin-Madison as an Assistant Professor in Fall 2019.
