Penalized regression methods are now popular in genome-wide association studies for identifying genetic markers associated with a disease or for generating genetic risk scores that can be useful for risk prediction. However, standard penalized regression methods do not account for population structure. Linear mixed models (LMM) can account for correlations due to relatedness but are not applicable in high-dimensional (HD) settings. In this work, we develop a general penalized LMM framework that performs variable selection for structured populations. The method can accommodate several sparsity patterns through penalties such as Lasso, Elastic Net, and group Lasso, and readily handles prior annotation information in the form of weights. Our algorithm is computationally efficient and scales to HD settings. Through simulations, we show advantages of the proposed approach against its competitors. We illustrate the use of our method via the analysis of the very large data set UK Biobank.