Effective feature representation is key to the predictive performance of any algorithm. We introduce a meta-procedure, called Non-Euclidean Upgrading (NEU), which learns feature maps that are expressive enough to embed the universal approximation property (UAP) into most model classes while only returning feature maps that preserve model class UAP. Properties of NEU are derived from a new deep neural model that is universal amongst all orientation-preserving homeomorphisms on the input space. We show that no deep feed-forward network with commonly used activation function has all of the properties of NEU. Performance is evaluated against competing machine learning methods on various regression and dimension reduction tasks both with financial market data and simulated data.