The Hierarchical Poincaré-Steklov (HPS) method is a recently developed spectral discretization for partial differential equations that naturally comes with an efficient direct solver. The HPS method has proved extremely effective for solving free space scattering problems with localized variable coefficients. For example, for a variable media problem where the support of the variation was 100 wavelengths in size, the HPS method required 3.6 million unknowns to achieve 9 digits of accuracy with the construction of the solver taking six minutes on a desktop computer. Each solve only required three seconds. In this talk, we present the first extension of the HPS method to inverse problems. The problem under consideration is that of reconstructing material properties from scattering measurements. We assume data is recovered from many plane waves impinging the medium from multiple frequencies is given. While this problem is ill-posed and nonlinear, it is turned into a sequence of well-posed problems via a recursive linearization technique. This approach requires the solution to a large number of forward scattering problems reaching well into the mid-frequency regime making the HPS method the ideal PDE solver.