Combining Programmable Potentials and Neural Networks for Materials Problems