Fixed it!
So after moving the App folder out from the build folder and into the root. I just had to fix all the 404s.
That meant changing the templateUrls on the three pages it starts with from "app/..." to "build/.." and then also remove one level of "../" in the scss files that call for the node_modules folder.