Hi @markdark - I ended up moving away form source maps and just having a dev and release build step. During dev and debugging etc I don't minify/uglify and just use the original files. Then for release I do the minify/uglify.
I did originally try using source maps but found it much more time consuming and it seemed inconsistent and flakey when using them with Angular due to the exceptions not showing the stack trace or correct source file properly - I can't remember exactly what the issue was, but it wasn't working as well as my current method.
My Gulp build file is here if you want to try it out. Obviously you'll have to edit it for your purposes. It has the source map generation in there, but just commented out.
Hope that helps