I figured it out!
I literally spent hours till I got it. All you have to do is set draggable to true on the markers like this:
var marker = new google.maps.Marker({
position: markerLatLng,
map: map,
title: tempItems[i].name,
draggable: true
});
The problem was only happening on my Nexus 5x, but it worked on an iPhone6, iPhone5c, and the Asus Zendphone2. If this doesn't work, comment out your markers and see if the maps come up at all. For me, it was ONLY the markers that where causing troubles.