Update:
I've managed to drill out the remaining easy out and surrounding dirty metal. Took it to a shop where a guy I know it's skilled at welding aluminum and he filled the hole back in. Now I just have to drill, then tap a new hole. He was surprised at how soft the aluminum was...whatever that means.
I tried to use alumiweld, but I could not get the manifold hot enough. I think that stuff is better suited for smaller work, like brackets and such.