On the front (window side) head, there are two heat shields.
There is one that wraps around the manifold, and would seem to be designed to keep stuff from getting against the manifold and getting burned.
There is also a flat one that goes against the head (then the gasket, then the manifold.)
It's designed to keep vital fluids off of the catalytic converter in the event that you happen to air-mail a rod through the front of the block, thus preventing a Fiero Flambe', and helping to prevent the furthering of the stereotype.
It also serves to funnel oil, that is leaked from the valve cover, away from the exhaust system.
The "outside" one is probably just held on by stamped nuts or maybe additional "regular" hardware.
In any event, you should be able to remove it, while leaving the manifold still bolted on.