I'm pretty sure it can be removed without loosening the trans but I think unless the hole over by the starter has been slotted, the starter has to be removed. There are only those 3 bolts that hold it on.
When Don worked on my first 85, he installed a clutch. When he put it back together, the inspection/dust cover was scrubbing, so he pulled it off without removing the transmission. Apparently it got bent during the process.