I'm being run out of my apartment by centipedes. Help!!!?

The construction work done on our building was kind of shoddy. There's a huge gap where the air conditioner was installed that looks straight to outside. I swear I can hear people outside (and down 3 floors) clear as day through the outlet by my bed. Anyway, the point is the centipedes have overtaken my apartment, presumably through these various gaps to the outside world. Can I caulk these spaces? What would you recommend? My realty company is sloooow and I want to take care of this problem quickly, but don't want to do something permanent to my apartment that I may get in trouble for (read: charged for) later. Any suggestions are much appreciated.