Almost ate a glider

So there we were in the Cardinal talking to center on our IFR flight plan in VFR weather and we about 5 miles off the departure end of the westbound runway at SWF. Center called and said "numerous targets ahead... we think they may be gliders as there is a school there." Geeezz... heads up, glaring out the window. We spotted a gaggle of them below us circling Orange County (NY) airport and thought we were in the clear. I was flying from the right seat and just happened to look towards Steve (the other pilot and owner) and holy *!xx there was a big yellow towplane with a glider behind him that looked like it took up the whole side window. I yanked the plane in a right dive and held my breath. He must of taken a left turn or didn't even see us. We called center and they said they either didn't have a transponder or didn't have it on and he couldn't see them. After that we got off the airway and gave the airport a large berth. We also took deep breathes to calm our heart rates down. Moral of the story... don't trust center to keep you from a midair when you are on an IFR flight plan in VFR.
-Nina Anderson and Steve Meyerowitz