Brake Rotor Build Up

Ever since I got this 2005 T&C Minivan I have had brake problems. From the factory the right front caliper had never operated. I fixed that easily. At that time I upgraded to drilled and slotted front rotors and and ceramic pads and got good results. However after just a few thousand miles the rotor faces developed a build up of material and had to be chipped off. I thought it was the ceramic pad material so I switched back to OEM pads and got the same build up. Should I plug the drilled holes? I changed rotor suppliers and now am considering the drilled holes and slots are my problem somehow. Pumping brake dust back onto the rotor faces.