It cant nic the clear coat or paint. Its much softer than either. If there was debris in the leftover glue then that could've caused the damage when the eraser wheel hit it.
I forgot to mention, keep the drill slow. I mean slow, half speed is too much. You're not trying to burn the glue off, the wheel needs to grab the glue peel it away then flake away from the wheel as it comes around.
The textured wheels are a bit easier to use as they have the eraser wheel equivalent of flutes on drill bits for chip clearance.
If it chipped something, just keep going and get something to touch it up afterwards. Using anything that will dissolve the glue is very likely eat partway or all the way through the clear coat and paint. In the long run, most solvents will cause the area to 'cloud' over time unless you either keep it waxed or hit it with more clear coat.
When you use the wheel i've found that going as slow as you can while moving it back and forth a bit while applying light to moderate pressure. It will take time, its nothing fast but its faster than most other things. It took me 5 minutes to get all the glue off from each badge on my truck when I was re-gluing them on. I managed about a foot a minute on some pinstriping on an old chevy van. (I say old but it was the same year as my truck
)