Sometimes the discs collect rust (from when it rains), and that will make
a "grinding" noise, which goes away after a few high-speed stops.
Since your grinding occurs "every now and then" and is not constant, you
probably have more life in the discs, and it could be rust.