Okay so...I am not going to claim to be an expert on this, but I am pretty sure that hemihustlin has it basically right.
You are trying to hold a speed under a speed that Cruise Control can be active (I believe 25mph is the cutoff), so the computer has no idea what speed you want to maintain. Gravity is pulling you down the hill, and gravity is going to cause you to accelerate if there's no real force opposing it (brakes, engine compression, friction, etc). Since you're under 25 mph you can't tell the computer 'hey I want to be at THIS speed' so it can't do calculations to keep you slow.
Couple this with the engine sensors trying to keep it running in a range of emissions factors, and you're going to end up in this scenario. The engine needs a certain air/fuel ratio to keep the O2 sensors happy, and with no other input coming in (i.e. a speed setpoint from the Cruise Control module) it is going to do whatever is necessary to keep those O2 sensors happy. Thus at low speed, as RPMs increase the A/F ratio changes, so the throttle plate needs to adjust slightly. Which, at such low speeds, will allow you to speed up because it isn't actively engine braking like it will with the cruise set.
Likely, if you were traveling at a higher speed, you wouldn't notice it as much. But at those speeds, you're going to need to work the brakes.