In a vacuum light of any wavelength will travel at a fixed speed, c. But light takes longer to travel through a different medium (such as glass or water) because it interacts with atoms in the medium, getting absorbed and re-emitted repeatedly. Light of shorter wavelengths (indigo) interacts stronger, so it will take longer to pass through a medium than light of longer wavelengths (red).
As white light enters a medium (in this case, the prism), each of its composing wavelengths will propagate at a different speed in the new medium, and this change in speed is what bends the path in which light is travelling. This is the phenomenon we call refraction. The ratio between the speed of light in vacuum and the speed of light in a medium is what we call index of refraction, and this value is specific for a given wavelength and medium.
It is important to realise that the speed of light is always constant, but only the interaction in the medium causes the light to take longer to travel through it. To say light travels slower is inaccurate.. between absorption and re-emission from atoms in the medium light still travels at the speed of light..