The compressor won't have a clutch, just a control valve that internally changes the swash plate angle and varies the displacement of the compressor. They don't click on/off as such.
When the ac is "OFF", the control valve sets the compressor displacement to minimum - almost off, though it does move some oil/refrigerant, generally.
The compressor could be noisier with higher load - at idle, the displacement has to be higher for a given amount of required cooling, than at higher speeds.
It could also be the additional load of the compressor making the diesel noise at idle, louder.
I know on our petrol cars, the L/H fuel readout at idel increase quite a bit with the AC on, rather than off.