The idea is that, since you borrowed someone's stock, YOU are getting paid the dividend, so you must therefore pass that dividend on to the rightful owner (the person you borrowed from). Dividends are not being paid out twice.

Another way to look at it:

When you BUY a stock and a dividend is paid, you get the dividend but your share price is adjusted down by the amount of the dividend. So it's a wash, overall.

On the other hand, when you SHORT a stock and a dividend is paid, the share price is still adjusted down by the amount of the dividend. If you didn't have to PAY that dividend, you would have a GAIN by virtue of the share price adjustment alone. This would mean you could simply short a stock right before the ex-date and make a gain on the share price adjustment, which would equate to a free lunch. There are no free lunches in the stock market, so you pay the dividend, and the event ends up being a wash overall, just like when you buy.