E. Cobham Brewer 1810–1897. Dictionary of Phrase and Fable. 1898.
Penny Wise.
Unwise thrift. The whole proverb is Penny wise-and pound foolish, like the man who lost his horse from his penny wisdom in saving the expense of shoeing it afresh when one of its shoes was loose.
