William Duff (1697-1763)

William Duff, inherited the wealth of the Duffs of Braco who were bankers, and accumulated a fortune, partly by the shrewd purchase of Jacobite forfeited estates. He was MP for Banffshire from 1727-1734, and became Lord Braco in 1735 and Earl of Fife in 1759.

In 1724 he commissioned James Gibb to design a new mansion at Balvenie, and then in 1735 he engaged William Adam to design Duff House. A legal dispute with the architect ensued which William Duff lost, and he refused to occupy the new mansion. Perhaps this was the reason for his acquisition of Rothiemay Castle in 1741, where he died. His second wife Jean Grant (the dowager Lady Fife), who had fourteen children, remained at Rothiemay until she died at the age of 84.

James Duff (1729-1809)

James Duff, 2nd son of William Duff, probably the ablest and most energetic of the Duffs, was brought up at Rothiemay. He became 2nd lord Fife, and MP for Banffshire and then Morayshire for 36 years in the time of William Pitt and Edmund Burke, through the period of the French revolution and American war of independence. He had a great passion for trees - a French biography described him as "un grand agronome" -  he was one of the pioneers of afforestation in Scotland. He planted over 600 acres  of mixed woodland in Rothiemay and the surrounding district. Many of these trees were felled in the mid 1960's by H.D. Ward & Co. after they took over the castle estate.