He died without issue in 1776 and was thus the last in the 2nd Creation Earl of Lichfield. The estate and title eventually reverted to his niece Lady Charlotte Lee, eldest surviving daughter of his brother the 2nd Earl of Lichfield.
In 1744 Charlotte had married Viscount Dillon. Their son Charles Dillon, 12th Viscount Dillon inherited the estate of Ditchley but not the title. Ditchley remained the home of the Viscounts Dillon until 1934.
The title was created for a third time when Thomas Anson was created Earl of Lichfield in the 1831 coronation honors of William IV.