In 1807, Aaron Burr proposed developing three new streets through his Richmond Hill property, but financial troubles prevented him from completing the project. After mortgaging 240 building lots, his creditors sold the estate to John Jacob Astor, who resold the lots with leases extending to 1864, securing future rights to the land.
In 1820, Astor moved the mansion to the corner of Varick and Charlton Streets, where it reopened in 1822 as a public resort. It later became a theater in 1831, briefly served as the Italian Opera House, hosted equestrian shows, and eventually operated as a saloon before being demolished in 1849. The surrounding neighborhood, developed by Astor with modest brick rowhouses, was also known as Richmond Hill and became connected to the growing city along the newly extended Canal Street.