During the 1863 Draft Riots, the house was attacked and set on fire due to the family's anti-slavery stance. James Sloan Gibbons and his daughter, Lucy Gibbons Morse, were in the house when the inferno began. Abigail Hopper Gibbons was in the South with a Union Army regiment serving as a volunteer nurse. The Gibbons family escaped across the rooftops of neighboring homes. The house remains a vital Civil War-era landmark and a symbol of resistance against slavery.