Sugar House Prison Window

At 5-7 stories, sugar houses were constructed with thick masonry walls and stood taller than most contemporary buildings. At least 5 sugar refineries were erected in lower Manhattan in colonial times. Later, refineries of 10 stories or taller were built near the waterfront. ^[1] 

During the Revolutionary War, Sugar Houses in New York City were used as prisons by the British. Out of 2,600 prisoners captured during the Battle of Fort Washington, 1,900 died in makeshift prisons across the city, with an estimated 17,500 perishing overall during the war, surpassing battle casualties. These warehouses were initially used to store molasses and were owned by prominent merchant families. Constructed by merchant families for sugar storage, notable ones included the Bayards, Cuylers, Livingstons, Rhinelanders, Roosevelts, and Van Cortlands. While some sources dispute their use as prisons, locals believed them haunted. Conditions in the prisons were dire, with overcrowding, moldy rations, and scarce medical supplies. Rations of pork and sea biscuits were often moldy and worm-infested, yet prisoners seldom refused food. Prison conditions were dire, initially housing 40 to 50 prisoners but swelling to 400 to 500.  Deceased prisoners were collected daily by a dead cart.