AT Stewart opened his first dry goods store with a $10,000 inheritance from his Irish grandfather. Stewart turned out to be a natural salesman. The store became known for good prices and customer service. Stewart became a millionaire after the Panic of 1837 when he was able to buy the inventory of other merchants who had financial reverses.By the mid 1840's the store was so successful that Stewart needed to expand. He assembled Broadway which became the first Department Store and the first building in New York clad in Tuckahoe Marble, from which it got its nickname. Over the next few years there were several 'modular' additions until it became the largest retail store in the world. Innovations included a women’s department where women could try on imported clothes, a fur department, large glass windows where people could see the merchandise, and mail order which he started in 1868.The store began on the corner of Broadway and Reade, and gradually expanded down Broadway and across Chambers Streets, with additions in 1850-51 and 1851–52, both designed by Trench & Snook, and 1872, designed by Frederick Schmidt.
A.T. Stewart store moved uptown, the building was converted into warehouses.
In 1884 Judge Henry Hilton, AT Stewart's Executor, got Edward D. Harris to design an addition, in which two floors were added to the original five, and the building was converted into offices.
It moved to a cast iron building on Broadway at the southern end of Ladies Mile. The Newspaper "The Sun" was moved into the building in 1917 until its end in 1952. The corner "Sun Clock" is a local landmark.
In 1966, New York City acquired the building to be demolished for a plan to redevelop the Civic Center which did not come to fruition.The building was rehabilitated from 1995 to 2002, overseen by Beyer Blinder Belle. The city's Department of Buildings uses the upper floors, and the first and second floors are used for retail stores.