The following was compiled by Mimi Miller, Historic Natchez Foundation, c.1986:

The Elms is associated with important territorial figures John Henderson and Lewis Evans and is a significant expression of territorial period architecture. Construction probably began on the The Elms shortly after the property was acquired by John Henderson in 1804 (Adams County Deed Book D, p. 50), since Henderson described the house as "nearly new" when he advertised it for sale in 1810 (Natchez Chronicle, April 10, 1810). The Elms was originally a two-and-a-half story brick residence with front and rear two-story galleries and access between the floors provided only by gallery stairs. Each story of the house contained two rooms, with the most formal rooms probably originally located on the second story. The house was soon enlarged by the enclosure of the upper and lower rear galleries, but it is not known whether the present encircling galleries were constructed during the territorial period.

The Elms was significantly enlarged in the 1850s, but the territorial character of the house is still intact. Most of the original Federal style millwork survives, with the third-story mantel pieces retaining granite marbleizing similar to the marbleizing on one of the early mantel pieces at Oak Grove in Church Hill, Mississippi. Like other brick houses built during the territorial period, the bricks, now obscured by stucco, were laid in Flemish bond.

John Henderson, a Scotchman who came to Natchez in 1787, was a merchant and an important figure during the territorial period. Henderson's literary work, PAINE DETECTED, OR THE UNREASONABLENESS OF PAINE'S AGE OF REASON, published in 1797, is the earliest known literary work in the Mississippi Territory. Henderson served as postmaster of Natchez during the first decade of the nineteenth century, the first auctioneer and general commission merchant in Natchez, and was instrumental in the organization of Pine Ridge Presbyterian Church and the First Presbyterian Church of Natchez (Goodspeed's BIOGRAPHICAL AND HISTORICAL MEMOIRS OF MISSISSIPPI, I, p. 906; THE WEEKLY CHRONICLE, May 6, 1809; and Claiborne's MISSISSIPPI AS A PROVINCE, TERRITORY, AND STATE, p. 202).

Lewis Evans, who purchased The Elms in 1815 (Deed Book H, p. 300), was a builder who may have made the alterations to the house during the territorial period. Evans was one of the leading men of wealth and influence in the territory, territorial assessor, and sheriff, and contractor for the 1818-1820 East Wing of Jefferson College (James, ANTEBELLUM NATCHEZ, p. 84; and Dawn Maddox, "Buildings and Grounds of Jefferson College in the Nineteenth Century," pp.40-41).

This building was individually listed on the National Register on 7 November 1976, and it was later included as element #469 in the Natchez-on-Top-of-the-Hill Historic District, which was placed on the National Register on 17 September 1979.

It is included in "A Guide to Early American Homes – South" (1956), "Shrines to Yesterday" (1968), "The Majesty of Natchez" (1969/1981/1986) (p. 51), "Old Homes of Mississippi, Volume I: Natchez and the South" (1977) (p. 40), the "Natchez Walking Guide" (1985) (#74, pp. 60-61), "The Great Houses of Natchez" (1986) (pp. 34-35), "Classic Natchez" (1996) (pp. 60-61), "Natchez Images, 1880-1960" (2002) (p. 52), and "Louisiana Architecture 1714-1820" (2004) (pp. 172-173).