The house was originally known as Captain Dow's house, a mean low dairy house. Robert Vining, a prominent local builder, rebuilt the house as Elwell.  Robert Vining was one of the original builders of the Esplanade and personally reconstructed the same after the great storm in 1824. In 1998 this was commemorated by a plaque on the esplanade after the original Tempest Stone was lost.. After his death the house was rebuilt as it is now in 1854 and called Elwell Lodge. Towards the end of the nineteenth century it was renamed Elwell Manor.

From 1880 to 1920 the house was occupied by the Speke family. Hugh Speke was a local JP. In 1927 it was bought by Vice Admiral Warren Hastings D'Oyly (Ret.). At this time Portland harbour was an important naval base. The house remained a family home up to the second world war when it became a nursery school. It reverted to a family home in 1958.

Most of the land was sold in the 1950's for building and the working area of the house, housing kitchens, stores and cellar, was sold in the early 1960's and converted to flats.

Elwell is number 192 on this map