Until 1661, the entire area on the north side of the river Jølstra was one large farm called "Førde med Skei". The owner was the Apostle Church in Bergen. In 1661, the wealthy Førde priest Peder Finde bought the farm and divided it into two separated farms: Skei and Førde Rectory. Skei was subsequently divided into two farms: Indre and Ytre Skei. The three farms were owned by Peder Finde and his descendants for a long time afterwards.

The Finde family owned Ytre Skei until 1741, when Captain William de Coucheron bought the farm. It marked the start of 200 years of military officials at Ytre Skei. This period ended when Captain, and County Governor, Ingolf Elster Christensen died in 1942 and it was taken over by his daughter Othilia and her husband, Nicolai Schei. They owned the farm until 1991, and Ytre Skei is still owned by their decendants.

Both Ingolf Christensen and his son-in-law Nikolai Schei were county governors of Sogn og Fjordane. The farm is therefore often misleadingly referred to as the county governors’ farm.

Another well-known individual at Ytre Skei was the writer Hjalmar Christensen (1869–1925). He was the brother of Ingolf. Even though he lived most of his life elsewhere, he was always strongly connected to the farm. In 1920 he set up a writer’s annex between the main building and Kyrkjevegen.  The building can still be seen. Hjalmar Christensen's position as a recognized author became evident when the publishing house Aschehoug gave Christensen a grant of 9,000 kroner to have the annex built.