To protect the monastery and the town, Abbot Kuno I had the "Burg ob Ellwangen" (Castle at Ellwangen) on what is today the palace mountain, probably around 1200. It served as a residence and a symbol of power. The residence was characterized by the worldly royal household of the abbots, who were raised to the rank of imperial princes in 1460. The castle is mentioned for the first time in a document in 1266 as "castrum Ellwangen".
In the 17th century the Prince Provost Johann Christoph I of Westerstetten (1603-1613) took the for structural alterations. The start of building of the four-wing complex with three corner towers on the main building and a courtyard with three-sided, triple arcades in the style of the Late Renaissance fell in the years 1603-1608.
In 1687-1689 the South Wing over the Chapel was decorated. In the third building phase, after a major palace fire, Ludwig of the Palatinate had the palace remodeled between 1720 and 1727 in the style of the Baroque under the architects of the Teutonic Order Franz Keller and Franz Josef Roth. It was given a uniform mansard roof and new representation rooms in the south wing with a Prince's Hall (Fürstensaal) and a staircase built onto the front.
The palace numbered over 83 rooms. The main rooms of the prince provosts lay in the south and east wing. The bedchamber of state with an antechamber and the cabinet used as a study were located in the east. Twelve portraits of the prince provosts and a ceiling fresco by the Asman student Christoph Thomas Scheffler (1728/29) adorn the banquet hall (Festsaal) in this wing. The Marshall Hall (Marschallsaal) located opposite is the worldly companion piece to the Banquet Hall. The Marshall, as the head of the princely royal retinue, resided here.
The entire arrangement of rooms followed the Vienna court ceremony: The staircase of state and the banquet hall are located in the center, flanked by the bedchamber, cabinet and private chambers.
On the lower floor and the mezzanine lay the silver chamber and archive, the bath chamber, kitchen, bakery and confectionery.
In the course of the Secularization of 1802/1803 Elector Friedrich made the palace his residence. The dinning hall of the prince provosts became the throne room of Friedrich, who had risen to King Friedrich I of Württemberg. Between 1803 and 1806 the government of New Württemberg was located in the palace. Then Friedrich united all offices of his kingdom in the unification of Old and New Württemberg in Stuttgart.
In 1815 and 1816 the banished King of Westphalia Jérôme Bonaparte, Napoleon's brother, and his wife Katharina, the daughter of the King of Württemberg, furnished individual rooms in the palace. To make palace life comfortable, they had the galleries fitted with windows. The resulting halls were heated with transportable ovens.
Beginning in 1842 a final change in the use of the palace was carried out. An agricultural school moved in, and individual state government agencies followed in 1849. The majority of its valuable furnishings and its art treasures were sold at auctions. The most completely preserved furnishings were those of the Court Chaple (Hofkapelle).
Important pieces like the "Ellwangen Cabinet Cupboard" (Ellwanger Kabinettschrank) from 1670 went to Stuttgart. In 1994 it was returned to Ellwangen again by the Administration of the State Palaces and Gardens and can be viewed today in the palace.