The first mention of the village of Skalice Hor Stříbrný dates back to 1361 under Ctibor of Skalice, who also had the patronage right to the church of St. James in Rovná. It is not documented whether the castle existed earlier (according to archaeological findings, however, there was a settlement on the castle hill as early as the 13th century) or whether it was built under this owner or his son Střížek of Skalice, mentioned in 1376, or Kuneš of Skalice, mentioned in 1377.
However, its origin is undoubtedly connected with the boom of silver mining in the vicinity of the town. The next owner of the castle was Ješek of Skalice from 1384 and at the turn of the 14th and 15th century the goods from Skalice came into the possession of Bohdal of Drahonice, who also used the predicate from Skalice. The first written mention of the castle is from 1402 and refers to its conquest and destruction by King Wenceslas IV's brother, Sigismund. This was during the second captivity of the king, because at that time the castle was controlled by the royal governor and Václav's favourite and adjutant, Racek Kobyla of Dvorc. After the siege of the castle, Racek Kobyla fled to Rataje nad Sázavou, and when the besiegers found the castle empty, they at least razed and burned it; most of the town also succumbed to the fire. This is also the end of the short history of the castle, because when in 1403 King Wenceslas II gave the castle to the town of Racek. Skalice to Jan Sokol of Lamberk, the castle was never rebuilt. The last mention of the remains of the castle is in the town directory of the Sázava Monastery in 1677. In 1730, the construction of the Church of St. John of Nepomuk was started in the area of the former castle courtyard.
The form and extent of the original castle forecourt cannot be determined, but thanks to the unambiguous layout, the actual form of the core can be determined. The castle was rectangular in shape, the NE corner of which was secured by a rounded bergfrit, the palace with the entrance gate was on the west side and the whole area was thoroughly fortified with a wall and a parcan. The remains found suggest a similar layout to the castle in nearby Talmberk. The last remnants of the castle gate at the entrance to the church, partial remains of the northern wall and the stone wall of the parkland with the entrance to the Gothic cellar are still preserved.