I attribute this to market forces, with both sides of the supply/demand equation potentially driven by different things,
On the demand side, I think some of it may be due to a tendency for a "rare" or expensive item to increase in price because of a collector's mentality. Some collectors are interested in an item simply because it is expensive, under the logic that if it is expensive, it must be good, and can only increase in price. For these people, ever-escalating prices are just further proof that an item is awesome, and no price is too much to pay. A great example of this is tulipomania, where tulip prices rose exponentially due to speculation, then crashed spectacularly. I wonder if we'll get a good test of this when the Goemon CD box set is released. With these CDs becoming more widely available, how will this affect the prices of the originals?
Another component on the demand side is popularity of a particular collecting hobby. As a hobby becomes trendy, more people become interested in it, and many are looking for "instant awesome". They don't want to fool around the experience of filtering through the drek, and are just looking for the "best" or the "hidden gems". As others respond with their favorites, a list begins to be formed which summarizes the community's favorites, which causes demand for these particular titles to skyrocket. Since these lists can persist forever on the internet, the demand can be constantly refreshed as more people enter the hobby, but the supply dwindles as those people who have the title are unwilling to sell it. This can be countered by re-releases, though for older releases, this is generally unlikely. Even if there is a re-release, some collectors will tout their original release as being superior for whatever reason.
Others can be driven by a focus on completionism, whereby the last few items needed to complete a set become ridiculously expensive, especially if print runs were low. A great example of this is the US TurboGrafx catalog. There are somewhere around 150 titles for it, but the print runs of the last few games were very low. Because the catalog is so small, trying to get a complete set seems like an attainable goal, and is a popular target for collectors. However, current market prices for complete copies of Legend of Hero Tonma are over $600; Bonk 3 CD, Super Air Zonk, and Dynastic Hero can be over $1,000; and a complete copy of Magical Chase on eBay right now has bids exceeding $5,000. Moonraiser sold his loose copy of Magical Chase for almost $1,500 a few weeks ago. If people lose interest in this goal or this hobby, the prices can decline,
On the supply side, large price increases may be driven by a reseller's mentality. When something already has a low supply, some people can try to "corner the market" by buying up every copy they can and holding on to it to resell it later at inflated prices. For recent examples, see the NES Classic console and the early days of amiibo. This can be countered by the producer releasing more copies, which eventually happened with amiibo. If it is out of production, that is obviously less likely to happen. Going back to Magical Chase, I remember reading a thread from 2001 or so which was describing someone buying up every single copy that came on eBay. At the time the thread was written, that person had already bought at least 20 copies, at prices in the $100 range. If he held on to them until now and unloaded them, he could make quite the score.
In any case, prices can rise and fall drastically at a moment's notice. Back when I first started buying VGM CDs, copies of Dracula New Classic were topping $1,000 on eBay (and YJA prices were several hundred dollars, as people were buying them on YJA to resell them on eBay). Now, you can get pristine copies for under 5000 yen.