One reason I find Japan fascinating is they do their own thing, they don't give an F what other people are doing. This aspect is simultaneously a good and bad thing.
Regarding the crappy quality downscaled item photos, that's probably because Japan doesn't really care about personal computers and monitors so the images have always been small and "optimised" and "good enough" for phone displays. Compared to other First World countries, PC ownership in Japan is relatively (or significantly) lower. I'd cite some statistics except these days, the line between phone and PC (plus tablets, laptops etc) is a lot more blurred and I can't be bothered doing the research and checking the methodology of the studies. Long story short, if Japan is viewing Y!J on phones, then there's probably not much demand for better quality photos. For the record, Y!J photos annoy the shit outta me. Though some sellers have the sense to embed better quality photos in the description section.
Regarding eBay's lack of success in Japan, I had a look at the Yahoo!Japan Auctions Wikipedia page which is surprisingly brief, considering the subject. According to that page, Y!JA was set up "in 1998 to compete against eBay", which implies that eBay started first and Y!JA came later. According to eBay's Wikipedia page, Yahoo! Auctions had a "head start" before eBay, and according to this article, eBay "entered the Japanese market in 2000" before giving up in 2002.
So that's the most likely reason. Y!JA started the online auctions market in Japan first, grabbed all the buyers and sellers, and by the time eBay arrived in two years later, it was too late. Having two sets of online market stalls (i.e. eBay and Yahoo!) is basically a waste - the whole point of auction sites is to have all buyers and sellers in the same place - so in this case, the natural monopoly dominates. eBay didn't just fail in Japan but other Asian countries too. Sure, there are millions of Chinese eBay sellers on eBay but when's the last time a Chinese buyer bought from you? I average like one or two sales to China a decade. Interestingly enough, eBay also failed in New Zealand (non-Asian first world country) where they have their own auctions site.
Also, let us not forget about PayPal, which is an integral, inseparable and compulsory part of eBay. Japan continues to not follow the trend by having very low PayPal adoption rates in comparison to other countries. According to a PayPal employee, Japan has "1 million active consumers" as of 2015 (out of a population of approximately 130 million) and will "grow" in "double digits" (hah!). In comparison, Australia (a reliable trend follower) has 5.6 million active accounts in the same year out of a population of only 23 million.
Based on my experience, Japanese Y!J Auction users seem to use mostly bank deposits and Yahoo!かんたん決済 (Yahoo! Easy Settlement), the latter of which I think is an umbrella for various types of PayPal-like online payments. So basically, since Yahoo!Japan Auctions was set up to suit Japanese tastes, preferences and sensibilities, and since they already had their own PayPal-like online payments service, this double whammy (established online auctions site plus established online payment methods) means that eBay never stood a chance.
Finally, how about the Japanese people themselves? I don't really get any feelings of Japanese people wanting to buy from overseas using eBay and to a lesser extent and one I have more experience with, most Japanese sellers don't seem to want to sell to overseas customers for reasons I can't imagine. They would even go as far so to block auction intermediaries from bidding on their products. It's like they don't want more money for their stuff. Language barrier maybe? Perhaps Japanese sellers don't want to deal with any post-sale BS from dirty gaijin who can't speak the language. Even if eBay did do everything right in the late 90ss/early 00s, what's the point if their was no demand for eBay in the first place?
"If it ain't broke, don't fix it" - A proverb that I think very much describes Japan/Japanese people in general. Or their localised version: 「触らぬ神に祟りなし」 "No curse from the God you don't touch".
Edit: Apparently Avex aren't allowing exports of anime to outside Japan so that further proves the point that Japanese sellers don't want money (unless there's some sort of rationale that makes this decision financially sound?).