His sword skills must be at least somewhat decent for him to have succeeded to the point of the final test, and not been disowned before taking the final test to pass onto the superior sword skills. Throughout his life on the mainland, as I have stated before, He didn't even attempt to better himself in any way. He knew what would happen if he failed, and he failed. On the island he could've chosen to be a farmer or something of different standard (I would assume not many trade ships go to the demon-inhabited island). Off the island, I'd assume he could not be a brat and do something other than picking flowers. Nothing you said has changed anything, in fact by you saying that THE MC IS A LITERAL FU%&ING RAPIST, you're lessening your point further. Not that you had one.
This manga is just a power fantasy made by some people who wanted to latch onto the success of revenge plots like "The rising of the shield hero" or something else. The problem with this blatant ripoff is that it's worldbuilding is so horrid it makes the MC to appear to just be a deranged, selfish, lunatic. He has no notable positives other than his, what you seem to be calling "mediocre" sword skills and doesn't try at anything or use what little skills he has to achieve anything.
The problem with the guild letting off citizens who do crime is the guild masters fault, he can run his guild however he pleases and he isn't the people doing these crimes. The MC may have a LVL 1 cap but what does that matter? Does he not even try to make himself smarter? Try to read books or gain skills not some dumb RPG skills, actual life skills for anything? No. Of course he doesn't, because he's a swordsman who made a vow to his dead mother, a mother who would totally be ok with this kid who is positively unqualified in anything picking flowers for a decade, going on a power trip and raping women weaker than him?
The people of this world are badly written. They exist solely to make the MC seem pathetic just so he can get super powerful and beat everyone around like a sociopathic nonce.
He is an objective bad person, and you should feel bad for trying to justify his psychotic actions.