His story begins with the royal family conspiring against him to destroy his life. Why would it be unreasonable for him to have zero trust in royals, and to suspect every move of them as some shady plan or trap?
It's not like Melty could lift the ban, anyway - that order was probably coming from the King, who outranks her.