April 6th, 2021
After putting in two and a half weeks of effort towards v5, we have a good sense of where we are in terms of progress towards getting the site back up. Things did not go as smoothly as we dared to hope, but significant progress has still been made.
The backend Symfony API that will support search (exciting), authentication, and the creation, retrieval, updating, and deletion of users, manga, chapters, follows etc is almost complete with optimistic estimates towards getting it live this weekend for testing. This gives mobile apps a chance to code for the new API if they'd like, but they should also be wary that the API is subject to change as we develop the frontend.
The VueJS SPA frontend won't be up this week, but getting it up within the next two weeks is our ideal goal. To accelerate this estimate, we are once again accepting offers for help. If you have experience with Vue 2, Nuxt, and Vuetify specifically, we would love to have you on board to help.
We're currently using Vuetify to hasten MVP development but future v5 design would use a CSS framework rather than a UI library like Vuetify as well as migrating to Vue 3. We have a fairly complete design document to follow, all you would need to do is implement it. If you're interested in assisting with development of the initial MVP, join our Discord server and DM Plykiya#1738. We'll likely only accept a few so as to avoid having too many people attempting to work on the same thing at the same time, but in the future the frontend will become open-source for all to contribute to.
Sorry for the continued wait, we're just trying to do things right the first time, not the second or third.
March 21st, 2021
Due to a recent hacking incident, MangaDex will be down until further notice.
Instead of keeping up a likely vulnerable website and wasting our time and efforts playing cat-and-mouse with constant attacks from DDoS to hacking, we have decided to take this opportunity to refocus and expedite our planned rewrite of the site, called v5. Contrary to our original plans, however, we will be launching this v5 as soon as the minimum essential features are ready.
As developing and maintaining MangaDex is nobody's actual job, it is difficult to give an accurate estimate as to when we'll be back up and running. It should go without saying that every one of us wants it to happen as soon as safely possible.
That said, if everything goes as smoothly as we dare to hope, we could be looking at a downtime of just a week or two. Or three.
For up-to-date news about our progress, please follow us on Twitter.
In the meantime, please take the time to read this full write-up of what happened, what our options for plans of action were, how the data breach may have affected you, and how you may be able to help by disclosing vulnerabilities.
All timings are in UTC time.
1. A brief recap:
Three days ago (2021-03-17), we correctly identified and reported that a malicious actor had managed to gain access to an admin account through the reuse of a session token found in an old database leak through faulty configuration of session management. Following that event, we moved to identify the vulnerable section of code and worked to patch it up, also clearing session data globally to thwart further attempts at exploitation through the same method.
After the breach, we started spending many hours reviewing the code for possible further vulnerabilities, and started to patch what we could find to the best of our capabilities. This ran parallel to us opening the site after the breach, as we had incorrectly assumed that the attacker would not be able to gain further access. However, as a precaution, we had started rolling out monitoring of our infrastructure and had remained vigilant in the event the attacker returned.
2. Why did we go down again?
At 2021-03-20 01:52:48, the attacker had managed to access the account of one of our developers who had been previously offline for four days. However, this time around we noticed this immediately and shut the site down at 01:53:40 to investigate further.
At 2021-03-20 02:10, the attacker had sent an email out to the first ten users with the message body, “MangaDex has a DB leak. I suggest you tell their staff about it.” abandoning any pretenses of ransom. Moving forward, while we have no clear evidence that a database breach had happened, for best security practices, we will assume it has happened.
At 2021-03-20 03:41, the attacker had updated the git repository containing the source code leak, claiming that we had successfully patched two out of three possible CVEs. Without any way to confirm the claims, we assumed the worst case scenario and kept the site down to further investigate.
3. What have we done since then?
As of writing, we have invited numerous volunteers to assist our developers with identifying the last possible CVE claimed by the attacker in the codebase. Thanks to our volunteers, we have identified a good number of potential security flaws and moved to rectify them. However, at time of writing, we have still yet to identify the last possible CVE claimed by the attacker.
With that knowledge in mind, we were confronted with a difficult decision. If we had assumed incorrectly that the web code is now secure, we could end up being compromised again by the attacker. As a result of that, in good conscience, we could not possibly re-open the website to users presently.
Lastly, our staff consists of volunteers. Volunteers with real life commitments and duties that do not earn a single cent from volunteering for MangaDex. While we aim to provide the best service we can to you, the repeated attacks were starting to take a toll on us all, having to repeatedly scan through thousands of lines of code trying to find a figurative needle in a haystack. We have evaluated our choices on hand and have decided this is unsustainable to both our users, and ourselves.
4. What are we planning to do now?
Seeing as the attacker has no intention of helping us to resolve the security issues and is instead more keen on causing maximum disruption to MangaDex, we have decided to keep the site offline till we are confident in its security. We considered a number of options on hand, namely:
- Bring the site back in its (potentially vulnerable) current state, and continue watching for signs of more attacks. We decided against this as it could lead to more emergency downtime, which would be frustrating for our users as well as our staff.
- Bring the site back in a nerfed/read-only state, making it impossible for the attacker to make any further changes. We decided against this because this would mean that the public would not be able to upload, and only our moderators could, which would place a large burden on them.
- Gut the site of most of its features such that only essential, non-abusable features remain. However, the time spent doing so would be better spent on v5, so this is not a sensible option.
- Close the site until v5 (the total site rewrite) is completely ready. As mentioned previously, the attacker has access to the v3 code, so this option would be relatively more secure. However, given the current progress of v5, this would mean that the hacker will have successfully deprived the community of manga for a longer period of time, which is most likely the hacker’s motive at this point (to force us offline).
- Close the site until a barebones version of v5 is complete. This would only contain the minimum essential features, namely to allow readers to read, follow, and groups to upload, much like how v1 of MangaDex was originally released (for those of you who have stuck by us since then) but using the same technologies we’ve planned for v5.
We have decided that option (e) would be the best approach, as it strikes a good balance between downtime and working to bring the site back up in a usable and (most importantly) secure state.
5. Data Breach & You
While we have numerous signs that the attacker had access to information not typically visible from the context of a normal user, we have not been able to confirm a full host compromised, or an up-to-date database breach. We intend to continue to keep a close eye on both and aim to update as we investigate and discover further. Moving forward however, it is in both our users’ interest and ourselves that we will consider the database breached.
As a user, we will encourage that you would assume that your data has been breached, and take precautions immediately, such as changing the passwords of any accounts that might share the same password as your MangaDex account. As a generally good security practice, password managers are highly recommended to keep your online identity secure.
In the meantime, we are still open to any suggestions or responsible disclosures of vulnerabilities found in the leaked v3 source code. While we have found numerous at time of writing, and have moved to patch most of it, we appreciate all attempts at helping us to find more. For more information, or for disclosures, please kindly approach a staff member on our Discord.
7. Bug Bounties
Moving forward from this incident, we sincerely intend to improve upon the security on existing and future infrastructure, and while some of our developers have experience in the security fields, we have decided that having some form of a bug bounty program for v5 will only prove to be beneficial to MangaDex. As means of backing that, we intend to consider payouts depending on the severity of reported bugs. More details to be released in the near future.