New servers and file optimisation request!

[ixlone]

New servers with increased bandwidth are en route and should be up and running some point next week - until then service will be sluggish, so just hold out :x


Another way uploaders can help us out is by improved file optimisation. Smaller filesizes means we can not only store more files on our servers, but reduces bandwidth per chapter.

@blackbunii allowed me to share an example of some files I was looking at that he uploaded.

The chapter in question was around 55% bloated with unneeded metadata and colour space info.

I ran the files through Pngout (Pnggauntlet) to lossless - no detail change - optimise them. The result is the total filesize dropped from 166mb to 77mb.

To give you an idea of the bandwidth savings at the chapter read count 12 days ago:
Original 163mb is 0.92TB bandwidth used.
Optimised it would only be 0.43TB.


There are 3 fairly simple ways to improve file optimisation.

1. Use the export as png-8 in Photoshop when saving the files, rather than save as.

2. Pnggauntlet. Install https://pnggauntlet.com then drag and drop files into it and start it up. Pnggauntlet is slow, so expect it to take a while on larger res files.

3. Pingo. Install PingaGUI, drag and drop files and click optimize. Pingo is much, much, much faster than Pnggauntlet.

This download link.

These settings work well.

Thanks

 

[aerozext]

Well, I’m using Gimp and this is what I do:
Change into grayscale
Change into 8-bit

File size reduced by 40-60%

I used to use pngauntlet as well but lately I just do them manually...

 

[Pika]

@aerozext
PNGgauntlet throws away different metadata, so using if will shrink the files even more.

 

[Holo]

Just to add to ix's post - we're currently in the process of pingoing all the .pngs on the image servers. This lossless optimisation process reduces filesize because it strips out the unnecessary metadata and does not result in any loss of quality. This is going to take a while, we're looking at around 16 weeks in total, because there are almost 10TB to go through. By the time it finishes, we expect to save around 1.5 TB of data!

 

[Kendama]

I stopped using PNGGauntlet when I realised it's not completely lossless depending on the colour features of the original file. I ran several tests, including asking other team members to compare pics before and after optimisation on their screens to rule out a differences between in hardware. Even screenshots of the pages side by side showed differences, though they're subtle.

Nowadays I use this script in Photoshop:

===

var files = openDialog();

for(var key in files) {
var inFile = files[key];
open(inFile);

var pngOptions = new PNGSaveOptions();
pngOptions.interlaced = false;
pngOptions.compression = true;
app.activeDocument.saveAs(inFile, pngOptions, true, Extension.LOWERCASE);
app.activeDocument.close();
}

===

It basically implements Photoshop's maximal lossless compression and allows one to batch a whole chapter.

I found it in the web and a team member added one line for compression.

Obviously, we start off with 8-bit greyscale (for manga) at the PSD level. Files generated by this script are at most 10% bigger than the output of PNGGauntlet. I have tested that.

Edit:

It appears some people have asked how to use this script in some Discord servers our Ambassador Achyif lurks at. I believe the small difference (about 1-5% to Pingo and other optimisers) is that it doesn't discard some important metadata like colour space, but I haven't checked that myself. It's just a chunk of Javascript, so just treat it as such:

i. Copy and paste the script (everything between the === above) onto MS Notepad or another simple word processor. If you have a JavaScript editor or whatever, so much the better, skip ii and iii and just save as JSX.
ii. Save as TXT.
iii. Change the extension of JSX.
iv. In Adobe Photoshop, go to File>Script>Browse, find where you saved the script and double-click it. It will open the window to select the PSDs for saving. When you select them (they must be all in one folder at a time), click Open and that's it. The PNGs will be automatically generated and saved in the same folder as the original PSDs, with the same name.

It works with PSDs of any size I've tried so far.

People who are into mass releases may find it inferior to external optimisers in that it doesn't allow for processing multiple folders. I don't find that to be an issue because I upload one chapter while batching the next. It might be possible to embed the script into a Photoshop action and then use File>Automate>Batch to process multiple folders, but I haven't tried. My experience with Batching is that it's not as versatile as one might expect, but I'm not authority.

Hope this helps.

 

[BraveDude8]

@aerozext Run it through pinga as well, and it'll shave off even more. It's a nice one-click solution for people who don't want to do it manually.

@Kendama Gauntlet by default (or rather, PNGOUT specifically) isn't lossless, there's a flag in there regarding color type you need to change. pinga/pingo, as far as anyone I've asked can tell, is lossless.

 

[DemonOVERLORD]

lol, my chapters size never increased to 10MB, i wonder why? 🙄 because i do Chinese manhua and they only have 10-20 pages max. 😅

 

[Kendama]

@BraveDude8 Pingo improves exactly 0.9% on the script I posted. I'll pass.

https://www.dropbox.com/s/e12yy048dxg9vtq/captura%20de%20tela%202020-05-16%2013.07.19.png?dl=0

 

[pandascepter]

i dont understnad, but i am intersted😂 Thanks scanlator T_T you give me a good day.

edit : Thanks for the knowledge, Dood

 

[hyoretsu]

Finally, PNG compression advice. @ixlone @Holo Pngyu's also a good .png compression software, using 256 colors at High Speed (custom settings) cuts chapter's size in half. Then if you choose 16 colors it gets reduced substantially, (Approximately 1/3 of original size before compression) though sometimes it causes unnoticeable quality loss in some pages. Posted a full ch comparison of Pngyu. Apparently it has that same unnoticeable quality drop on colored pages though
There's also SuperPNG, a photoshop .png compression plugin, though I've never used that (Both of these use PNGQuant)

@Kendama Try Pngyu on 256 colors+High Speed (or High Quality, High Speed is really only needed for 2MB+ files). If you wanna go lower than that (16 colors for around 70% file size reduction) you'll have to use an image comparison software like Diffchecker to make sure there's no quality loss (There's an overlapping mode, if it's completely black they're identical)

 

[Kendama]

@hyoretsu Tested PNGyu at 256 highest quality, with the same file as above. It reduced to less than half the size… but screwed up the colours. Pass.

 

[ixlone]

@hyoretsu

Was trying to only recommend lossless or close-to-lossless-before-you-can-notice-less. Sounds like your ones are lossy compression.

 

[hyoretsu]

@ixlone All the 285 pages I ran through it on chs i released were lossless using 16/32/64 colors, all fully black on an image comparison software. (if 16 colors had any quality drop, restore backup and up a level) Although, again, all black and white pages. Only seen it fail given proper usage on colored pages, like Star Martial God Technique ch345 (Also .jpg's lossy on most settings afaik, some unnoticeable but still)

 

[Kendama]

@hyoretsu Well, I can imagine how it works well with B&W. Then again, it's unlikely to be superior to any other compressor for B&W. But a sizeable part of what gets released here is colour. It might work with simple webtoons that use solid colours, but it certainly isn't a universal tool.

 

[XXXXXXXXXIII]

Wouldn't it be easier to just convert and store lossy JPGs when user upload the chapter? (And face the outrage from scan teams

 

[Pika]

@XXXXXXXXXIII
That just defeats the purpose of mangadex, no?

 

[biribiri]

I've been using the right PS way pretty much since I switched to png for releases. I tested it a few times, but pngouting those take a lot of time, but only reduce 1-3% at best, if anything at all

 

[Lymus]

@biribiri pngu/pinga is remarkably faster, not sure what kind of black magic they use, but it's much better.

 

[Teasday]

_{@XXXXXXXXXIII posted:}

Wouldn't it be easier to just convert and store lossy JPGs when user upload the chapter? (And face the outrage from scan teams

Easier yes

Advisable no

 

[ixlone]

@XXXXXXXXXIII

Outrage!

I spend a lot of time cleaning my scans, adding all those jpg artefacts in would ruin them