A Downside to large HDD's 2tb+... Wasted space!!!

[nikeymikey]

As part of a new build/upgrade I am doing on one of my OG's I have recently bought a 6tb HDD. This is great for having lots of space for games, emulators etc etc etc.... 

However due to the cluster sizes need on these large drives (the bigger the drive, the bigger cluster size needed) you can end up with a LOT of wasted space! 

The drive was formatted using FatXplorer and the auto suggested cluster size for the 6tb drive. I foramatted with F&G partitions (old habits die hard lol) and they both had approx 2.7TB of space on each, lovely

I have F with the entire NTSC set along with all PAL/Jap exclusives all as CCI minus the incompatible few, they are extracted folders instead. Also have a folder for translated games and another for modded games. These are all xiso apart few a few that would not load as xiso. This is all good and I do not have much wasted space on this partition due to ISO/CCi being larger files. 

The issue comes with my G partition. I am using this for Emulators and Apps. 

So far I have a couple of Coinops sets plus a Ninja set on there along with a huge PS1 set. Below are the figures that windows properties displays for file size and actual size taken on disk.... 

Filesize first followed by size on disk.

Coinops 6 Adult - 1.79gb - 3.10gb - diff 1.31gb

Coinops 8 Adult -  4.25gb -  5.72gb - diff 1.47gb

PCSX-Xtras 1450 gamepack  - 392gb  - 394gb - diff 2gb

Emustation  - 358gb - 411gb - diff 53gb!!!

DosBOX Ultimate v9 - 94.2gb - 114gb - diff 19.8gb

Ninja - 250gb - 269gb - diff 19gb

Total wasted space = 96.58gb!!!!!

Before anyone says it, I know that there's a lot of duplicate roms etc between those sets.. As I had the space on my HDD I thought I'd add them all and see which ones I want to keep after testing  

So for obvious reasons the Playstation set does not waste that much space, the PS1 isos are bigger than the cluster size. But on the other sets where there are LOTS of tiny files there is much more wasted space. 

The recommended cluster sizes as i understand it are as follow...

EDIT - These are PARTITION sizes.

1-2TB - 128kb

2-4TB - 256kb

4-8TB - 512kb 

8-16TB - 1024kb

This means the bigger the HDD the more wasted space will occur when installing emulators/games with lots of smaller files.

My 6tb uses 512kb. SO every file that is less than 512kb in size will always take 512kb on the disk.  

EDIT - Although my drive is 6tb my F&G are 2.7tb each so are using 256kb clusters. 

Now I understand why this happens and I know there is nothing that can be done about it. I just thought I would post this as it was interesting to me how much space was actually being wasted, its a shame we cant format different partitions with differing cluster sizes (or can we???) as that would help keep space wasted down to minimum. 

EDIT - Added the latest C64 EmuXtras set v3.0 - Filesize 32.3gb - Size on disk - 44.9gb - diff 12.6gb

 

[HDShadow]

Are you using CerBIOS?

[nikeymikey]

27 minutes ago, HDShadow said:

Are you using CerBIOS?

Yes I am. The same issue will happen with the Titan patches for large HDD's too tho. 

[nikeymikey]

4 minutes ago, Thairanny said:

 

Ok........ but I was already aware of PrometheOS. Don't see what that has to do with my original post tho??? Its an alternate os for Xenium modchips, nothing it does so far changes the fact that larger HDD's need larger cluster sizes. 

[nikeymikey]

1 hour ago, Thairanny said:

I don't see the point in posting at all if you are not seeking to find a solution to the problem...

Mate you are missing the point.

That is an OS for a modchip. The subject I was talking about is to do with cluster sizes and large hdds. Smaller cluster sizes have issues with the larger partitions. It was the same when the max we could use was a 2tb drive. XBpartitioner would need to be used to format F&G with 64k clusters. 

I wasn't posting to find a solution because I know that there currently isn't one. I posted because it was interesting to see just how much space is wasted when you start going to clusters of 512k and above for 4TB+ hard drives. If a fix was to arrive then it would more likely come as part of a bios not a modchip OS. PrometheOS is similar to what X3 Config Live was/is for X3 chips but for Xenium based chips instead.

Tbh this is one thing that I would actually give in and buy Stellar for if they managed to implement a fix. In reality I don't think its actually possible though. If anybody actually knows different then please correct me

Kaos does a great job explaining it ion the 2 post in this thread...

This line says it all really..

"Smaller cluster sizes don't support access to the entire partition and will cause data corruption when you reach the last cluster and loop back to the beginning of the partition overwriting data."

[MadMartigan]

Not sure if this shines any light on this for you or not. You can always reach out to Kaos and see what he recommends. 

[KaosEngineer]

10 hours ago, nikeymikey said:

The recommended cluster sizes as i understand it are as follow...

1-2TB - 128kb

2-4TB - 256kb

4-8TB - 512kb 

8-16TB - 1024kb

You are off on the required cluster sizes.

Max supported Partition size | Cluster Size

256 GB | 16 KB

512 GB | 32 KB

1024 GB | 64 KB

----------These next few I'm not certain of since I've not used M8 Titan patched or CerBIOS----------

2048 GB | 128 KB

4096 GB | 256 KB

8192 GB | 512 KB

[KaosEngineer]

3 hours ago, MadMartigan said:

Not sure if this shines any light on this for you or not. You can always reach out to Kaos and see what he recommends. 

You beat me to posting the info.  However I'd not read down this far in the thread before I posted.

[Bomb Bloke]

11 hours ago, nikeymikey said:

SO every file that is less than 512kb in size will always take 512kb on the disk. 

There's a little more to it than that - rather, every file will consume a multiple of 512KB on the disk. So a 513KB file would eat 1024KB, for example, as that's the smallest multiple it'd fit in to.

You still end up wasting less space on average as you increase your file sizes, though, as larger files means fewer files, and fewer files means fewer partially-filled clusters being wasted.

 

Back in the day, ZipFS was considered to help deal with the slack space problem. (And the filename length problem, and the character restriction problem...) I can't remember which emulators ended up getting support coded in, but the basic idea is that you archive up a bunch of files in "store" mode (no actual compression) and mount that up as a drive. It's still not 100% efficient, but it's certainly better than dealing with the sort of slack you see with FATX. StepMania's "smzip" files are an example of this technique being put to use.

More recently it was pointed out to me that we could just cram all our ROM files or whatever into ISOs, and then mount those. This would allow us to use them with emulators that don't actually have ZipFS support. Also it'd probably allow us to mount much, much larger packages of combined files, as ISOs can be split into multiple 4GB parts, and I'm not sure if we can do that with ZipFS.

The catch to ZipFS and ISO mounting is that the volumes end up being read-only, so saves and config files still need to go separately into a FATX file system.

Many emulators can also load ROMs from SMB shares, so that's another work around to consider.

11 hours ago, nikeymikey said:

its a shame we cant format different partitions with differing cluster sizes (or can we???)

Sure we can. If you want to have a stonking great big volume with large clusters, sitting alongside a tiny little volume with small clusters, then there's nothing stopping you. In fact your C/E/X/Y/Z volumes should each already be running 16KB clusters anyway.

XBMC can also mount some of the extended volumes past number 7 (G), I think something like up to six extra past that? You should be able to limit your maximum cluster size down to 64KB by spreading things out a bit more.

[Bowlsnapper]

36 minutes ago, Bomb Bloke said:

There's a little more to it than that - rather, every file will consume a multiple of 512KB on the disk. So a 513KB file would eat 1024KB, for example, as that's the smallest multiple it'd fit in to.

You still end up wasting less space on average as you increase your file sizes, though, as larger files means fewer files, and fewer files means fewer partially-filled clusters being wasted.

 

Back in the day, ZipFS was considered to help deal with the slack space problem. (And the filename length problem, and the character restriction problem...) I can't remember which emulators ended up getting support coded in, but the basic idea is that you archive up a bunch of files in "store" mode (no actual compression) and mount that up as a drive. It's still not 100% efficient, but it's certainly better than dealing with the sort of slack you see with FATX. StepMania's "smzip" files are an example of this technique being put to use.

More recently it was pointed out to me that we could just cram all our ROM files or whatever into ISOs, and then mount those. This would allow us to use them with emulators that don't actually have ZipFS support. Also it'd probably allow us to mount much, much larger packages of combined files, as ISOs can be split into multiple 4GB parts, and I'm not sure if we can do that with ZipFS.

The catch to ZipFS and ISO mounting is that the volumes end up being read-only, so saves and config files still need to go separately into a FATX file system.

Many emulators can also load ROMs from SMB shares, so that's another work around to consider.

Sure we can. If you want to have a stonking great big volume with large clusters, sitting alongside a tiny little volume with small clusters, then there's nothing stopping you. In fact your C/E/X/Y/Z volumes should each already be running 16KB clusters anyway.

XBMC can also mount some of the extended volumes past number 7 (G), I think something like up to six extra past that? You should be able to limit your maximum cluster size down to 64KB by spreading things out a bit more.

I was thinking you could just have multiple 1GB partitions and spread your assets that way... although I don't know if it would really be worth it. I would be confused as hell.

[Bomb Bloke]

2 hours ago, Bowlsnapper said:

I was thinking you could just have multiple 1GB partitions and spread your assets that way... although I don't know if it would really be worth it. I would be confused as hell.

Multiple ~1TB partitions would be fine in my book. 1GB partitions are out of the question - the minimum cluster size is 16KB, allowing volumes of up to 256GB, so in terms of slack avoidance there's no point in making any extended partitions that're smaller than that.

 

[Bowlsnapper]

1 hour ago, Bomb Bloke said:

Multiple ~1TB partitions would be fine in my book. 1GB partitions are out of the question - the minimum cluster size is 16KB, allowing volumes of up to 256GB, so in terms of slack avoidance there's no point in making any extended partitions that're smaller than that.

 

Origins has it's content split across two partitions. That's easy to keep track of,.. but If we're talking about like, a 16TB HDD, that would be pretty overwhelming... Why is this an issue with the Xbox and not on PC for instance? Probably a very dumb question. Maybe space just IS wasted these days on our PCs and nobody talks about it. Does this happen on the 2TB HDD in my PS2, for instance?

[nikeymikey]

2 hours ago, Bomb Bloke said:

Multiple ~1TB partitions would be fine in my book. 1GB partitions are out of the question - the minimum cluster size is 16KB, allowing volumes of up to 256GB, so in terms of slack avoidance there's no point in making any extended partitions that're smaller than that.

 

That would be manageable if all useful Homebrew could see more than the F&G partition. A 6tb would need to go to I or J, but I suppose it could work as long as only games/emus went on the later partitions then XBMC could still see them and use the data inside. Although emus might struggle to see roms or saves etc that are on these later partitions. Anything needed by other apps would have to stay on E/F/G. I might have to buy another 6tb drive and test out the theory. 

I do like the idea of adding emu roms to an ISO file tho, Having just the saves etc as small files wouldn't be so bad. I may give it a try with a couple of emus and see whats what. 

[Bomb Bloke]

2 hours ago, Bowlsnapper said:

Why is this an issue with the Xbox and not on PC for instance? Probably a very dumb question. Maybe space just IS wasted these days on our PCs and nobody talks about it.

It does happen, and the Windows File Explorer very helpfully points it out: open the Properties panel on a given file or folder, and you'll see a "Size" figure (representing the total amount of data selected) and a "Size on Disk" figure (representing the total size of the clusters that data is allocated across). The difference - the "wastage" - is your slack space.

The trick is that most consumer PCs these days have their hard drives formatted using modern file tables such as NTFS, which can handle a lot more clusters per volume than FATX can. This means that larger volumes can still have smaller clusters - pretty much always just 4KB. You don't get so much slack that way.

For really small files - like a few bytes - NTFS doesn't even bother to assign any clusters at all, instead simply writing the data directly where the addresses for the clusters would go! In these cases, the reported "Size on Disk" is 0! (Though technically your file index still gets larger - Windows just doesn't bother to report on that.)

1 hour ago, nikeymikey said:

That would be manageable if all useful Homebrew could see more than the F&G partition.

Good point, I went and forgot about that. I'm not entirely sure how well the average emu handles ROMs stored on volumes 7+, even if they're in the emu's own folder.

[nikeymikey]

9 hours ago, KaosEngineer said:

You are off on the required cluster sizes.

Max supported Partition size | Cluster Size

256 GB | 16 KB

512 GB | 32 KB

1024 GB | 64 KB

----------These next few I'm not certain of since I've not used M8 Titan patched or CerBIOS----------

2048 GB | 128 KB

4096 GB | 256 KB

8192 GB | 512 KB

I went of a discord post that I believe was in the Cerbios discord.... When formatting my drive with FatXplorer it automatically selected the cluster size before formatting and that matched with the discord post so I went with it. 

 

13 hours ago, MadMartigan said:

Not sure if this shines any light on this for you or not. You can always reach out to Kaos and see what he recommends. 

Also unless I'm having a mental block today and going mad. The last 4 in the info above match up with what I said in my first post? I didn't mention the smaller partition sizes as that should be common knowledge by now lol

My drive is 6tb and is using 512kb clusters as it seems it should be

[Bomb Bloke]

38 minutes ago, nikeymikey said:

My drive is 6tb and is using 512kb clusters as it seems it should be

You would indeed require 512KB clusters for a volume 6TB in size.

But your largest volumes are only 2.7TB, right? Those two would only require 256KB clusters.

[nikeymikey]

13 minutes ago, Bomb Bloke said:

You would indeed require 512KB clusters for a volume 6TB in size.

But your largest volumes are only 2.7TB, right? Those two would only require 256KB clusters.

Ah yes you are correct, Hmm I will need to check what size they actually have. FatXplorer did all the setup and formatted the partitions with its suggested cluster sizes so maybe my F&G are using 256k as you stated. 

 

EDIT - As expected my 2.7gb F&G are both using 256kb clusters. I edited the first post to reflect this info

 

[Bowlsnapper]

11 hours ago, Bomb Bloke said:

It does happen, and the Windows File Explorer very helpfully points it out: open the Properties panel on a given file or folder, and you'll see a "Size" figure (representing the total amount of data selected) and a "Size on Disk" figure (representing the total size of the clusters that data is allocated across). The difference - the "wastage" - is your slack space.

The trick is that most consumer PCs these days have their hard drives formatted using modern file tables such as NTFS, which can handle a lot more clusters per volume than FATX can. This means that larger volumes can still have smaller clusters - pretty much always just 4KB. You don't get so much slack that way.

For really small files - like a few bytes - NTFS doesn't even bother to assign any clusters at all, instead simply writing the data directly where the addresses for the clusters would go! In these cases, the reported "Size on Disk" is 0! (Though technically your file index still gets larger - Windows just doesn't bother to report on that.)

Good point, I went and forgot about that. I'm not entirely sure how well the average emu handles ROMs stored on volumes 7+, even if they're in the emu's own folder.

NOW I finally understand what "Space on Disk" means. I ALWAYS wondered that. Thank you.