Optimize and/or trim CPR images

[Solo Kazuki]

Is possible to optimize and/or trim CPR images? Because almost all have 256 or 512 kB when data is sometimes far less.

[GUNHED]

Yes, they can be truncated. Somewhere there is software in this forum.

[Solo Kazuki]

Yes, they can be truncated. Somewhere there is software in this forum.

Can You say more about this?

[eto]

Is possible to optimize and/or trim CPR images? Because almost all have 256 or 512 kB when data is sometimes far less.

I used to do it a few times to put more than one game on a (multi-)cartridge and the following process worked for me:

• convert CPR back to a pure BIN with CPR Tools
• check with a hex editor that this file doesn't have any data above 128K or 256K. 
• Once that is verified, just cut the BIN to the right size (if not possible with the hex editor, then using a file splitter)

I think when you convert the reduced BIN now back again to CPR the file size should stay close to 128K or 256K (with a bit of overhead).


May I ask why you want to do that? 

[Solo Kazuki]

May I ask why you want to do that?

To trim CPRs that they would be closer in size to exact data contained.

[eto]

To trim CPRs that they would be closer in size to exact data contained.

Sure... but where does this help? Even a 4GB USB Stick or SD-Card is providing much more space than a full set of all available CPRs would ever need. 

[Solo Kazuki]

To trim CPRs that they would be closer in size to exact data contained.

Sure... but where does this help? Even a 4GB USB Stick or SD-Card is providing much more space than a full set of all available CPRs would ever need.

And tell me what does help completely useless empty data in CPR images? Less is better in this case, if trim is possible.

[Solo Kazuki]

I can even pay someone for such tool, if it's needed. Some Patreon, or kickstarter, or even direct payment when required.

[robcfg]

What is it that you cannot do with the tools Eto mentioned above?

Besides, there's no automatic way to detect where's the end of the data. Suppose you have a block of empty data that is used by the program, and that crosses the 128KB boundary.

If you trim that and put it into a 128KB chip, it wouldn't work because it would be missing data supposed to be there.

Sure, those are edge cases, but also there's no reliable way to do that.

[GUNHED]

Yes, they can be truncated. Somewhere there is software in this forum.

Can You say more about this?

As eto told: CPR tools (couldn't remember the name before).  Just give it a try. 

[eto]

seems it's not that easy to find with Google:

http://www.cpcmania.com/cprtools/cprtools.htm

[Solo Kazuki]

seems it's not that easy to find with Google:

http://www.cpcmania.com/cprtools/cprtools.htm

I found this already, but it's only converter, not optimizer/trimmer to CPRs. But thanks anyway for Your efforts.

[Fessor]

If it's just a matter of storage space on the medium, you can simply enable compression in the file system. (At least on Windows)

[Solo Kazuki]

If it's just a matter of storage space on the medium, you can simply enable compression in the file system. (At least on Windows)

No, it's not matter of storage space. It's matter of empty useless data in CPRs.

[robcfg]

I was taking a look at the Masarat Khatira .cpr file, and it contains large areas of 0xE5 bytes (which are probably because the file was created from a disk image), and 0xFF bytes, which would be the empty space.

Maybe we could trim those 0xFF bytes, but the question is, would these files still be recognizable and usable by emulators and hardware devices?

[eto]

I found this already, but it's only converter, not optimizer/trimmer to CPRs. But thanks anyway for Your efforts.

That's why the process I told you has several steps. Step 2 is to use a HEX editor and identify, if you can trim the file. If that's the case, you can save a subset of the file or - if your hex editor is not capable of this - use a file splitter (e.g. https://github.com/dubasdey/File-Splitter sounds promising).

The last step then is to create a CPR again. 

[Solo Kazuki]

I was taking a look at the Masarat Khatira .cpr file, and it contains large areas of 0xE5 bytes (which are probably because the file was created from a disk image), and 0xFF bytes, which would be the empty space.

Maybe we could trim those 0xFF bytes, but the question is, would these files still be recognizable and usable by emulators and hardware devices?

Why not? Good example is Hexavirus v0, where CPR have 49kB
https://www.cpc-power.com/index.php?page=detail&num=17197

If You made such trimmed image, I can check it with M4 board if this works.

[andycadley]

I was taking a look at the Masarat Khatira .cpr file, and it contains large areas of 0xE5 bytes (which are probably because the file was created from a disk image), and 0xFF bytes, which would be the empty space.

Maybe we could trim those 0xFF bytes, but the question is, would these files still be recognizable and usable by emulators and hardware devices?

Why not? Good example is Hexavirus v0, where CPR have 49kB
https://www.cpc-power.com/index.php?page=detail&num=17197

If You made such trimmed image, I can check it with M4 board if this works.

I suspect Hexavirus was written specifically as a Plus cartridge, whereas the vast majority of CPRs are auto conversions from disk. It's possible they need to replicate all the disk structure to be guaranteed to work (or it might just have been the most convenient mechanism for faking disks). It's also possible the data is scattered across ROMs as a result of this and some emulators will fail to load a CPR if not all ROMs are in sequence (even though the CPR format allows it).

[eto]

t's possible they need to replicate all the disk structure to be guaranteed to work (or it might just have been the most convenient mechanism for faking disks)

While theoretically possible it is really unlikely that you will ever have an issue. 

The sectors of the DSK are copied sequentially to the CPR. If there is no data in a sector it is unlikely that it will ever be accessed at any time by the game. Especially with the cracked versions used in those CPRs which just load the game via firmware and not via their own routines. 

I shrunk several CPRs from 512K to 128K or 256K and - as long as they did only contain empty sectors beyond the 128K/256K barrier, they perfectly worked. 

[robcfg]

Hey, @Solo Kazuki , care to try this one?

[Solo Kazuki]

Hey, @Solo Kazuki , care to try this one?

It works ok on M4 board. Someone with C4CPC should also check this.

[Solo Kazuki]

t's possible they need to replicate all the disk structure to be guaranteed to work (or it might just have been the most convenient mechanism for faking disks)

While theoretically possible it is really unlikely that you will ever have an issue.

The sectors of the DSK are copied sequentially to the CPR. If there is no data in a sector it is unlikely that it will ever be accessed at any time by the game. Especially with the cracked versions used in those CPRs which just load the game via firmware and not via their own routines.

I shrunk several CPRs from 512K to 128K or 256K and - as long as they did only contain empty sectors beyond the 128K/256K barrier, they perfectly worked.

This is nice step forward, these CPRs are half or even quarter of previous version.
Maybe further optimization is possible?

Could You share with those trimmed CPRs?

[eto]

Could You share with those trimmed CPRs?

I did not make CPRs but created binaries to build onto multi-game cartridges, so unfortunately the last step (BIN to CPR) was not done. 

But if you follow my explanation you can create a reduced CPR easily within a few minutes from the original CPR. 

[robcfg]

The test I send him is of a small command-line program I wrote some years ago, to which I added the trimming of the unused part of the rom. 

Now, I can try to tell it to accept a certain trim value as parameter, like 0xFF for the unused rom space.

Obviously up to the user's discretion. I'm not gonna try to develop heuristics or any fancy algorithm.

[Solo Kazuki]

Ok guys, probably I get it how trim could be done. I just cut some CPRs with hxd editor and... it works! Without any converting or so.
It was done by cutting from first empty cb?? block (where ?? is number) to the end. By empty it can be either FF or E5 hexadecimal value.