CPCWiki forum

Is possible to optimize and/or trim CPR images? Because almost all have 256 or 512 kB when data is sometimes far less.

Yes, they can be truncated. Somewhere there is software in this forum.

Quote from: GUNHED on 10:49, 25 July 24Yes, they can be truncated. Somewhere there is software in this forum.

Can You say more about this?

Quote from: Solo Kazuki on 10:04, 24 July 24Is possible to optimize and/or trim CPR images? Because almost all have 256 or 512 kB when data is sometimes far less.

I used to do it a few times to put more than one game on a (multi-)cartridge and the following process worked for me:

• convert CPR back to a pure BIN with CPR Tools
• check with a hex editor that this file doesn't have any data above 128K or 256K. 
• Once that is verified, just cut the BIN to the right size (if not possible with the hex editor, then using a file splitter)

I think when you convert the reduced BIN now back again to CPR the file size should stay close to 128K or 256K (with a bit of overhead).


May I ask why you want to do that? 

Quote from: eto on 13:07, 25 July 24May I ask why you want to do that?

To trim CPRs that they would be closer in size to exact data contained.

Quote from: Solo Kazuki on 16:50, 25 July 24To trim CPRs that they would be closer in size to exact data contained.

Sure... but where does this help? Even a 4GB USB Stick or SD-Card is providing much more space than a full set of all available CPRs would ever need. 

Quote from: eto on 17:25, 25 July 24

Quote from: Solo Kazuki on 16:50, 25 July 24To trim CPRs that they would be closer in size to exact data contained.

Sure... but where does this help? Even a 4GB USB Stick or SD-Card is providing much more space than a full set of all available CPRs would ever need.

And tell me what does help completely useless empty data in CPR images? Less is better in this case, if trim is possible.

I can even pay someone for such tool, if it's needed. Some Patreon, or kickstarter, or even direct payment when required.

What is it that you cannot do with the tools Eto mentioned above?

Besides, there's no automatic way to detect where's the end of the data. Suppose you have a block of empty data that is used by the program, and that crosses the 128KB boundary.

If you trim that and put it into a 128KB chip, it wouldn't work because it would be missing data supposed to be there.

Sure, those are edge cases, but also there's no reliable way to do that.

Quote from: Solo Kazuki on 12:16, 25 July 24

Quote from: GUNHED on 10:49, 25 July 24Yes, they can be truncated. Somewhere there is software in this forum.

Can You say more about this?

As eto told: CPR tools (couldn't remember the name before).  :) Just give it a try.  :)

seems it's not that easy to find with Google:

http://www.cpcmania.com/cprtools/cprtools.htm

Quote from: eto on 17:24, 26 July 24seems it's not that easy to find with Google:

http://www.cpcmania.com/cprtools/cprtools.htm

I found this already, but it's only converter, not optimizer/trimmer to CPRs. But thanks anyway for Your efforts.

If it's just a matter of storage space on the medium, you can simply enable compression in the file system. (At least on Windows)

Quote from: Fessor on 18:19, 26 July 24If it's just a matter of storage space on the medium, you can simply enable compression in the file system. (At least on Windows)

No, it's not matter of storage space. It's matter of empty useless data in CPRs.

I was taking a look at the Masarat Khatira .cpr file, and it contains large areas of 0xE5 bytes (which are probably because the file was created from a disk image), and 0xFF bytes, which would be the empty space.

Maybe we could trim those 0xFF bytes, but the question is, would these files still be recognizable and usable by emulators and hardware devices?

Quote from: Solo Kazuki on 17:31, 26 July 24I found this already, but it's only converter, not optimizer/trimmer to CPRs. But thanks anyway for Your efforts.

That's why the process I told you has several steps. Step 2 is to use a HEX editor and identify, if you can trim the file. If that's the case, you can save a subset of the file or - if your hex editor is not capable of this - use a file splitter (e.g. https://github.com/dubasdey/File-Splitter sounds promising).

The last step then is to create a CPR again. 

Quote from: robcfg on 22:35, 26 July 24I was taking a look at the Masarat Khatira .cpr file, and it contains large areas of 0xE5 bytes (which are probably because the file was created from a disk image), and 0xFF bytes, which would be the empty space.

Maybe we could trim those 0xFF bytes, but the question is, would these files still be recognizable and usable by emulators and hardware devices?

Why not? Good example is Hexavirus v0, where CPR have 49kB
https://www.cpc-power.com/index.php?page=detail&num=17197

If You made such trimmed image, I can check it with M4 board if this works.

Quote from: Solo Kazuki on 06:35, 27 July 24

Quote from: robcfg on 22:35, 26 July 24I was taking a look at the Masarat Khatira .cpr file, and it contains large areas of 0xE5 bytes (which are probably because the file was created from a disk image), and 0xFF bytes, which would be the empty space.

Maybe we could trim those 0xFF bytes, but the question is, would these files still be recognizable and usable by emulators and hardware devices?

Why not? Good example is Hexavirus v0, where CPR have 49kB
https://www.cpc-power.com/index.php?page=detail&num=17197

If You made such trimmed image, I can check it with M4 board if this works.

I suspect Hexavirus was written specifically as a Plus cartridge, whereas the vast majority of CPRs are auto conversions from disk. It's possible they need to replicate all the disk structure to be guaranteed to work (or it might just have been the most convenient mechanism for faking disks). It's also possible the data is scattered across ROMs as a result of this and some emulators will fail to load a CPR if not all ROMs are in sequence (even though the CPR format allows it).

Quote from: andycadley on 08:18, 27 July 24t's possible they need to replicate all the disk structure to be guaranteed to work (or it might just have been the most convenient mechanism for faking disks)

While theoretically possible it is really unlikely that you will ever have an issue. 

The sectors of the DSK are copied sequentially to the CPR. If there is no data in a sector it is unlikely that it will ever be accessed at any time by the game. Especially with the cracked versions used in those CPRs which just load the game via firmware and not via their own routines. 

I shrunk several CPRs from 512K to 128K or 256K and - as long as they did only contain empty sectors beyond the 128K/256K barrier, they perfectly worked. 

Hey, @Solo Kazuki , care to try this one?

Quote from: robcfg on 22:02, 27 July 24Hey, @Solo Kazuki , care to try this one?

It works ok on M4 board. Someone with C4CPC should also check this.

Quote from: eto on 15:01, 27 July 24

Quote from: andycadley on 08:18, 27 July 24t's possible they need to replicate all the disk structure to be guaranteed to work (or it might just have been the most convenient mechanism for faking disks)

While theoretically possible it is really unlikely that you will ever have an issue.

The sectors of the DSK are copied sequentially to the CPR. If there is no data in a sector it is unlikely that it will ever be accessed at any time by the game. Especially with the cracked versions used in those CPRs which just load the game via firmware and not via their own routines.

I shrunk several CPRs from 512K to 128K or 256K and - as long as they did only contain empty sectors beyond the 128K/256K barrier, they perfectly worked.

This is nice step forward, these CPRs are half or even quarter of previous version.
Maybe further optimization is possible?

Could You share with those trimmed CPRs?

Quote from: Solo Kazuki on 09:42, 28 July 24Could You share with those trimmed CPRs?

I did not make CPRs but created binaries to build onto multi-game cartridges, so unfortunately the last step (BIN to CPR) was not done. 

But if you follow my explanation you can create a reduced CPR easily within a few minutes from the original CPR. 

The test I send him is of a small command-line program I wrote some years ago, to which I added the trimming of the unused part of the rom. 

Now, I can try to tell it to accept a certain trim value as parameter, like 0xFF for the unused rom space.

Obviously up to the user's discretion. I'm not gonna try to develop heuristics or any fancy algorithm.

Ok guys, probably I get it how trim could be done. I just cut some CPRs with hxd editor and... it works! Without any converting or so.
It was done by cutting from first empty cb?? block (where ?? is number) to the end. By empty it can be either FF or E5 hexadecimal value.

I'll implement the byte value to trim and make the code available.

Ok, did a version of the CPR that takes away the trailing 0xFF bytes of empty rom and also the trailing 0xE5 bytes from the embedded disk image.

Please let me know if it works.

Quote from: robcfg on 22:41, 28 July 24Ok, did a version of the CPR that takes away the trailing 0xFF bytes of empty rom and also the trailing 0xE5 bytes from the embedded disk image.

Please let me know if it works.

No, You cut too much. Version which works should have 209KB. Check version which I cut manually, it should work ok.

Ok, so I fixed a small error in my program where it would skip the las full chunk of data and seems to be working.

If you trim the 0xE5 bytes from the disk image, the Masarat Khatira file goes under 200KB.

Please try the attached file.

Nope, not working. I have just "Bad command" error after start cart.

Interesting. It does work on RetroVirtualMachine...

But this highlights the problem I told you earlier we could have.

So it seems that trimming the disk image itself yields errors. Would be good to know why.

If I just trim the 0xFF bytes does it work? (Try the attached file, please)

It depends on the emulator I would say; the CPR has a header for example; @robcfg you say "trim", so I don't know how are you doing it.

I'm not sure I see the point of all this TBH :D but if I wanted to make a smaller CPR, with my tools you can: https://github.com/reidrac/cpr-tools

1. use cprdump.py to generate 16kb bin chunks
2. remove any bin files that are all 0xff (or 0x00; that's what my tools use for padding), starting from the last and going backwards -- stop when a block has other data that is not 0x00 or 0xff!
3. use mkcpr.py to build the CPR back with the remaining bin files (by default the tool won't pad the CPR)

The resulting CPR should be valid and work everywhere; although I recall that the padded version (which is without trimming) is recommended for higher compatibility. That's what I used in Hyperdrive, for example.

(if this is how you did it, apologies for the noise!)

Hi @reidrac , thanks for your feedback!

What I do is take the binary and just trim the 0xFF bytes at the end and, if you tell me to, the following 0xE5 bytes at the end after that.

I don't think it makes much more sense than to use the closest rom size (64,128,256,512KB) to the data, but I just had this small program of mine and could easily add the required functionality.

Quote from: robcfg on 13:34, 01 August 24I don't think it makes much more sense than to use the closest rom size (64,128,256,512KB) to the data, but I just had this small program of mine and could easily add the required functionality.

I believe it is important. The CPR is a RIFF file, if you don't follow the spec, some emulators may decide the file is corrupt instead of processing it properly

See: https://cpctech.cpcwiki.de/docs/cprdef.html

QuoteIf a emulator encounters a block with less than 16k, it should fill the rest of the range with 0's, if a emulator encounters a block with more than 16k, it should stop reading at 16k and ignore the remaining data.

Do all emulators follow this? I guess they don't! :)

Thing is, I trim the source data, and then I write the proper RIFF structure.

That's why it works on RVM.

For me the interesting thing would be knowing why that would fail on a real machine. Maybe it's an issue with the device being used to supply the CPR?

Hi,
    joining the party a bit late, but anyway... What CPR are you trying to trim? Was it a pure ROM / Cartridge dump file before it got made into a CPR? And what format do you need it to be (ROM for ROMboard or Cartridge)?
I've always done this manually with a HEX editor. If you can send me the file in question, I can look at why a real machine doesn't like it.

Bryce.

I'm playing with the Masarat Khatira CPR conversion from this post: https://www.cpcwiki.eu/forum/index.php?msg=241585

I don't remember what device is @Solo Kazuki using.

Quote from: robcfg on 18:26, 01 August 24I'm playing with the Masarat Khatira CPR conversion from this post: https://www.cpcwiki.eu/forum/index.php?msg=241585

I don't remember what device is @Solo Kazuki using.

Ok, so I've just looked at the (UK).CPR file: The part I've marked in red needs to be removed, but... The Byte I've marked green is also wrong. The value 00 would indicate to a real CPC, that the ROM is a background ROM (such as BASIC). For a game that needs to be started with a bar command, this value should be 01 and the value right after it should be the address of the command table for the ROM. So you not only need to remove bits, but also correct these values for it to work properly.

Bryce.

A CPR is not a straight binary dump of ROMs, it's a RIFF based file and that has it's own header (which appears to be what you're looking at)

I mentioned earlier that:

QuoteThing is, I trim the source data, and then I write the proper RIFF structure.

That's why it works on RVM.

I wrote the program to convert from raw binary data to cpr format and viceversa in 2008, and the cpr file specs have been out there since forever.

What is being discussed is that if the unused data, whether empty rom space or empty embedded disk image space, thus making the resulting files smaller, and if that would work on several devices that accept this kind of files.

@Bryce : Thanks for the info! I wasn't aware of that bit (no pun intended). Does that work the same on a Plus machine?

Quote from: robcfg on 21:30, 01 August 24@Bryce : Thanks for the info! I wasn't aware of that bit (no pun intended). Does that work the same on a Plus machine?

No, that's kind of what I meant. You can't assume a cartridge ROM is going to follow the firmware specs. And, if you want to be compatible with all the emulators out there, you need the ROMs to be in order and without any gaps (despite one of the points of using RIFF in the first place was to allow that) and if you're doing that the first ROM in the file would be the firmware ROM itself, which also doesn't follow the rules of upper ROMS (because it isn't one)

Quote from: robcfg on 18:26, 01 August 24I don't remember what device is @Solo Kazuki using.

M4 board with 6128 plus.

Quote from: robcfg on 21:30, 01 August 24I mentioned earlier that:

QuoteThing is, I trim the source data, and then I write the proper RIFF structure.

That's why it works on RVM.

I wrote the program to convert from raw binary data to cpr format and viceversa in 2008, and the cpr file specs have been out there since forever.

What is being discussed is that if the unused data, whether empty rom space or empty embedded disk image space, thus making the resulting files smaller, and if that would work on several devices that accept this kind of files.

@Bryce : Thanks for the info! I wasn't aware of that bit (no pun intended). Does that work the same on a Plus machine?

There is an older thread somewhere here, where multiple people were converting tape games to work on a standard ROMBoard, some of which required the game to be spread across more than one ROM IC. There was a lot of discussion there about the formats and compatibility back then, maybe you can find some useful info there? From memory (which mine isn't always great), they all worked on Plus cartridges too. I think they were mostly using the noca$h converter to create the files.

Bryce.

Thanks for the tip!

Though I suspect it may be more some strict handling of the cpr files by the M4 (or any other device).

The two issues that crop up the most , I think, are:

Cartridge blocks out of order or missing - a few implementations assume that the first block will be for cb00, the second for cb01 etc without bothering to actually parse the block headers. So stripping out a ROM midway through or reassembling the cart in the "wrong" order can cause failures.

ROMs that aren't exactly 16KB - again the spec is supposed to allow for this, filling the remainder of the ROM with 0, but there are implementations that assume every ROM will be exactly 16KB and (again) ignore what the header actually says (including potentially assuming the header of the next block is part of the ROM etc)

Nice!

The order of the chunks is not an issue, because I make sure they are always ordered, and I also have a note on the code reminding me of it  :D

That the chunks are not exactly 16KB in size could definitely be a problem. I think the devices should support the full spec of the files, but it's way easier to make my code just fill up (or not trim) the last chunk, which is always the one that could get in that situation.