Optimize and/or trim CPR images

[robcfg]

I'll implement the byte value to trim and make the code available.

[robcfg]

Ok, did a version of the CPR that takes away the trailing 0xFF bytes of empty rom and also the trailing 0xE5 bytes from the embedded disk image.

Please let me know if it works.

[Solo Kazuki]

Ok, did a version of the CPR that takes away the trailing 0xFF bytes of empty rom and also the trailing 0xE5 bytes from the embedded disk image.

Please let me know if it works.

No, You cut too much. Version which works should have 209KB. Check version which I cut manually, it should work ok.

[robcfg]

Ok, so I fixed a small error in my program where it would skip the las full chunk of data and seems to be working.

If you trim the 0xE5 bytes from the disk image, the Masarat Khatira file goes under 200KB.

Please try the attached file.

[Solo Kazuki]

Nope, not working. I have just "Bad command" error after start cart.

[robcfg]

Interesting. It does work on RetroVirtualMachine...

But this highlights the problem I told you earlier we could have.

So it seems that trimming the disk image itself yields errors. Would be good to know why.

If I just trim the 0xFF bytes does it work? (Try the attached file, please)

[reidrac]

It depends on the emulator I would say; the CPR has a header for example; @robcfg you say "trim", so I don't know how are you doing it.

I'm not sure I see the point of all this TBH but if I wanted to make a smaller CPR, with my tools you can: https://github.com/reidrac/cpr-tools

1. use cprdump.py to generate 16kb bin chunks
2. remove any bin files that are all 0xff (or 0x00; that's what my tools use for padding), starting from the last and going backwards -- stop when a block has other data that is not 0x00 or 0xff!
3. use mkcpr.py to build the CPR back with the remaining bin files (by default the tool won't pad the CPR)

The resulting CPR should be valid and work everywhere; although I recall that the padded version (which is without trimming) is recommended for higher compatibility. That's what I used in Hyperdrive, for example.

(if this is how you did it, apologies for the noise!)

[robcfg]

Hi @reidrac , thanks for your feedback!

What I do is take the binary and just trim the 0xFF bytes at the end and, if you tell me to, the following 0xE5 bytes at the end after that.

I don't think it makes much more sense than to use the closest rom size (64,128,256,512KB) to the data, but I just had this small program of mine and could easily add the required functionality.

[reidrac]

I don't think it makes much more sense than to use the closest rom size (64,128,256,512KB) to the data, but I just had this small program of mine and could easily add the required functionality.

I believe it is important. The CPR is a RIFF file, if you don't follow the spec, some emulators may decide the file is corrupt instead of processing it properly

See: https://cpctech.cpcwiki.de/docs/cprdef.html

If a emulator encounters a block with less than 16k, it should fill the rest of the range with 0's, if a emulator encounters a block with more than 16k, it should stop reading at 16k and ignore the remaining data.

Do all emulators follow this? I guess they don't!

[robcfg]

Thing is, I trim the source data, and then I write the proper RIFF structure.

That's why it works on RVM.

For me the interesting thing would be knowing why that would fail on a real machine. Maybe it's an issue with the device being used to supply the CPR?

[Bryce]

Hi,
    joining the party a bit late, but anyway... What CPR are you trying to trim? Was it a pure ROM / Cartridge dump file before it got made into a CPR? And what format do you need it to be (ROM for ROMboard or Cartridge)?
I've always done this manually with a HEX editor. If you can send me the file in question, I can look at why a real machine doesn't like it.

Bryce.

[robcfg]

I'm playing with the Masarat Khatira CPR conversion from this post: https://www.cpcwiki.eu/forum/index.php?msg=241585

I don't remember what device is @Solo Kazuki using.

[Bryce]

I'm playing with the Masarat Khatira CPR conversion from this post: https://www.cpcwiki.eu/forum/index.php?msg=241585

I don't remember what device is @Solo Kazuki using.

Ok, so I've just looked at the (UK).CPR file: The part I've marked in red needs to be removed, but... The Byte I've marked green is also wrong. The value 00 would indicate to a real CPC, that the ROM is a background ROM (such as BASIC). For a game that needs to be started with a bar command, this value should be 01 and the value right after it should be the address of the command table for the ROM. So you not only need to remove bits, but also correct these values for it to work properly.

Bryce.

[andycadley]

A CPR is not a straight binary dump of ROMs, it's a RIFF based file and that has it's own header (which appears to be what you're looking at)

[robcfg]

I mentioned earlier that:

Thing is, I trim the source data, and then I write the proper RIFF structure.

That's why it works on RVM.

I wrote the program to convert from raw binary data to cpr format and viceversa in 2008, and the cpr file specs have been out there since forever.

What is being discussed is that if the unused data, whether empty rom space or empty embedded disk image space, thus making the resulting files smaller, and if that would work on several devices that accept this kind of files.

@Bryce : Thanks for the info! I wasn't aware of that bit (no pun intended). Does that work the same on a Plus machine?

[andycadley]

@Bryce : Thanks for the info! I wasn't aware of that bit (no pun intended). Does that work the same on a Plus machine?

No, that's kind of what I meant. You can't assume a cartridge ROM is going to follow the firmware specs. And, if you want to be compatible with all the emulators out there, you need the ROMs to be in order and without any gaps (despite one of the points of using RIFF in the first place was to allow that) and if you're doing that the first ROM in the file would be the firmware ROM itself, which also doesn't follow the rules of upper ROMS (because it isn't one)

[Solo Kazuki]

I don't remember what device is @Solo Kazuki using.

M4 board with 6128 plus.

[Bryce]

I mentioned earlier that:

Thing is, I trim the source data, and then I write the proper RIFF structure.

That's why it works on RVM.

I wrote the program to convert from raw binary data to cpr format and viceversa in 2008, and the cpr file specs have been out there since forever.

What is being discussed is that if the unused data, whether empty rom space or empty embedded disk image space, thus making the resulting files smaller, and if that would work on several devices that accept this kind of files.

@Bryce : Thanks for the info! I wasn't aware of that bit (no pun intended). Does that work the same on a Plus machine?

There is an older thread somewhere here, where multiple people were converting tape games to work on a standard ROMBoard, some of which required the game to be spread across more than one ROM IC. There was a lot of discussion there about the formats and compatibility back then, maybe you can find some useful info there? From memory (which mine isn't always great), they all worked on Plus cartridges too. I think they were mostly using the noca$h converter to create the files.

Bryce.

[robcfg]

Thanks for the tip!

Though I suspect it may be more some strict handling of the cpr files by the M4 (or any other device).

[andycadley]

The two issues that crop up the most , I think, are:

Cartridge blocks out of order or missing - a few implementations assume that the first block will be for cb00, the second for cb01 etc without bothering to actually parse the block headers. So stripping out a ROM midway through or reassembling the cart in the "wrong" order can cause failures.

ROMs that aren't exactly 16KB - again the spec is supposed to allow for this, filling the remainder of the ROM with 0, but there are implementations that assume every ROM will be exactly 16KB and (again) ignore what the header actually says (including potentially assuming the header of the next block is part of the ROM etc)

[robcfg]

Nice!

The order of the chunks is not an issue, because I make sure they are always ordered, and I also have a note on the code reminding me of it 

That the chunks are not exactly 16KB in size could definitely be a problem. I think the devices should support the full spec of the files, but it's way easier to make my code just fill up (or not trim) the last chunk, which is always the one that could get in that situation.