Optimize and/or trim CPR images

[norecess464]

Well, I never understood why to use CPRs instead of ROMs directly. A CPR file has no advantage (or has it one?), but just fills in some bytes for no gain. What do you think?
Suggesting to move to read ROM content in general! 

IMO You are viewing this from a developer's perspective.

From a user's point of view, it makes perfect sense to me to have a dedicated file format for cartridges.

One for tapes, one for discs, one for cartridges, one for ROMs.. all good.

[Prodatron]

And it's much much easier to handle one single CPR instead of dealing with 8, 16, 32 or what ever number of single 16K ROM files.

[andycadley]

And CPR contains information about how the ROMs need to fit into slots, which pure binary ROMs require the user to understand.

[roudoudou]

Well, I never understood why to use CPRs instead of ROMs directly. A CPR file has no advantage (or has it one?), but just fills in some bytes for no gain. What do you think?

Suggesting to move to read ROM content in general! 

Advantage, you can gainz some bytes on a 16 cores / 64Gb RAM machine with 32TB HDD ...

[eto]

Compared to single 16K ROMs agreed.

But for cartridges what is the benefit of a CPR compared to simple, binary 128K, 256K or 512K ROM file? For what was the format established in the first place?

The only hypothetical benefit I see is that unused parts do not "waste" space on a hard drive. And this benefit is barely used as almost all CPRs I have ever seen are 512K CPRs even if the content is a lot less. Even if it would be used, on a typical 4GB SD Card, we still can put 8000 512K ROMs.

Or are there any potential future updates of the format?
Btw: where is the format defined? The one on e the Wiki is quite vague and the link to the original is dead.

[andycadley]

There's no requirement for cartridges to be linear (and there are potential advantages to them not being linear). Furthermore, since it's based on RIFF, it's a lot easier to extend if people wanted to, say to include instructions or to support NES style "mapper" arrangements.

And the overhead of RIFF compared to a straight binary is minimal. Admittedly a lot of DSK->CPR conversions are probably massively bloated, but that's really the fault of the tools used to do the conversion rather than the CPR format, you'd still end up with a massively unnecessary file if they'd been writing raw binary.

[eto]

There's no requirement for cartridges to be linear (and there are potential advantages to them not being linear).

Do you have an example in mind? From what I know about physical cartridges and how they have been built I don't see how cartridges would not be linear or what advantages that could have.

[andycadley]

There's no requirement for cartridges to be linear (and there are potential advantages to them not being linear).

Do you have an example in mind? From what I know about physical cartridges and how they have been built I don't see how cartridges would not be linear or what advantages that could have.

Off the top of my head, the GX and Plus treat Physical Page 3 differently: A Plus will page it in when ROM 7 is requested, a GX will not. So a small cartridge might only need a lower ROM (Physical page 0) and a second ROM that can be used to identify the model (Physical page 3). 

More importantly, at the time the format was defined, nobody really knew what "real" GX cartridges might be like - so the format was defined to be flexible enough to cope with anything and extensible enough should the need arise (to avoid the problems with a myriad of other emulator "formats" that actually have multiple different specs to cope with weirdness that was discovered later).

[Solo Kazuki]

Well, I never understood why to use CPRs instead of ROMs directly. A CPR file has no advantage (or has it one?), but just fills in some bytes for no gain. What do you think?

Suggesting to move to read ROM content in general! 

Because CPR is much easier to handle for me than ROM in M4board. I can have ROM (M4FE in my case) and CPR (games) at once. Changing from M4FE to game and back to M4FE after is not comfortable - You can just "eject" CPR and You have M4FE back then.
So CPR HAS advantage for me.

[GUNHED]

And it's much much easier to handle one single CPR instead of dealing with 8, 16, 32 or what ever number of single 16K ROM files.

A ROM file for the 6128plus consists ouf of one(sic!) file exactly.  It's super easy!

[GUNHED]

And CPR contains information about how the ROMs need to fit into slots, which pure binary ROMs require the user to understand.

There is nothing special to understand, The ROM file address &000000 starts at ROM &80. That's it. 

[GUNHED]

Well, I never understood why to use CPRs instead of ROMs directly. A CPR file has no advantage (or has it one?), but just fills in some bytes for no gain. What do you think?

Suggesting to move to read ROM content in general! 

Advantage, you can gainz some bytes on a 16 cores / 64Gb RAM machine with 32TB HDD ...

Oh, you got a neat CPC setup then.   

[andycadley]

And CPR contains information about how the ROMs need to fit into slots, which pure binary ROMs require the user to understand.

There is nothing special to understand, The ROM file address &000000 starts at ROM &80. That's it. 

And how does that work when you have gaps in the slots? Or mapper chips in future carts? What would be the point of making a new cartridge format that is less flexible when the current one is already trivial to support?

[eto]

And how does that work when you have gaps in the slots? Or mapper chips in future carts? What would be the point of making a new cartridge format that is less flexible when the current one is already trivial to support?

CPR is not flexible at all. At least not in a useful way. Sure, you can say "this 16K block goes at position 12 and this at position 1" but what's the point of this? I have a (up to) 512K ROM in a cartridge which I dump to a CPR - why should I rearrange the 16K blocks in the CPR?

CPR just maps up to 32 16K blocks into a 512K ROM space of a physical cartridge ROM. Nothing else. Treatment of pyhsical page 3? Not part of the CPR. Mapping to ROM &80 => not done in the CPR.  The only "benefit" is if inside of those 512K there would be a gap of let's say 16K so that you don't "waste" space. But a) this is completely pointless with nowadays mass storage solutions, b) if you compress the CPR there is close to none difference to a compressed ROM and c) it's theoretical as the only scenario I have seen in cartridges are empty blocks at the end of a 128k or 256K ROM in a 512K CPR which could be solved by just having a 128K or 256K ROM (and weirdly enough you still find 16K blocks with only 0s in the CPR - so the CPR is still a full 512K+).

CPR as it's being used today imho only adds overhead over pure ROM cartridge files. I don't even think that mapper chips in future cards might change that as it doesn't matter if the cartridge holds 512KB or 1MB - it's the hardware that will take care of switching to the right 16K rom bank in the 1MB EPROM. 

[andycadley]

The only "benefit" is if inside of those 512K there would be a gap of let's say 16K so that you don't "waste" space. But a) this is completely pointless

The argument against CPR was that it "wastes space". If you take the point of view that disk space is cheap (which is a perfectly valid argument) then that argument no longer makes sense and you might just stick to CPR.

CPR as it's being used today imho only adds overhead over pure ROM cartridge files. I don't even think that mapper chips in future cards might change that as it doesn't matter if the cartridge holds 512KB or 1MB - it's the hardware that will take care of switching to the right 16K rom bank in the 1MB EPROM.

The hardware would do the actual bank switching, but for emulation you need to describe what that mapper chip is. And so you need a file format that can contain some variety of metadata - a single linear dump of ROM space is not that, which is why basically every cartridge format on every system doesn't just use that.

[GUNHED]

And CPR contains information about how the ROMs need to fit into slots, which pure binary ROMs require the user to understand.

There is nothing special to understand, The ROM file address &000000 starts at ROM &80. That's it. 

And how does that work when you have gaps in the slots? Or mapper chips in future carts? What would be the point of making a new cartridge format that is less flexible when the current one is already trivial to support?

Ah... Not a new format... Just having a 1:1 copy of the chip inside the Cartridge, and that's a (EP)ROM with linear address.

Well, of course I'm wrong in case Amstrad (somebody) will release something on a Cartridge with new technology - but in this case new computers will be needed too. 

Nice to have some fantasies, but let's stay realistic. 

[eto]

The hardware would do the actual bank switching, but for emulation you need to describe what that mapper chip is. And so you need a file format that can contain some variety of metadata - a single linear dump of ROM space is not that, which is why basically every cartridge format on every system doesn't just use that.

But that's exactly my point. Unlike other cartridge based systems on the GX4000 there is nothing you need to add. By definition the target of a CPRs content is a linear 512K address space. Nothing else is specified.
 
I did a bit of research a while ago and I tried to find any CPR that would use the block number feature, as I wondered what it could be used for. But I couldn't find any. Even worse ... the typical CPR file is >512K while they often only had 128K or 256K of ROM in it. So even the argument that it saves some space is moot as obviously nobody ever cared about the saving space feature. It's actually funny that the only "feature" of a CPR that it can save space is not used and leads to the situation that having used ROM files instead would have required less space on disk.

Agreed, in the future we might need a format to cover special mapping chips (let's see if that will ever happen - and if it happens, if it happens more than once) but why on earth was that format defined as it is 20 years ago?

PS: to be clear I don't  argue to abandon the CPR file format I only try to understand why it was created in the first place. There must have been some idea what it could be used for. Somebody must have thought that it's a good idea to shift around the 16K blocks. And there must have been a reason why this was thought to be a good idea. I'd love to understand what those reasons were.

[andycadley]

But that's exactly my point. Unlike other cartridge based systems on the GX4000 there is nothing you need to add. By definition the target of a CPRs content is a linear 512K address space. Nothing else is specified.

Bubble Quest, if it ever gets released, was targeting a 2MB cartridge. The extra paging inside the cartridge is, essentially, a form of mapper chip. If other larger cartridges get released, they may follow the BQ scheme but they may use something else. To emulated these you will need some way in the format to indicate what additional hardware is in the cartridge (you don't necessarily need the logic there just a byte or two to say "use this mapping scheme - like NES ROMs do)

PS: to be clear I don't  argue to abandon the CPR file format I only try to understand why it was created in the first place. There must have been some idea what it could be used for. Somebody must have thought that it's a good idea to shift around the 16K blocks. And there must have been a reason why this was thought to be a good idea. I'd love to understand what those reasons were.

1) Nobody had done any analysis on GX cartridges, the only one ripped at the time was Burnin' Rubber / BASIC. Maybe every other cartridge had "holes" in the address space. Best to start with a format that could handle it.

2) Things like Gamebase were becoming popular. Being able to embed Screenshots, Artwork, Instructions, POKEs etc in the same file format as the game might have become useful (such things have fallen from favour so never got defined)

3) Nobody thought generic disk->CPR tools had any use. People openly mocked the idea anyone would build programmable carts for the GX. Thus the target usage was preserving the 27 original cartridges and possibly supporting the odd crazy developer who wanted to use Plus features in emulator only programs. So worrying about minor overheads for automated tools was pointless.

4) Disk space was less cheap 20 years ago, the idea your complete GX game collection was the best part of 15MB seemed nuts. The assumption was the carts would get ripped, analysed, reduced down to what was necessary and then nobody would ever produce any more.

5) The original tool for ripping carts was built for the Amiga and IFF is basically how the Amiga handles all its native file formats, so it became the easiest format to use (since it is inherently extensible). Just be thankful it wasn't XML.

[Prodatron]

I only try to understand why it was created in the first place. There must have been some idea what it could be used for.

A CPR file replaces many individual 16K ROM files. It's much more practical to use a single container file for larger software than to have to mess around with dozens of small ROM files.
Is there another format that bundles multiple ROMs into one file? I only know of CPR.

Or have I misunderstood something?

And if people convert cartridges to CPRs and don't pay attention to remove unused 16K slots, that's not the format's problem I would say.

[eto]

A CPR file replaces many individual 16K ROM files. It's much more practical to use a single container file for larger software than to have to mess around with dozens of small ROM files.

If that is possible: immediately agreed. I've never seen something like that, do you have an example? 

I might have not fully understood the specification but imho there's no information about mapping a 16K ROM into a specific ROM slot. Just as an example: try to create a CPR where a ROM is in slot 5 or slot 12. Afaik this is not possible. The CPR only allows to map the 16K ROM to an address inside of a (virtual or physical) 512K ROM chip. So if you are using a CPR for an application your program would appear (as with all other CPRs) at slot 0,1, 7 and then at 128. You have to assemble your app specifically for use in a CPR so it finds the ROM content at ROM slot 128 and above (and this is again not even visible in the CPR as in the CPR you have to define it to be in block 0-31). 

[andycadley]

A CPR file replaces many individual 16K ROM files. It's much more practical to use a single container file for larger software than to have to mess around with dozens of small ROM files.

If that is possible: immediately agreed. I've never seen something like that, do you have an example?

I might have not fully understood the specification but imho there's no information about mapping a 16K ROM into a specific ROM slot. Just as an example: try to create a CPR where a ROM is in slot 5 or slot 12. Afaik this is not possible. The CPR only allows to map the 16K ROM to an address inside of a (virtual or physical) 512K ROM chip. So if you are using a CPR for an application your program would appear (as with all other CPRs) at slot 0,1, 7 and then at 128. You have to assemble your app specifically for use in a CPR so it finds the ROM content at ROM slot 128 and above (and this is again not even visible in the CPR as in the CPR you have to define it to be in block 0-31).

That's because it specifies a Plus cartridge and the Plus cartridge cannot do that.

However you can choose which slots in the cartridge ROMs go into and three of them are "special" because they get mapped onto the Lower ROM, the BASIC ROM and AMSDOS ROM respectively. Also the first 8 can be paged differently to subsequent cartridge slots.

I think what you are missing is that there is no requirement in the Plus hardware for the ROM space to be contiguous. It's entirely possible to have a cartridge with three physical 16K ROMs in slots 0, 1 and 130 if you really wanted and the CPR format was designed to cope with this (and it is not the same as having a bunch of 0 bytes throughout the 'gaps')

[eto]

That's because it specifies a Plus cartridge and the Plus cartridge cannot do that.

Exactly my point. CPR is not a good format for ROM applications as it does not support standard ROM applications like in a ROM board. If you go for CPR to distribute your application it's again not different from using a linear binary. 

[andycadley]

That's because it specifies a Plus cartridge and the Plus cartridge cannot do that.

Exactly my point. CPR is not a good format for ROM applications as it does not support standard ROM applications like in a ROM board. If you go for CPR to distribute your application it's again not different from using a linear binary.

CPR is not intended for ordinary sideways ROMs, it's for cartridges which completely replace the OS.

That's a bit like saying DSK isn't a suitable replacement for CDT tape images, because it can't represent a tape file. It's not the intended purpose.

[GUNHED]

That's because it specifies a Plus cartridge and the Plus cartridge cannot do that.

Exactly my point. CPR is not a good format for ROM applications as it does not support standard ROM applications like in a ROM board. If you go for CPR to distribute your application it's again not different from using a linear binary.

Exactly!!! 

[eto]

CPR is not intended for ordinary sideways ROMs,

That's what I said in #69 and #71.