purpose of the CPR format?

[eto]

Over the last days I was creating some multi-ROM cartridges for which I had to switch between pure binary ROMS and CPRs a lot. I was wondering what purpose the CPRs serve as using the CPRtools utility was slightly annoying.

According to the docs, I understand, that it might save some space if parts of the ROM banks are empty. But looking at real-life CPRs that's not really the case - even worse, many CPRs are full blown 512K although the ROMs inside are only 128K or 256K.


Is there anything I am missing for which it is useful, or is it just a relic that we have to deal with because it became standard?


btw: Is there a command line tool (Windows) which will do that? I only found CPRtools which are painful to use.

[andycadley]

I think the purpose was just a reasonable representation of a cartridge. Being based on RIFF means there are standardised ways of reading the data out whilst retaining some level of extensibility (you could easily include manuals, artwork, pokes etc by just agreeing new chunk definitions). 

That hasn't really been necessary or utilized so far, although with games like Bubble Quest now pushing the limits of what can be stored in a cartridge it'll almost certainly become necessary to both hold more data and meta definitions of internal mapping logic (even if it's a simple as NES ROMs which just assign numeric codes to different paging chips) and that's not something a straightforward binary ROM dump really allows. 

[GUNHED]

Thanks @eto : 1000% agreed!!! 

We really should all switch to pure binary files, today there is no reason to use that .cpr format (it there was any anyway).

[roudoudou]

i guess CPR was somehow designed with extended chunks in mind, like snapshots

about me, i already added some chunks to snapshots. Common snapshots deals with RAM, i simply added ROM and extra-info => http://rasm.wikidot.com/dev:supersnap

i (also) guess for simple hardware it's easier to deal with CPR instead of a super snapshot but you made a point. Dealing with a single huge ROM file is even easier

[Bryce]

I always assumed that the CPR format was created for emulators. It has no advantages if you are working with real hardware, it just complicates things.

Bryce.

[andycadley]

I don't think the possibility of using it on hardware was much of a consideration at the time, Plus emulation was scarce enough and there were about 27 possible cartridges to worry about.

I don't really see an enormous advantage in trying to change it now though, there are already hundreds of files in CPR format and trying to change them to something else for only minimal benefits seems fruitless. Especially given forthcoming titles like Bubble Quest are absolutely going to require either an extension to CPR (which is relatively trivial) or a whole other file format which is going to be as complicated to manage in hardware if not more so anyway.

[eto]

I always assumed that the CPR format was created for emulators. It has no advantages if you are working with real hardware, it just complicates things.

Bryce.

Is there an advantage for an emulator?

I mean, sure it's the standard now and I doubt we will get rid of it ever. I'm just wondering why it was introduced in the first place. Somebody must have thought "I have to add metadata to the ROM and make reading and writing more complicated as otherwise I can't support XYZ". I'd love to get an idea what XYZ could be.

[eto]

I don't really see an enormous advantage in trying to change it now though, there are already hundreds of files in CPR format and trying to change them to something else for only minimal benefits seems fruitless. Especially given forthcoming titles like Bubble Quest are absolutely going to require either an extension to CPR (which is relatively trivial) or a whole other file format which is going to be as complicated to manage in hardware if not more so anyway.

please don't get me wrong, I don't want to get rid of the format, I was just wondering, why CPR was invented at all.

btw: why will Bubble Quest need an extension to the CPR format?

[andycadley]

I always assumed that the CPR format was created for emulators. It has no advantages if you are working with real hardware, it just complicates things.

Bryce.

Is there an advantage for an emulator?

I mean, sure it's the standard now and I doubt we will get rid of it ever. I'm just wondering why it was introduced in the first place. Somebody must have thought "I have to add metadata to the ROM and make reading and writing more complicated as otherwise I can't support XYZ". I'd love to get an idea what XYZ could be.

Off the top of my head:

• Embedded artwork, so that emulators could display images in a selection front end (I'm think RetroPi-esque)
• Advanced ROM mapping schemes in the cartridge (as seen in Bubble Quest)
• POKE files to allow selectable cheats etc

But probably just as likely is "We might need to extend this format in future, as pretty much every emulator file format has required and so we might just as well use a common and relatively simple extensible base for it from the get go"

Bubble Quest will end up needing it because it's over the 512K cartridge limit, which it accomplishes by embedding some hardware logic inside the cartridge to provide some additional ROM selection logic (similar to the way NES mapper chips work but slightly more hacky because write signals aren't exposed to the cartridge). That sort of removes any assumption that you can treat the cartridge as a linear ROM, especially given the possibility that future cartridge games might also implement their own similar, but fundamentally different, mapping scheme to increase storage (or sneak some other hardware functionality into the card).

[Prodatron]

Btw, I don't understand what is the problem with the CPR format?

When you read this discussion you have the feeling, that it would be complex, which it isn't at all.
Creating a CPR file is so simple, just add 8 bytes in front of every 16K rom and place another little 12 byte header at the beginning, and that's it.

[eto]

When you read this discussion you have the feeling, that it would be complex, which it isn't at all.
Creating a CPR file is so simple, just add 8 bytes in front of every 16K rom and place another little 12 byte header at the beginning, and that's it.

There is no problem, I was just wondering why it was created in the first place, if it doesn't have advantages.

If it's really just that, then it's adding complexity for no purpose. And if it's more complex (like Andycadley explained) then the logic easily becomes more complex and the assumption that it's simple is no longer valid.

• Embedded artwork, so that emulators could display images in a selection front end (I'm think RetroPi-esque)
• Advanced ROM mapping schemes in the cartridge (as seen in Bubble Quest)
• POKE files to allow selectable cheats etc

That's just ideas for the future but have not been defined yet, right? But I get it: the CPR format could be easily enhanced to add that information. Tools and emulators could use or ignore additional information as long as they read and interpret the CBxx chunks properly (and don't expect them to be there one after the other).

[roudoudou]

I don't really see an enormous advantage in trying to change it now though, there are already hundreds of files in CPR format and trying to change them to something else for only minimal benefits seems fruitless. Especially given forthcoming titles like Bubble Quest are absolutely going to require either an extension to CPR (which is relatively trivial) or a whole other file format which is going to be as complicated to manage in hardware if not more so anyway.

please don't get me wrong, I don't want to get rid of the format, I was just wondering, why CPR was invented at all.

btw: why will Bubble Quest need an extension to the CPR format?

BubbleQuest is supposed to be 512k x 4 which is 128 roms

I could not keep regular CPR format using cb<%02d> (limited to 100 roms...)

There is already some evolutions in Rasm and a few emulators but we will decide what we do later (i do not think you will see such a game for years)

Listen, let the police do the job, be sure i give you answers as soon as possible, okayyyyyyy

[GUNHED]

I always assumed that the CPR format was created for emulators. It has no advantages if you are working with real hardware, it just complicates things.

Bryce.

True. And probably true for emulators too.

[GUNHED]

So..... Bubble Quest will only run on emulators? No real Cartridges?

[andycadley]

So..... Bubble Quest will only run on emulators? No real Cartridges?

On the contrary. Bubble Quest currently only runs on hardware, none of the emulator file formats are currently capable of storing it.

[SyX]

please don't get me wrong, I don't want to get rid of the format, I was just wondering, why CPR was invented at all.
btw: why will Bubble Quest need an extension to the CPR format?

Ok, it looks like it is time for a history lesson.

It was the year 94 or 95 (pardon my memory but it was more than 30 years ago, then surely I will misremember some details). Well, arnoldemu was working on the first version of the Arnold emulator for Amiga computers, I helped him with tests, coding, preserving, making snapshots, ...

One important thing for us was to preserve CPC software and hardware. And after we got the CPC side in a decent state, we both thought that we should try to preserve CPC+ cartridges. You need to know that there was not any  CPC+ emulator yet, the first one was Multimachine by Paul Hogdson around 98. And the CPC community was nonexistent, aside from news or irc or  email, we didn't have another way of  contacting other cpc interested people, and only using the text terminals during laboratory hours in our respective faculties.

Our first try was preserving the games using a small program that would save all the cartridge pages to a floppy disk and then we would build a cartridge file manually joining the files. But for games where it was not Basic included, we used a multiface 2. Although later arnoldemu attached a cartridge to an amiga parallel port and he was able to read those pesky cartridges.

But if I remember well, years later all the official CPC cartridges were preserved by my friend deepfb.

But returning to CPR format, well, we were Amiga users and in the Amiga world, there was a standard for files, the IFF format. Then we took inspiration in that format for developing the CPR format (PNG format was based in IFF too).

We defined 2 main chunks:
1.- The main chunk or the container of the other chunks:
ID_CHUNK (L): RIFF (ROM IFF)
SIZE_CHUNK (L): Size of the data in little ending *
DATA: SIZE_CHUNK bytes (the full file length - this 8 bytes header)

2.- The cartridge page chunk, where every cartridge page would be defined like this chunk:
ID_CHUNK(L): CBxx (Cartridge Block + number cartridge page)
SIZE_CHUCK(L): Size of the data in little ending
DATA (B): cartridge page data

(*) We thought that we could parse those files in the CPC, then it we let in CPC native endianess.

The use of 32 bits (or L, for Long Word) for defining the sizes was inherited from the Amiga IFF format too.

Sincerely I don't remember now why that "AMS!" marker between the main chunk and the cartridge page chunks, it looks so awkward... maybe there was some file format using RIFF already and we wanted to identify our file format easily.

Basing the format in different types of chunks, it would let us easily extend the format in the case that we discovered a cartridge that wasn't a simple rom dump. Remember, that we didn't have every commercial CPC+ cartridge, I had 2 and arnoldemu had a similar number of cartridges. And of course, we didn't even dream about new cartridges being produced in the future.

But nothing prevents updating and cleaning the CPR format, because it was designed for being updated. A possible proposal for discussion would be:
1.- Main Chunk remains the same:
ID_CHUNK (L): "RIFF" (ascii upper case)
SIZE_CHUNK (L): Size of the data in little ending
DATA: SIZE_CHUNK bytes (the full file length - this 8 bytes header)

2.- We can ignore the "AMS!" string, because RIFF is enough for identifying as a CPC+ cartridge.

3.- For a standard cartridge rom page, we could continue using the CBxx chunk:
ID_CHUNK(L): "CBxx" (the number cartridge page in hexadecimal)
SIZE_CHUCK(L): Size of the data in little ending
DATA (B): cartridge page data

4.- But if a cartridge uses more than 256 pages, then we can use a new chunk instead of the old "CBxx", something like this:
ID_CHUNK(L): "Pxxx" (cartridge page number 000-FFF, 4096 pages should be enough, no? )
SIZE_CHUCK(L): Size of the cartridge page data in little ending
DATA (B): cartridge page data

5.- And if people need to specify a mapper type or any extra feature that is something else than a cartridge page, then we can define new chunks.

Then, speak!!!

The typical Burning Rubber cartridge would be something like this using the "Pxxx"chunk:
RIFF
0x00020040
P000
0x00004000
0xC3,0x86,0x01, ...
P001
0x00004000
0x00,0x00,0x00, ...
P002
0x00004000
0x00,0x00,0x00, ...

P003
0x00004000
0x00,0x00,0x00, ...

P004
0x00004000
0x00,0x00,0x00, ...

P005
0x00004000
0x00,0x00,0x00, ...

P006
0x00004000
0x00,0x00,0x00, ...

P007
0x00004000
0x00,0x00,0x00, ...

And I can convert every CPR archive released to use the new format or the format that we can agree on 5 minutes, then don't worry about compatibility.

[eto]

Ok, it looks like it is time for a history lesson.

great history lesson, thanks!

[andycadley]

I don't think there's any need to change existing cartridges. We can leave CB00-CB99 for existing cartridges and support P000-P999 just for cartridges that actually need it (or P100-P999 as it doesn't really matter if we mix and match anyway)

[PulkoMandy]

>  2.- We can ignore the "AMS!" string, because RIFF is enough for identifying as a CPC+ cartridge.

Please, no

The RIFF format is used by many other things, for example WAV files, AVI, WEBP, ...

The format is very well specified and starts with RIFF, then a size, then a file type identifier. This allows to identify very clearly what type of file it is.
https://en.wikipedia.org/wiki/Resource_Interchange_File_Format

And that's useful in various situations.

For example, let's say your emulator supports loading files by drag and drop on its window. You receive only the file content and not the filename from system APIs. In this case, you can very easily identify if the dropped file is a cartridge, a disk file, a snapshot or maybe a ROM image, and act accordingly. That's why these headers are for.

In my operating system here (Haiku), we never use file extensions for this, and always rely on the file contents. So, you can change the chunk format to have a single big chunk with all the data, sure, but please keep the header a standard RIFF one. It also makes it easy to edit the files with standard tools, for example Python has built-in support for it.

[SyX]

>  2.- We can ignore the "AMS!" string, because RIFF is enough for identifying as a CPC+ cartridge.

Please, no

The RIFF format is used by many other things, for example WAV files, AVI, WEBP, ...

The format is very well specified and starts with RIFF, then a size, then a file type identifier. This allows to identify very clearly what type of file it is.
https://en.wikipedia.org/wiki/Resource_Interchange_File_Format

And that's useful in various situations.

For example, let's say your emulator supports loading files by drag and drop on its window. You receive only the file content and not the filename from system APIs. In this case, you can very easily identify if the dropped file is a cartridge, a disk file, a snapshot or maybe a ROM image, and act accordingly. That's why these headers are for.

In my operating system here (Haiku), we never use file extensions for this, and always rely on the file contents. So, you can change the chunk format to have a single big chunk with all the data, sure, but please keep the header a standard RIFF one. It also makes it easy to edit the files with standard tools, for example Python has built-in support for it.

Thanks Pulko, I sincerely forget about the RIFF in the wav format, certainly that was the reason for having the "AMS!" string... ayyyysss, feeling old... hehehe

[eto]

Quick question: does the number in the chunk identify the page number or should the number be ignored? So does CB03 always identify page 4, or could it also be page 3, if e.g. CB02 is missing?

Could CB03 appear before CB02 in the CPR file but would still be mapped to page 4?

Background: I have a CPR file where CB02 is completely missing. Emulators and C4CPC read it correctly and map CP03 to page 4. But when I convert it back to BIN with CPRTools, then I get a 48KB BIN file instead of a 64KB bin file. 

[roudoudou]

if you have only 48K then it's only an extractor, not a converter. i guess it should be fixed easily

[andycadley]

It should always map to the page number, RIFF chunks are identified by the code and technically don't have to be in a given order IIRC. That said, o think there are emulators and other tools that don't cope if you miss chunks out or put them in the wrong order.