[SOLVED] CPCTelera 1.5 (de)compression problem

[Norman]

Greetings everyone!

So I'm currently making a game (my first time with CPCTelera) and I'm facing a problem. Using Tiled to create levels (or rather screens), I'm converting .tmx files to C arrays using the amazing tools CPCTelera provides (TMX2C here). A problem is that these arrays are taking ridiculously high amounts of memory, e.g. 1.2kB per screen, and if one level has ~10 screens... you can see where this is going. This data is easy to compress though, so I'm trying to utilize the ZX7B compression algorithm, integrated into CPCTelera. It took quite a bit of digging in old forum threads to come up with a way of...

   
• converting .tmx files to a C file with an array
• compiling that (this creates a .rel file, among others)
• creating a .ihx file using `$CPCT_PATH/tools/sdcc-3.6.8-r9946/bin/sdcc file.rel`
• creating a .bin file using `$CPCT_PATH/tools/hex2bin-2.0/bin/hex2bin file.ihx`
• finally converting that to another C file which contains the crunched data using `cpct_pack file file.bin`

However, this doesn't yield correct results when decrunching the data (following the 1.5 docs), namely the rest of the C file also gets carried over in above conversion, not only the array, but I only need the latter. The explanation by @ronaldo in the thread linked above makes it seem like everything from the C file gets stripped away except the array data... though that seems actually strange looking at it now.

For example, the .rel file looks like this:

Code Select Expand


XL2
H 9 areas 2 global symbols
M map1
O -mz80
S .__.ABS. Def0000
A _CODE size 528 flags 0 addr 0
S _g_map1 Def0000
A _DATA size 0 flags 0 addr 0
A _INITIALIZED size 0 flags 0 addr 0
A _DABS size 0 flags 8 addr 0
A _HOME size 0 flags 0 addr 0
A _GSINIT size 0 flags 0 addr 0
A _GSFINAL size 0 flags 0 addr 0
A _INITIALIZER size 0 flags 0 addr 0
A _CABS size 0 flags 8 addr 0
T 00 00
R 00 00 00 00
T 00 00 0E 0E 0E 0E 0E 0E 0E 0E 0E 0E 0E 0E 0E 0E
R 00 00 00 00
T 0E 00 0E 0E 0E 0E 0E 0E 0E 0E 0E 0E 0E 0E 0E 0E[...]
R 00 00 00 00
T 46 00 0E 0E 0E 0E 0E 0E 0E 0E 0E 0E 09 0E 0E 0E
[...]
(notice the 0Es and 09s)
The actual tilemap data is at the same position in the .bin file:

Code Select Expand

$ hexdump file.bin
0000000 00c3 ff01 ffff ffff 4ded ffff ffff ffff
0000010 4ded ffff ffff ffff 4ded ffff ffff ffff
*
0000040 ffff ffff ffff ffff ffff ffff ffff ffff
*
0000100 0031 cd00 0732 00cd c300 0204 ffff ffff
0000110 ffff ffff ffff ffff ffff ffff ffff ffff
*
0000200 023e c9cf 003e 76cf fd18 0e0e 0e0e 0e0e
0000210 0e0e 0e0e 0e0e 0e0e 0e0e 0e0e 0e0e 0e0e
*
0000250 0e0e 0e0e 0e0e 0e0e 0e0e 0e09 0e0e 0e09
0000260 0e0e 0e09 0e0e 0e09 0e0e 0e09 0e0e 0e09
*
0000280 0e0e 090e 090e 090e 090e 090e 090e 090e
0000290 090e 090e 090e 090e 090e 090e 090e 090e[...](same 0Es and 09s as above)So, am I missing something? How do I strip the C file of everything except the data? And would this be the correct (or at least a good) way to compress Tiled map files? Thanks in advance for any help.

Cheers
P.S.: I'd like to integrate the whole conversion process into the Makefiles later on, so tips on how to go about that would be a bonus 

[Arnaud]

Hi,
here the sh script i use to compress my tmx map.

I don't use backward compression with zx7 but it's just an option to add in script.

Code Select Expand

## Paths and files required for this script to work
INCLUDES=${CPCT_PATH}/src
SDCC=${CPCT_PATH}/tools/sdcc-3.6.8-r9946/bin/sdcc
HEX2BIN=${CPCT_PATH}/tools/hex2bin-2.0/bin/hex2bin
ZX7=$(pwd)/tools/zx7.exe

## Function that converts and compress 1 REL file
function convertAndCompress {
   cpct_tmx2csv "${1}.tmx" -gc -of ../temp >/dev/null 2>&1
   $SDCC -I$INCLUDES -mz80 -c "${1}.c"
   $SDCC "${1}.rel" --no-std-crt0 >/dev/null 2>&1
   $HEX2BIN  "${1}.ihx" >/dev/null 2>&1
   $ZX7 -f "${1}.bin" "${1}.zx7" >/dev/null 2>&1
}

ex :
convertAndCompress c:/cpc/map/forest.tmx

[ronaldo]

Greetings everyone!

So I'm currently making a game (my first time with CPCTelera) and I'm facing a problem. Using Tiled to create levels (or rather screens), I'm converting .tmx files to C arrays using the amazing tools CPCTelera provides (TMX2C here). A problem is that these arrays are taking ridiculously high amounts of memory, e.g. 1.2kB per screen, and if one level has ~10 screens... you can see where this is going. This data is easy to compress though, so I'm trying to utilize the ZX7B compression algorithm, integrated into CPCTelera. It took quite a bit of digging in old forum threads to come up with a way of...

This post was previous to present CPCtelera version. In CPCtelera 1.5 zx7 is integrated and you can easily ask CPCtelera to do the whole process for you. You don't have to create any script nor use manual commands.

In the cfg/ folder you have a file named compression.mk. Inside this file you have simple commands to create ZX7 compressed binaries including anything you like in them. For example

Code Select Expand


$(eval $(call ADD2PACK,scoreboard_pack,img/scoreboard.bin))
$(eval $(call PACKZX7B,scoreboard_pack,src/sprites))

These 2 lines create a PACK file named scoreboard_pack, that includes and compresses the file img/scoreboard.bin and generates compressed data in src/sprites. Generated compressed data will then be picked up by SDCC, assembled and included in your game.

The only think you have to take into account is that anything you want to compress should originally be in the same format you will use it once decompressed. Therefore, if you want to use data inside the CPC, you need the binary format of that data. You cannot compress a C array or a REL file: you need to compress the binary. For that reason, most of the tools from CPCtelera, and TMX2C in particular, can output binary data instead of C/ASM files. So, that scoreboard.bin from the previous example could be a tilemap generated by TMX2C in binary.

Have a look at the compression.mk file (it has detailed explanations inside). You may also have a look at some videos of my lessons, like this one (
https://www.youtube.com/watch?v=PdP40h7DH_k)

Hope this helps

[funkheld]

Hi good afternoon.
here is written by cpctelera 1.5


I do not find it.


thank you.
greeting

[Arnaud]

here is written by cpctelera 1.5
I do not find it.

cpctelera 1.5 is the development branch under git not yet a release.

[Norman]

Thanks for the quick answers.

Hi, here the sh script i use to compress my tmx map. I don't use backward compression with zx7 but it's just an option to add in script. [...]

How do you use the resulting zx7 files in your game? Perhaps you'd also benefit from the new CPCtelera features?

This post was previous to present CPCtelera version. In CPCtelera 1.5 zx7 is integrated and you can easily ask CPCtelera to do the whole process for you. You don't have to create any script nor use manual commands. In the cfg/ folder you have a file named compression.mk. Inside this file you have simple commands to create ZX7 compressed binaries including anything you like in them. For example

Code Select Expand

$(eval $(call ADD2PACK,scoreboard_pack,img/scoreboard.bin)) $(eval $(call PACKZX7B,scoreboard_pack,src/sprites)) These 2 lines create a PACK file named scoreboard_pack, that includes and compresses the file img/scoreboard.bin and generates compressed data in src/sprites. Generated compressed data will then be picked up by SDCC, assembled and included in your game. The only think you have to take into account is that anything you want to compress should originally be in the same format you will use it once decompressed. Therefore, if you want to use data inside the CPC, you need the binary format of that data. You cannot compress a C array or a REL file: you need to compress the binary. For that reason, most of the tools from CPCtelera, and TMX2C in particular, can output binary data instead of C/ASM files. So, that scoreboard.bin from the previous example could be a tilemap generated by TMX2C in binary. Have a look at the compression.mk file (it has detailed explanations inside). You may also have a look at some videos of my lessons, like this one [%u2026] Hope this helps

Thank you, that looks much better than what I tried to do, will test it now. The integration of ZX7 into CPCtelera is even better than I thought it was 

The memory map (like in the video) that gets generated automatically is nice to have.

Edit: TMX2DATA and compression now works great as well, but I've got another question regarding that: If a pack is created from multiple files, it probably has to be unpacked as a whole again? So if I want to compress maps and want to decompress one or more of them on the fly (e.g. 5 maps for one level), individual packs have to be created?

Cheers

[Norman]

Yeah, so everything is working now, one screen is currently using 50-80 bytes instead of 1320 which is amazing (it might go up a bit once the tiles have more variation). Decrunching screens on the fly works too, the algorithm is actually quite fast, so there's no noticeable delay. Maximum memory saved! 
Thanks!

[ronaldo]

Nice, @Norman . From your latest post I assume you have understood how everything works and now you have your workflow automated. That's really nice

In fact, everything you compress as a whole, you have to unpack it as a whole. There is no easy way to decompress part of a compressed file. The thing to keep in mind is to compress everything in the same exact state you want it decompressed. So, you may compress levels one by one to have 1 decompressed and the rest compressed, or you may compress levels in packs of 5 if you want to have 5 of them decompressed at the same time. With levels, it is quite normal to pack them one by one. However, one level could be composed of many files (tiles, a tilemap, specific graphics, a music, etc.). In this case, you would create one pack per level, but composed of many files. Everything depends on your needs. Only restriction: each pack is decompressed as a whole. With that in mind, you can easily distribute your assets at will.

If you have any other doubt, just ask. Hope it helps, and I'm curious to see what you are doing

[Norman]

Your assumption is correct, I have understood how everything works (related to compression ) and the workflow is also automated as it just requires an entry in the tilemap_conversion.mk and compression.mk files.
A pack with specific data for each level sounds good, will keep that in mind.

I'm participating in CPCRetroDev 2020, or at least trying to, hope I meet the deadline! It will be a platform game (my first CPC game, only dabbled with assembly before) where you solve little puzzles and fight enemies, and it will contain a reference to Prince of Persia .

Thanks for creating this great open source tool!

Cheers