CPCWiki forum

Hi fellows, Einar Saukas made a new promising cruncher!
Average ratio like exomizer, very slow crunching (i mean, VERY) but faster decrunch than exomizer
Just tried it on Boulder Dash binary
ZX0      : 10718
AP-Lib   : 10892
Exomizer : 11026
LZSA2    : 11415
ZX7      : 11651




reference: https://spectrumcomputing.co.uk/forums/viewtopic.php?f=6&t=4119
sources+exe: https://github.com/einar-saukas/ZX0

An 82 byte fast and efficient decompressor is pretty nice, and I learned something - elias encoding - from reading about it.


Edit: Actually 69 bytes (same as ZX7) or 20% faster in 128 bytes in a more recent version.

This looks impressive. Just out of curiosity, how many seconds does it take to decode the boulder dash binary?

i broke my setup (because i'm doing ZX0 integration to rasm :D ) but they post a GFX of this
ZX0 seems to be as fast as ZX7 which is 4x slower than LDIR (exomizer is 15x slower than LDIR and fastest decrunch like LZ4 are 1.5x slower than LDIR)

(https://i.postimg.cc/rmPz0r5G/image.png)

This is really impressive !! thank you @roudoudou (https://www.cpcwiki.eu/forum/index.php?action=profile;u=1714) for sharing this with us  ;)

Quote from: eto on 18:44, 02 February 21
This looks impressive. Just out of curiosity, how many seconds does it take to decode the boulder dash binary?

i'v done a test on very crunchable screen data (32K => 6K) and the decrunch speed is very good, 2x slower than LDIR (where LZSA2 is 1.5x slower than LDIR)
on boulder dash (28K => 10K), ZX0 is 2.7x slower than LDIR where LZSA2 is 2.1x slower than LDIR
so obviously when there is less compression, you lose more time in the decrunch management routine, i mean, there is more loss than LZSA2

Quote from: roudoudou on 08:47, 03 February 21i'v done a test on very crunchable screen data (32K => 6K) and the decrunch speed is very good,

I am sorry if this sounds dumb, but I have never dealt with compression on the CPC, so I have no idea what "very good" could mean. How many seconds will the CPC need to decrunch that screen or boulder dash?

Quote from: eto on 12:05, 03 February 21
I am sorry if this sounds dumb, but I have never dealt with compression on the CPC, so I have no idea what "very good" could mean. How many seconds will the CPC need to decrunch that screen or boulder dash?

LDIR is a memory copy. You can consider this as the fastest memory transfer (even it's not true) => 6us per byte

A standard screen of 16K is copied in 98304us and the CPC has 998400us clocks (312x64x50) per second
So, a standard screen is copied in 1/10s which is fast
Exomizer is 15x times slower than LDIR => 1.5s to depack a screen
ZX0 is 2.5x times slower in average => 0.25s to depack a screen

OK, this I understand ;-) thank you so much for the explanation.

is it worth writing a PI version to see if it is faster than the CPC (transfer binary from CPC to PI then uncrunch then transfer back) - perhaps the library can autocheck if the PI is plugged into the CPC?

Quote from: Sykobee (Briggsy) on 18:07, 02 February 21
An 82 byte fast and efficient decompressor is pretty nice, and I learned something - elias encoding - from reading about it.


Edit: Actually 69 bytes (same as ZX7) or 20% faster in 128 bytes in a more recent version.

69 bytes decruncher, damn good compression ratio, good speed, it's a kind of magic  8)

Quote from: roudoudou on 19:51, 02 February 21
i broke my setup (because i'm doing ZX0 integration to rasm :D ) but they post a GFX of this
ZX0 seems to be as fast as ZX7 which is 4x slower than LDIR (exomizer is 15x slower than LDIR and fastest decrunch like LZ4 are 1.5x slower than LDIR)

That plot was computed for the original data format and original decompressors, which were modified and additionally re-optimized to be made more efficient.

This is the current state of things:

(https://introspec.retroscene.org/compression/pareto_20210128.png)


There are 4 ZX0 decompressors on this plot. They are as follows (from left to right):

   
• 69-byte "standard" decompressor, which is about 4 times slower than LDIR copy (this is faster than, say, the fastest ApLib can do and more than twice as fast as the best Exomizer can do.
• 128-byte "turbo" decompressor is already faster than every other Z80 decompressor using mixed bit/byte stream. OK, every one that I know of. It decompresses less than 3 times slower than LDIR copy. Only byte-based compressors such as LZ4, LZF or LZSA1, or semi-byte compressors such as LZSA2 can compete in terms of the speed. Of course, their compression ratio is much lower.
• ~150-200-byte experimental compressor that is even faster than "turbo" that I am working on at the moment. Hopefully, it can become the more-or-less default speed-optimized decompressor.
• 562-byte "mega" decompressor is even faster, but its memory requirements are bit extreme.

Einar made somewhat smaller (and slower) version of the "mega" decompressor, as well as even bigger (and slightly faster) version too; I did not have time to test them on the same data. So, the fast decompression options are still evolving. What you can be assured is that, basically, ZX0 has cancelled the majority of other compression options. There are almost no reasons to use most other compressors. Hrust, MegaLZ, NRV2b, ZX7, Hrum, Pletter, ApLib - they will usually lose in most practical situations, both in terms of ratio and in terms of decompression speed. The only compressors that survive this onslaught are:

   
• The results for Exomizer on my figure relate to old(ish) Exomizer ver.3.0.2. The newest version of Exomizer (ver.3.1.0) compresses better and you should expect it to be a better compressor in situations where only the compression ratio matters. The difference is not large, but it is definitely there.
• LZ4, LZSA1 and LZSA2 are still relevant in situations where you mostly care about decompression speed and can sacrifice the ratio. However, the difference in speed is not as massive at it once was, so these compressors need to get better to stay relevant (and by getting better I mean that they probably need to both increase their ratio and to keep or to increase the decompression speed).

Would be nice to have Madrams Turbocruncher in this picture too.
A while ago I testet about a dozen of crunchers on CPC, Madram's tool won (significantly).
Here just some 'outtake' of the result:

Original file: &7920 Bytes long
--------------------------------
Exomizer 2.07 ....(PC)...: &3210  / PC
Bit Buster 2.1....(PC)...: &359F /
---------------------------------
MadRAM CPC Turbo Cruncher: &3DBB \
Cheese Cruncher 2.2......: &405D  \ CPC
Elmsoft Turbo Cruncher...: &44F1
Turbo Cruncher CPCAI.....: &44F1
Crown Cruncher 1.4.......: &4D5B
Crown Turbo Imploder 1.1.: &523E
Flower Cruncher..........: file too long
S-DAF....................: file too long

Quote from: GUNHED on 17:23, 04 February 21
Would be nice to have Madrams Turbocruncher in this picture too.
A while ago I testet about a dozen of crunchers on CPC, Madram's tool won (significantly).
Here just some 'outtake' of the result:

do you still have your test file? i do not see LZ48 neither LZ49 in the results. I may modify my crunchers to output LZ4 too

Quote from: GUNHED on 17:23, 04 February 21
Would be nice to have Madrams Turbocruncher in this picture too.
A while ago I testet about a dozen of crunchers on CPC, Madram's tool won (significantly).

There is not much point having it in the picture. First, I only use PC compressors as the matter of principle (not using PC compressors is simply wasting some compression potential, cannot possibly benefit you).


Second, in any case, BitBuster is approximately the same thing as ZX7. I.e. you know that the Turbocruncher's ratio is going to be quite a bit behind. I cannot assess its speed because I've never seen the decompressor, but given how much decompressor technology improved during the last 5-10 years, I'd be shocked to find if it is at least competitive with something like LZSA.

Quote from: introspec on 19:17, 04 February 21
There is not much point having it in the picture. First, I only use PC compressors as the matter of principle (not using PC compressors is simply wasting some compression potential, cannot possibly benefit you).


Second, in any case, BitBuster is approximately the same thing as ZX7. I.e. you know that the Turbocruncher's ratio is going to be quite a bit behind. I cannot assess its speed because I've never seen the decompressor, but given how much decompressor technology improved during the last 5-10 years, I'd be shocked to find if it is at least competitive with something like LZSA.

LZ48 and LZ49 do have Z80 crunchers. They beat LZ4/LZSA1 ratio but LZSA1 has really a tremendous speed

Quote from: roudoudou on 17:53, 04 February 21
do you still have your test file? i do not see LZ48 neither LZ49 in the results. I may modify my crunchers to output LZ4 too

Sorry, I forgot the file. But if you drop a file here I can of course repeat the tests and provide crunched size and seconds.  :)

Quote from: introspec on 19:17, 04 February 21
There is not much point having it in the picture. First, I only use PC compressors as the matter of principle (not using PC compressors is simply wasting some compression potential, cannot possibly benefit you).

I like to be able to do things on the real machine (CPC/Plus). Of course pretty much all CPC users probably have PC too (nowadays). But PCs will fail one day - CPCs not. ;D

Quote from: roudoudou on 20:02, 04 February 21
LZ48 and LZ49 do have Z80 crunchers. They beat LZ4/LZSA1 ratio but LZSA1 has really a tremendous speed

I only tested PC compressors for LZ48 and LZ49, but in my tests their average compression ratio is usually lower than the compression ratio of LZ4 or LZSA1. I can see that for some categories of files LZ48 and LZ49 overtake LZ4 (e.g. for text files). However, I am not seeing this for LZSA1. It would be interesting to compare the notes. Which compressor is the best compressor for LZ48 and LZ49?

Quote from: GUNHED on 02:09, 05 February 21
I like to be able to do things on the real machine (CPC/Plus). Of course pretty much all CPC users probably have PC too (nowadays). But PCs will fail one day - CPCs not. ;D

I am into compression (a lot), but I don't care much for platform wars. You can probably guess this from the fact that I am hanging around on a CPC forum while I am actually a ZX Spectrum guy :)


So, if you want best compression - use the PC, it will definitely help. Otherwise, it becomes an exercise in something different from compression per se.

Quote from: introspec on 11:02, 05 February 21
I only tested PC compressors for LZ48 and LZ49, but in my tests their average compression ratio is usually lower than the compression ratio of LZ4 or LZSA1. I can see that for some categories of files LZ48 and LZ49 overtake LZ4 (e.g. for text files). However, I am not seeing this for LZSA1. It would be interesting to compare the notes. Which compressor is the best compressor for LZ48 and LZ49?

My reference is always boulder dash executable :D
LZ48 was designed for very small data to crunch, in fact doing real time decrunch of tiles
LZ49 perform better on larger files because the offset can reach 512 instead of 256
the main purpose of LZ4* is to use it for developpment because all other crunchers are slow, or very slow

Quote from: introspec on 10:54, 04 February 21That plot was computed for the original data format and original decompressors, which were modified and additionally re-optimized to be made more efficient.

Which file is used for these measures?

Quote from: m_dr_m on 08:36, 22 February 21
Which file is used for these measures?

I'm also interested as i do not see Shrinkler without parity context  :P

Quote from: m_dr_m on 08:36, 22 February 21
Which file is used for these measures?

The figures are generated for the corpus that I prepared for the (now, very dated) review of compression for Z80 (https://hype.retroscene.org/blog/933.html) that I wrote back in 2017. It is part of the pack at the end of the article, see http://introspec.retroscene.org/hype/compression2017.7z (http://introspec.retroscene.org/hype/compression2017.7z).


This corpus tends to have somewhat large file size by the standards of 8-bit computers and it does not have enough graphics to be fully representitative for completely average data. However, I am still using this corpus because I have no time to re-create all of the datapoints in my figure from scratch, and also because most Z80 compressors are usually tested on much smaller sets of the files, so even though my corpus is not perfect, it may well be the best one currently available online.

Quote from: roudoudou on 09:35, 22 February 21
I'm also interested as i do not see Shrinkler without parity context  :P

Yes, it is absolutely high up on my TODO list, I just do not have much time these days.

Quote from: introspec on 11:05, 05 February 21. You can probably guess this from the fact that I am hanging around on a CPC forum while I am actually a ZX Spectrum guy

Boo, hiss!! Lol :D

Wow, great job @introspec (https://www.cpcwiki.eu/forum/index.php?action=profile;u=2783) , thanks!
If I've understood well (I should say "did't read too fast"), you got your ratio for the graph as total_uncrunched/total_crunched.
That is, you concatenated all the corpus.

I just quickly added this new compressor on real project and result is:


compared to exomizer2: ZX0 saved me 318 bytes (not very much but hey :) )


If I use shrinkler I gain 625 bytes but I don't like decompression speed so I will stick with ZX0 for now.

Thanks for sharing this and of course the author!


Sizes:


EXO2:  00009F30
ZX0:   00009DF2 (-318 bytes)
SHR:   00009CBF (-625 bytes)

Quote from: m_dr_m on 03:51, 03 March 21
If I've understood well (I should say "did't read too fast"), you got your ratio for the graph as total_uncrunched/total_crunched.
That is, you concatenated all the corpus.

I would be careful to use word concatenated. For my total uncompressed data - yes, I used the size of all files added together, which is the same size I would get if I was to concatenate the files. However, if I was to take such a massive concatenated file and compress it - my compressed size would become much smaller compared to the compressed size I am getting by compressing every file independently. Of course, this would not be possible, as many compressors in the test have a hard limit of 64K for the largest possible file, but this is another issues. So, overall, I would not say it is concatenated, I'd say it is just total. Also, the article shows the split for the individual file categories, so you can see the relative performance of compressors on specific types of data.

I'm struggling to make this work from BASIC. MAybe someone can help me to understand what needs to be done here.

I must admit, I still do not know assembler, so probably this is just a newbie mistake (Assembler book is alreeady here and I started to read...). I thought it should work if I add a few lines to the code, so HL and DE are properly populated. But when I call the code, I only get garbage on screen (and it returns to BASIC) or the computer halts or resets. So clearly I am missing something.

I want to store a compressed image and then decrunch it to screen memory. Decompressor starts at &4000, compressed data is at &9200, target is &c000. In memory, the data looks like I would expect it (compared with a hex editor).


These are the lines I added to the source code from github:



org $4000


ld    hl, $9200  ; source address
ld    de, $c000  ; target address
;

I call the decompressor with "call &4000" from BASIC.

i guess you need to use the AMSDOS compatible version (without AF' usage) or

DI : EX AF,AF' : PUSH AF : ... : POP AF : EX AF,AF' : EI

Quote from: roudoudou on 11:25, 07 May 21
i guess you need to use the AMSDOS compatible version (without AF' usage) or

Code Select Expand

DI : PUSH AF : ... : POP AF : EI

Ok, so I can't find a AMSDOS compatible version, but the hint with AF now helped (despite my lack of knowledge). Seems however that I have to preserve AF' and not AF.

this finally works:


di
ex af,af'
push af


ld    hl, &9000  ; source address (put "Cobra.scr.zx0" there)
ld    de, &c000  ; target address (screen memory in this case)


call dzx0_turbo


pop af
ex af,af'


ei
ret


thanks!

oups, i forgot the EX, i edited my previous post, well done!

Just in case someone else needs this, here is the code to use the zx0 decruncher in BASIC:




10 SYMBOL AFTER 256
20 MEMORY &9000
30 FOR i=&A600 TO &A695: REM can be any other feasible address
40 READ p:POKE i,p
50 NEXT
60 REM source=&9000:LOAD "screen.zx0",source
70 REM target=&C000
80 REM CALL &A600,source,target
90 END
100 DATA &DD,&66,&3,&DD,&6E,&2,&DD,&56,&1,&DD,&5E,&0,&F3,&8,&F5,&CD
110 DATA &16,&A6,&F1,&8,&FB,&C9,&1,&FF,&FF,&ED,&43,&46,&A6,&3,&3E,&80
120 DATA &18,&2D,&C,&87,&C2,&2A,&A6,&7E,&23,&17,&D4,&6D,&A6,&8,&AF,&91
130 DATA &C8,&47,&8,&4E,&23,&CB,&18,&CB,&19,&ED,&43,&46,&A6,&1,&1,&0
140 DATA &D4,&6D,&A6,&3,&E5,&21,&0,&0,&19,&ED,&B0,&E1,&87,&38,&D3,&C
150 DATA &87,&C2,&57,&A6,&7E,&23,&17,&D4,&6D,&A6,&ED,&B0,&87,&38,&C3,&C
160 DATA &87,&C2,&67,&A6,&7E,&23,&17,&D4,&6D,&A6,&C3,&44,&A6,&87,&CB,&11
170 DATA &87,&30,&FA,&C0,&7E,&23,&17,&D8,&87,&CB,&11,&87,&D8,&87,&CB,&11
180 DATA &87,&D8,&87,&CB,&11,&87,&D8,&87,&CB,&11,&CB,&10,&87,&30,&F8,&C0
190 DATA &7E,&23,&17,&30,&F2,&C9

It would be very good if you could compress several files into one, and when decompressing add one more parameter:

ld hl, source
ld de, destination
ld a, number_of_file
call zx0decompress

the usefulness is that being very similar and small files (like sprites), when compressing from the second it can make use of previously compressed chains or patterns.

Or you can just combine all your resources together into one file as it's going to be organised when in memory, then compress that. That's far simpler to deal with on the CPC side - decompress to final location, and you're done.

not if you don't need all of them at same time

Then you have a problem, because that is something you need to deal with even if you got exactly what you originally asked for.
It's also solved the same way.

ZX0 is incredible. Depending on the source it really helps to save a lot of space. Even for BASIC programs this can be helpful, e.g. for screens, music files or custom fonts. Or just to create disks with LOTS of stuff on it. (I just created some collections with around 20 games on a single disk - AMSDOS format 178K, 2 sides).

ZX0 hast received an update to v2. I have updated my code now too and also took the chance to make it more flexible as it's sometimes annoying if a binary is bound to a specific address in RAM. Therefore here 3 approaches that give more freedom where you want your decruncher to be placed in memory.

First some fully relocatable code. You can poke it to any RAM address and it should work:
1 REM *** ZX0 v2 decruncher / full relocatable / works at any ram address
2 REM ***
3 REM *** usage: poke data to the required ram address (base)
4 REM ***        call base,compressedSourceAddress,targetAddress,execAddress
5 REM ***     
6 REM ***        base can be any address in ram
7 REM ***        if execAdress is 0, decruncher will return to BASIC,
8 REM ***        otherwise, will call execAddress
9 REM ***
10 MEMORY &2FFF:dzxAddr=&A500:REM take any address you need
20 MODE 1:PRINT "loading and decrunching..."
30 RESTORE 60
40 idx=0
50 READ a$:IF a$<>"x" THEN POKE dzxAddr+idx,VAL("&"+a$):idx=idx+1:GOTO 50
60 DATA 18,0,F3,D9,D5,D9,FB,FD,E1,FD,E5,E1,11,4D,0,19,FD,75,41,FD,74,42,11,35,0,19,FD,75,55,FD,74,56,FD,75,5D,FD,74,5E,11,1,0,19,FD,75,6D,FD,74,6E,11,7,0,19,FD,75,7D,FD,74,7E,FD,36,1,3C,18,51,CD,4D,0,DD,66,1,DD,6E,0,7C,B5,C8,E9,1,FF,FF
70 DATA C5,3,3E,80,CD,82,0,ED,B0,87,38,D,CD,82,0,E3,E5,19,ED,B0,E1,E3,87,30,EB,C1,E,FE,CD,83,0,C,C8,41,4E,23,CB,18,CB,19,C5,1,1,0,D4,8A,0,3,18,DD,C,87,20,3,7E,23,17,D8,87,CB,11,CB,10,18,F2,DD,66,5,DD,6E,4,DD,56,3,DD,5E,2,18,A1,x
80 REM **** usage  call base,source,target,exec
90 REM *** if exec is 0, decruncher will return to BASIC, otherwise will call exec
100 sourceAddr=&3000:targetAddr=&5800:execAddr=&63F7
110 LOAD"rolahoy.zx0",sourceAddr
120 CALL dzxAddr,sourceAddr,targetAddr,execAddr
Disadvantage is of course that it requires more RAM than the pure decruncher and that you keep the data in your BASIC code too.

So this will create a BIN file that can be executed from a specific RAM address:
1 REM *** ZX0 v2 decruncher / relocate to any ram address and save as bin
2 REM ***
3 REM *** usage: change cruncherAddr to the desired target Address where
4 REM ***        the decruncher shall be located and run the program
5 REM ***     
6 REM ***        call cruncherAddr,sourceAddress,targetAddress,execAddress
7 REM ***             sourceAddress: address of compress data
8 REM ***             targetAddress: target adress for decompressed data
9 REM ***             execAddress will be called after decompression if not 0
10 MEMORY &2FFF
20 cruncherAddr=&A500:REM this is the address where dzx will later work
30 delta=&3E:dzxAddr=cruncherAddr-delta
40 idx=0
50 READ a$:IF a$<>"x" THEN POKE dzxAddr+idx,VAL("&"+a$):idx=idx+1:GOTO 50
60 DATA 18,0,F3,D9,D5,D9,FB,FD,E1,FD,E5,E1,11,4D,0,19,FD,75,41,FD,74,42,11,35,0,19,FD,75,55,FD,74,56,FD,75,5D,FD,74,5E,11,1,0,19,FD,75,6D,FD,74,6E,11,7,0,19,FD,75,7D,FD,74,7E,c9,0,0,0,18,51,CD,4D,0,DD,66,1,DD,6E,0,7C,B5,C8,E9,1,FF,FF
70 DATA C5,3,3E,80,CD,82,0,ED,B0,87,38,D,CD,82,0,E3,E5,19,ED,B0,E1,E3,87,30,EB,C1,E,FE,CD,83,0,C,C8,41,4E,23,CB,18,CB,19,C5,1,1,0,D4,8A,0,3,18,DD,C,87,20,3,7E,23,17,D8,87,CB,11,CB,10,18,F2,DD,66,5,DD,6E,4,DD,56,3,DD,5E,2,18,A1,x
75 CALL dzxAddr
80 filename$="dzx"+HEX$(cruncherAddr)+".bin"
90 SAVE filename$,b,cruncherAddr,idx-delta

And finally a very special edge case version that runs from invisible parts of the screen RAM. I once had the situation that I did not have enough space for the decruncher. So this version will be poked into parts of the screen RAM that are not visible. Of course this only works for standard screen modes where the screen starts at &c000.

1 REM *** ZX0 v2 decruncher / works from hidden screen ram
2 REM ***
3 REM *** usage: make sure, screen starts at &c000, normal screen mode
4 REM ***        (executing mode command will ensure this)
5 REM ***        call &c7d0,compressedSourceAddress,targetAddress,execAddress
6 REM ***
7 REM ***        if execAdress is 0, decruncher will return to BASIC,
8 REM ***        otherwise, will call execAddress
9 REM ***
10 SYMBOL AFTER 256:MEMORY &2FFF
20 MODE 1:PRINT "loading and decrunching..."
30 RESTORE 70
40 FOR i=0 TO 2:idx=0
50 READ a$:IF a$<>"x" THEN POKE &C7D0+&800*i+idx,VAL("&"+a$):idx=idx+1:GOTO 50
60 NEXT
70 DATA DD,66,5,DD,6E,4,DD,56,3,DD,5E,2,CD,D0,CF,DD,66,1,DD,6E,0,7C,B5,C8,E9,x
80 DATA 1,FF,FF,C5,3,3E,80,CD,EA,D7,ED,B0,87,DA,D0,D7,CD,EA,D7,E3,E5,19,ED,B0,E1,E3,87,D2,D7,CF,C3,D0,D7,x
90 DATA C1,E,FE,CD,EB,D7,C,C8,41,4E,23,CB,18,CB,19,C5,1,1,0,D4,F2,D7,3,C3,E3,CF,C,87,20,3,7E,23,17,D8,87,CB,11,CB,10,18,F2,x
100 dzxAddr=&C7D0:sourceAddr=&3000:targetAddr=&5800:execAddr=&63F7
110 LOAD"rolahoy.zx0",sourceAddr
120 CALL dzxAddr,sourceAddr,targetAddr,execAddr
All files are on the attached DSK:

• dzxreloc.bas
• dzxgen.bas
• dzxc000.bas

Plus a few extras:

• dzxrelo2.bas + dzxre.bin - fully relocatable version as binary, so you don't need the DATA statements in BASIC
• dzxa600.bin + dzxa600.bas - binary created with dzxgen + example how to use
• rolahoy.zx0 - just to see it all in action

I hope this is of some use for someone.

Thanks eto, that's really cool! :) :) :)

This ZX0 compressor is just fantastic.
When Eto asked me, if it's possible to have SymbOS in a CPR (https://www.cpcwiki.eu/forum/applications/symbos-3-1-released/msg228281/#msg228281) (CPC Plus catridge) I asked myself if it's possible to have as much apps and media stored into such a 512K media as well (like a "ROM disc").
So I discovered the ZX0 compressor, was completely amazed and fully integrated it into SymbOS.

The next SymbOS version supports loading of ZX0 compressed executables (EXE, COM etc.) but also provides functions for applications for decompressing data. You can load compressed data from an opened file in a transparent way like if it would be uncompressed, which makes it very easy to update even existing applications for handling compressed data.

All EXE (GUI executables) and COM (command line executables) files can be compressed, as well as some special executables "*.WDG" (desktop widgets) and "*.SAV" (screen saver).
Picture files (SGX (http://www.symbos.de/appinfo.htm?00004)) can be compressed, too, including desktop background pictures, as well as help files (HLP (http://www.symbos.de/appinfo.htm?00026)).
Last but not least I decided to add a header for compressed music files, which are supported in the SymAmp music player (http://www.symbos.de/appinfo.htm?00005). These are PT3, ST2 (Amstrad Soundtrakker 128), SKM (Amstrad Starkos Tracker) and SA2 (Adlib Tracker 2), which can all be ZX0-compressed now.

The compression ratio is just fantastic, in most cases it's better than ZIP! Now it's possible again to place the whole operating system with all system apps, background picture and additional stuff which is loaded during booting on one Amstrad standard disc side (178K).
I am using the Turbo decompressor, which is still small and has a great speed. It's directly integrated into the SymbOS kernel, which makes it possible to decompress data with the full linear size of up to 63K!

I hope to be able to release the next SymbOS version later this year.

Thanks so much to Einar Saukas (for the whole thing) and @introspec (for the Turbo decompressor)!

PS: Bye bye Crown Cruncher. I was using this tool since the beginning of the 90ies for demos and tools, and later for compressing the SymbOS core boot binaries (ROM and disc loader). It is now replaced by the ZX0 compressor. I loved the Crown Cruncher, thanks to Crown again, it was fast and familiar for me, but now after 30 years it's time to use much more improved stuff. E.g. Crown Cruncher compressed the SymbOS core from about 80K to 62K, while ZX0 reaches 47K, which is a lot.

Wonderful! This is probably more advanced than some modern operating systems :) 

How fast is the turbo decompressor?

E.g., is it faster to load from disk the zx0 compressed files, including decompression, than the uncompressed original files - i.e., the decompressor is faster than the disc read speed, given you are reading far less data now?

Quote from: Sykobee (Briggsy) on 10:17, 07 June 23E.g., is it faster to load from disk the zx0 compressed files, including decompression, than the uncompressed original files - i.e., the decompressor is faster than the disc read speed, given you are reading far less data now?

yes. it depends on the compression ratio of course, but loading 5KB screen data + decompressing is MUCH faster than loading 16KB from disc. Things might be different when you load from mass storage but for normal discs, it's worth it. The time required to decompress is about 3-4x the time a similar LDIR requires. More can be seen here: https://www.cpcwiki.eu/forum/programming/new-cruncher-zx0/msg197727/#msg197727

If it's still too slow, you can sacrifice another 1.5% of compression for 15% more performance with the ZX1 method: https://github.com/einar-saukas/ZX1 

However I didn't use that yet as my primary goal was to reduce space to get as much stuff on a disc or tape as possible. And compared to normal loading routines, it doesn't matter if the decompression is done in .5 or .4 seconds. 

E.g. I just made my "game tape collection" where I used ZX0 for many games. Loading times at 2000 baud shrink to 1.5 - 3.5 minutes, depending on the original game size (and if I could remove the loading screen). Over 50 games fit on two 60 minute tapes. I also made a disc collection where I can fit up to 13 games on a single side. Considering that discs are rare but I still like to load from real media, this is amazing.

Eto is right, you will save time when loading from floppy disc, but you will lose time, when you load from mass storage.

As mass storage on the CPC is about as fast as LDIR it is not possible to increase loading speed with compression, as any decompressor is always slower than LDIR.
But floppy is much slower than LDIR, so you will save time, when using a very good compressor with a very fast decompressor (like ZX0).

I made a test with Lymings, which loads 62K uncompressed or 29K compressed, when you start the app:

	62K uncompressed	29K compressed
floppy disc	11 s	9 s
mass storage	4 s	5 s

Tested only in WinApe, and when you start an application in SymbOS much more things are done like relocating the code, preparing the memory etc., which explains, why the difference with floppy disc is not that big. But I wanted to get a feeling how much time I lose, when loading from mass storage, and I think this is ok.

Quote from: Prodatron on 14:28, 07 June 23But I wanted to get a feeling how much time I lose, when loading from mass storage, and I think this is ok.

Compared to the space saved, the additional second is well worth it.

By chance, a mass storage has enough room to store unpacked files from folders if the speed is critical.

For FutureOS I'm using the Exomizer to handle compressed data since quite some years. This makes especially sense when working with XROMs (eXpansion ROMs for FutureOS). The compression rate of exomizer is fantastic and also the decompression speed is great.

Now I'm asking myself if it would make any sense to move from Exomizer to ZX0 compression. Of would the difference be just minor?

Quote from: TotO on 15:08, 07 June 23By chance, a mass storage has enough room to store unpacked files from folders if the speed is critical.

It will still be helpful if you're dealing with disk images or, in this case, with a Plus cartridge, where the ROM acts like a storage device. Space is still limited there and - considering I am sitting in front of an 8 bit machine - waiting 5 instead of 4 seconds seems like a good choice to store twice as much files on a DSK or on ROM.

Quote from: GUNHED on 15:57, 07 June 23Now I'm asking myself if it would make any sense to move from Exomizer to ZX0 compression. Of would the difference be just minor?

The compression rate of Exomizer seems to be slightly better than ZX0. However the speed of ZX0 is much better. 

If speed doesn't matter, there is no need to switch to ZX0. 

Quote from: eto on 17:28, 07 June 23It will still be helpful if you're dealing with disk images or, in this case, with a Plus cartridge, where the ROM acts like a storage device. Space is still limited there and - considering I am sitting in front of an 8 bit machine - waiting 5 instead of 4 seconds seems like a good choice to store twice as much files on a DSK or on ROM.

That is the reason the R-Type (2012) File System automatically unpack data when loading them from the floppy disc to speed-up all the access. About the ROM usage, it is sad to use it like a mass storage but if it is the case, it is faster to left data unpacked.

Quote from: eto on 17:28, 07 June 23The compression rate of Exomizer seems to be slightly better than ZX0. However the speed of ZX0 is much better.

The compression rate is more or less the same (both very perfect). I am not sure, if Exomizer is based on bit-trees like Huffman, but the decompression of Exomizer is very slow compared to ZX0, which is completely byte-based + using Elias encoding for offset and length, which is still very fast. It's crazy, that ZX0 is able to have such a great compression ratio but is still byte-based.
For me Exomizer wasn't an option as loading speed was always very important for me. So ZX0 is like a dream, as it has an ultra-short, ultra-fast decompressor and (nearly) the best compression-rate.

ZX0 is just unbelievable and ZX1 even faster to balance usages.

Quote from: TotO on 08:07, 08 June 23ZX0 is just unbelievable and ZX1 even faster to balance usages.

The only difference is, that ZX0 is using elias gamma coding for the upper (MSB) offset byte, which maybe only a few bits, while ZX1 is always using 1 (if within 127 bytes) or 2 (if more) full bytes for the offset.
Sometimes even ZX1 files are smaller (if a lot of offsets are <=127), but usually ZX0 wins in file size.

Sure, I'm not speaking about the size but the speed.

Quote from: Prodatron on 23:06, 06 June 23I loved the Crown Cruncher, thanks to Crown again, it was fast and familiar for me,

Are you aware that Crown stole most of "his" crunching related code? See http://memoryfull.net/articles.php?id=18 for details.

Probably this one from Richard Aplin: https://www.cpc-power.com/index.php?page=detail&num=12176

Haha, yes, but in this case Antoine is talking about the Crown Imploder, not the Crown Cruncher.
I didn't use the Crown Imploder, as decrunching speed was slow (probably very bit based huffman stuff), and I like to have a fast booting speed.
I am not sure, but I guess the Crown Cruncher code wasn't stolen as well, as this kind of compression is easier to implement compared to huffman.

Let me do a real life test... (5 to 7 tries for every part done)

- Loading of an application of 39 KB size from floppy disc: 2,7 seconds
- Loading of an application of 39 KB size from M4 SD card: 0,2 seconds (*)
- Exomizer-Uncompression of an application of 39 KB size from ROM: 2,9 seconds

(*) Maybe faster, but my fingers are not.

Comparing that with the before posted values, the decompression speed of Exomzer vs. ZX0 seems not too bad. Well, i guess at the moment I can stay with Exomizer without having a too bad consciousness.  ;)

Any idea how quick a 39 KB file can be uncompressed using ZX0?

From my experience, Exomizer 3 can be 10% better than ZX0 on GFX (but never on code) and it's also slow to decrunch whereas ZX0 is fast everytime
ZX0 decrunch is very small and do not need an additionnal buffer where Exo3 is big as 260 bytes + 256 aligned buffer

Quote from: GUNHED on 18:08, 09 June 23Any idea how quick a 39 KB file can be uncompressed using ZX0?

if you share the binary, I can try it. 

Quote from: eto on 19:14, 09 June 23

Quote from: GUNHED on 18:08, 09 June 23Any idea how quick a 39 KB file can be uncompressed using ZX0?

if you share the binary, I can try it.

Yes, of course. Here it comes...

What DSK is that? Winape can't properly read it and neither does ManageDSK.

I could extract it with RetroVM. Hopefully that worked properly.

Original size: 39808 bytes
Compressed size: 16540 bytes
time to decompress (standard decompressor): 0.93s

WoW! Less than a second!!! Great timing!!!
And thank you very much for testing.  :) :) :)

BTW: The DSK was from Caprice, WinApe likes no DSKs anyway  ;) Therefore I never use it. But maybe it was a problem of the extended file header.