Author Topic: New cruncher ZX0  (Read 3376 times)

0 Members and 1 Guest are viewing this topic.

Offline roudoudou

  • 6128 Plus
  • ******
  • Posts: 1.078
  • Country: fr
    • urban exploration
  • Liked: 1481
  • Likes Given: 860
New cruncher ZX0
« on: 16:42, 02 February 21 »
Hi fellows, Einar Saukas made a new promising cruncher!
Average ratio like exomizer, very slow crunching (i mean, VERY) but faster decrunch than exomizer
Just tried it on Boulder Dash binary
Code: [Select]
ZX0      : 10718
AP-Lib   : 10892
Exomizer : 11026
LZSA2    : 11415
ZX7      : 11651




reference: https://spectrumcomputing.co.uk/forums/viewtopic.php?f=6&t=4119
sources+exe: https://github.com/einar-saukas/ZX0
use RASM, the best assembler ever made :p

I will survive

Offline Sykobee (Briggsy)

  • 6128 Plus
  • ******
  • Posts: 828
  • Country: gb
  • Liked: 313
  • Likes Given: 489
Re: New cruncher ZX0
« Reply #1 on: 19:07, 02 February 21 »
An 82 byte fast and efficient decompressor is pretty nice, and I learned something - elias encoding - from reading about it.


Edit: Actually 69 bytes (same as ZX7) or 20% faster in 128 bytes in a more recent version.
« Last Edit: 19:09, 02 February 21 by Sykobee (Briggsy) »

Offline eto

  • Supporter
  • CPC6128
  • *
  • Posts: 268
  • Country: de
  • Liked: 183
  • Likes Given: 161
Re: New cruncher ZX0
« Reply #2 on: 19:44, 02 February 21 »
This looks impressive. Just out of curiosity, how many seconds does it take to decode the boulder dash binary?

Offline roudoudou

  • 6128 Plus
  • ******
  • Posts: 1.078
  • Country: fr
    • urban exploration
  • Liked: 1481
  • Likes Given: 860
Re: New cruncher ZX0
« Reply #3 on: 20:51, 02 February 21 »

i broke my setup (because i'm doing ZX0 integration to rasm :D ) but they post a GFX of this
ZX0 seems to be as fast as ZX7 which is 4x slower than LDIR (exomizer is 15x slower than LDIR and fastest decrunch like LZ4 are 1.5x slower than LDIR)


use RASM, the best assembler ever made :p

I will survive

Offline norecess

  • CPC6128
  • ****
  • Posts: 217
  • Country: ca
  • Liked: 420
  • Likes Given: 235
Re: New cruncher ZX0
« Reply #4 on: 21:55, 02 February 21 »
This is really impressive !! thank you @roudoudou for sharing this with us  ;)

Offline roudoudou

  • 6128 Plus
  • ******
  • Posts: 1.078
  • Country: fr
    • urban exploration
  • Liked: 1481
  • Likes Given: 860
Re: New cruncher ZX0
« Reply #5 on: 09:47, 03 February 21 »
This looks impressive. Just out of curiosity, how many seconds does it take to decode the boulder dash binary?
i'v done a test on very crunchable screen data (32K => 6K) and the decrunch speed is very good, 2x slower than LDIR (where LZSA2 is 1.5x slower than LDIR)
on boulder dash (28K => 10K), ZX0 is 2.7x slower than LDIR where LZSA2 is 2.1x slower than LDIR
so obviously when there is less compression, you lose more time in the decrunch management routine, i mean, there is more loss than LZSA2




use RASM, the best assembler ever made :p

I will survive

Offline eto

  • Supporter
  • CPC6128
  • *
  • Posts: 268
  • Country: de
  • Liked: 183
  • Likes Given: 161
Re: New cruncher ZX0
« Reply #6 on: 13:05, 03 February 21 »
i'v done a test on very crunchable screen data (32K => 6K) and the decrunch speed is very good,

I am sorry if this sounds dumb, but I have never dealt with compression on the CPC, so I have no idea what "very good" could mean. How many seconds will the CPC need to decrunch that screen or boulder dash?

Offline roudoudou

  • 6128 Plus
  • ******
  • Posts: 1.078
  • Country: fr
    • urban exploration
  • Liked: 1481
  • Likes Given: 860
Re: New cruncher ZX0
« Reply #7 on: 13:14, 03 February 21 »
I am sorry if this sounds dumb, but I have never dealt with compression on the CPC, so I have no idea what "very good" could mean. How many seconds will the CPC need to decrunch that screen or boulder dash?
LDIR is a memory copy. You can consider this as the fastest memory transfer (even it's not true) => 6us per byte

A standard screen of 16K is copied in 98304us and the CPC has 998400us clocks (312x64x50) per second
So, a standard screen is copied in 1/10s which is fast
Exomizer is 15x times slower than LDIR => 1.5s to depack a screen
ZX0 is 2.5x times slower in average => 0.25s to depack a screen


use RASM, the best assembler ever made :p

I will survive

Offline eto

  • Supporter
  • CPC6128
  • *
  • Posts: 268
  • Country: de
  • Liked: 183
  • Likes Given: 161
Re: New cruncher ZX0
« Reply #8 on: 13:20, 03 February 21 »
OK, this I understand ;-) thank you so much for the explanation.

Offline zhulien

  • 6128 Plus
  • ******
  • Posts: 650
  • Country: au
  • aka Vorax
    • 8bitology
  • Liked: 266
  • Likes Given: 260
Re: New cruncher ZX0
« Reply #9 on: 08:02, 04 February 21 »
is it worth writing a PI version to see if it is faster than the CPC (transfer binary from CPC to PI then uncrunch then transfer back) - perhaps the library can autocheck if the PI is plugged into the CPC?

Offline roudoudou

  • 6128 Plus
  • ******
  • Posts: 1.078
  • Country: fr
    • urban exploration
  • Liked: 1481
  • Likes Given: 860
Re: New cruncher ZX0
« Reply #10 on: 11:52, 04 February 21 »
An 82 byte fast and efficient decompressor is pretty nice, and I learned something - elias encoding - from reading about it.


Edit: Actually 69 bytes (same as ZX7) or 20% faster in 128 bytes in a more recent version.
69 bytes decruncher, damn good compression ratio, good speed, it's a kind of magic  8)
use RASM, the best assembler ever made :p

I will survive

Offline introspec

  • CPC464
  • **
  • Posts: 27
  • Country: gb
  • Liked: 42
  • Likes Given: 10
Re: New cruncher ZX0
« Reply #11 on: 11:54, 04 February 21 »
i broke my setup (because i'm doing ZX0 integration to rasm :D ) but they post a GFX of this
ZX0 seems to be as fast as ZX7 which is 4x slower than LDIR (exomizer is 15x slower than LDIR and fastest decrunch like LZ4 are 1.5x slower than LDIR)
That plot was computed for the original data format and original decompressors, which were modified and additionally re-optimized to be made more efficient.

This is the current state of things:




There are 4 ZX0 decompressors on this plot. They are as follows (from left to right):
  • 69-byte "standard" decompressor, which is about 4 times slower than LDIR copy (this is faster than, say, the fastest ApLib can do and more than twice as fast as the best Exomizer can do.
  • 128-byte "turbo" decompressor is already faster than every other Z80 decompressor using mixed bit/byte stream. OK, every one that I know of. It decompresses less than 3 times slower than LDIR copy. Only byte-based compressors such as LZ4, LZF or LZSA1, or semi-byte compressors such as LZSA2 can compete in terms of the speed. Of course, their compression ratio is much lower.
  • ~150-200-byte experimental compressor that is even faster than "turbo" that I am working on at the moment. Hopefully, it can become the more-or-less default speed-optimized decompressor.
  • 562-byte "mega" decompressor is even faster, but its memory requirements are bit extreme.
Einar made somewhat smaller (and slower) version of the "mega" decompressor, as well as even bigger (and slightly faster) version too; I did not have time to test them on the same data. So, the fast decompression options are still evolving. What you can be assured is that, basically, ZX0 has cancelled the majority of other compression options. There are almost no reasons to use most other compressors. Hrust, MegaLZ, NRV2b, ZX7, Hrum, Pletter, ApLib - they will usually lose in most practical situations, both in terms of ratio and in terms of decompression speed. The only compressors that survive this onslaught are:
  • The results for Exomizer on my figure relate to old(ish) Exomizer ver.3.0.2. The newest version of Exomizer (ver.3.1.0) compresses better and you should expect it to be a better compressor in situations where only the compression ratio matters. The difference is not large, but it is definitely there.
  • LZ4, LZSA1 and LZSA2 are still relevant in situations where you mostly care about decompression speed and can sacrifice the ratio. However, the difference in speed is not as massive at it once was, so these compressors need to get better to stay relevant (and by getting better I mean that they probably need to both increase their ratio and to keep or to increase the decompression speed).
« Last Edit: 12:23, 04 February 21 by introspec »

Offline GUNHED

  • 6128 Plus
  • ******
  • Posts: 2.591
  • Country: de
  • Reincarnation of TFM
    • FutureOS - The quickest OS for the CPC and Plus
  • Liked: 1226
  • Likes Given: 2886
Re: New cruncher ZX0
« Reply #12 on: 18:23, 04 February 21 »
Would be nice to have Madrams Turbocruncher in this picture too.
A while ago I testet about a dozen of crunchers on CPC, Madram's tool won (significantly).
Here just some 'outtake' of the result:

Original file: &7920 Bytes long
--------------------------------
Exomizer 2.07 ....(PC)...: &3210  / PC
Bit Buster 2.1....(PC)...: &359F /
---------------------------------
MadRAM CPC Turbo Cruncher: &3DBB \
Cheese Cruncher 2.2......: &405D  \ CPC
Elmsoft Turbo Cruncher...: &44F1
Turbo Cruncher CPCAI.....: &44F1
Crown Cruncher 1.4.......: &4D5B
Crown Turbo Imploder 1.1.: &523E
Flower Cruncher..........: file too long
S-DAF....................: file too long
http://futureos.de --> Get the revolutionary FutureOS (Recent update: 2021.01.24)
http://futureos.cpc-live.com/files/LambdaSpeak_RSX_by_TFM.zip --> Get the RSX-ROM for LambdaSpeak :-) (Updated: 2021.05.24)

Offline roudoudou

  • 6128 Plus
  • ******
  • Posts: 1.078
  • Country: fr
    • urban exploration
  • Liked: 1481
  • Likes Given: 860
Re: New cruncher ZX0
« Reply #13 on: 18:53, 04 February 21 »
Would be nice to have Madrams Turbocruncher in this picture too.
A while ago I testet about a dozen of crunchers on CPC, Madram's tool won (significantly).
Here just some 'outtake' of the result:
do you still have your test file? i do not see LZ48 neither LZ49 in the results. I may modify my crunchers to output LZ4 too
use RASM, the best assembler ever made :p

I will survive

Offline introspec

  • CPC464
  • **
  • Posts: 27
  • Country: gb
  • Liked: 42
  • Likes Given: 10
Re: New cruncher ZX0
« Reply #14 on: 20:17, 04 February 21 »
Would be nice to have Madrams Turbocruncher in this picture too.
A while ago I testet about a dozen of crunchers on CPC, Madram's tool won (significantly).
There is not much point having it in the picture. First, I only use PC compressors as the matter of principle (not using PC compressors is simply wasting some compression potential, cannot possibly benefit you).


Second, in any case, BitBuster is approximately the same thing as ZX7. I.e. you know that the Turbocruncher's ratio is going to be quite a bit behind. I cannot assess its speed because I've never seen the decompressor, but given how much decompressor technology improved during the last 5-10 years, I'd be shocked to find if it is at least competitive with something like LZSA.

Offline roudoudou

  • 6128 Plus
  • ******
  • Posts: 1.078
  • Country: fr
    • urban exploration
  • Liked: 1481
  • Likes Given: 860
Re: New cruncher ZX0
« Reply #15 on: 21:02, 04 February 21 »
There is not much point having it in the picture. First, I only use PC compressors as the matter of principle (not using PC compressors is simply wasting some compression potential, cannot possibly benefit you).


Second, in any case, BitBuster is approximately the same thing as ZX7. I.e. you know that the Turbocruncher's ratio is going to be quite a bit behind. I cannot assess its speed because I've never seen the decompressor, but given how much decompressor technology improved during the last 5-10 years, I'd be shocked to find if it is at least competitive with something like LZSA.
LZ48 and LZ49 do have Z80 crunchers. They beat LZ4/LZSA1 ratio but LZSA1 has really a tremendous speed
use RASM, the best assembler ever made :p

I will survive

Offline GUNHED

  • 6128 Plus
  • ******
  • Posts: 2.591
  • Country: de
  • Reincarnation of TFM
    • FutureOS - The quickest OS for the CPC and Plus
  • Liked: 1226
  • Likes Given: 2886
Re: New cruncher ZX0
« Reply #16 on: 03:07, 05 February 21 »
do you still have your test file? i do not see LZ48 neither LZ49 in the results. I may modify my crunchers to output LZ4 too
Sorry, I forgot the file. But if you drop a file here I can of course repeat the tests and provide crunched size and seconds.  :)
http://futureos.de --> Get the revolutionary FutureOS (Recent update: 2021.01.24)
http://futureos.cpc-live.com/files/LambdaSpeak_RSX_by_TFM.zip --> Get the RSX-ROM for LambdaSpeak :-) (Updated: 2021.05.24)

Offline GUNHED

  • 6128 Plus
  • ******
  • Posts: 2.591
  • Country: de
  • Reincarnation of TFM
    • FutureOS - The quickest OS for the CPC and Plus
  • Liked: 1226
  • Likes Given: 2886
Re: New cruncher ZX0
« Reply #17 on: 03:09, 05 February 21 »
There is not much point having it in the picture. First, I only use PC compressors as the matter of principle (not using PC compressors is simply wasting some compression potential, cannot possibly benefit you).
I like to be able to do things on the real machine (CPC/Plus). Of course pretty much all CPC users probably have PC too (nowadays). But PCs will fail one day - CPCs not. ;D
http://futureos.de --> Get the revolutionary FutureOS (Recent update: 2021.01.24)
http://futureos.cpc-live.com/files/LambdaSpeak_RSX_by_TFM.zip --> Get the RSX-ROM for LambdaSpeak :-) (Updated: 2021.05.24)

Offline introspec

  • CPC464
  • **
  • Posts: 27
  • Country: gb
  • Liked: 42
  • Likes Given: 10
Re: New cruncher ZX0
« Reply #18 on: 12:02, 05 February 21 »
LZ48 and LZ49 do have Z80 crunchers. They beat LZ4/LZSA1 ratio but LZSA1 has really a tremendous speed
I only tested PC compressors for LZ48 and LZ49, but in my tests their average compression ratio is usually lower than the compression ratio of LZ4 or LZSA1. I can see that for some categories of files LZ48 and LZ49 overtake LZ4 (e.g. for text files). However, I am not seeing this for LZSA1. It would be interesting to compare the notes. Which compressor is the best compressor for LZ48 and LZ49?

Offline introspec

  • CPC464
  • **
  • Posts: 27
  • Country: gb
  • Liked: 42
  • Likes Given: 10
Re: New cruncher ZX0
« Reply #19 on: 12:05, 05 February 21 »
I like to be able to do things on the real machine (CPC/Plus). Of course pretty much all CPC users probably have PC too (nowadays). But PCs will fail one day - CPCs not. ;D
I am into compression (a lot), but I don't care much for platform wars. You can probably guess this from the fact that I am hanging around on a CPC forum while I am actually a ZX Spectrum guy :)


So, if you want best compression - use the PC, it will definitely help. Otherwise, it becomes an exercise in something different from compression per se.

Offline roudoudou

  • 6128 Plus
  • ******
  • Posts: 1.078
  • Country: fr
    • urban exploration
  • Liked: 1481
  • Likes Given: 860
Re: New cruncher ZX0
« Reply #20 on: 14:05, 05 February 21 »
I only tested PC compressors for LZ48 and LZ49, but in my tests their average compression ratio is usually lower than the compression ratio of LZ4 or LZSA1. I can see that for some categories of files LZ48 and LZ49 overtake LZ4 (e.g. for text files). However, I am not seeing this for LZSA1. It would be interesting to compare the notes. Which compressor is the best compressor for LZ48 and LZ49?
My reference is always boulder dash executable :D
LZ48 was designed for very small data to crunch, in fact doing real time decrunch of tiles
LZ49 perform better on larger files because the offset can reach 512 instead of 256
the main purpose of LZ4* is to use it for developpment because all other crunchers are slow, or very slow
use RASM, the best assembler ever made :p

I will survive

Offline m_dr_m

  • CPC6128
  • ****
  • Posts: 233
  • Country: se
  • http://orgams.wikidot.com/
    • OrgaMS!
  • Liked: 159
  • Likes Given: 170
Re: New cruncher ZX0
« Reply #21 on: 09:36, 22 February 21 »
That plot was computed for the original data format and original decompressors, which were modified and additionally re-optimized to be made more efficient.

Which file is used for these measures?

Offline roudoudou

  • 6128 Plus
  • ******
  • Posts: 1.078
  • Country: fr
    • urban exploration
  • Liked: 1481
  • Likes Given: 860
Re: New cruncher ZX0
« Reply #22 on: 10:35, 22 February 21 »
Which file is used for these measures?
I'm also interested as i do not see Shrinkler without parity context  :P
use RASM, the best assembler ever made :p

I will survive

Offline introspec

  • CPC464
  • **
  • Posts: 27
  • Country: gb
  • Liked: 42
  • Likes Given: 10
Re: New cruncher ZX0
« Reply #23 on: 19:11, 02 March 21 »
Which file is used for these measures?
The figures are generated for the corpus that I prepared for the (now, very dated) review of compression for Z80 that I wrote back in 2017. It is part of the pack at the end of the article, see http://introspec.retroscene.org/hype/compression2017.7z.


This corpus tends to have somewhat large file size by the standards of 8-bit computers and it does not have enough graphics to be fully representitative for completely average data. However, I am still using this corpus because I have no time to re-create all of the datapoints in my figure from scratch, and also because most Z80 compressors are usually tested on much smaller sets of the files, so even though my corpus is not perfect, it may well be the best one currently available online.

Offline introspec

  • CPC464
  • **
  • Posts: 27
  • Country: gb
  • Liked: 42
  • Likes Given: 10
Re: New cruncher ZX0
« Reply #24 on: 19:12, 02 March 21 »
I'm also interested as i do not see Shrinkler without parity context  :P
Yes, it is absolutely high up on my TODO list, I just do not have much time these days.