New cruncher ZX0

Started by roudoudou, 16:42, 02 February 21

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

roudoudou

Hi fellows, Einar Saukas made a new promising cruncher!
Average ratio like exomizer, very slow crunching (i mean, VERY) but faster decrunch than exomizer
Just tried it on Boulder Dash binary
ZX0      : 10718
AP-Lib   : 10892
Exomizer : 11026
LZSA2    : 11415
ZX7      : 11651





reference: https://spectrumcomputing.co.uk/forums/viewtopic.php?f=6&t=4119
sources+exe: https://github.com/einar-saukas/ZX0
use RASM, the best assembler ever made :p

I will survive

Sykobee (Briggsy)

#1
An 82 byte fast and efficient decompressor is pretty nice, and I learned something - elias encoding - from reading about it.


Edit: Actually 69 bytes (same as ZX7) or 20% faster in 128 bytes in a more recent version.

eto

This looks impressive. Just out of curiosity, how many seconds does it take to decode the boulder dash binary?

roudoudou


i broke my setup (because i'm doing ZX0 integration to rasm :D ) but they post a GFX of this
ZX0 seems to be as fast as ZX7 which is 4x slower than LDIR (exomizer is 15x slower than LDIR and fastest decrunch like LZ4 are 1.5x slower than LDIR)


use RASM, the best assembler ever made :p

I will survive

norecess

This is really impressive !! thank you @roudoudou for sharing this with us  ;)

roudoudou

Quote from: eto on 19:44, 02 February 21
This looks impressive. Just out of curiosity, how many seconds does it take to decode the boulder dash binary?
i'v done a test on very crunchable screen data (32K => 6K) and the decrunch speed is very good, 2x slower than LDIR (where LZSA2 is 1.5x slower than LDIR)
on boulder dash (28K => 10K), ZX0 is 2.7x slower than LDIR where LZSA2 is 2.1x slower than LDIR
so obviously when there is less compression, you lose more time in the decrunch management routine, i mean, there is more loss than LZSA2




use RASM, the best assembler ever made :p

I will survive

eto

Quote from: roudoudou on 09:47, 03 February 21i'v done a test on very crunchable screen data (32K => 6K) and the decrunch speed is very good,

I am sorry if this sounds dumb, but I have never dealt with compression on the CPC, so I have no idea what "very good" could mean. How many seconds will the CPC need to decrunch that screen or boulder dash?

roudoudou

Quote from: eto on 13:05, 03 February 21
I am sorry if this sounds dumb, but I have never dealt with compression on the CPC, so I have no idea what "very good" could mean. How many seconds will the CPC need to decrunch that screen or boulder dash?
LDIR is a memory copy. You can consider this as the fastest memory transfer (even it's not true) => 6us per byte

A standard screen of 16K is copied in 98304us and the CPC has 998400us clocks (312x64x50) per second
So, a standard screen is copied in 1/10s which is fast
Exomizer is 15x times slower than LDIR => 1.5s to depack a screen
ZX0 is 2.5x times slower in average => 0.25s to depack a screen


use RASM, the best assembler ever made :p

I will survive

eto

OK, this I understand ;-) thank you so much for the explanation.

zhulien

is it worth writing a PI version to see if it is faster than the CPC (transfer binary from CPC to PI then uncrunch then transfer back) - perhaps the library can autocheck if the PI is plugged into the CPC?

roudoudou

Quote from: Sykobee (Briggsy) on 19:07, 02 February 21
An 82 byte fast and efficient decompressor is pretty nice, and I learned something - elias encoding - from reading about it.


Edit: Actually 69 bytes (same as ZX7) or 20% faster in 128 bytes in a more recent version.
69 bytes decruncher, damn good compression ratio, good speed, it's a kind of magic  8)
use RASM, the best assembler ever made :p

I will survive

introspec

#11
Quote from: roudoudou on 20:51, 02 February 21
i broke my setup (because i'm doing ZX0 integration to rasm :D ) but they post a GFX of this
ZX0 seems to be as fast as ZX7 which is 4x slower than LDIR (exomizer is 15x slower than LDIR and fastest decrunch like LZ4 are 1.5x slower than LDIR)
That plot was computed for the original data format and original decompressors, which were modified and additionally re-optimized to be made more efficient.

This is the current state of things:




There are 4 ZX0 decompressors on this plot. They are as follows (from left to right):

       
  • 69-byte "standard" decompressor, which is about 4 times slower than LDIR copy (this is faster than, say, the fastest ApLib can do and more than twice as fast as the best Exomizer can do.
  • 128-byte "turbo" decompressor is already faster than every other Z80 decompressor using mixed bit/byte stream. OK, every one that I know of. It decompresses less than 3 times slower than LDIR copy. Only byte-based compressors such as LZ4, LZF or LZSA1, or semi-byte compressors such as LZSA2 can compete in terms of the speed. Of course, their compression ratio is much lower.
  • ~150-200-byte experimental compressor that is even faster than "turbo" that I am working on at the moment. Hopefully, it can become the more-or-less default speed-optimized decompressor.
  • 562-byte "mega" decompressor is even faster, but its memory requirements are bit extreme.
Einar made somewhat smaller (and slower) version of the "mega" decompressor, as well as even bigger (and slightly faster) version too; I did not have time to test them on the same data. So, the fast decompression options are still evolving. What you can be assured is that, basically, ZX0 has cancelled the majority of other compression options. There are almost no reasons to use most other compressors. Hrust, MegaLZ, NRV2b, ZX7, Hrum, Pletter, ApLib - they will usually lose in most practical situations, both in terms of ratio and in terms of decompression speed. The only compressors that survive this onslaught are:

       
  • The results for Exomizer on my figure relate to old(ish) Exomizer ver.3.0.2. The newest version of Exomizer (ver.3.1.0) compresses better and you should expect it to be a better compressor in situations where only the compression ratio matters. The difference is not large, but it is definitely there.
  • LZ4, LZSA1 and LZSA2 are still relevant in situations where you mostly care about decompression speed and can sacrifice the ratio. However, the difference in speed is not as massive at it once was, so these compressors need to get better to stay relevant (and by getting better I mean that they probably need to both increase their ratio and to keep or to increase the decompression speed).

GUNHED

Would be nice to have Madrams Turbocruncher in this picture too.
A while ago I testet about a dozen of crunchers on CPC, Madram's tool won (significantly).
Here just some 'outtake' of the result:

Original file: &7920 Bytes long
--------------------------------
Exomizer 2.07 ....(PC)...: &3210  / PC
Bit Buster 2.1....(PC)...: &359F /
---------------------------------
MadRAM CPC Turbo Cruncher: &3DBB \
Cheese Cruncher 2.2......: &405D  \ CPC
Elmsoft Turbo Cruncher...: &44F1
Turbo Cruncher CPCAI.....: &44F1
Crown Cruncher 1.4.......: &4D5B
Crown Turbo Imploder 1.1.: &523E
Flower Cruncher..........: file too long
S-DAF....................: file too long
http://futureos.de --> Get the revolutionary FutureOS (Update: 2022.03.09)
http://futureos.cpc-live.com/files/LambdaSpeak_RSX_by_TFM.zip --> Get the RSX-ROM for LambdaSpeak :-) (Updated: 2021.12.26)

roudoudou

Quote from: GUNHED on 18:23, 04 February 21
Would be nice to have Madrams Turbocruncher in this picture too.
A while ago I testet about a dozen of crunchers on CPC, Madram's tool won (significantly).
Here just some 'outtake' of the result:
do you still have your test file? i do not see LZ48 neither LZ49 in the results. I may modify my crunchers to output LZ4 too
use RASM, the best assembler ever made :p

I will survive

introspec

Quote from: GUNHED on 18:23, 04 February 21
Would be nice to have Madrams Turbocruncher in this picture too.
A while ago I testet about a dozen of crunchers on CPC, Madram's tool won (significantly).
There is not much point having it in the picture. First, I only use PC compressors as the matter of principle (not using PC compressors is simply wasting some compression potential, cannot possibly benefit you).


Second, in any case, BitBuster is approximately the same thing as ZX7. I.e. you know that the Turbocruncher's ratio is going to be quite a bit behind. I cannot assess its speed because I've never seen the decompressor, but given how much decompressor technology improved during the last 5-10 years, I'd be shocked to find if it is at least competitive with something like LZSA.

roudoudou

Quote from: introspec on 20:17, 04 February 21
There is not much point having it in the picture. First, I only use PC compressors as the matter of principle (not using PC compressors is simply wasting some compression potential, cannot possibly benefit you).


Second, in any case, BitBuster is approximately the same thing as ZX7. I.e. you know that the Turbocruncher's ratio is going to be quite a bit behind. I cannot assess its speed because I've never seen the decompressor, but given how much decompressor technology improved during the last 5-10 years, I'd be shocked to find if it is at least competitive with something like LZSA.
LZ48 and LZ49 do have Z80 crunchers. They beat LZ4/LZSA1 ratio but LZSA1 has really a tremendous speed
use RASM, the best assembler ever made :p

I will survive

GUNHED

Quote from: roudoudou on 18:53, 04 February 21
do you still have your test file? i do not see LZ48 neither LZ49 in the results. I may modify my crunchers to output LZ4 too
Sorry, I forgot the file. But if you drop a file here I can of course repeat the tests and provide crunched size and seconds.  :)
http://futureos.de --> Get the revolutionary FutureOS (Update: 2022.03.09)
http://futureos.cpc-live.com/files/LambdaSpeak_RSX_by_TFM.zip --> Get the RSX-ROM for LambdaSpeak :-) (Updated: 2021.12.26)

GUNHED

Quote from: introspec on 20:17, 04 February 21
There is not much point having it in the picture. First, I only use PC compressors as the matter of principle (not using PC compressors is simply wasting some compression potential, cannot possibly benefit you).
I like to be able to do things on the real machine (CPC/Plus). Of course pretty much all CPC users probably have PC too (nowadays). But PCs will fail one day - CPCs not. ;D
http://futureos.de --> Get the revolutionary FutureOS (Update: 2022.03.09)
http://futureos.cpc-live.com/files/LambdaSpeak_RSX_by_TFM.zip --> Get the RSX-ROM for LambdaSpeak :-) (Updated: 2021.12.26)

introspec

Quote from: roudoudou on 21:02, 04 February 21
LZ48 and LZ49 do have Z80 crunchers. They beat LZ4/LZSA1 ratio but LZSA1 has really a tremendous speed
I only tested PC compressors for LZ48 and LZ49, but in my tests their average compression ratio is usually lower than the compression ratio of LZ4 or LZSA1. I can see that for some categories of files LZ48 and LZ49 overtake LZ4 (e.g. for text files). However, I am not seeing this for LZSA1. It would be interesting to compare the notes. Which compressor is the best compressor for LZ48 and LZ49?

introspec

Quote from: GUNHED on 03:09, 05 February 21
I like to be able to do things on the real machine (CPC/Plus). Of course pretty much all CPC users probably have PC too (nowadays). But PCs will fail one day - CPCs not. ;D
I am into compression (a lot), but I don't care much for platform wars. You can probably guess this from the fact that I am hanging around on a CPC forum while I am actually a ZX Spectrum guy :)


So, if you want best compression - use the PC, it will definitely help. Otherwise, it becomes an exercise in something different from compression per se.

roudoudou

Quote from: introspec on 12:02, 05 February 21
I only tested PC compressors for LZ48 and LZ49, but in my tests their average compression ratio is usually lower than the compression ratio of LZ4 or LZSA1. I can see that for some categories of files LZ48 and LZ49 overtake LZ4 (e.g. for text files). However, I am not seeing this for LZSA1. It would be interesting to compare the notes. Which compressor is the best compressor for LZ48 and LZ49?
My reference is always boulder dash executable :D
LZ48 was designed for very small data to crunch, in fact doing real time decrunch of tiles
LZ49 perform better on larger files because the offset can reach 512 instead of 256
the main purpose of LZ4* is to use it for developpment because all other crunchers are slow, or very slow
use RASM, the best assembler ever made :p

I will survive

m_dr_m

Quote from: introspec on 11:54, 04 February 21That plot was computed for the original data format and original decompressors, which were modified and additionally re-optimized to be made more efficient.

Which file is used for these measures?

roudoudou

Quote from: m_dr_m on 09:36, 22 February 21
Which file is used for these measures?
I'm also interested as i do not see Shrinkler without parity context  :P
use RASM, the best assembler ever made :p

I will survive

introspec

Quote from: m_dr_m on 09:36, 22 February 21
Which file is used for these measures?
The figures are generated for the corpus that I prepared for the (now, very dated) review of compression for Z80 that I wrote back in 2017. It is part of the pack at the end of the article, see http://introspec.retroscene.org/hype/compression2017.7z.


This corpus tends to have somewhat large file size by the standards of 8-bit computers and it does not have enough graphics to be fully representitative for completely average data. However, I am still using this corpus because I have no time to re-create all of the datapoints in my figure from scratch, and also because most Z80 compressors are usually tested on much smaller sets of the files, so even though my corpus is not perfect, it may well be the best one currently available online.

introspec

Quote from: roudoudou on 10:35, 22 February 21
I'm also interested as i do not see Shrinkler without parity context  :P
Yes, it is absolutely high up on my TODO list, I just do not have much time these days.

Powered by SMFPacks Menu Editor Mod