LZSA compressor

[Targhan]

I don't think I heard about it here: LZSA, a new compression tool, including a z80 decompressor. Seems quite good.

[teopl]

Introspec is mentioning it in the last sentence here: http://www.cpcwiki.eu/forum/programming/compression-of-text-(and-other)/msg175902/#msg175902

In the post there are some other compressors mentioned also.

I personally didn't try any except ZX7 but I guess I will try eventually.

[reidrac]

I don't know, what I look for mostly is high compression ration and not that much speed. I started using LZ4 in a project and had to move to ZX7 because the compression ratio was awful for the data I was compressing.

Looking at the graph in https://github.com/emmanuel-marty/lzsa I would say ZX7 hits the sweet spot for me (although for Z80 based projects I still use UCL because for the type of data I usually compress it gives me a bit more than ZX7).

EDIT: OK, scrap that. LZSA2 seems very promising!

[introspec]

Hi, I helped to design LZSA and wrote Z80 decompressors. I'd be happy to answer any questions you might have. Oh, and if you have any optimizations for size/speed optimized decompressors, please let me know too.

@reidrac, if you are mainly after compression ratio, I'd suggest checking out Exomizer, ApLib, Hrust, MegaLZ (in order of decreasing compression ratio / increasing decompression speed).

In fact, I'd very interested in your uses of UCL. I only know of one UCL decompressor for Z80, found here: http://icculus.org/~aspirin/uclz80/
Unfortunately, I've never managed to get it to work. Could you please share what is your setup for using UCL on Z80?

[teopl]

I just tried it on binary file and ZX7 won.

I used random english text which is "pre compressed" by encoding most frequently used words with bytes not used in the text.

So this encoding alone made almost 40% smaller file.

Then I did compression on that encoded binary of size 6797 and results are:

Code Select Expand

LZSA2 = 4978
ZX7   = 4645

Attached is the endoded binary used for compression.

I will try exomizer and others later.

[introspec]

I just tried it on binary file and ZX7 won.

I used random english text which is "pre compressed" by encoding most frequently used words with bytes not used in the text. So this encoding alone made almost 40% smaller file.

Then I did compression on that encoded binary of size 6797 and results are:

Code Select Expand

LZSA2 = 4978
ZX7   = 4645

This is perfectly normal. Every compressor is guaranteed to lose out at least on some files. The test results used to generate plot at the github are produced on the corpus 1.2Mb (77 files, mostly from ZX Spectrum). I am currently moving to another, bigger corpus with 2.6Mb of files (267 files).

LZSA2 will not always beat ZX7 on ratio. They are mostly the same on average, but it does not compress, for example, graphics as well as ZX7. It has the edge where the data size is bigger, because it has much larger window size. On texts I usually see LZSA2 having a slight advantage compared to ZX7, but it is very slim, and all bets are off on specific files.

In fact, your mixing of two compression methods may be unhelpful; if I were you, I'd compare these compression results with the results obtained by ZX7 and LZSA2 on the original full text file.

I also did quick runs by other relevant compressors. LZSA2: 4975, ZX7: 4645, ApLib: 4580, MegaLZ: 4554, Hrust 1.3: 4546, Exomizer 3: 4449.

[reidrac]

@reidrac, if you are mainly after compression ratio, I'd suggest checking out Exomizer, ApLib, Hrust, MegaLZ (in order of decreasing compression ratio / increasing decompression speed).

In fact, I'd very interested in your uses of UCL. I only know of one UCL decompressor for Z80, found here: http://icculus.org/~aspirin/uclz80/
Unfortunately, I've never managed to get it to work. Could you please share what is your setup for using UCL on Z80?

That's the decompressor I use I believe! I converted it to SDCC assembler and I don't think I fixed any bugs or anything like that, but that was long time ago and perhaps I don't remember. It should be straightforward to use!

To compress the data I wrote a simple tool in C that uses libucl.

EDIT: ah, I suspect I know what is your problem... the UCL library is a pain to compile. I use Debian packages, but you can get the source and apply the same patches they use for it to compile with a modern GCC compiler.

EDIT2: see this: http://www.cpcwiki.eu/forum/programming/ucl-compressor/

[introspec]

Yes, you are right, I struggled to compile the UCL library. In addition, I knew that UCL library compressors are not very optimal - the author was selling better compressors as part of his commercial NRV library. This made me even less motivated to get UCL library to compile. At the same time, I found an alternative compressor that did not work at all with Z80 decompressor and it was difficult to see why it did not work without a working compressor. So, many thanks for sharing the compressor - it will help me to figure out why another (better) compressor is not compatible.


I'll prepare the results of some compression ratio/decompression speed tests in the other thread. What I can see already is that UCL is pretty slow to decompress and that its ratio is actually quite similar to LZSA2. In situations where compression ratio is more important, you would benefit from using ApLib, Hrust or even MegaLZ - they are all better than the standard UCL library compressor.

[reidrac]

I'm interested in your results!

The reason why I've been using UCL for a while is complex, but long story short could be that... I didn't know better, although it has served me well. It all depends!

For example, in large files it tends to be more efficient than ZX7 (e.g. compressing the game on tape or disk), and for small chunks of data it gave me acceptable compression ratio and OK speed (e.g. uncompressing map data entering a room; I've used it in all my CPC games).

It is very likely I could have done the same with ZX7, but I remember running some tests and getting slightly worse results in some cases and about the same in others. On top of that once I understood the UCL library is was slightly better to integrate in my compilation; so I kept using UCL because it was OK-ish and definitely easier than changing.

I run some tests last night and I've decided I'm moving to ZX7 for most of my use cases, but I wanted to share my UCL tools anyway.