News:

Printed Amstrad Addict magazine announced, check it out here!

Main Menu
avatar_roudoudou

New cruncher ZX0

Started by roudoudou, 15:42, 02 February 21

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Animalgril987

Quote from: introspec on 11:05, 05 February 21. You can probably guess this from the fact that I am hanging around on a CPC forum while I am actually a ZX Spectrum guy
Boo, hiss!! Lol :D

m_dr_m

Wow, great job @introspec , thanks!
If I've understood well (I should say "did't read too fast"), you got your ratio for the graph as total_uncrunched/total_crunched.
That is, you concatenated all the corpus.

teopl


I just quickly added this new compressor on real project and result is:


compared to exomizer2: ZX0 saved me 318 bytes (not very much but hey :) )


If I use shrinkler I gain 625 bytes but I don't like decompression speed so I will stick with ZX0 for now.

Thanks for sharing this and of course the author!


Sizes:


EXO2:  00009F30
ZX0:   00009DF2 (-318 bytes)
SHR:   00009CBF (-625 bytes)


introspec

Quote from: m_dr_m on 03:51, 03 March 21
If I've understood well (I should say "did't read too fast"), you got your ratio for the graph as total_uncrunched/total_crunched.
That is, you concatenated all the corpus.
I would be careful to use word concatenated. For my total uncompressed data - yes, I used the size of all files added together, which is the same size I would get if I was to concatenate the files. However, if I was to take such a massive concatenated file and compress it - my compressed size would become much smaller compared to the compressed size I am getting by compressing every file independently. Of course, this would not be possible, as many compressors in the test have a hard limit of 64K for the largest possible file, but this is another issues. So, overall, I would not say it is concatenated, I'd say it is just total. Also, the article shows the split for the individual file categories, so you can see the relative performance of compressors on specific types of data.

eto

I'm struggling to make this work from BASIC. MAybe someone can help me to understand what needs to be done here.

I must admit, I still do not know assembler, so probably this is just a newbie mistake (Assembler book is alreeady here and I started to read...). I thought it should work if I add a few lines to the code, so HL and DE are properly populated. But when I call the code, I only get garbage on screen (and it returns to BASIC) or the computer halts or resets. So clearly I am missing something.

I want to store a compressed image and then decrunch it to screen memory. Decompressor starts at &4000, compressed data is at &9200, target is &c000. In memory, the data looks like I would expect it (compared with a hex editor).


These are the lines I added to the source code from github:



org $4000


ld    hl, $9200  ; source address
ld    de, $c000  ; target address
;


I call the decompressor with "call &4000" from BASIC.

roudoudou

#30

i guess you need to use the AMSDOS compatible version (without AF' usage) or

DI : EX AF,AF' : PUSH AF : ... : POP AF : EX AF,AF' : EI
My pronouns are RASM and ACE

eto

Quote from: roudoudou on 11:25, 07 May 21
i guess you need to use the AMSDOS compatible version (without AF' usage) or DI : PUSH AF : ... : POP AF : EI

Ok, so I can't find a AMSDOS compatible version, but the hint with AF now helped (despite my lack of knowledge). Seems however that I have to preserve AF' and not AF.

this finally works:


di
ex af,af'
push af


ld    hl, &9000  ; source address (put "Cobra.scr.zx0" there)
ld    de, &c000  ; target address (screen memory in this case)


call dzx0_turbo


pop af
ex af,af'


ei
ret



thanks!

roudoudou

oups, i forgot the EX, i edited my previous post, well done!
My pronouns are RASM and ACE

eto

Just in case someone else needs this, here is the code to use the zx0 decruncher in BASIC:




10 SYMBOL AFTER 256
20 MEMORY &9000
30 FOR i=&A600 TO &A695: REM can be any other feasible address
40 READ p:POKE i,p
50 NEXT
60 REM source=&9000:LOAD "screen.zx0",source
70 REM target=&C000
80 REM CALL &A600,source,target
90 END
100 DATA &DD,&66,&3,&DD,&6E,&2,&DD,&56,&1,&DD,&5E,&0,&F3,&8,&F5,&CD
110 DATA &16,&A6,&F1,&8,&FB,&C9,&1,&FF,&FF,&ED,&43,&46,&A6,&3,&3E,&80
120 DATA &18,&2D,&C,&87,&C2,&2A,&A6,&7E,&23,&17,&D4,&6D,&A6,&8,&AF,&91
130 DATA &C8,&47,&8,&4E,&23,&CB,&18,&CB,&19,&ED,&43,&46,&A6,&1,&1,&0
140 DATA &D4,&6D,&A6,&3,&E5,&21,&0,&0,&19,&ED,&B0,&E1,&87,&38,&D3,&C
150 DATA &87,&C2,&57,&A6,&7E,&23,&17,&D4,&6D,&A6,&ED,&B0,&87,&38,&C3,&C
160 DATA &87,&C2,&67,&A6,&7E,&23,&17,&D4,&6D,&A6,&C3,&44,&A6,&87,&CB,&11
170 DATA &87,&30,&FA,&C0,&7E,&23,&17,&D8,&87,&CB,&11,&87,&D8,&87,&CB,&11
180 DATA &87,&D8,&87,&CB,&11,&87,&D8,&87,&CB,&11,&CB,&10,&87,&30,&F8,&C0
190 DATA &7E,&23,&17,&30,&F2,&C9




Fran123


It would be very good if you could compress several files into one, and when decompressing add one more parameter:

ld hl, source
ld de, destination
ld a, number_of_file
call zx0decompress

the usefulness is that being very similar and small files (like sprites), when compressing from the second it can make use of previously compressed chains or patterns.

pelrun

Or you can just combine all your resources together into one file as it's going to be organised when in memory, then compress that. That's far simpler to deal with on the CPC side - decompress to final location, and you're done.

Fran123

not if you don't need all of them at same time

pelrun

Then you have a problem, because that is something you need to deal with even if you got exactly what you originally asked for.
It's also solved the same way.

eto

ZX0 is incredible. Depending on the source it really helps to save a lot of space. Even for BASIC programs this can be helpful, e.g. for screens, music files or custom fonts. Or just to create disks with LOTS of stuff on it. (I just created some collections with around 20 games on a single disk - AMSDOS format 178K, 2 sides).

ZX0 hast received an update to v2. I have updated my code now too and also took the chance to make it more flexible as it's sometimes annoying if a binary is bound to a specific address in RAM. Therefore here 3 approaches that give more freedom where you want your decruncher to be placed in memory.

First some fully relocatable code. You can poke it to any RAM address and it should work:
1 REM *** ZX0 v2 decruncher / full relocatable / works at any ram address
2 REM ***
3 REM *** usage: poke data to the required ram address (base)
4 REM ***        call base,compressedSourceAddress,targetAddress,execAddress
5 REM ***     
6 REM ***        base can be any address in ram
7 REM ***        if execAdress is 0, decruncher will return to BASIC,
8 REM ***        otherwise, will call execAddress
9 REM ***
10 MEMORY &2FFF:dzxAddr=&A500:REM take any address you need
20 MODE 1:PRINT "loading and decrunching..."
30 RESTORE 60
40 idx=0
50 READ a$:IF a$<>"x" THEN POKE dzxAddr+idx,VAL("&"+a$):idx=idx+1:GOTO 50
60 DATA 18,0,F3,D9,D5,D9,FB,FD,E1,FD,E5,E1,11,4D,0,19,FD,75,41,FD,74,42,11,35,0,19,FD,75,55,FD,74,56,FD,75,5D,FD,74,5E,11,1,0,19,FD,75,6D,FD,74,6E,11,7,0,19,FD,75,7D,FD,74,7E,FD,36,1,3C,18,51,CD,4D,0,DD,66,1,DD,6E,0,7C,B5,C8,E9,1,FF,FF
70 DATA C5,3,3E,80,CD,82,0,ED,B0,87,38,D,CD,82,0,E3,E5,19,ED,B0,E1,E3,87,30,EB,C1,E,FE,CD,83,0,C,C8,41,4E,23,CB,18,CB,19,C5,1,1,0,D4,8A,0,3,18,DD,C,87,20,3,7E,23,17,D8,87,CB,11,CB,10,18,F2,DD,66,5,DD,6E,4,DD,56,3,DD,5E,2,18,A1,x
80 REM **** usage  call base,source,target,exec
90 REM *** if exec is 0, decruncher will return to BASIC, otherwise will call exec
100 sourceAddr=&3000:targetAddr=&5800:execAddr=&63F7
110 LOAD"rolahoy.zx0",sourceAddr
120 CALL dzxAddr,sourceAddr,targetAddr,execAddr

Disadvantage is of course that it requires more RAM than the pure decruncher and that you keep the data in your BASIC code too.

So this will create a BIN file that can be executed from a specific RAM address:
1 REM *** ZX0 v2 decruncher / relocate to any ram address and save as bin
2 REM ***
3 REM *** usage: change cruncherAddr to the desired target Address where
4 REM ***        the decruncher shall be located and run the program
5 REM ***     
6 REM ***        call cruncherAddr,sourceAddress,targetAddress,execAddress
7 REM ***             sourceAddress: address of compress data
8 REM ***             targetAddress: target adress for decompressed data
9 REM ***             execAddress will be called after decompression if not 0
10 MEMORY &2FFF
20 cruncherAddr=&A500:REM this is the address where dzx will later work
30 delta=&3E:dzxAddr=cruncherAddr-delta
40 idx=0
50 READ a$:IF a$<>"x" THEN POKE dzxAddr+idx,VAL("&"+a$):idx=idx+1:GOTO 50
60 DATA 18,0,F3,D9,D5,D9,FB,FD,E1,FD,E5,E1,11,4D,0,19,FD,75,41,FD,74,42,11,35,0,19,FD,75,55,FD,74,56,FD,75,5D,FD,74,5E,11,1,0,19,FD,75,6D,FD,74,6E,11,7,0,19,FD,75,7D,FD,74,7E,c9,0,0,0,18,51,CD,4D,0,DD,66,1,DD,6E,0,7C,B5,C8,E9,1,FF,FF
70 DATA C5,3,3E,80,CD,82,0,ED,B0,87,38,D,CD,82,0,E3,E5,19,ED,B0,E1,E3,87,30,EB,C1,E,FE,CD,83,0,C,C8,41,4E,23,CB,18,CB,19,C5,1,1,0,D4,8A,0,3,18,DD,C,87,20,3,7E,23,17,D8,87,CB,11,CB,10,18,F2,DD,66,5,DD,6E,4,DD,56,3,DD,5E,2,18,A1,x
75 CALL dzxAddr
80 filename$="dzx"+HEX$(cruncherAddr)+".bin"
90 SAVE filename$,b,cruncherAddr,idx-delta


And finally a very special edge case version that runs from invisible parts of the screen RAM. I once had the situation that I did not have enough space for the decruncher. So this version will be poked into parts of the screen RAM that are not visible. Of course this only works for standard screen modes where the screen starts at &c000.

1 REM *** ZX0 v2 decruncher / works from hidden screen ram
2 REM ***
3 REM *** usage: make sure, screen starts at &c000, normal screen mode
4 REM ***        (executing mode command will ensure this)
5 REM ***        call &c7d0,compressedSourceAddress,targetAddress,execAddress
6 REM ***
7 REM ***        if execAdress is 0, decruncher will return to BASIC,
8 REM ***        otherwise, will call execAddress
9 REM ***
10 SYMBOL AFTER 256:MEMORY &2FFF
20 MODE 1:PRINT "loading and decrunching..."
30 RESTORE 70
40 FOR i=0 TO 2:idx=0
50 READ a$:IF a$<>"x" THEN POKE &C7D0+&800*i+idx,VAL("&"+a$):idx=idx+1:GOTO 50
60 NEXT
70 DATA DD,66,5,DD,6E,4,DD,56,3,DD,5E,2,CD,D0,CF,DD,66,1,DD,6E,0,7C,B5,C8,E9,x
80 DATA 1,FF,FF,C5,3,3E,80,CD,EA,D7,ED,B0,87,DA,D0,D7,CD,EA,D7,E3,E5,19,ED,B0,E1,E3,87,D2,D7,CF,C3,D0,D7,x
90 DATA C1,E,FE,CD,EB,D7,C,C8,41,4E,23,CB,18,CB,19,C5,1,1,0,D4,F2,D7,3,C3,E3,CF,C,87,20,3,7E,23,17,D8,87,CB,11,CB,10,18,F2,x
100 dzxAddr=&C7D0:sourceAddr=&3000:targetAddr=&5800:execAddr=&63F7
110 LOAD"rolahoy.zx0",sourceAddr
120 CALL dzxAddr,sourceAddr,targetAddr,execAddr

All files are on the attached DSK:

  • dzxreloc.bas
  • dzxgen.bas
  • dzxc000.bas

Plus a few extras:

  • dzxrelo2.bas + dzxre.bin - fully relocatable version as binary, so you don't need the DATA statements in BASIC
  • dzxa600.bin + dzxa600.bas - binary created with dzxgen + example how to use
  • rolahoy.zx0 - just to see it all in action

I hope this is of some use for someone.



GUNHED

Thanks eto, that's really cool! :) :) :)
http://futureos.de --> Get the revolutionary FutureOS (Update: 2023.11.30)
http://futureos.cpc-live.com/files/LambdaSpeak_RSX_by_TFM.zip --> Get the RSX-ROM for LambdaSpeak :-) (Updated: 2021.12.26)

Prodatron

This ZX0 compressor is just fantastic.
When Eto asked me, if it's possible to have SymbOS in a CPR (CPC Plus catridge) I asked myself if it's possible to have as much apps and media stored into such a 512K media as well (like a "ROM disc").
So I discovered the ZX0 compressor, was completely amazed and fully integrated it into SymbOS.

The next SymbOS version supports loading of ZX0 compressed executables (EXE, COM etc.) but also provides functions for applications for decompressing data. You can load compressed data from an opened file in a transparent way like if it would be uncompressed, which makes it very easy to update even existing applications for handling compressed data.

All EXE (GUI executables) and COM (command line executables) files can be compressed, as well as some special executables "*.WDG" (desktop widgets) and "*.SAV" (screen saver).
Picture files (SGX) can be compressed, too, including desktop background pictures, as well as help files (HLP).
Last but not least I decided to add a header for compressed music files, which are supported in the SymAmp music player. These are PT3, ST2 (Amstrad Soundtrakker 128), SKM (Amstrad Starkos Tracker) and SA2 (Adlib Tracker 2), which can all be ZX0-compressed now.

The compression ratio is just fantastic, in most cases it's better than ZIP! Now it's possible again to place the whole operating system with all system apps, background picture and additional stuff which is loaded during booting on one Amstrad standard disc side (178K).
I am using the Turbo decompressor, which is still small and has a great speed. It's directly integrated into the SymbOS kernel, which makes it possible to decompress data with the full linear size of up to 63K!

I hope to be able to release the next SymbOS version later this year.

Thanks so much to Einar Saukas (for the whole thing) and @introspec (for the Turbo decompressor)!

GRAPHICAL Z80 MULTITASKING OPERATING SYSTEM

Prodatron

PS: Bye bye Crown Cruncher. I was using this tool since the beginning of the 90ies for demos and tools, and later for compressing the SymbOS core boot binaries (ROM and disc loader). It is now replaced by the ZX0 compressor. I loved the Crown Cruncher, thanks to Crown again, it was fast and familiar for me, but now after 30 years it's time to use much more improved stuff. E.g. Crown Cruncher compressed the SymbOS core from about 80K to 62K, while ZX0 reaches 47K, which is a lot.

GRAPHICAL Z80 MULTITASKING OPERATING SYSTEM

Sykobee (Briggsy)

Wonderful! This is probably more advanced than some modern operating systems :) 

How fast is the turbo decompressor?

E.g., is it faster to load from disk the zx0 compressed files, including decompression, than the uncompressed original files - i.e., the decompressor is faster than the disc read speed, given you are reading far less data now?

eto

Quote from: Sykobee (Briggsy) on 10:17, 07 June 23E.g., is it faster to load from disk the zx0 compressed files, including decompression, than the uncompressed original files - i.e., the decompressor is faster than the disc read speed, given you are reading far less data now?
yes. it depends on the compression ratio of course, but loading 5KB screen data + decompressing is MUCH faster than loading 16KB from disc. Things might be different when you load from mass storage but for normal discs, it's worth it. The time required to decompress is about 3-4x the time a similar LDIR requires. More can be seen here: https://www.cpcwiki.eu/forum/programming/new-cruncher-zx0/msg197727/#msg197727

If it's still too slow, you can sacrifice another 1.5% of compression for 15% more performance with the ZX1 method: https://github.com/einar-saukas/ZX1 

However I didn't use that yet as my primary goal was to reduce space to get as much stuff on a disc or tape as possible. And compared to normal loading routines, it doesn't matter if the decompression is done in .5 or .4 seconds. 

E.g. I just made my "game tape collection" where I used ZX0 for many games. Loading times at 2000 baud shrink to 1.5 - 3.5 minutes, depending on the original game size (and if I could remove the loading screen). Over 50 games fit on two 60 minute tapes. I also made a disc collection where I can fit up to 13 games on a single side. Considering that discs are rare but I still like to load from real media, this is amazing.

Prodatron

Eto is right, you will save time when loading from floppy disc, but you will lose time, when you load from mass storage.

As mass storage on the CPC is about as fast as LDIR it is not possible to increase loading speed with compression, as any decompressor is always slower than LDIR.
But floppy is much slower than LDIR, so you will save time, when using a very good compressor with a very fast decompressor (like ZX0).

I made a test with Lymings, which loads 62K uncompressed or 29K compressed, when you start the app:

62K uncompressed29K compressed
floppy disc11 s9 s
mass storage 4 s5 s

Tested only in WinApe, and when you start an application in SymbOS much more things are done like relocating the code, preparing the memory etc., which explains, why the difference with floppy disc is not that big. But I wanted to get a feeling how much time I lose, when loading from mass storage, and I think this is ok.

GRAPHICAL Z80 MULTITASKING OPERATING SYSTEM

eto

Quote from: Prodatron on 14:28, 07 June 23But I wanted to get a feeling how much time I lose, when loading from mass storage, and I think this is ok.
Compared to the space saved, the additional second is well worth it.

TotO

By chance, a mass storage has enough room to store unpacked files from folders if the speed is critical.
"You make one mistake in your life and the internet will never let you live it down" (Keith Goodyer)

GUNHED

For FutureOS I'm using the Exomizer to handle compressed data since quite some years. This makes especially sense when working with XROMs (eXpansion ROMs for FutureOS). The compression rate of exomizer is fantastic and also the decompression speed is great.

Now I'm asking myself if it would make any sense to move from Exomizer to ZX0 compression. Of would the difference be just minor?
http://futureos.de --> Get the revolutionary FutureOS (Update: 2023.11.30)
http://futureos.cpc-live.com/files/LambdaSpeak_RSX_by_TFM.zip --> Get the RSX-ROM for LambdaSpeak :-) (Updated: 2021.12.26)

eto

Quote from: TotO on 15:08, 07 June 23By chance, a mass storage has enough room to store unpacked files from folders if the speed is critical.
It will still be helpful if you're dealing with disk images or, in this case, with a Plus cartridge, where the ROM acts like a storage device. Space is still limited there and - considering I am sitting in front of an 8 bit machine - waiting 5 instead of 4 seconds seems like a good choice to store twice as much files on a DSK or on ROM.

Quote from: GUNHED on 15:57, 07 June 23Now I'm asking myself if it would make any sense to move from Exomizer to ZX0 compression. Of would the difference be just minor?
The compression rate of Exomizer seems to be slightly better than ZX0. However the speed of ZX0 is much better. 

If speed doesn't matter, there is no need to switch to ZX0. 

TotO

Quote from: eto on 17:28, 07 June 23It will still be helpful if you're dealing with disk images or, in this case, with a Plus cartridge, where the ROM acts like a storage device. Space is still limited there and - considering I am sitting in front of an 8 bit machine - waiting 5 instead of 4 seconds seems like a good choice to store twice as much files on a DSK or on ROM.
That is the reason the R-Type (2012) File System automatically unpack data when loading them from the floppy disc to speed-up all the access. About the ROM usage, it is sad to use it like a mass storage but if it is the case, it is faster to left data unpacked.
"You make one mistake in your life and the internet will never let you live it down" (Keith Goodyer)

Powered by SMFPacks Menu Editor Mod