Quick update on the state-of-the-art compression using ApLib

[introspec]

After being asked about this in another thread, I thought I'll make a quick update on what constitutes the state-of-the-art compression using ApLib compression library, in application to Z80 of course.

First, please do not use the official compressors anymore, they do not give you the full compression possible nowadays. Instead, use open source compressor by Sven-Åke Dahl: https://github.com/svendahl/cap
A compiled version of the compressor with a workaround for the bug discussed below is attached to this post.

This new compressor will give you about 1-1.5% higher compression ratio compared to the official closed source compressor. The compressor is still not optimal, so if you are fighting for every byte, you may also attempt compressing data using another closed source compressor by r57shell: http://gendev.spritesmind.net/forum/viewtopic.php?p=32548#p32548 (please note that it can get VERY slow on some files...)
Most of the time the compressor by S.-A. Dahl will win; however, on some files, compressor by r57shell would compress better.

Now, assuming you compressed your data, here are two brand new decompressors for Z80. First, this is the size-optimized decompressor:

Code Select Expand


;
;  size-optimized aplib decompressor (v.1 21/08/2018-01/09/2018, 140 bytes)
;
;  original decompressor was written by Dan Weiss (Dwedit),
;  tweaked by utopian, and optimized by Metalbrain
;
;  re-optimized for size by spke (v1, 21-23/08/2018)
;  (it is 16 bytes shorter and 20% faster than the 156b version by Metalbrain)
;
;  the decompression is done in the standard way:
;
;  ld hl,CompressedData
;  ld de,WhereToDecompress
;  call Decompress
;
;  the decompressor modifies AF, AF', BC, DE, HL, IXH, IY
;
;  drop me an email if you have any comments/ideas/suggestions: zxintrospec@gmail.com
;


;
;  Uncomment the next line for the additional 20% speed-up at the cost of extra 4 bytes


;   DEFINE FasterGetBit


   IFDEF FasterGetBit
      MACRO   GET_BIT
         add a : call z,ReloadByte
      ENDM
   ELSE
      MACRO   GET_BIT
         call GetOneBit
      ENDM
   ENDIF


@Decompress:      ld a,128


;
;  case "0"+BYTE: copy a single literal


CASE0:         ldi               ; first byte is always copied as literal
ResetLWM:      ld ixh,1            ; LWM = 0 (LWM stands for "Last Was Match"; a flag that we did not have a match)


;
;  main decompressor loop


MainLoop:      GET_BIT : jr nc,CASE0         ; "0"+BYTE = copy literal
         GET_BIT : jr nc,CASE10         ; "10"+gamma(offset/256)+BYTE+gamma(length) = the main matching mechanism


         ld bc,%11100000
         GET_BIT : jr nc,CASE110         ; "110"+[oooooool] = matched 2-3 bytes with a small offset


;
;  case "111"+"oooo": copy a byte with offset -1..-15, or write zero to dest


CASE111:
ReadFourBits      GET_BIT               ; read short offset (4 bits)
         rl c : jr c,ReadFourBits
         ex de,hl : jr z,WriteZero      ; zero offset means "write zero" (NB: B is zero here)


         ; "write a previous byte (1-15 away from dest)"
         push hl               ; BC = offset, DE = src, HL = dest
         sbc hl,bc            ; HL = dest-offset (SBC works because branching above ensured NC)
         ld b,(hl)
         pop hl


WriteZero      ld (hl),b : ex de,hl
         inc de : jr ResetLWM         ; write one byte, reset LWM


;
;  branch "110"+[oooooool]: copy two or three bytes (bit "l") with the offset -1..-127 (bits "ooooooo"), or stop


CASE110:      ; "use 7 bit offset, length = 2 or 3"
         ; "if a zero is found here, it's EOF"
         ld c,(hl) : rr c : ret z      ; process EOF
         inc hl


         push hl               ; save src
         ld h,b : ld l,c            ; HL = offset


         ; flag NC means len=2, flag C means len=3
         ld c,1 : rl c : jr SaveLWMOffset
         
;
;  branch "10"+gamma(offset/256)+BYTE+gamma(length): the main matching mechanism


CASE10:         ; "use a gamma code * 256 for offset, another gamma code for length"
         call GetGammaCoded


         ; the original decompressor contains
         ;
         ; if ((LWM == 0) && (offs == 2)) { ... }
         ; else {
         ;   if (LWM == 0) { offs -= 3; }
         ;   else { offs -= 2; }
         ; }
         ;
         ; so, the idea here is to use the fact that GetGammaCoded returns (offset/256)+2,
         ; and to split the first condition by noticing that C-1 can never be zero
         dec ixh : jr nz,NoLWM
         dec c
NoLWM         ; "if gamma code is 2, use old r0 offset"
         dec c : jr z,KickInLWM
         dec c
         ld b,c : ld c,(hl) : inc hl      ; BC = offset


         push bc               ; (SP) = offset
         call GetGammaCoded         ; BC = len*
         ex (sp),hl            ; HL = offset, (SP) = src


         ; interpretation of length value is offset-dependent
         ; NB offsets over 32000 need one more check, but other
         ; Z80 decompressors seem to ignore it. is it not needed?
         exa : ld a,h : cp 5 : jr nc,.Add1
         or a : jr nz,.Add0
         bit 7,l : jr nz,.Add0
.Add2         inc bc
.Add1         inc bc
.Add0         exa


SaveLWMOffset:      push hl : pop iy         ; save offset for future LWMs


CopyMatch:      ; this assumes that BC = len, DE = dest, HL = offset
         ; and also that (SP) = src, while having NC
         push de
         ex de,hl : sbc hl,de         ; HL = dest-offset
         pop de               ; DE = dest
         ldir
         pop hl               ; recover src
         ld ixh,b            ; LWM = 0
         jr MainLoop


;
;  the re-use of the previous offset (LWM magic)


KickInLWM:      ; "and a new gamma code for length"
         call GetGammaCoded         ; BC = len
         push hl : push iy
;         pop hl               ; DE = dest, HL = prev offset
         jr CopyMatch-1            ; this jumps into POP HL of the POP IY


;
;  interlaced gamma code reader
;  x0 -> 1x
;  x1y0 -> 1xy
;  x1y1z0 -> 1xyz etc
;  (technically, this is a 2-based variation of Exp-Golomb-1)


GetGammaCoded:      ld bc,1
ReadGamma      GET_BIT : rl c : rl b
         GET_BIT : jr c,ReadGamma
         ret


;
;  pretty usual getbit for mixed datastreams


   IFNDEF FasterGetBit
GetOneBit:      add a : ret nz
   ENDIF
ReloadByte:      ld a,(hl) : inc hl
         rla : ret

This decompressor is 140 bytes long; it will spend approximately 130 Z80 cycles per each decompressed byte.

If this is not enough and you need faster decompression, here is the speed-optimized version:

Code Select Expand


;
;  speed-optimized aplib decompressor (v.2 12/11/2018-06/08/2019, 234 bytes)
;
;  original decompressor was written by Dan Weiss (Dwedit),
;  tweaked by utopian, and optimized by Metalbrain
;
;  completely re-written for speed by spke
;  (it is 13 bytes shorter and 12% faster than the 247b version by Metalbrain)
;
;  the decompression is done in the standard way:
;
;  ld hl,CompressedData
;  ld de,WhereToDecompress
;  call Decompress
;
;  the decompressor modifies AF, AF', BC, DE, HL, IX
;
;  drop me an email if you have any comments/ideas/suggestions: zxintrospec@gmail.com
;


@Decompress:      ld a,128 : jr LWM0.CASE0


;==================================================================================================================
;==================================================================================================================
;==================================================================================================================


LWM0:         ;LWM = 0 (LWM stands for "Last Was Match"; a flag that we did not have a match)


.ReloadByteC0      ld a,(hl) : inc hl : rla
         jr c,.Check2ndBit


;
;  case "0"+BYTE: copy a single literal


.CASE0:         ldi                  ; first byte is always copied as literal


;
;  main decompressor loop


.MainLoop:      add a : jr z,.ReloadByteC0 : jr nc,.CASE0   ; "0"+BYTE = copy literal
.Check2ndBit      add a : call z,ReloadByte : jr nc,.CASE10   ; "10"+gamma(offset/256)+BYTE+gamma(length) = the main matching mechanism
         add a : call z,ReloadByte : jp c,LWM1.CASE111   ; "110"+[oooooool] = matched 2-3 bytes with a small offset


;
;  branch "110"+[oooooool]: copy two or three bytes (bit "l") with the offset -1..-127 (bits "ooooooo"), or stop


.CASE110:      ; "use 7 bit offset, length = 2 or 3"
         ; "if a zero is found here, it's EOF"
         ld c,(hl) : rr c : ret z         ; process EOF
         inc hl
         ld b,0


         ld iyl,c : ld iyh,b            ; save offset for future LWMs


         push hl                  ; save src
         ld h,d : ld l,e               ; HL = dest
         jr c,.LengthIs3


.LengthIs2      sbc hl,bc
         ldi : ldi
         jr .PreMainLoop


.LengthIs3      or a : sbc hl,bc
         ldi : ldi : ldi
         jr .PreMainLoop


;
;  branch "10"+gamma(offset/256)+BYTE+gamma(length): the main matching mechanism


.CASE10:      ; "use a gamma code * 256 for offset, another gamma code for length"
         call GetGammaCoded


         ; the original decompressor contains
         ;
         ; if ((LWM == 0) && (offs == 2)) { ... }
         ; else {
         ;   if (LWM == 0) { offs -= 3; }
         ;   else { offs -= 2; }
         ; }
         ;
         ; so, the idea here is to use the fact that GetGammaCoded returns (offset/256)+2,
         ; and to split the first condition by noticing that C-1 can never be zero
         dec c : dec c : jr z,LWM1.KickInLWM


.AfterLWM      dec c : ld b,c : ld c,(hl) : inc hl   ; BC = offset


         ld iyl,c : ld iyh,b : push bc


         call GetGammaCoded         ; BC = len*


         ex (sp),hl


         ; interpretation of length value is offset-dependent:
         ; if (offs >= 32000) len++; if (offs >= 1280) len++; if (offs < 128) len+=2;
         ; in other words,
         ; (1 <= offs < 128) +=2
         ; (128 <= offs < 1280) +=0
         ; (1280 <= offs < 31999) +=1
         ; NB offsets over 32000 need one more check, but other Z80 decompressors seem to ignore it. is it not needed?
         exa : ld a,h : cp 5 : jr nc,.Add1
         or a : jr nz,.Add0
         bit 7,l : jr nz,.Add0
.Add2         inc bc
.Add1         inc bc
.Add0         ; for offs<128 : 4+4+7+7 + 4+7 + 8+7 + 6+6 = 60t
         ; for offs>=1280 : 4+4+7+12 + 6 = 33t
         ; for 128<=offs<1280 : 4+4+7+7 + 4+12 = 38t OR 4+4+7+7 + 4+7+8+12 = 53t


.CopyMatch:      ; this assumes that BC = len, DE = offset, HL = dest
         ; and also that (SP) = src, while having NC
         ld a,e : sub l : ld l,a
         ld a,d : sbc h
.CopyMatchLDH      ld h,a : ldir : exa
.PreMainLoop      pop hl               ; recover src


;==================================================================================================================
;==================================================================================================================
;==================================================================================================================


LWM1:         ; LWM = 1


;
;  main decompressor loop


.MainLoop:      add a : jr z,.ReloadByteC0 : jr nc,LWM0.CASE0      ; "0"+BYTE = copy literal
.Check2ndBit      add a : call z,ReloadByte : jr nc,.CASE10      ; "10"+gamma(offset/256)+BYTE+gamma(length) = the main matching mechanism
         add a : call z,ReloadByte : jr nc,LWM0.CASE110      ; "110"+[oooooool] = matched 2-3 bytes with a small offset


;
;  case "111"+"oooo": copy a byte with offset -1..-15, or write zero to dest


.CASE111:      ld bc,%11100000
         DUP 4
         add a : call z,ReloadByte : rl c      ; read short offset (4 bits)
         EDUP
         ex de,hl : jr z,.WriteZero      ; zero offset means "write zero" (NB: B is zero here)


         ; "write a previous byte (1-15 away from dest)"
         push hl               ; BC = offset, DE = src, HL = dest
         sbc hl,bc            ; HL = dest-offset (SBC works because branching above ensured NC)
         ld b,(hl)
         pop hl


.WriteZero      ld (hl),b : ex de,hl
         inc de : jp LWM0.MainLoop            ; 10+4*(4+10+8)+4+7 + 11+15+7+10 + 7+4+6+10 = 179t


.ReloadByteC0      ld a,(hl) : inc hl : rla
         jp nc,LWM0.CASE0
         jr .Check2ndBit


;
;  branch "10"+gamma(offset/256)+BYTE+gamma(length): the main matching mechanism


.CASE10:      ; "use a gamma code * 256 for offset, another gamma code for length"
         call GetGammaCoded


         ; the original decompressor contains
         ;
         ; if ((LWM == 0) && (offs == 2)) { ... }
         ; else {
         ;   if (LWM == 0) { offs -= 3; }
         ;   else { offs -= 2; }
         ; }
         ;
         ; so, the idea here is to use the fact that GetGammaCoded returns (offset/256)+2,
         ; and to split the first condition by noticing that C-1 can never be zero
         dec c : jp LWM0.AfterLWM


;
;  the re-use of the previous offset (LWM magic)


.KickInLWM:      ; "and a new gamma code for length"
         call GetGammaCoded         ; BC = len
         push hl
         exa : ld a,e : sub iyl : ld l,a
         ld a,d : sbc iyh
         jp LWM0.CopyMatchLDH


;==================================================================================================================
;==================================================================================================================
;==================================================================================================================


;
;  interlaced gamma code reader
;  x0 -> 1x
;  x1y0 -> 1xy
;  x1y1z0 -> 1xyz etc
;  (technically, this is a 2-based variation of Exp-Golomb-1)


GetGammaCoded:      ld bc,1
ReadGamma      add a : jr z,ReloadByteRG1
         rl c : rl b
         add a : jr z,ReloadByteRG2
         jr c,ReadGamma : ret


ReloadByteRG1      ld a,(hl) : inc hl : rla
         rl c : rl b
         add a : jr c,ReadGamma : ret


ReloadByteRG2      ld a,(hl) : inc hl : rla
         jr c,ReadGamma : ret


;
;  pretty usual getbit for mixed datastreams


ReloadByte:      ld a,(hl) : inc hl : rla : ret

It is 234 bytes long. It may look a bit crazy, but it will spend less than 90 cycles per byte of decompressed data. Just to give you something to compare against; this is almost 20% faster than the standard ZX7 decompressor and it is less than 10% slower than ZX7 "turbo" decompressor.

On average, ApLib has about 8% better compression ratio than ZX7 and only 1.5% worse compression ratio than the latest Exomizer. It now has an excellent open source compressor, does not need any memory buffers during decompression and its decompression speed is highly competitive, esp. given how many people still use old compressors with old decompressors. In short, ApLib is currently one of the best compressors on Z80. Definitely consider using it for your next project.

[SyX]

Thank you very much @introspec!!! 

I installed mono in my msys2 (mingw64) development machine and i compiled the two branches of the compressor (Villena added a few modifications to the Sven code) without any problem. I only made a fast test and both branches generated a compressed file with the same lenght, but not equal content (and both decrunched the compressed file).

Later i will test better everything and add support to appack in my project

[SyX]

Ok, i made a test using the racing game data, i am using zx7 in this project.
And appack saved 8 KBs with respect to zx7... but this saving was only in code, level data and music. All the files with graphics in native cpc format (mainly those XXXX_intermission.apk/zx7 files) were compressed a lot worst than zx7. Meaning that at the end the saving of appack was negligible (139.815 bytes vs 140.314 bytes). 
The ZX7 results:

Code Select Expand

-rw-r--r-- 1 Mauricio None 12.851 ago. 13 18:25 antartida_data.zx7
-rw-r--r-- 1 Mauricio None  2.831 ago. 13 18:25 antartida_intermission.zx7
-rw-r--r-- 1 Mauricio None 14.264 ago. 13 18:25 blob.zx7
-rw-r--r-- 1 Mauricio None  1.604 ago. 13 18:25 car_sprites.zx7
-rw-r--r-- 1 Mauricio None 13.088 ago. 13 18:25 cuenca_data.zx7
-rw-r--r-- 1 Mauricio None  4.542 ago. 13 18:25 cuenca_intermission.zx7
-rw-r--r-- 1 Mauricio None 14.721 ago. 13 18:25 egipto_data.zx7
-rw-r--r-- 1 Mauricio None  3.061 ago. 13 18:25 egipto_intermission.zx7
-rw-r--r-- 1 Mauricio None 11.687 ago. 13 18:25 escocia_data.zx7
-rw-r--r-- 1 Mauricio None  4.167 ago. 13 18:25 escocia_intermission.zx7
-rw-r--r-- 1 Mauricio None 11.302 ago. 13 18:25 florida_data.zx7
-rw-r--r-- 1 Mauricio None  2.494 ago. 13 18:25 florida_intermission.zx7
-rw-r--r-- 1 Mauricio None  2.342 ago. 13 18:25 game_state.zx7
-rw-r--r-- 1 Mauricio None  2.097 ago. 13 18:25 hud.zx7
-rw-r--r-- 1 Mauricio None  1.273 ago. 13 18:25 intermission_state.zx7
-rw-r--r-- 1 Mauricio None    341 ago. 13 18:25 intro_logo.zx7
-rw-r--r-- 1 Mauricio None  9.529 ago. 13 18:25 menu_data.zx7
-rw-r--r-- 1 Mauricio None  4.366 ago. 13 18:25 menu_state.zx7
-rw-r--r-- 1 Mauricio None  3.910 ago. 13 18:25 outro_data.zx7
-rw-r--r-- 1 Mauricio None    188 ago. 13 18:25 outro_state.zx7
-rw-r--r-- 1 Mauricio None 15.211 ago. 13 18:25 pascua_data.zx7
-rw-r--r-- 1 Mauricio None  4.445 ago. 13 18:25 pascua_intermission.zx7
Total 140.314 bytes

The appack results:

Code Select Expand

-rw-r--r-- 1 Mauricio None 11.943 ago. 13 18:25 antartida_data.apk
-rw-r--r-- 1 Mauricio None  3.543 ago. 13 18:25 antartida_intermission.apk
-rw-r--r-- 1 Mauricio None 13.583 ago. 13 18:25 blob.apk
-rw-r--r-- 1 Mauricio None  1.575 ago. 13 18:25 car_sprites.apk
-rw-r--r-- 1 Mauricio None 11.924 ago. 13 18:25 cuenca_data.apk
-rw-r--r-- 1 Mauricio None  5.170 ago. 13 18:25 cuenca_intermission.apk
-rw-r--r-- 1 Mauricio None 13.684 ago. 13 18:25 egipto_data.apk
-rw-r--r-- 1 Mauricio None  3.799 ago. 13 18:25 egipto_intermission.apk
-rw-r--r-- 1 Mauricio None 10.677 ago. 13 18:25 escocia_data.apk
-rw-r--r-- 1 Mauricio None  4.628 ago. 13 18:25 escocia_intermission.apk
-rw-r--r-- 1 Mauricio None 10.481 ago. 13 18:25 florida_data.apk
-rw-r--r-- 1 Mauricio None  3.081 ago. 13 18:25 florida_intermission.apk
-rw-r--r-- 1 Mauricio None  2.301 ago. 13 18:25 game_state.apk
-rw-r--r-- 1 Mauricio None  3.016 ago. 13 18:25 hud.apk
-rw-r--r-- 1 Mauricio None  1.204 ago. 13 18:25 intermission_state.apk
-rw-r--r-- 1 Mauricio None  2.095 ago. 13 18:25 intro_logo.apk
-rw-r--r-- 1 Mauricio None  9.230 ago. 13 18:25 menu_data.apk
-rw-r--r-- 1 Mauricio None  3.917 ago. 13 18:25 menu_state.apk
-rw-r--r-- 1 Mauricio None  3.472 ago. 13 18:25 outro_data.apk
-rw-r--r-- 1 Mauricio None    179 ago. 13 18:25 outro_state.apk
-rw-r--r-- 1 Mauricio None 13.938 ago. 13 18:25 pascua_data.apk
-rw-r--r-- 1 Mauricio None  6.375 ago. 13 18:25 pascua_intermission.apk
Total 139.815 bytes
After this, i have started to test the appack compressor from the r57 shell, but it is superslow, you were not joking about it. I only tested two files, one with data and other with a native graphic. And the results were fun:

Code Select Expand

-rw-r--r-- 1 Mauricio None 14.120 ago. 13 18:35 pascua_data.r57
-rw-r--r-- 1 Mauricio None  4.407 ago. 13 18:45 pascua_intermission.r57
The data file was compressed worst than the normal appack compressor, but the file with native graphics was compressed even better than zx7. That means that i could use both appack compressors for saving a few KBs. Although my reason for not using appack anymore, it was because the closed source compressor... at least, i know that the open source appack compressor could improve more 

Lucky for me, those files with native graphics only need to be compressed once, although it will be a long wait, hehehe.
But for now, i think that i am going to continue using zx7 while i develop this project. Because it is FASTER, everything gets compressed in five seconds (i use gnu parallel for compressing all the files in parallel, putting all those cores in my pc to do something useful ). Maybe at the end, if i need extra space in the floppy... at least, now i have the option of using appack, thanks again 
Ahhh, there is another reason, i customized the zx7 compressor for putting in the error stream the delta value for every file compressed. After the compression job is finished, the information of the biggest delta value is passed to the disk loader/decruncher routine during its assembly. And the delta information is very important in this project because everything is decrunched in place and even those extra delta bytes are used as variables later (that it would be a nice feature that appack should print in the shell).

[introspec]

Ok, i made a test using the racing game data, i am using zx7 in this project.

Very interesting results, SyX! However, one thing bothers me a little. You keep saying that you are using the appack to compress the data. However, the new (the good!) compressor is actually called APC. Are you sure you actually scrolled down to the bottom of the thread at www.amstrad.es to get it?

Otherwise, it is of course annoying that the compression can be so unreliable on some files. I've never seen differential of this magnitude on my data. Do you have any files with huge differences in compressed sizes between the compressors that you can share so that I can try to investigate what is going on? (e.g. "pascua_intermission" looks very exciting in this respect).

[SyX]

Very interesting results, SyX! However, one thing bothers me a little. You keep saying that you are using the appack to compress the data. However, the new (the good!) compressor is actually called APC. Are you sure you actually scrolled down to the bottom of the thread at www.amstrad.es to get it?

Well, I went to the GitHub of Sven, I downloaded the source code of his version and the Antonio's version, that is in the same github repository.

Antonio's version makes the same modification that I made to the original appack, skip the 2 first bytes from the compressed file.

Then I downloaded mono and recompiled both versions, and I made my tests using these two versions (the compressed file lengths are always the same).

I could use that executable in the spanish forum, but in theory we should not see any difference. I will confirm tomorrow.

[SyX]

Ok, i just tested with the apc.exe in the spanish forum and the result is the same and the data identical to the data generated by the Antonio's appack version that you can get in the Antonio's branch in the Sven github repository.

Otherwise, it is of course annoying that the compression can be so unreliable on some files. I've never seen differential of this magnitude on my data. Do you have any files with huge differences in compressed sizes between the compressors that you can share so that I can try to investigate what is going on? (e.g. "pascua_intermission" looks very exciting in this respect).

No problem, i am sending you a private message for sending those interesting files for you can test and discover why we get those huge differences between zx7 and appack.

[introspec]

@SyX , I think I found a way around the bug in the compressor. The fixed compressor is now attached to the starting post and the results on your problem files are as follows:

Code Select Expand



17/08/2019  01:28             2,780 antartida_intermission.apc
12/07/2019  17:48             2,831 antartida_intermission.zx7


17/08/2019  01:28             4,474 cuenca_intermission.apc
12/07/2019  17:48             4,542 cuenca_intermission.zx7


17/08/2019  01:28             3,036 egipto_intermission.apc
12/07/2019  17:48             3,061 egipto_intermission.zx7


17/08/2019  01:28             3,998 escocia_intermission.apc
12/07/2019  17:48             4,167 escocia_intermission.zx7


17/08/2019  01:29             2,400 florida_intermission.apc
12/07/2019  17:48             2,494 florida_intermission.zx7


17/08/2019  01:29             1,934 hud.apc
12/07/2019  17:48             2,097 hud.zx7


17/08/2019  01:29               306 intro_logo.apc
12/07/2019  17:48               341 intro_logo.zx7


17/08/2019  01:29             4,327 pascua_intermission.apc
12/07/2019  17:48             4,445 pascua_intermission.zx7

I hope this convinces you to switch to ApLib for good! 

[roudoudou]

I hope this convinces you to switch to ApLib for good! 

Only opensource and portable C code may convince me 

[reidrac]

Only opensource and portable C code may convince me 

Or a Python implementation , but C# is definitely not it. I rather not add .NET (or Mono) to my tools (and same for Java).

It sounds like a silly reason to not use it, but there you are.

[introspec]

Only opensource and portable C code may convince me 

Well, then you've got something to port

[introspec]

Only opensource and portable C code may convince me 

I must admit, I did not take your comment too seriously. However, here I am, about half a year later, with a link to a new ApLib compressor, which has almost the same compression ratio as APC, but is open source and written in C:
https://github.com/emmanuel-marty/apultra
It has been developed by Emmanuel Marty who you might remember from LZSA project.


So, you have no more excuses to avoid using ApLib in your next project.

[ervin]

This sounds very interesting.
How fast is decompression?

[roudoudou]

I must admit, I did not take your comment too seriously. However, here I am, about half a year later, with a link to a new ApLib compressor, which has almost the same compression ratio as APC, but is open source and written in C:
https://github.com/emmanuel-marty/apultra
It has been developed by Emmanuel Marty who you might remember from LZSA project.


So, you have no more excuses to avoid using ApLib in your next project.

Yeeeeepeeeeeeeeeeeeeeeeeeeeeeee!!! Will take a look very soon!!!!

[introspec]

This sounds very interesting.
How fast is decompression?

With my "fast"  decompressor in the top post - about 90 t-states per decompressed byte. This is half the speed of LZSA2 or twice the speed of the fastest decompressor for Exomizer.

[Eugene85]

For academic purposes, an optimal aPACK compressor was developed. If interested, check https://gitlab.com/eugene77/oapack

[Urusergi]

For academic purposes, an optimal aPACK compressor was developed.

MAGCODE.SBF 34.560 bytes

MAGCODE.OAP 17.816 bytes

MAGCODE.EXO 17.804 bytes

Very impressive!!!

Would it be very complicated if you added the backward option?  please please 

[Eugene85]

Urusergi

You mean unpack option? You can use apc12spke or apultra for unpacking. Or do you want it all in one? No, it's not hard.

[Arnaud]

Urusergi

You mean unpack option? You can use apc12spke or apultra for unpacking. Or do you want it all in one? No, it's not hard.

No, backward is decrunching from the end of the compressed and destination buffer.

[GUNHED]

MAGCODE.OAP 17.816 bytes
MAGCODE.EXO 17.804 bytes

Is aPack quicker than Exomizer? Significantly? Oh, I mean decrunching of course. 

[Eugene85]

Urusergi
Arnaud

I need relevant decompressor on PC for testing. Is there one?

[roudoudou]

So, you have no more excuses to avoid using ApLib in your next project.

Sources seems to be only for educational purpose, you cannot compile as there is files missing, wrong includes, ...
I guess that's because Emmanuel M. is using libdiv on LZSA and APultra
On the contrary i succeed to compile LZSA repository
I will contact the author because i think exomizer is today deprecated (slow compression, slow decompression, ratio similar to APultra)

[reidrac]

Sources seems to be only for educational purpose, you cannot compile as there is files missing, wrong includes, ...
I guess that's because Emmanuel M. is using libdiv on LZSA and APultra
On the contrary i succeed to compile LZSA repository
I will contact the author because i think exomizer is today deprecated (slow compression, slow decompression, ratio similar to APultra)

Not sure what are you talking about. I didn't have any problems compiling it. Kitsune's Curse uses apultra to compress all data that can be compressed.

For the decompressor in Z80, I used some code not distributed by apultra, but 1.2.0 now comes with a fast Z80 decompressor too.

[roudoudou]

Not sure what are you talking about. I didn't have any problems compiling it. Kitsune's Curse uses apultra to compress all data that can be compressed.

i'm talking about wrong includes and/or missing files
FILE shrink_context.c => include "shrink_context.h" which is not in the project i cloned just a few minutes ago
requesting file shrink_block missing too
i dont know how to contact the author, i registered github but it seems there is no message system

[reidrac]

i'm talking about wrong includes and/or missing files
FILE shrink_context.c => include "shrink_context.h" which is not in the project i cloned just a few minutes ago
requesting file shrink_block missing too
i dont know how to contact the author, i registered github but it seems there is no message system

I got 1.2.0 and just run make with success (?). He just released 1.2.1. Give it a go.

[GUNHED]

For the decompressor in Z80, I used some code not distributed by apultra, but 1.2.0 now comes with a fast Z80 decompressor too.

Is there a link to the compressor and Z80 decompressor?