CPCWiki forum

General Category => Programming => Topic started by: SRS on 00:06, 30 September 16

Title: "Tokenizer" for A-lot-of-Text-games ..
Post by: SRS on 00:06, 30 September 16
Transfering BASIC-Adeventures from other platforms often leads to the "low memory" problem on CPC in Basic.

Sometime I do a manual "tokenizing" if I see some big chunk of repeated text in several lines, i.e." You are watching a" I'll reduce to some string ... or "token"

Do we have some software out there that scans let'S say 60 room desriptions and so on for "common text parts", (above 3 ore four chars) and gives a table of replacement ? Like "you do" becomes "T01" ... so we could compress / recylce parst of text "On the fly" maybe with an RSx and using second RAM bank for adventures ?

Hope you get my point ...

Example :

Code: [Select]
"You are standing at the bar inside the tavern. There ..."

gets

Aat the bar inside the tavern. There ..."

With decoding
Code: [Select]
IF LEFT$(hoel$,1)="A" THEN PRINT"You are standing ";:GOTO 998

Title: Re: "Tokenizer" for A-lot-of-Text-games ..
Post by: Prodatron on 01:14, 30 September 16
Better use something like
#A
instead of just A
so you can replace phrases anywhere in the sentences. With the help of INSTR and MID it's easy to write a string replace function.
Regarding "common text parts" it's nothing else than using a compression algorithm which is based on a dictionary.
Title: Re: "Tokenizer" for A-lot-of-Text-games ..
Post by: Docent on 16:27, 03 October 16
Transfering BASIC-Adeventures from other platforms often leads to the "low memory" problem on CPC in Basic.

Sometime I do a manual "tokenizing" if I see some big chunk of repeated text in several lines, i.e." You are watching a" I'll reduce to some string ... or "token"

Do we have some software out there that scans let'S say 60 room desriptions and so on for "common text parts", (above 3 ore four chars) and gives a table of replacement ? Like "you do" becomes "T01" ... so we could compress / recylce parst of text "On the fly" maybe with an RSx and using second RAM bank for adventures ?

Hope you get my point ...

Example :

Code: [Select]
"You are standing at the bar inside the tavern. There ..."

gets

Aat the bar inside the tavern. There ..."

With decoding
Code: [Select]
IF LEFT$(hoel$,1)="A" THEN PRINT"You are standing ";:GOTO 998

Hey, you've just invented LZW compression algorithm :)

Title: Re: "Tokenizer" for A-lot-of-Text-games ..
Post by: SRS on 23:19, 03 October 16
IF it would be the SRS algorithm :)

But did not see something like this for single PRINT commands until nowadays. :)