CPCWiki forum

General Category => Programming => Topic started by: zhulien on 19:11, 24 January 22

Title: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 19:11, 24 January 22
I have been looking at Locomotive BASIC to see what could be done to extend it - internally rather than via RSXs.


There is a good CPC Wiki page here: https://www.cpcwiki.eu/index.php/Technical_information_about_Locomotive_BASIC#BASIC_tokens (https://www.cpcwiki.eu/index.php/Technical_information_about_Locomotive_BASIC#BASIC_tokens)


and a good disassembly of the BASIC ROM here: http://cpctech.cpc-live.com/docs/basic.asm (http://cpctech.cpc-live.com/docs/basic.asm)


The ROM is very packed with not a lot of space in it, but as a test I could setup a command in RAM and add a command to the command table, eg: QBEEP and point it to a routine in #be80 just to make it beep.


There are only 3 main tokens that are unused, and it appears the first 128 tokens are essentially paged (prefixed by &ff) between immediate tokens for non-commands (numbers and letters for strings etc) as well as functions.


Ignoring the fact the ROM is jam packed already, it looks trivial to add new functions as there are lots of unused paged tokens.


One thing stuck out to me though... How does "LINE INPUT" work?  is it just two command tokens following each other without a : between?  It seems INPUT is not a function but still a command.


I was thinking BASIC could be extended via the unused &e2, but ultimately that would mean introducing variable-width tokens for commands - but with a little luck it still 'could' work.  It could allow another 128 paged commands that are not functions.


What I wanted to look at more-so, was how to make BASIC use more memory than 42kb.  Either move all variables to the 2nd 64kb bank and have every variable access a (slow) far fetch... perhaps put the entire BASIC program behind the BASIC ROM, meaning instructions would also need to be (slow) far fetches.  That could give a very good 63kb HIMEM or such with an additional 64kb variable RAM.  Another possibility is to remove BASIC from the ROM and make it an executable that runs at &0040 with the BASIC program once again in the 2nd 64kb RAM.  Then... make the BASIC ROM into a proper OS shell.


Someone here said Vortex BASIC did use extra RAM for larger programs - so it should be possible right?  Does anyone have a disassembly of Vortex BASIC?


Locomotive Shell 1.0

I recommend we call shell programs with a different extension so they don't get confused with BASIC ones for the time being... unless we can work out a header system for them.
Title: Re: Extending Locomotive BASIC
Post by: zhulien on 19:32, 24 January 22
This is version 1.0 of the Locomotive Shell, it is currently less functional, than Locomotive BASIC.  I hope the original developers don't mind the name, it is a tribute to them for their fantastic work.


You can either replace BASIC with this ROM and put your BASIC in a different slot, e.g. 14 - or you can put this one in a different slot.  Access this using |shell, or your basic via |basic.



; tokens spare:      29
; bytes reclaimed:    1027
; current himem:   up 128 bytes

; commands removed
;   after, auto, clg, defint, defreal, defstr, deg, draw, drawr, move, mover, on sq, origin, plot, plotr, rad, release, symbol
;   tag, tagoff, troff, tron, wait, width, window, window swap, zone, fill, graphics paper, graphics pen, mask, cursor

; to replace with better alternatives
;   border, ent, env, ink, paper, pen, sound

; commands added
;   dir
Title: Re: Extending Locomotive BASIC
Post by: Animalgril987 on 19:59, 24 January 22
Hi  zhulien.
I don't think this will work. BASIC is a foreground ROM not background, and the OS will only initialise the first foreground ROM that finds, afaik.


Alan
Title: Re: Extending Locomotive BASIC
Post by: zhulien on 20:02, 24 January 22
Quote from: Animalgril987 on 19:59, 24 January 22
Hi  zhulien.
I don't think this will work. BASIC is a foreground ROM not background, and the OS will only initialise the first foreground ROM that finds, afaik.


Alan


Using EmuCPC, I have setup the BASIC ROM 1.1 as usual but in socket 5 put BASIC ROM 1.0 with |BASIC renamed to |JASIC - and it does initialise all the way to the cursor (then pauses), so I is definitely trying.


SUCCESS!!!


Yes it is very easy to make lots of BASICs, only 1 obviously is invoked at a time, eg, my BASIC 1.1 and my JASIC 1.2 work fine together, swap between them via |BASIC (with GOSUB & RETURN) and |JASIC (with GOSUB & REBURN).


All background ROMS work fine in both too, so... now a matter of reassembling rather than editing the ROM image to see how compatible a re-assembled one works.
Title: Re: Extending Locomotive BASIC
Post by: zhulien on 22:11, 24 January 22
What's cool, is I mapped the ZX Spectrum tokens within locomotive basic I get about 90% coverage of Spectrum BASIC, so besides AMSDOS headers / Spectrum headers (if they have one) - it should be possible to run some Spectrum BASIC code in a 2nd modified Locomotive BASIC ROM.  Also, Locomotive BASIC has quite a LOT more commands than the Spectrum which means some freed up ROM space to implement those missing 10%.


I've also mapped ZX81 tokens (for a possible 3rd BASIC ROM).  However, ZX81 doesn't use ASCII and I didn't yet work out how the character set is mapped.  Needs a little more research.  Besides the non-ASCII of the ZX81 we also have close to 90% coverage in commands.
Title: Re: Extending & Modifying Locomotive BASIC
Post by: Sykobee (Briggsy) on 13:06, 25 January 22
That would be neat, having a language-level compatible version of spectrum basic running on top of the locomotive basic interpreter and editor, which are superior! And CPC typeface - don't import that awful Sinclair typeface :D.


I guess you'd need to handle the screen size difference and other aspects, and anything that relied on peek, poke, out and in to spectrum specific addresses would still fail.

Title: Re: Extending & Modifying Locomotive BASIC
Post by: Animalgril987 on 19:34, 25 January 22
I stand corrected. Well done zhulien.  :D
Title: Re: Extending & Modifying Locomotive BASIC
Post by: zhulien on 16:36, 04 February 22
Quote from: zhulien on 19:11, 24 January 22
and a good disassembly of the BASIC ROM here: http://cpctech.cpc-live.com/docs/basic.asm


Would anyone know an assembler that would assemble that source?  It is way too big for MAXAM1.5 on CPC.
Title: Re: Extending & Modifying Locomotive BASIC
Post by: Bread80 on 12:04, 10 February 22
I take it you've not seen my reversed assembled BASIC 1.1 at https://github.com/Bread80/Amstrad-CPC-BASIC-Source


Run it through RASM and it'll assemble straight away.


I'd like to modify it to be spread across two ROMs. Probably one for immediate mode and one for run mode. But that's a lot of work and I don't have time right now. Until then if you want to add stuff then you'll probably need to remove some stuff you don't use.


As to your questions:
Functions are prefixed with &ff. There's plenty of space to add more, but bear in mind they're in two blocks: one which takes a single numeric(?) parameter; another which takes any type of parameters. (Internally the single parameter ones are read by the interpreter and passed to the function. The other are read by the function itself).


For statements, yes, there's very few slots available. My preference would be to use one of those slots as a prefix for a second block of extended tokens. Failing that you can always replace some tokens you never use.


Re LINE INPUT: commands with multiple tokens are usually stored as multiple tokens. The LINE command will read the following token and process (or raise an error) as appropriate. Off the top of my head, SPEED INK/WRITE, GRAPHICS PEN/INK are two others that do the same.


If you want to extend the language then I'd recommend starting with the source of a similar command/function. It's not too complex once you understand it but you'll need to know the routines to call to read tokens, raise errors etc.
Title: Re: Extending & Modifying Locomotive BASIC
Post by: GUNHED on 16:04, 10 February 22
To work with two BASIC ROMs you can install both of them and have one of them parked. To park a ROM you can use any good ROM management software (ROManager f.e.). Just keep only one of them active.
BUT you MUST add the correct lower ROM = firmware to the corresponding BASIC ROM. This can be done with an external Lower ROM (using X-MEM, MegaFlash).
Title: Re: Extending & Modifying Locomotive BASIC
Post by: zhulien on 11:21, 14 February 22
Quote from: Bread80 on 12:04, 10 February 22
I take it you've not seen my reversed assembled BASIC 1.1 at https://github.com/Bread80/Amstrad-CPC-BASIC-Source


Run it through RASM and it'll assemble straight away.


I'd like to modify it to be spread across two ROMs. Probably one for immediate mode and one for run mode. But that's a lot of work and I don't have time right now. Until then if you want to add stuff then you'll probably need to remove some stuff you don't use.


As to your questions:
Functions are prefixed with &ff. There's plenty of space to add more, but bear in mind they're in two blocks: one which takes a single numeric(?) parameter; another which takes any type of parameters. (Internally the single parameter ones are read by the interpreter and passed to the function. The other are read by the function itself).


For statements, yes, there's very few slots available. My preference would be to use one of those slots as a prefix for a second block of extended tokens. Failing that you can always replace some tokens you never use.


Re LINE INPUT: commands with multiple tokens are usually stored as multiple tokens. The LINE command will read the following token and process (or raise an error) as appropriate. Off the top of my head, SPEED INK/WRITE, GRAPHICS PEN/INK are two others that do the same.


If you want to extend the language then I'd recommend starting with the source of a similar command/function. It's not too complex once you understand it but you'll need to know the routines to call to read tokens, raise errors etc.


Awesome, I am about 60% through manually unassembling BASIC 1.1 - but seems you have saved the work.  Amazing how many subroutines they have created in there that are purely to save a byte her and there - it certainly is one of the most jam-packed 16kb ROMS.  I am glad I spent the time doing this though as I have learned quite a lot of new things too.


I will take a look at yours because you have already done the work!  I have proven multiple ROMS can work, so my thoughts were:


1. shell ROM, remove unwanted keywords and make a OS Shell, this would be the default that we could configure our CPC with and it would have hooks to be native to UniDOS RSXs among other OS-type features, multi tasking and memory management.


2. of course Standard CPC BASIC, but it will be with a renamed RSX.


3. spectrum BASIC - token substitution and reclaim space from unused keywords to implement some spectrum ones that don't exist.


4. a Game-oriented BASIC?


I have looked at the tokens, and I need to experiment somewhat to see if we can introduce additional ones - the issue is all 256 tokens (minus a couple) are used, and are even paged for the function usage and character set - 8 bit paged token set already in there.


Splitting across 2 ROMS could certainly mean we can extend existing keywords in some ways with additional code.
Title: Re: Extending & Modifying Locomotive BASIC
Post by: zhulien on 11:24, 14 February 22
Quote from: GUNHED on 16:04, 10 February 22
To work with two BASIC ROMs you can install both of them and have one of them parked. To park a ROM you can use any good ROM management software (ROManager f.e.). Just keep only one of them active.
BUT you MUST add the correct lower ROM = firmware to the corresponding BASIC ROM. This can be done with an external Lower ROM (using X-MEM, MegaFlash).


So far my tests with multiple BASICS (not parked) have been successful, and unless the lower ROM calls the middle of BASIC directly, there shouldn't be any issues.


Upper ROM make the BASIC you want to default to, then use an RSX to get to any other BASIC you may have in your ROMboard.  It works nicely in my experiments.
Title: Re: Extending & Modifying Locomotive BASIC
Post by: zhulien on 11:56, 14 February 22
There are a lot of functions that flow to the next function to save bytes too so lots of code, the order that it is assembled in is very important.


It is interesting the way it also loops through the alphabet and individually looks through each first letter - saving lots of bytes for keywords of the same letter while also balancing the timeframe to find a keyword somewhat.


I am now trying to find one error in the original disassembly i found - where it jumps into the middle of a command... is that likely an error in the original code or just the disassembly?
Title: Re: Extending & Modifying Locomotive BASIC
Post by: Bread80 on 18:30, 14 February 22
There were one or two data areas which hadn't been identified as such, and had been disassembled as data. In the BASIC ROM ISTR the infix_maths_table and infix_comparison_table at &cfed. I don't remember anywhere with a jump/call into mid-opcode though.


In the fp maths section of the firmware there is a function (process_inline_parameters at &3404) that takes the code after a call as parameters (and returns to the code after them). These data blocks certainly had some JRs landing mid-opcode.


As part of the unassembly process I wrote a utility to log code areas, data areas and jumps/calls which did 'odd' things. There shouldn't be any of them left.


The thing I can't really guarantee is addresses which have been calculated. Usually this is for data areas stored in RAM. I've looked for them, and fixed any I've found but it's impossible to know if I've caught them all. Most of these would only be a problem if you moved or modified the data storage areas.




And as for the optimisations, yes, they are a great lesson in writing dense assembler. There's a lot of JRs to a JP which save a single byte. And the compression of error messages. And there's a call to raise an error which uses the byte after the call as the error code to save having to load a register. I did find one JP which could have been replaced by a JR though. But the jump was into a neighbouring module so I think I can forgive them.
Title: Re: Extending & Modifying Locomotive BASIC
Post by: zhulien on 11:21, 16 February 22
just an update:


I can build fine, I can now create a custom shell and of course the original BASIC 1.1 just works out of the box as another ROM number (e.g. 14).


The shell I can remove or add commands without issues, other than storage within the ROM itself, but... I can put code elsewhere - such as other expansion RAM.


I made 2 lengthy attempts to split the code into two ROMS.  Actually I succeeded but it wasn't happy with them.  My first approach was to move all functions to the second ROM, but due to the way it is so tightly coupled with all theses byte saving measures - the 2nd ROM still had 8kb of baggage to support those functions.  In fact, the entire parser was in both the first and 2nd ROM out of necessity unless major reworkings are made.


The 2nd attempt was similar except I had put every command in the 2nd ROM instead of the functions - with just shells, the first ROM goes down to about 3kb (just the editor), but - as these were just POCs, I didn't patch every command to work in the other ROM, but since the baggage is in both (execution code memory management error handler etc...) it doesn't matter which ROM is actually paged in at any time for the functions to work.


I am now back to a separate shell and BASIC, I think that is the way to go - as mentioned BASIC can be extended, but - it's best to put additional code in the 2nd ROM (or RAM) for new functions, perhaps some of the easy to move functions but we need their tokens for our usage.


Now, one other option I have been playing with is removing the majority of the editor since protext can also edit BASIC programs.  Now, the shell is supposed to be a shell even though it is based on locomotive BASIC - removal of the graphics commands (plot, draw, tag, window, streams would all be a global stream etc) - the purpose being the BASIC would be enhanced for better creation of tools (batch processes and utilities).


Once I have trimmed the shell of unwanted commands / functions (and perhaps the editor?) I will share the code.
Title: Re: Extending & Modifying Locomotive BASIC
Post by: zhulien on 14:43, 16 February 22

questions and ideas


i want a better startup sequence, ideally with better hardware detection... or should we just output a custom message that is user maintained?


such a custom message could alternatively be created by a separate hardware detection utility


fast text patch? and what is the best default colour scheme?


more tight integration with unidos?


companion second MCP rom POC currently allows for multi-tasking of machine code utilities in the background


some random read / write commands for files?


remove the sound commands (ent, env, sound) for a more powerful sound command?


remove border, ink, paper, pen?  perhaps just get rid of border and ink and make the border always change with the paper?


do we want to keep the line editor or use protext? we could perhaps gain another kilobyte and the following keywords:
     edit, list, load


is the default aspect ratio what we all want? 80 x 25? 
   
we could make a microfont like in the batman demo and have more columns in the same space?
   
graphics have been removed, we can add more powerful graphics if needed... but... it is a shell, this could be the basis of some
more powerful commands for easier coding of utils, and a separate 'gamebasic' for more powerful graphics?

does anyone use the remain function?

do we want to retain printer support in the shell?

new commands should use a driver system where possible, e.g. a new sound command, perhaps a play command could take    the form play "<drivername>", <other parameters> so that the same code could work parameterized across different audio    hardware.
Title: Re: Extending & Modifying Locomotive BASIC
Post by: zhulien on 14:48, 16 February 22
We have just over a kilobyte of reclaimed ROM space in this ROM meaning that newly created commands can be paged into another ROM for a much more powerful command set.


I am thinking, full memory support with commands for tiles & sprites based on my existing library - unless someone has a better code set, maybe we can work together and add commands for them?  Our BASIC doesn't only have to be 16kb anymore either, we can even use more than 2 ROMS in addition to 4mb of real RAM.


Thinking of adding the following commands:


Status - Displays system statistics.
Tasks - Lists current tasks.
Spawn - Loads and runs a new task.
Kill - Kills the specified task and frees up resources.
Start - Manually start tasks.
Stop - Manually stop all tasks.
Reboot - Reboot the computer preserving the running status of tasks.
Recover - Attempts to recover the system.
Snapshot - Manually save the computer state.
Restore - Manually restore the computer state and resume tasks.



Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: Bread80 on 12:19, 21 February 22
I'd suggest a lot of the changes your making are more a firmware thing rather than a BASIC thing. If you put them in the firmware then there available for any software to use. And it's a lot easier to strip out space in the firmware - the floating point maths stuff can move to a sideways ROM. And the printer stuff sounds like a safe bet to get rid of. Maybe even the cassette routines depending on how you like to load and save stuff.
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 18:49, 21 February 22
I have modified the tokenizer, the detokenizer and the executor to cater for double-byte tokens prefixed by #e9.


It is still somewhat buggy but on a good path I think.


We have a second set of tokens almost 100 tokens from #80.  The second set of tokens can be dual purpose just as the first.  Interestingly, the standard set of tokens are actually overloaded, eg: line input, graphics pen, graphics paper - the input, pen and paper are dual purpose, stand alone and as dual words - the tokens are the same but the code is different.  So, using the 100 new tokens, we can have either new keywords that never previously existed, they can be multi-words as previously and overloaded too, however if a word is previously used, it is not an extended token - but... can form part of that overloading.


For example:  take 3 tokens


MCP
STATUS
LIST


and the following arrangements:


MCP - new token with 1 word provided
MCP STATUS - new dual word command
MCP LIST - overloading of the existing LIST command / token
STATUS - new 1 word command that is overloaded by MCP STATUS
LIST - existing list command


Of course we don't need to code all the overloadings, only the ones we actually want to use.  Effectively the CPC BASIC is a series of words that can be strung together any number of words deep to form a single command.  I think it's pretty cool what Locomotive Software did.




Current bugs:


- the detokenizer is working fine, i can verify this by manually keying the correct series of tokens into memory


- the tokenizer, it is working sometimes, I must have missed something because if I edit a code line, I still get some garbage at the end, likely missed a line size or similar


- stray syntax error, sometimes even when the command is coded and runs it still gives a syntax error, not sure why yet


- execution within a program sometimes stops at a command even though immediate mode the commands work, likely something to do with the modification to fetch the next command vs next extended command




Happy to provide the buggy source if you'd like to run a second set of eyes over it.
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 18:30, 15 August 22
Just a short update.  I have it now reliably able to extend commands and edit them, list etc.  The only issue currently is that for extended commands, I have to modify where it works out line numbers so that while, wend and other loop constructs work (goto).  These actually have calculated and stored addresses and since I've now got support for multi-byte commands, i need to modify where it calculates these addresses.

As much as I think the Locomotive BASIC is ingenenious in the way it is coded, it is equally terrible.  I believe they were aiming for the best BASIC within 16kb - and I think they really did achieve that, but now we need to live with oddities because of that even though we now can extend it to 64kb or perhaps more.
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: Bread80 on 12:34, 18 August 22
If I were you I'd search for anywhere where the $ff token (prefix for functions) is processed and duplicate that for your own token.

However, I suspect your issue is within skip_next_tokenised_item at &e9fd. Modify as suggested above.
;;=skip next tokenised item
;Advances over the next tokenised item,
;including stepping over strings, comments, bar commands etc.
skip_next_tokenised_item:        ;{{Addr=$e9fd Code Calls/jump count: 9 Data use count: 0}}
        call    get_next_token_skipping_space; get next token skipping space
        ret    z               

        cp      $0e              ;Tokens $02 to $0d are variables
        jr      c,skip_over_variable; (+$25)
        cp      $20              ; space
        jr      c,skip_over_numbers; Tokens $0e to $19 are number constants
        cp      $22              ; double quote
        jr      z,skip_over_string         
        cp      $7c              ;'|'
        jr      z,skip_over_bar_command; (+$1b)
        cp      $c0              ;"'" comment
        jr      z,skip_over_comment; (+$30)
        cp      $c5              ;REM
        jr      z,skip_over_comment; (+$2c)
        cp      $ff              ; Extended/function tokens
        ret    nz               

        inc    hl               
        ret     
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 11:40, 06 October 22
Old Tokenizer
tokenise_identifiers:            ;{{Addr=$e03a Code Calls/jump count: 1 Data use count: 0}}
        push    bc                ;{{e03a:c5}}
        push    de                ;{{e03b:d5}}
        push    hl                ;{{e03c:e5}}

        ld      a,(hl)            ;{{e03d:7e}} ; get initial character of BASIC keyword
        inc    hl                ;{{e03e:23}}
        call    convert_character_to_upper_case;{{e03f:cdabff}} ; convert character to upper case
        call    get_keyword_table_for_letter;{{e042:cda8e3}} ; get list of keywords beginning with this letter
        call    keyword_to_token_within_single_table;{{e045:cdebe3}}
        jr      nc,tokenise_variable;{{e048:3026}} ;not found? - it's a variable!

        ld      a,c              ;{{e04a:79}}
        and    $7f              ;{{e04b:e67f}}
        call    test_if_letter_period_or_digit;{{e04d:cd9cff}}
        jr      nc,_tokenise_identifiers_18;{{e050:3009}}  (+$09)
        ld      a,(de)            ;{{e052:1a}} get prev token
        cp      $e4              ;{{e053:fee4}} FN token
        ld      a,(hl)            ;{{e055:7e}}
        call    nz,test_if_letter_period_or_digit;{{e056:c49cff}}
        jr      c,tokenise_variable;{{e059:3815}}  (+$15) tokenise variable name after FN

New Tokenizer
tokenise_identifiers:            ;{{Addr=$e03a Code Calls/jump count: 1 Data use count: 0}}
        push    bc                ;{{e03a:c5}}
        push    de                ;{{e03b:d5}}
        push    hl                ;{{e03c:e5}}

        ld      a,(hl)            ;{{e03d:7e}} ; get initial character of BASIC keyword
        inc    hl                ;{{e03e:23}}
        call    convert_character_to_upper_case;{{e03f:cdabff}} ; convert character to upper case

push af ; to be discarded or restored

        call    get_keyword_table_for_letter;{{e042:cda8e3}} ; get list of keywords beginning with this letter

;push af
;ld a, e
;ld ($be80), a
;ld a, d
;ld ($be81), a
;pop af

        call    keyword_to_token_within_single_table;{{e045:cdebe3}}
        ;jr      nc,tokenise_variable;{{e048:3026}} ;not found? - it's a variable!
jr c, tokenise_found ; found it first time

pop af ; restored


; try again in extension table
        call    get_extension_table_for_letter;get list of keywords beginning with this letter

;push af
;ld a, e
;ld ($be82), a
;ld a, d
;ld ($be83), a
;pop af

        call    keyword_to_token_within_single_table
        jp      nc,tokenise_variable;not found? - it's a variable!

; tokenise $e9 (extension)
;ld a, 7
;call $bb5a
        pop    af                ;{{e05b:f1}}
ld a, (de) ; a is the extension command token
        pop    de                ;{{e061:d1}} tokens <&80 = functions
        pop    bc                ;{{e062:c1}}
        push bc
push de
push af

;push af
;ld a, e
;ld ($be80), a
;ld a, d
;ld ($be81), a
;pop af

ld a, $e9 ; extension indicator token
ld (de), a
pop af
pop de
pop bc
inc de
dec bc
ld (de), a ; extension command token
inc hl
dec bc
push bc
push de
push af

; then resume with found token
push af   ; to be discarded

tokenise_found:
pop af   ; discard this

        ld      a,c              ;{{e04a:79}}
        and    $7f              ;{{e04b:e67f}}
        call    test_if_letter_period_or_digit;{{e04d:cd9cff}}
        jr      nc,_tokenise_identifiers_18;{{e050:3009}}  (+$09)
        ld      a,(de)            ;{{e052:1a}} get prev token
        cp      $e4              ;{{e053:fee4}} FN token
        ld      a,(hl)            ;{{e055:7e}}
        call    nz,test_if_letter_period_or_digit;{{e056:c49cff}}
        jr      c,tokenise_variable;{{e059:3815}}  (+$15) tokenise variable name after FN
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 11:43, 06 October 22
Old detokenizer
_detokenise_keyword_7:            ;{{Addr=$e303 Code Calls/jump count: 1 Data use count: 0}}
        push    af                ;{{e303:f5}}
        push    hl                ;{{e304:e5}}
        call    convert_token_to_keyword_text_ptr;{{e305:cdb8e3}}
        or      a                ;{{e308:b7}}
        jr      z,_detokenise_keyword_16;{{e309:2808}}  (+$08)
        push    af                ;{{e30b:f5}}
        call    detokenise_append_space_if_needed;{{e30c:cde6e2}}
        pop    af                ;{{e30f:f1}}

New detokenizer
_detokenise_keyword_7:            ;{{Addr=$e303 Code Calls/jump count: 1 Data use count: 0}}
        push    af                ;{{e303:f5}}
        push    hl                ;{{e304:e5}}

push af
cp $e9 ; do we have an extension token?
jr nz, _detokenise_patch_2

pop af

pop hl
ld a, (hl) ; since we had an extension token, get next token
inc hl ; point to the next token
push hl
        call    convert_token_to_extension_text_ptr;{{e305:cdb8e3}}

jr _detokenise_patch_3

_detokenise_patch_2:
pop af
        call    convert_token_to_keyword_text_ptr;{{e305:cdb8e3}}

_detokenise_patch_3:
        or      a                ;{{e308:b7}}
        jr      z,_detokenise_keyword_16;{{e309:2808}}  (+$08)
        push    af                ;{{e30b:f5}}
        call    detokenise_append_space_if_needed;{{e30c:cde6e2}}
        pop    af                ;{{e30f:f1}}
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 11:49, 06 October 22
Old keyword LUT
get_keyword_table_for_letter:    ;{{Addr=$e3a8 Code Calls/jump count: 1 Data use count: 0}}
        push    hl                ;{{e3a8:e5}}
        sub    $41              ;{{e3a9:d641}}  initial letter - 'A'
                                  ; number in range 0->27
        add    a,a              ;{{e3ab:87}}  x2 (two bytes per table entry)
                                  ; A = offset into table

        add    a,(keyword_table_per_letter) and $ff;{{e3ac:c618}} $18  table starts at $e418 Low byte of keyword table address
        ld      l,a              ;{{e3ae:6f}}
        adc    a,(keyword_table_per_letter >> 8);{{e3af:cee4}} $e4  high byte of keyword table address
        sub    l                ;{{e3b1:95}}
        ld      h,a              ;{{e3b2:67}}

        ld      e,(hl)            ;{{e3b3:5e}}  get address of keyword list from table
        inc    hl                ;{{e3b4:23}}
        ld      d,(hl)            ;{{e3b5:56}}
        pop    hl                ;{{e3b6:e1}}
        ret                      ;{{e3b7:c9}}

;;========================================================================
;;convert token to keyword text ptr
convert_token_to_keyword_text_ptr:;{{Addr=$e3b8 Code Calls/jump count: 1 Data use count: 0}}
        push    bc                ;{{e3b8:c5}}
        ld      c,a              ;{{e3b9:4f}}
        ld      b,$1a            ;{{e3ba:061a}}  Table count
        ld      hl,keyword_table_Z;{{e3bc:214ce4}}

_convert_token_to_keyword_text_ptr_4:;{{Addr=$e3bf Code Calls/jump count: 1 Data use count: 0}}
        call    search_within_a_single_table;{{e3bf:cdd7e3}}  Loop through each table
        jr      c,_convert_token_to_keyword_text_ptr_12;{{e3c2:380e}}  (+$0e)
        inc    hl                ;{{e3c4:23}}
        djnz    _convert_token_to_keyword_text_ptr_4;{{e3c5:10f8}}  (-$08)

        ld      hl,symbols_table  ;{{e3c7:2136e7}}  Also search symbols table
        call    search_within_a_single_table;{{e3ca:cdd7e3}}
        jp      nc,Error_Syntax_Error;{{e3cd:d249cb}}  Not found: Syntax Error
        ld      b,$c0            ;{{e3d0:06c0}} "'" comment

New keyword LUT
get_keyword_table_for_letter:    ;{{Addr=$e3a8 Code Calls/jump count: 1 Data use count: 0}}
        push    hl                ;{{e3a8:e5}}
        sub    $41              ;{{e3a9:d641}}  initial letter - 'A'
                                  ; number in range 0->27
        add    a,a              ;{{e3ab:87}}  x2 (two bytes per table entry)
                                  ; A = offset into table

        ;add    a,(keyword_table_per_letter) and $ff;{{e3ac:c618}} $18  table starts at $e418 Low byte of keyword table address
        ;ld      l,a              ;{{e3ae:6f}}
        ;adc    a,(keyword_table_per_letter >> 8);{{e3af:cee4}} $e4  high byte of keyword table address
        ;sub    l                ;{{e3b1:95}}
        ;ld      h,a              ;{{e3b2:67}}
push bc
ld b, 0
ld c, a
ld hl, keyword_table_per_letter
add hl, bc
pop bc

        ld      e,(hl)            ;{{e3b3:5e}}  get address of keyword list from table
        inc    hl                ;{{e3b4:23}}
        ld      d,(hl)            ;{{e3b5:56}}
        pop    hl                ;{{e3b6:e1}}
        ret                      ;{{e3b7:c9}}

;;========================================================================
;;convert token to extension text ptr
convert_token_to_extension_text_ptr:;Code Calls/jump count: 1 Data use count: 0}}
        push    bc
        ld      c,a
        ld      b,26 ; Table count
        ld      hl,extension_table_Z
jr _convert_token_to_keyword_text_ptr_4

convert_token_to_keyword_text_ptr:;{{Addr=$e3b8 Code Calls/jump count: 1 Data use count: 0}}
        push    bc                ;{{e3b8:c5}}
        ld      c,a              ;{{e3b9:4f}}
        ld      b,26            ;{{e3ba:061a}}  Table count
        ld      hl,keyword_table_Z;{{e3bc:214ce4}}

_convert_token_to_keyword_text_ptr_4:;{{Addr=$e3bf Code Calls/jump count: 1 Data use count: 0}}
        call    search_within_a_single_table;{{e3bf:cdd7e3}}  Loop through each table
        jr      c,_convert_token_to_keyword_text_ptr_12;{{e3c2:380e}}  (+$0e)
        inc    hl                ;{{e3c4:23}}
        djnz    _convert_token_to_keyword_text_ptr_4;{{e3c5:10f8}}  (-$08)

        ld      hl,symbols_table  ;{{e3c7:2136e7}}  Also search symbols table
        call    search_within_a_single_table;{{e3ca:cdd7e3}}
        jp      nc,Error_Syntax_Error;{{e3cd:d249cb}}  Not found: Syntax Error
        ld      b,$c0            ;{{e3d0:06c0}} "'" comment
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 11:52, 06 October 22
Old execution code
execute_command_token:            ;{{Addr=$de8f Code Calls/jump count: 2 Data use count: 0}}
        add    a,a              ;{{de8f:87}}
        jp      nc,BAR_command_or_implicit_LET;{{de90:d289d6}} token < &80: either a bar command or a variable (implicit LET)
        cp tokenise_a_BASIC_line - command_to_code_address_LUT - 1;{{de93:fec3}} version with formula;WARNING: Code area used as literal
;OLD de93 fec3      cp      $c3              ;the last valid token is &e1 which doubles to &c2, so >= &c3 is error
        jr      nc,raise_syntax_error_E;{{de95:3010}}  (+$10)
        ex      de,hl            ;{{de97:eb}}
        add    a,command_to_code_address_LUT and $ff;{{de98:c6e0}} $e0 lookup token in table
        ld      l,a              ;{{de9a:6f}}
        adc    a,command_to_code_address_LUT >> 8;{{de9b:cede}} $de
        sub    l                ;{{de9d:95}}
        ld      h,a              ;{{de9e:67}}
        ld      c,(hl)            ;{{de9f:4e}} code address into BC
        inc    hl                ;{{dea0:23}}
        ld      b,(hl)            ;{{dea1:46}}
        push    bc                ;{{dea2:c5}} push so we'll return to code with next token
        ex      de,hl            ;{{dea3:eb}}
        jp      get_next_token_skipping_space;{{dea4:c32cde}}  get next token skipping space

;;=raise syntax error
raise_syntax_error_E:            ;{{Addr=$dea7 Code Calls/jump count: 3 Data use count: 0}}
        jp      Error_Syntax_Error;{{dea7:c349cb}}  Error: Syntax Error

New execution code
execute_command_token:            ;{{Addr=$de8f Code Calls/jump count: 2 Data use count: 0}}

cp #e9 ; do we have an extension?
jr nz, execute_command_token2

; if so, skip to the next token
call get_next_token_skipping_space
jr execute_extension_token

execute_command_token2:
        add    a,a              ;{{de8f:87}}
        jp      nc,BAR_command_or_implicit_LET;{{de90:d289d6}} token < &80: either a bar command or a variable (implicit LET)
        cp command_to_code_address_LUTEnd - command_to_code_address_LUT - 1;the last valid token is &e1 which doubles to &c2, so >= &c3 is error
        jr      nc,raise_syntax_error_E;{{de95:3010}}  (+$10)
        ex      de,hl            ;{{de97:eb}}
        add    a,command_to_code_address_LUT and $ff;{{de98:c6e0}} $e0 lookup token in table
        ld      l,a              ;{{de9a:6f}}
        adc    a,command_to_code_address_LUT >> 8;{{de9b:cede}} $de
        sub    l                ;{{de9d:95}}
        ld      h,a              ;{{de9e:67}}
        ld      c,(hl)            ;{{de9f:4e}} code address into BC
        inc    hl                ;{{dea0:23}}
        ld      b,(hl)            ;{{dea1:46}}
        push    bc                ;{{dea2:c5}} push so we'll return to code with next token
        ex      de,hl            ;{{dea3:eb}}
        jp      get_next_token_skipping_space;{{dea4:c32cde}}  get next token skipping space

;;============================================
;;execute extension token
;A=token
;Tokens >= &80 are tokenised words
;the only token < &80 we should have here are for bar extensions or variable names (implicit LET)
execute_extension_token:            ;{{Addr=$de8f Code Calls/jump count: 2 Data use count: 0}}
        add    a,a              ;{{de8f:87}}
        cp extension_to_code_address_LUTEnd - extension_to_code_address_LUT - 1;the last valid token is &e1 which doubles to &c2, so >= &c3 is error
        jr      nc,raise_syntax_error_E;{{de95:3010}}  (+$10)
        ex      de,hl            ;{{de97:eb}}
        add    a,extension_to_code_address_LUT and $ff;{{de98:c6e0}} $e0 lookup token in table
        ld      l,a              ;{{de9a:6f}}
        adc    a,extension_to_code_address_LUT >> 8;{{de9b:cede}} $de
        sub    l                ;{{de9d:95}}
        ld      h,a              ;{{de9e:67}}
        ld      c,(hl)            ;{{de9f:4e}} code address into BC
        inc    hl                ;{{dea0:23}}
        ld      b,(hl)            ;{{dea1:46}}
        push    bc                ;{{dea2:c5}} push so we'll return to code with next token
        ex      de,hl            ;{{dea3:eb}}
        jp      get_next_token_skipping_space;{{dea4:c32cde}}  get next token skipping space

;;=raise syntax error
raise_syntax_error_E:            ;{{Addr=$dea7 Code Calls/jump count: 3 Data use count: 0}}
        jp      Error_Syntax_Error;{{dea7:c349cb}}  Error: Syntax Error
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 11:55, 06 October 22
new_commands.asm for new LUT lookups
newcommand_DIR: jp command_CAT


; command_GRAPHICS_PAPER__GRAPHICS_PEN_and_set_graphics_draw_mode:;{{Addr=$c59a Code Calls/jump count: 0 Data use count: 1}}
        ; cp      $ba              ;{{c59a:feba}}  token for "PAPER"
        ; jr      z,eval_and_set_graphics_paper;{{c59c:2813}}  set graphics paper
     
        ; call    next_token_if_equals_inline_data_byte;{{c59e:cd25de}}
        ; defb $bb                  ; token for "PEN"
        ; cp      $2c              ;{{c5a2:fe2c}}  ','
        ; call    nz,validate_and_set_graphics_pen;{{c5a4:c4bac5}}  set graphics pen


newcommand_MCP:
;ret nz
push af
push bc
push de
push hl

push af
ld a, 0
ld ($be80), a
pop af

        ;cp $0d                  ;CR
;jr z, newcommand_MCPOnly

        cp $00                  ;zero
jr z, newcommand_MCPOnly

; check overloaded tokens that are not extensions before we read the next token
        cp $a7                  ;extension token to test "LIST"
jr z, newcommand_MCPLIST

; do we have extensions?
        cp $e9                  ;extension indicator token
jr nz, newcommand_MCPEnd

; read the next token
call get_next_token_skipping_space

        cp $80                  ;extension token to test "STATUS"
jr z, newcommand_MCPSTATUS

newcommand_MCPEnd:
ld ($be80), a
pop hl
pop de
pop bc
pop af
;jr newcommand_MCPOnly
ret

newcommand_MCPOnly:
;ld a, 1
ld ($be80), a
ld hl, mcp_message
call output_ASCIIZ_string

pop hl
pop de
pop bc
pop af
        ret

newcommand_MCPLIST:
;ld a, 2
ld ($be80), a
ld hl, mcp_list_message
call output_ASCIIZ_string

pop hl
pop de
pop bc
pop af
;xor a
        ret

newcommand_MCPSTATUS:
;ld a, 3
ld ($be80), a
ld hl, mcp_status_message
call output_ASCIIZ_string

pop hl
pop de
pop bc
pop af
;xor a
        ret

mcp_message:
defb "MCP",10,13,0  ; base mcp

mcp_list_message:
defb "MCP LIST",10,13,0  ; base mcp and overloaded list

mcp_status_message:
defb "MCP STATUS",10,13,0  ; base mcp, but extended status

; commands made of more than 1 word seemed previously to always have two variants implemented
; for example: symbol after.  symbol is a command that checks for a subsequent after, but after is also a valid command on it's own.
; similar for graphics pen, graphics paper, line input
; ultimately we get dual use of the 2nd keyword if it makes sense
; note: the overloaded second keyword must have the same token as the original use keyword
;
; for example, below is status, but above is also mcp status separately
newcommand_STATUS:
;ret nz
push af
push bc
push de
push hl

ld a, 4
ld ($be80), a
ld hl, status_message
call output_ASCIIZ_string

pop hl
pop de
pop bc
pop af
;xor a
        ret

status_message:
defb "STATUS",10,13,0 

; does nothing as we already have a list command, however we are using this token for extended list commands
newcommand_LIST:
ret

;; get extension table for letter
;; A = initial letter of BASIC extension
get_extension_table_for_letter:
        push    hl
        sub    $41              ;{initial letter - 'A'
                                  ; number in range 0->27
        add    a,a              ; x2 (two bytes per table entry)
                                  ; A = offset into table

        ;add    a,(extension_table_per_letter) and $ff;table starts at Low byte of keyword table address
        ;ld      l,a
        ;adc    a,(extension_table_per_letter >> 8);high byte of keyword table address
        ;sub    l
        ;ld      h,a

push bc
ld b, 0
ld c, a
ld hl, extension_table_per_letter
add hl, bc
pop bc

        ld      e,(hl)            ;get address of keyword list from table
        inc    hl
        ld      d,(hl)
        pop    hl
        ret

extension_to_code_address_LUT:
                                 
defw newcommand_STATUS ;STATUS $80
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;
defw doNothing ;

extension_to_code_address_LUTEnd:

extension_table_per_letter:

        defw extension_table_A      ;
        defw extension_table_B      ;
        defw extension_table_C      ;
        defw extension_table_D      ;
        defw extension_table_E      ;
        defw extension_table_F      ;
        defw extension_table_G      ;
        defw extension_table_H      ;
        defw extension_table_I      ;
        defw extension_table_J      ;
        defw extension_table_K      ;
        defw extension_table_L      ;
        defw extension_table_M      ;
        defw extension_table_N      ;
        defw extension_table_O      ;
        defw extension_table_P      ;
        defw extension_table_Q      ;
        defw extension_table_R      ;
        defw extension_table_S      ; STATUS
        defw extension_table_T      ;
        defw extension_table_U      ;
        defw extension_table_V      ;
        defw extension_table_W      ;
        defw extension_table_X      ;
        defw extension_table_Y      ;
        defw extension_table_Z      ;

defb "EXTENSION_TABLE:"
;;======================================================================
;; Extension table
;; list of extension as text followed by extension byte (token?)
;; end of list signalled with a 0 byte
;;
;; - BASIC extension stored excluding initial letter

;;=extension table Z
extension_table_Z:
                                 
        defb 0                   

;;=extension table Y
extension_table_Y:
                                 
        defb 0                   

;;=extension table X
extension_table_X:
                                 
        defb 0                   

;;=extension table W
extension_table_W:
                                 
        defb 0                   

;;=extension table V
extension_table_V:
                                 
        defb 0                   

;;=extension table U
extension_table_U:
                                 
        defb 0                   

;;=extension table T
extension_table_T:
                                 
        defb 0                   

;;=extension table S
extension_table_S:
                                 
        defb "TATU","S"+$80,$80    ; STATUS
        defb 0                   

;;=extension table R
extension_table_R:
                                 
        defb 0                   

;;=extension table Q
extension_table_Q:
                                 
        defb 0                   

;;=extension table P
extension_table_P:
                                 
        defb 0                   

;;=extension table O
extension_table_O:
                                 
        defb 0                   

;;=extension table N
extension_table_N:
                                 
        defb 0                   

;;=extension table M
extension_table_M:
                                 
        defb 0                   

;;=extension table L
extension_table_L:
                                 
        defb 0                   

;;=extension table K
extension_table_K:
                                 
        defb 0                   

;;=extension table J
extension_table_J:
                                 
        defb 0                   

;;=extension table I
extension_table_I:
                                 
        defb 0                   

;;=extension table H
extension_table_H:
                                 
        defb 0                   

;;=extension table G
extension_table_G:
                                 
        defb 0                   

;;=extension table F
extension_table_F:
                                 
        defb 0                   

;;=extension table E
extension_table_E:
                                 
        defb 0                   

;;=extension table D
extension_table_D:
                                 
        defb 0                   

;;=extension table C
extension_table_C:
                                 
        defb 0                   

;;=extension table B
extension_table_B:
                                 
        defb 0                   

;;=extension table A
extension_table_A:
                                 
        defb 0                   

defb "EXTENSION_END:"

Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: Bread80 on 22:02, 06 October 22
I hadn't thought before about how extra tokens would affect the keyword lookups.

It looks like your creating a whole second set of keyword tables. But you'll need to know which keyword (ASCII) you have in order to know which table to look in, and you won't know which table to look in until you've done the lookup.

My suggestions would be to take the current lookup tables and add a second byte (after the existing token byte) to specify the table. The ASCII to token mapping would need to skip over (or return) the extra byte.

For token to keyword mapping you could retain the existing subroutine for original code to use, but updating it to check the second token byte and create a separate routine for the new table, which would be a near copy except for the second byte checking.

(Or you could do this in one routine which checks both bytes. That would depend on how easy it is to adapt the calling code, and the availability of registers both for the call and the code itself).
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 08:16, 07 October 22
Actually the keywords string together to form sentences, that how how it is by default, currently up to 3 words I think (on error goto blah).  what is intersting, is that each keyword can be functional by itself, or with 2, or with 3 etc and each keyword that is strung together takes up a token, the only way to get more tokens is to sacrifice existing or as I have done, create a new lookup of tokens triggered by an extension token.  To make matters worst, for expressions, they use the same token space as the non-expression tokens, effectively overloaded so we don't want to mess that up too.

To keep changes to a minimum, the extension token $e9, if present does go lookup a 2nd table for which we can have all new keywords or use existing tokens as extension, eg.

extension tokens:

MCP
STATUS

standard tokens:

LIST - lists a basic program usually

combinations:

LIST - as usual
MCP STATUS - extension token followed by another extension token
MCP LIST - list behaviour specific to the MCP, perhaps list tasks, using extension token followed by standard token

I didn't want to complicate the existing code too much, but even to introduce a new list or keywords, we need to take space from somewhere.

We can do that by either somehow splitting the existing 16kb ROM into 2 ROMS trying to make the code organised with a minimal number of jumps - I started looking at this and it is very very difficult because almost all the logic for keywords, errors, etc seems to be needed from everywhere.  The only way to do this with a bit of effort would be to put the actual implementations of the keywords into a 2nd ROM, but then we are swapping ROMs for every invokation - might be a bit slow (compared to usual)?

I thought it is easier with the extension lookup because, we can essentially move that to the 2nd ROM (duplicating only the 1st keyword lookup table in the 2nd ROM purely for lookups) if we want to continue stringing words together.

The most logical way to extend it would be to introduce keywords that string together related subsystems of commands.

like 

GRAPHICS PEN, GRAPHICS PAPER
which have 3 keywords involved: GRAPHICS, PEN, PAPER (note PEN & PAPER can be used by themselves too).

So, we could introduce new GRAPHICS commands, new AUDIO commands, new TASK (or MCP) commands, new RAM access commands etc.

Sadly, having to sacrifice existing keywords from the ROM to make the job easier to get any space to even pass control to a 2nd ROM for the 2nd lookup table, that is why I thought to create Locomotive Shell.  Already proven that we can launch different flavours of BASIC via their RSXs, |BASIC, or |SHELL or |BASICPLUS, and they all co-exist nicely and RSXs seem to work perfectly in all.  Just be sure to choose the default one you like to have in the upper half of the system ROM.
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: Bread80 on 18:23, 07 October 22
So, if I understand you, certain keywords cause the system to switch to an alternate keyword table for the tokens/keywords which follow them. That's actually quite neat.

I was tempted to mention that any new keywords will cause any old code using that word as a variable name will fail, but using a separate table, presumably, means no such problems.

As for splitting the ROM in two, I have that in my future plans, but plenty of other projects to do first. I'll need to create some tools to parse the code and find links to see what to put in each ROM (or both), and then to replace any links between ROMs with the appropriate system call.
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: TotO on 18:48, 07 October 22
I can imagine that people using a shell instead of the BASIC to run programs expect to only replace the ROM 0.
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 02:23, 02 May 23
moved... to RASM thread
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 19:44, 09 June 23
I was looking through the lower rom disassembly here:

http://cpctech.cpc-live.com/docs/os.asm

A few thoughts.  Quite a lot of space can be recovered if some functionality is removed. 

Does anyone still use a printer on cpc?

Does anyone with disc machines still use tapes?

Would there be many games that actually use firmware graphics and maths functions or are they almost exclusively used by BASIC?

Machine startup messages, any need to keep them all?

Then I was thinking can any of it be optimized for speed if other code was removed.

Of course if stuff required by BASIC is removed it needs to be somewhere, perhaps in a 2nd BASIC ROM, although it means a slower far call to get to the 2nd ROM. But the 2nd ROM gives a full 16k, and we ultimately can have a 3rd or 4th ROM.

What if CPM programs could just be run from AMSDOS on 128k machines like any other program? 

What if the following were all combined into the CPC OS ROM...

MOS (multitasking across banked RAM)
AMSDOS Software support
CPM Software support
Primal Software support
Multiple CPU support
Extra Memory support

A bit off topic, with extra memory, would programmers prefer 16mb support with 256 byte (minimum memory access then translation) or 24 bit addressing?   Or 4mb support only?
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: GUNHED on 13:32, 13 June 23
And the power of the Z80 can be nearly doubled (in quite some cases) if the Firmware does NOT use the 2nd register file of the Z80 any longer. They just get wasted for interrupt handling. So instead of saving some values continuously in A' and BC' just put them to the RAM.
- This will reduce the overall system speed by approximately 0,001%
- But this will speed up most of the routine quite a well degree, of course they will need to be overworked to use 2nd register file instead of RAM. A complete rewrite would be indeed more beneficial.

- All this just in case you don't want to create you own completely new OS -  ;) ;D :) :) :)
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 15:00, 11 September 23
Currently on CPC we get approxi 42kb of BASIC program space using the current memory map:

EXISTING:

+--------+ 0000
|LOWJB   |
|PROGRAM |
+--------+ 4000
|PROGRAM |
|PROGRAM |
+--------+ 8000
|VARIABLE|
|HIGHJB  |
+--------+ C000
|SCREEN  |
|        |
+--------+ FFFF

Possible new memory map:

+--------+ 0000
|LOWJB   |
|VARIABLE|
+--------++--------++--------++--------++--------+ 4000
|PROGRAM ||PROGRAM ||PROGRAM ||PROGRAM ||PROGRAM |
|PROGRAM ||PROGRAM ||PROGRAM ||PROGRAM ||PROGRAM |
+--------++--------++--------++--------++--------+ 8000
|STACK   |
|HIGHJB  |
+--------+ C000
|SCREEN  |
|        |
+--------+ FFFF


If the above possible new memory map was used, on a standard CPC6128 would give 80kb himem PLUS 16kb (-0100H) variable space.  And 4mb+ on a 4mb CPC (still with 16kb variable space for now).

Looking at the lower ROM disassembly, it seems that the main firmware handles the loading of the BASIC program, and not the BASIC interpreter itself.  Somewhere in or before MC_START_PROGRAM is called.  I am not quite sure where it get's the load address from.  Ultimately I didn't want to change the lower ROM, but I seem to need to change it to load BASIC programs to #4000 instead of #0100 - and of course, enhance the loader to split the BASIC into lots of 16kb banks sideways during the load.  

Does anyone know where in the lower ROM this actually happens?  Or is it purely read from the load address stored in the header of the .BAS file vs .BIN files?

Why do this?  To cater for better software with more memory, I belive the memory map is better utilised sideways for single program (non-multitasking) - I have failed to convince everyone so far here though.  

I have proven however, 2 ways to expand functionality of BASIC - that is through the use of more keywords in BASIC, as well as through UniDOS Accessory ROMS.  Both have their own benefits.  UniDOS is unconventional for CPC way of doing things but does not take up any additional RSXs therefore not reducing HIMEM it seems regardless of how many BASIC enhancements implemented - this is more using a Linux way of doing things using virtual files to do stuff.
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: andycadley on 16:29, 11 September 23
Interesting ideas. As far as Sinclair BASIC goes, it shares a lot of keywords with Locomotive BASIC but the specific syntax and meaning is very different, I'm not sure automatic translation would necessarily work without a lot of effort.

If I were going to do something like this (and I considered it once upon a time) I'd be inclined to just use 24 bit pointers for everything (variables, code lines etc) and have the BASIC interpreter manage swapping the memory around as needed (including things like allowing PEEK and POKE to address all the RAM) - the Sam Coupé does that and it works really well. It is slower, but BASIC is interpreted anyway and the time impact is marginal. Space considerations might be more of an issue, but I'd assume it wouldn't be used on machines with less than 128K anyway.
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: McArti0 on 17:48, 11 September 23
Basic has CHAIN and CHAIN MERGE ,DELETE. 
This is a natural method of changing the Basic code and accumulating it to a size larger than 42kb.
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: andycadley on 19:30, 11 September 23
Quote from: McArti0 on 17:48, 11 September 23Basic has CHAIN and CHAIN MERGE ,DELETE.
This is a natural method of changing the Basic code and accumulating it to a size larger than 42kb.

For code, sure. But there are advantages (to a point) to being able to store larger arrays etc. Up to about 512K is probably reasonable to do in that fashion, much more than that and the structures needed to manage things start getting incompatible with running on 128K.
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 20:34, 19 September 23
For marge data sets like arrays i would create linked list commands that are in the central ram after the basic program, this bit is trivial. I have already coded RSXs linked list commands that support 512kb ram.  Alyssa Database just used extra ram like fixed size array elements which suited that application but isn't very flexible. 
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: Bread80 on 23:25, 25 September 23
I'll point out one potential problem with storing BASIC code in sideways RAM: Where a variable stores a string constant the variable  (string pointer) will reference the string stored in the BASIC code (which saves memory). Thus the line:
s$="Hello!"
the Hello! is not copied to the strings area. This would create problems if the variable is referenced whilst that bank of RAM is paged out.

As for LOMEM and HIMEM, these are loaded into registers and passed to the foreground ROM. Consult the firmware manual. MC_START_PROGRAM should be the appropriate routine.

The start address of BASIC is part of the specification for foreground ROMs. It's the reason you can install other foreground ROM(s) in place of BASIC.

If you don't want to modify the firmware ROM, it might be possible to hack this via the RSX/ROM initialisation procedure. I.e. when your ROM gets initialised set the registers appropriately and call MC_START_PROGRAM to restart BASIC. You'll obviously need to detect the second call. Which may not be easy as memory gets cleared.

Having said that, IIRC, the ROM initialisation procedure allows you to modify LOMEM and HIMEM. If that's correct this will be easy :-)

BTW the RSX/ROM initialisation is called by BASIC at startup.
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: Bread80 on 23:28, 25 September 23
BTW if you ask me it would be cooler to be able to store multiple BASIC programs in memory and switch between them. The Elan Enterprise had built in support for that.

Realistically that would limit you to 16k code and data. And you'd need a way to preserve the BASIC data area between programs. Maybe using the spare non-banked memory?
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 07:23, 26 September 23
Actually it may be possible to have 
Quote from: Bread80 on 23:28, 25 September 23BTW if you ask me it would be cooler to be able to store multiple BASIC programs in memory and switch between them. The Elan Enterprise had built in support for that.

Realistically that would limit you to 16k code and data. And you'd need a way to preserve the BASIC data area between programs. Maybe using the spare non-banked memory?
Entire 64k chunks swapped around, it seems if I manually swap memory around in basic, basic is still working... I just cannot see it, but I can type and press del to make it beep.

My MOS kernel originally was going to allow multiple shells but I think its better as drivers for basic.  Just I failed to convince anyone of conforming to any driver standard previously. Nobody has any interest it seems in standardization on cpc. 
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 00:51, 04 February 24
Quote from: Bread80 on 12:19, 21 February 22I'd suggest a lot of the changes your making are more a firmware thing rather than a BASIC thing. If you put them in the firmware then there available for any software to use. And it's a lot easier to strip out space in the firmware - the floating point maths stuff can move to a sideways ROM. And the printer stuff sounds like a safe bet to get rid of. Maybe even the cassette routines depending on how you like to load and save stuff.
I think this might be a better idea - i will investigate this too
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: GUNHED on 14:26, 07 February 24
Time for ZhulienOS!  8)
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 01:46, 28 February 24
changes made by zhulien:

- removed hardcoded 128k from startup message
- added dynamic memory check (placeholder for now)
- shortened the startup error message to gain some space
- removed all the brand names and name lookup in favour of a hardcoded Amstrad
- removed 60hz support as this is seldom used
- moved graphics, text and screen operations (including FONT) to ROM 31 so that it could be considered a video driver extension (perhaps this could be made to work like OffseT's UniDOS extension ROMs to support different graphics cards down the track)

todo:

- modify the huge FAR jumpblock to be calculated (perhaps) to take less bytes, need to see if it lowers performance too greatly though

issues:

- a couple of functions relating to RAMLAM LDIR and RAMLAM CLEAR should be recoded to be faster
- inks are not correct and colour cycling event is not working
- some badly coded software doesn't work

ROM sizes:

lower: 10894 bytes, 5490 bytes claimed
upper: 6589 bytes , 9795 bytes left

how does it work:

2 ROMS, the moved functions are now in ROM 31 and are accessed by the lower ROM via a jumpblock of RES 3s.  The upper ROM uses the normal jumpblock
where necessary (only a few functions) to call the lower ROM. A couple of functions in the upper ROM are slow due to using RAMLAM but they could be
improved either to use a buffer at BE80 or in extra RAM. The RES 3 jumpblock in the lower ROM could be shrunk to be calculated - but will that be
faster or slower? note sure.

next:

- Examine the best ways to extend functionality in the lower ROM, perhaps adding new commands? or perhaps jumpblock entries in the ROM 31 could
be expanded... I still think that a mixture of this experment and the UniDOS streamer ROM can work well together also - streamer could use this newer
ROM. 

- API extensions that could benefit.  Memory management! So d*head Jason can't get his way...
- maybe to make it more compatible with badly written software, i will need to put as many entry points back in place as possible - and work within the gaps.
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 01:49, 28 February 24
You can build them with RASM, or put them in WinAPE (newlower.rom goes in ROM 0, newupper.rom goes in ROM 31).
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 04:37, 28 February 24
Here is another build with source with most of the original entry points in-tact (I hope). Of course the font is not in the ROM.

I need to see if RASM will produce a symbol table and I need to verify it.

I still haven't figured out why the inks don't work.
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: XeNoMoRPH on 07:13, 28 February 24
(https://i.ibb.co/JKNhn5x/Sin-t-tulo.png)

Ok, and now what can I do, how can I test this?
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 08:51, 28 February 24
At the moment it is to test and see how compatible it is with software. I really do believe in the 2nd build there i must have some of the entry points wrong.

I was surprised that the following work:

Discology 6 rom
Turbo esprit by domain 
Unidos
Orgams (mostly)
Basic
Alyssa database with fast text patch removed 

What didn't work for me yet

Cpm 2.2
Cpm plus
My old fast text patch for mode 2
Protext rom (corrupted fonts)
Maxam 1.5 rom (invisible text sometimes)
Alien (game, I can hear it but the inks are broken)

I don't really understand how inks work at the hardware level on cpc.  It looks in the code to do the following.  There is an interrupt that cycles through 2 sets of inks, it does this always even if they are 2 sets of the same inks.  But events can't work currently in the 2nd rom so I changed them to be in the first rom.  But oddly they don't show right colours often making all text invisible.

If I verify the symbol table (if that is the cause of some crashes) and fix the inks and prove its compatible then... we have room in the roms for enhancements such as history in the editor perhaps, as mentioned, memory management will be a priority for me.  

Also before I was looking at adding commands to locomotive basic but the issue it is a very full rom, and it is hardntonsplit into 2, however... its much easier to put some of the basic logic in the lower rom that is visible at the same time as basic, than to move it to another sideways rom.
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: Bread80 on 15:10, 07 March 24
The two sets of ink colours are to allow flashing inks. The flashing rate can be varied, so there's also a counter somewhere to trigger the switch between sets.

The interrupt, at frame flyback, simply OUTs each ink to the gate array.

If text is invisible then I assume that means the gate array has the same inks for both the foreground and background colours. Possibly an issue with the setting of pen/paper colours?
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: McArti0 on 08:35, 08 March 24
Quote from: Bread80 on 15:10, 07 March 24trigger the switch between sets
do you know how to check which set is currently loaded? I tried to follow this interrupt but got lost several times and failed.
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: ZorrO on 12:52, 08 March 24
I don't know what you are doing here, but maybe you have a problem with inks because they are at a different address in 464 and 6128.

IF PEEK(&BB5B)=0 THEN p=45530 ELSE p=47061 'this check inks address for 464 or 6128
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: zhulien on 17:14, 11 March 24
Quote from: ZorrO on 12:52, 08 March 24I don't know what you are doing here, but maybe you have a problem with inks because they are at a different address in 464 and 6128.

IF PEEK(&BB5B)=0 THEN p=45530 ELSE p=47061 'this check inks address for 464 or 6128
It is a recompilation of the OS ROM to be split into two ROMs for further space and possible enhancements.  If all in a single ROM, the interrupt will work, but if in an upper ROM, it will be extremely difficult because interrupts aren't always disabled when upper ROMs are executing, so I moved the interrupt handler to the lower ROM, but... still the hardware colours are mucked up with the ink command.  it's a bit bizzare - but most definitely due to me know knowing exactly how the inks work.  I have built 2 versions, one where entry points are in a single large jumpblock and another where all known official entry points are at their original positions within the lower ROM - although the single jumpblock is cleaner and makes newly added code a lot nicer, i was hoping the original entry locations (if other code jumps to ROMs directly) would increase compatability - but doesn't appear to make much difference.

The 2 main problems I haven't resolved so far is:

- the ink issue, submit an ink command and chances are your text will disappear or be the wrong colour, but it is the correct colour upon reset (note: i did disable the interrupt handler for now to at least get 1 set of inks to work, but no luck yet)
- then figure out why some software breaks and others don't.  Of the ones that don't break, I am surprised: discology 6 for example, then others that do break, CP/M 2.2/Plus (not sure why yet, unless they do extensive jumps to undocumented locations in the lower ROM).
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: andycadley on 17:30, 11 March 24
It's worth noting that the INK command doesn't directly set the ink, it just sets the relevant values for the interrupt handler to actually set the hardware colours at the next flyback.

So if the INK command isn't aware of where you've moved the data to, or the interrupt handler is looking in the wrong place, you'll get unexpected colours. Disabling the interrupt won't help, because it just means the colour values never actually get configured.
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: McArti0 on 21:12, 11 March 24
https://cpctech.cpc-live.com/docs/os.asm

OS-LowROM 6128
0d87 11e5b7    ld      de,$b7e5  'hard address palette 1
0d8a 3af6b7    ld      a,($b7f6)
0d8d b7        or      a
0d8e 3ad3b7    ld      a,($b7d3)
0d91 c8        ret     z
;;===========================================================================
0d92 11d4b7    ld      de,$b7d4  'hard address palette 2
0d95 3ad2b7    ld      a,($b7d2)
0d98 c9        ret
Title: Re: Locomotive Shell 1.0 - Extending & Modifying Locomotive BASIC
Post by: Bread80 on 23:51, 13 March 24
Quote from: McArti0 on 08:35, 08 March 24
Quote from: Bread80 on 15:10, 07 March 24trigger the switch between sets
do you know how to check which set is currently loaded? I tried to follow this interrupt but got lost several times and failed.
I've had a dig in the source code, and updated the firmware variables list at https://github.com/Bread80/CPC6128-Firmware-Source/blob/main/Includes/MemoryFirmware.txt

On a '6128 you need the byte at &b7f6. It should be &00 if the first set of inks is active, &ff for the second.
&b7f8 is the current value of the counter (counts down).

For '464 and other related addresses check the link :)
Powered by SMFPacks Menu Editor Mod