SDCC question - how to deal with program being too large?

[ervin]

Hi folks.

My program has now got to the stage where it goes past 0xA6FC (I'm loading the code at 0x8000 because I'm using double buffering).
As a result, the following error message appears when loading: *** PROGRAM LOAD FAILED ***

Now, I'm using cpctelera's cpct_disableFirmware function, but that doesn't kick in until the program is loaded, which of course it doesn't.


So now I'm stuck.

I can think of the following solutions:
- store some functions below 0x4000, once I'm certain those functions won't change in size and I can hardcode their locations.
- load part of the program to below 0x4000 using a BASIC loader, then load the rest at 0x8000.
- load part of the program in at 0x8000, disable the firmware, then load more in the newly freed up memory space (including after 0xA6FC). Except that without the firmware, I don't know how to load stuff.
- exomizer

Has anyone got any ideas or advice they can give me?


Thanks.

[ronaldo]

If's normal that you get that error when firmware variables get overwritten when loading your program. cpct_disableFirmware has no use here, as your program isn't running yet, and the Firmware routines are carrying the loading process.

An easy solution, at your progress stage, would be this one:

   
• Use 0x8000-0xBFFF as double buffer section
• For that, you'll need to move the stack to other place at the start of the program
• You can load your code at 0x200 and put the stack at 0x1FF, for instance (beware! ROM character functions must be loaded above 0x4000 or they will crash when Low ROM gets enabled for reading the characters).
• Our you may load your code at, say, 0x4000 and use 0x040 - 0x3FFF for stack and data.

To move the stack at the start of your code, you can use @lachlank's code

Code Select Expand


// in c code
extern void SetStack();


void main() {
   SetStack();
   ...
}


; in assembly (.s file).

.area _CODE


_SetStack::
pop hl ; preserve return address in main()
ld sp, #0x01FF ; whatever address you want stack to start at
jp (hl) ; return

[ronaldo]

Just for your reference, latest push to master branch of CPCtelera includes cpct_setStackLocation for changing the stack and a small demo on how to change it and have 0x8000-0xBFFF as double buffer area .

[ervin]

It works!
Thank you so much for that brilliant (and incredibly easy) solution!

Fantastic timing too.
I can't believe that after asking for help, I have a full, easy solution just a few hours later.


However, the latest cpctelera-master causes warnings about the following on compilation:
cpct_memcpy.s
memutils.s

The warnings are about duplicate declarations or something like that (similar to the sort of thing I had a month ago).
I've deleted the cpctelera folder, and I'm reinstalling, to see if it fixes it.

[EDIT] ... and, it is fixed.

[ervin]

Hmmm... seems that moving the stack below 0x4000 has some side-effects.

If I do this:

Code Select Expand

cpct_setStackLocation(NEW_STACK);
cpct_setVideoMode(0);
pscr=cpct_getScreenPtr(SCR_VMEM,0,0);
cpct_drawStringM0("HELLO",pscr,1,0);

and NEW_STACK is set above 0x4000, then I get "HELLO" on the screen.
But if NEW_STACK is set below 0x4000, I only get the first raster line of "HELLO" appearing.

(SCR_VMEM points to 0xC000).

[ervin]

Oops, double-post.

[mr_lou]

I have to say that I'm a bit surprised at how fast my game has grown in kb too.

So 0xa6fc is the limit because there are firmware functions there?
So that means a program can be 25kb big if loading from 0x4000?
My game is 23kb at the moment, and I'm about to add music that takes up more than 2kb. So I guess I "just" have to change my loading address to 0x200?
That way I should have a whole 41kb available for the whole binary program, right?

EDIT: Changing the Z80CODELOC entry in the cfg/build_config.mk file to 0x200 seems to be all that's required?

[ronaldo]

Hmmm... seems that moving the stack below 0x4000 has some side-effects.

Yes, it has 1 side effect: ROM character functions do not work because they use the stack and also enable Lower ROM for reading ROM characters. This means that when they push, values are stored in RAM, but when they pop, values are recovered from ROM, which means unexpected behaviour.

So, at present, when you are calling String and Char functions from CPCtelera, you need the stack to be above 0x4000. I'll add it to the documentation.

However, the latest cpctelera-master causes warnings about the following on compilation:
cpct_memcpy.s
memutils.s

The warnings come from old code inside the .lib. You don't need to reinstall: you just need to type this from inside cpctelera folder

Code Select Expand


make -C cpctelera/ cleanall
./setup.sh

That will ensure that all changes are applied. To the .lib.

I have to say that I'm a bit surprised at how fast my game has grown in kb too.
So 0xa6fc is the limit because there are firmware functions there?

Not functions, but variables. Firmware routines are stored in ROM, but they need to use variables (and jump tables), so they have to store them in RAM.

So, 0xA6F is not "the limit": is the limit for the firmware loading routine that you are using to load your binary into memory (run"game"). You can always create your own loader (as most games did) to get rid of firmware. Other alternatives are moving code or data, using that zone as dynamic memory, generating part of your data, etc.

EDIT: Changing the Z80CODELOC entry in the cfg/build_config.mk file to 0x200 seems to be all that's required?

Yes, changing that moves your the loading point of your code. Please, take into account code restrictions (as functions using Lower ROM must have their code above 0x4000)

[arnoldemu]

I have to say that I'm a bit surprised at how fast my game has grown in kb too.

So 0xa6fc is the limit because there are firmware functions there?
So that means a program can be 25kb big if loading from 0x4000?
My game is 23kb at the moment, and I'm about to add music that takes up more than 2kb. So I guess I "just" have to change my loading address to 0x200?
That way I should have a whole 41kb available for the whole binary program, right?

EDIT: Changing the Z80CODELOC entry in the cfg/build_config.mk file to 0x200 seems to be all that's required?

I don't mind taking a look to see where the memory has all gone.

[ervin]

Yes, it has 1 side effect: ROM character functions do not work because they use the stack and also enable Lower ROM for reading ROM characters. This means that when they push, values are stored in RAM, but when they pop, values are recovered from ROM, which means unexpected behaviour.

So, at present, when you are calling String and Char functions from CPCtelera, you need the stack to be above 0x4000. I'll add it to the documentation.
The warnings come from old code inside the .lib. You don't need to reinstall: you just need to type this from inside cpctelera folder

Code Select Expand


make -C cpctelera/ cleanall
./setup.sh

That will ensure that all changes are applied. To the .lib.

Excellent, thanks ronaldo.


If I relocate the stack to 0x8000, will it grow down to 0x7fff, 0x7ffe, 0x7ffd etc?
Is that a good place for it if I'm using 0x8000 and 0xC000 for my screens?
Or would it grow up from 0x8000, 0x8001, 8x8002 etc, thereby destroying my double buffer?

In general, how many bytes would be safe to leave reserved for the stack (i.e. to ensure that data doesn't touch the stack area)? Would 64 bytes be enough?
How much stack space does a function like cpct_memset_f64 need to operate correctly?

[mr_lou]

Yes, changing that moves your the loading point of your code. Please, take into account code restrictions (as functions using Lower ROM must have their code above 0x4000)

I'm a complete noob.
Can you give an example of a lower ROM function that I cannot use when loading my game at 0x200 instead of 0x4000?
Since my game still loads and plays fine, I'm assuming I'm not using anything "forbidden".

[arnoldemu]

I'm a complete noob.
Can you give an example of a lower ROM function that I cannot use when loading my game at 0x200 instead of 0x4000?
Since my game still loads and plays fine, I'm assuming I'm not using anything "forbidden".

As discussed, keep your stack between &4000-&c000 to play nice with the firmware.

Certain firmware functions require data to be within the central 32k (&4000-&c000).

kl add frame fly, kl new fast ticker, kl add ticker,kl init event, km exp buffer, txt set m table, sound queue.

They can all be called by code that is at any address because the code jumps through the jumpblocks and pages the rom in and then out again.
So it's probably a stack position problem if you see any problems.

[mr_lou]

As discussed, keep your stack between &4000-&c000 to play nice with the firmware.

Hm, well I'm assuming CPCtelera sets stack to 0xc000 - 6, judging from previous posts.
I'm also assuming that stack is set to 0xc000 - 6 regardless of my starting point being 0x4000 or 0x200.
Is that wrong?

Certain firmware functions require data to be within the central 32k (&4000-&c000).

I can see how this might be a problem in some cases, since data is usually the first in the binary produced by CPCtelera.
But since I'm not using firmware, I guess I don't have to worry? I'm calling cpct_disableFirmware().

kl add frame fly, kl new fast ticker, kl add ticker,kl init event, km exp buffer, txt set m table,>>> sound queue. <<<

Um.... another stoopid noob question for @ronaldo :
Does the Arkos music functions require that I enable firmware? I hope not.

They can all be called by code that is at any address because the code jumps through the jumpblocks and pages the rom in and then out again.
So it's probably a stack position problem if you see any problems.

mkay, but since I'm not touching stack-anything, I should be fine, right?
Could I mess up stack just by changing my start address to 0x200?

[ervin]

mkay, but since I'm not touching stack-anything, I should be fine, right?
Could I mess up stack just by changing my start address to 0x200?

I had set the stack to 0x0200, and everything worked fine, until I tried to use cpct_drawStringM0, which itself calls cpct_drawCharM0.

Firstly (though this is not related directly to moving the stack), we have this info in the docs for these 2 functions.
I didn't take heed of this warning initially, and caused lots of crashes. 

   
• Do not put this function's code below 0x4000 in memory.  In order to read characters from ROM, this function enables Lower ROM (which is located 0x0000-0x3FFF), so CPU would read code from ROM instead of RAM in first bank, effectively shadowing this piece of code.  This would lead to undefined results (typically program would hang or crash).

However, directly related to moving the stack is this, mentioned by ronaldo earlier in this thread:

Yes, it has 1 side effect: ROM character functions do not work because they use the stack and also enable Lower ROM for reading ROM characters. This means that when they push, values are stored in RAM, but when they pop, values are recovered from ROM, which means unexpected behaviour.
So, at present, when you are calling String and Char functions from CPCtelera, you need the stack to be above 0x4000.

I'm now experimenting with putting the stack at 0x8000, though I don't know how safe this is, considering I am using 0x8000 as a double-buffer.

[ronaldo]

Can you give an example of a lower ROM function that I cannot use when loading my game at 0x200 instead of 0x4000?

All CPCtelera functions that use Lower ROM have it reported in their documentation, as @ervin points out. Check, for instance, cpct_drawCharM1. At present, all character and string functions read characters from Lower ROM.

If I relocate the stack to 0x8000, will it grow down to 0x7fff, 0x7ffe, 0x7ffd etc?
Is that a good place for it if I'm using 0x8000 and 0xC000 for my screens?
Or would it grow up from 0x8000, 0x8001, 8x8002 etc, thereby destroying my double buffer?

Stack always grows backwards. When you put stack at 0x8000 you are telling the CPC that the top of the stack lies at 0x8000. Next push operation would write 0x7FFE,0x7FFF, making SP=0x7FFE, and a pop operation would read 0x8000-0x8001, making SP=0x8002.

Stack at 0x8000 is as safe as stack at 0x752A or 0xC000: stack has no problem if you know how to manage it. You have to left enough space for it and take care of not shadowing it with Lower or Upper ROM (that's why @arnoldemu suggest keeping the stack between 0x4000-0xC000).

As discussed, keep your stack between &4000-&c000 to play nice with the firmware.

They are not using the firmware. The first thing they do is disabling it with cpct_disableFirmware. The problem comes from using CPCtelera character drawing functions, which read from Lower ROM and may shadow the stack.

Hm, well I'm assuming CPCtelera sets stack to 0xc000 - 6, judging from previous posts.

CPCtelera sets the stack at 0xC000, same as any standard SDCC program using its standard crt0.s file.

The 0xC000-6 value comes from the example I posted before. In that example, the stack has grown 6 bytes from 0xC000, and is storing 6 bytes. What the example does is copying that 6 bytes to the new location (0xBFFA-0xC000 to 0x1FA-0x200), and setting the stack to the top of that 6 bytes copied (0x200-6 = 0x1FA).

I can see how this might be a problem in some cases, since data is usually the first in the binary produced by CPCtelera.

CPCtelera does no do anything regarding to your data or code. That is managed by SDCC. In fact, the crt0.s layout file sets the DATA area after the CODE area. However, you may be inserting your data as code.

Um.... another stoopid noob question for @ronaldo :
Does the Arkos music functions require that I enable firmware? I hope not.

No. Arkos tracker does not use Firmware. As we were discussing, all these details are reported in the documentation. You can check these things whenever in doubt.

mkay, but since I'm not touching stack-anything, I should be fine, right?
Could I mess up stack just by changing my start address to 0x200?

You can place your code wherever you wanted. There is nothing wrong with placing it at 0x200 or 0x100. You only have to take care of things like Lower ROM/Upper ROM issues (if you call Charater/String functions), not overwritting firmware values when they are being used, not overwritting the stack, not overwritting RST table (0x000-0x039) and not writing to video memory (unless you wanted to see your code as pixels ).

In fact, several CPCtelera examples place the code at 0x100 or 0x040.

I'm now experimenting with putting the stack at 0x8000, though I don't know how safe this is, considering I am using 0x8000 as a double-buffer.

Completely safe. Just take into account what the stack stores: local variables, function parameters and return addresses, and temporal values that do not fit into registers. Depending on how you program, your stack will grow more or less. Keep an eye on it.

[ervin]

Thanks once again for the great advice ronaldo!

[mr_lou]

I'm still somewhat confused.
So far I get that we can't place the stack below 0x4000.
But code is fine to place below 0x4000 - except if it's the code for the CPCtelera drawString methods.....
One thing that confuses me, is that my talk about placing my code at 0x200 spawns a lot of talk about stack. As if the stack is also automatically moved when I change placement of my code.
I'm using drawStringM1 alot, and it all seems to run fine. I'm not experiencing the problems with drawString methods despite my code being at 0x200.
Is this just luck? Or is it because the first 14-16kb of my code is really just data, resulting in the code for drawString coming after 0x4000?
Sorry for these noob questions.

[ronaldo]

@mr_lou: Don't worry if your questions spawn more talk. We need to talk a lot about things to understand each other .

cpct_drawCharM? functions need to be above 0x4000 no to crash (cpct_drawString do not, but they call cpct_drawCharM? functions). To understand this, imagine that cpct_drawCharM1 was at 0x3000 and gets called. The CPU is reading bytes from 0x3000,0x3001... and executing them. Lets say that the byte at 0x3020 is an OUT instruction that enables Lower ROM (required to read character definitions that are stored there). After that instruction, CPU reads byte 0x3021 to continue executing the function. However, when reading byte 0x3021 it does not come from RAM (where the function code is stored) but from ROM, because instruction at 0x3020 enabled lower ROM, shadowing first RAM bank 0x0000-0x4000: now all reads at that bank come from ROM instead of RAM. This means that the next instruction to be executed (0x3021) will yield unexpected behaviour, as it is not our code but a pseudo-random byte from ROM.

The same happens to the stack, as cpct_drawCharM? functions use it doing push and pop operations while lower ROM is enabled. If the stack is below 0x4000, push operations are done in RAM, but pop operations (which are reads) access ROM.

If you want to check where your cpct_drawCharM functions are located, thats easy. Check your obj/ folder. Inside, you will find a yourproject.map file. That file contains all the symbols and their RAM locations. Check for cpct_drawM function location. Say that your file is myproj.map:

Code Select Expand


cat obj/myproj.map | grep cpct_drawChar

That will do

[mr_lou]

If you want to check where your cpct_drawCharM functions are located, thats easy. Check your obj/ folder. Inside, you will find a yourproject.map file. That file contains all the symbols and their RAM locations. Check for cpct_drawM function location. Say that your file is myproj.map:

Code Select Expand


cat obj/myproj.map | grep cpct_drawChar

That will do

Results:

00005871  _cpct_drawCharM1_f                 cpct_strings
0000587E  cpct_drawCharM1_f_asm              cpct_strings

So I guess that's why I'm having no problems.
All my sprites data push down the code. 

As long as you don't get tired of answering noob questions, then all is well. 

[pacomix]

Hi! My two cents here.

By having a custom crtc.s file where you set the stack to 0x8000 and disable the firmware at start time you can have the starting address of your program at 0x100 or so (I don't remember exactly where).
You will need to have the double buffering in the 0x8000 && 0xC000 addresses. This way you can have almost 32KB in a row to divide it into code & data which makes everything easier to manage. You could even set the stack to an address of your screen data if you don't fully use it but then you will have to be careful about the amount of data to stack.

True is that you will need to have code that barely changes in the lower memory addresses. That's what I was doing with CPC Bros. I had the library functions loaded as low as possible and then the main code immediately after.

This includes Sprite drawing routines, character drawing function, tiny disk loading function and Wyztracker. You really don't need more than this to make your own game. Perhaps some sprite animation functions or basic tile functions but no more. That depends then a lot on the videogame type you develop.
This way you don't have to fight against the f***g firmware that eats unnecessary memory.

[ervin]

Hi! My two cents here.

By having a custom crtc.s file where you set the stack to 0x8000 and disable the firmware at start time you can have the starting address of your program at 0x100 or so (I don't remember exactly where).
You will need to have the double buffering in the 0x8000 && 0xC000 addresses. This way you can have almost 32KB in a row to divide it into code & data which makes everything easier to manage. You could even set the stack to an address of your screen data if you don't fully use it but then you will have to be careful about the amount of data to stack.

True is that you will need to have code that barely changes in the lower memory addresses. That's what I was doing with CPC Bros. I had the library functions loaded as low as possible and then the main code immediately after.

This includes Sprite drawing routines, character drawing function, tiny disk loading function and Wyztracker. You really don't need more than this to make your own game. Perhaps some sprite animation functions or basic tile functions but no more. That depends then a lot on the videogame type you develop.
This way you don't have to fight against the f***g firmware that eats unnecessary memory.

That's great advice - thankyou.

[ronaldo]

@pacomix: Thank you very much for your much more than two cents, man .

The good news is that you actually don't have to deal with crt0.s to get the same result. With the latest version of CPCtelera you can do the same, as we suggested, changing the stack and loading your code even as low as 0x0040 . Propperly distributing code functions is something we always have to keep in mind, and it's always nice to have an easy interface like cpct_setStackLocation and the prefix declarator __at(XXXX) from SDCC.

How's your CPC Bros going? I'm quite anxious to have it in my hands and invest hours playing with it . It's a really nice job you are doing there! Please, keep it up!

[ervin]

@mr_lou...

I'm wondering, are you using SDCC's time.h in your program?

Code Select Expand

#include <time.h>

I just discovered that it bloats the size of my compiled binary by well over 4K!!!
Instead of using the default SDCC time.h, I'm now using a trimmed down version in the same folder as the rest of my code.

I've got trimmed versions of time.h and time.c, which I have renamed to time_small.h and time_small.c.
I only need the time() function in order to provide a random number seed.

In main.c, I have removed the normal time.h include, and replaced it with:

Code Select Expand

#include "time_small.h"

My current binary has shrunk from 9,965 bytes down to 5,515 bytes, and it still works correctly! 
This little optimisation has really taken a lot of pressure off of memory requirements! It will make a *HUGE* difference to my program, as right now I'm trying to plan where to put all my sprite and tile data.

If you're interested, I've attached my trimmed time_small files.

[ronaldo]

If you need sources of entropy, like time(), I recommend you to construct your own sources. For instance, these are a few examples:

   
• Count the number of iterations (call this NI) of your main loop during menu screen until the user starts the game.
• Add up the values of NI at each time the user presses a key, and divide the sum by 2 or 4.
• Use the x or y location of a given character at the momment of producing next random value.

These are some ideas, but you may find much more. I think these entropy sources are neater than time(). Moreover, take into account that some implementations of time might rely on firmware or on interrupts, as CPC hasn't got any user hardware clock.

[FloppySoftware]

   
• Count the number of iterations (call this NI) of your main loop during menu screen until the user starts the game.

Very good advice, I use it a lot and works very nice.