Why on earth, the firmware routine CAS_OUT_CHAR is soooo slow?

[ikonsgr]

Hi everyone,


I took a look to this code, for a project i'm working on (it's a cheap and easy to use ,MCU PIC based, universal serial port for amstrad cpc ), and i wanted to test the writing speed  to disk, using this "byte by byte" method. So,i used a simplified version of the code, where a 16KB dummy file of '1's is created,and the "core" loop of .next_byte, was minimized to:


.next_byte
LD A,1
PUSH BC
CALL CAS_OUT_CHAR
POP BC
DEC BC
LD A,B
OR C
JR NZ,next_byte


Τhe actual time needed for the 16kb dump file to be saved to disk, was ~17sec,that is less than 1kb/sec. As ~99% of time is consumed to the above loop, it's easy to caclulate that each turn (where one byte is written), needed ~1msec =1000us. That's ~1000machine cycles for Z80, and since all the above instructions (LD,PUSH,POP etc) needed a few dozens of machine cycles/loop, it's easy to conclude that, the CALL to firmware routine "cas_out_char", is the main "culprit", needing more than 950us to be executed just once!!
So my question is, why writing of only one byte to a file to disk, is sooooo slow, this is what amstrad cpc hardware can really give?
Or the firmware function is poorly optimized and maybe there is some alternative function to use, in order to achieve better speed?

[Docent]

Hi everyone,


I took a look to this code, for a project i'm working on (it's a cheap and easy to use ,MCU PIC based, universal serial port for amstrad cpc ), and i wanted to test the writing speed  to disk, using this "byte by byte" method. So,i used a simplified version of the code, where a 16KB dummy file of '1's is created,and the "core" loop of .next_byte, was minimized to:


.next_byte
LD A,1
PUSH BC
CALL CAS_OUT_CHAR
POP BC
DEC BC
LD A,B
OR C
JR NZ,next_byte


Τhe actual time needed for the 16kb dump file to be saved to disk, was ~17sec,that is less than 1kb/sec. As ~99% of time is consumed to the above loop, it's easy to caclulate that each turn (where one byte is written), needed ~1msec =1000us. That's ~1000machine cycles for Z80, and since all the above instructions (LD,PUSH,POP etc) needed a few dozens of machine cycles/loop, it's easy to conclude that, the CALL to firmware routine "cas_out_char", is the main "culprit", needing more than 950us to be executed just once!!
So my question is, why writing of only one byte to a file to disk, is sooooo slow, this is what amstrad cpc hardware can really give?
Or the firmware function is poorly optimized and maybe there is some alternative function to use, in order to achieve better speed?

CAS OUT CHAR is a buffered write - each byte is stored in the 2kb buffer you provided when calling CAS OUT OPEN. When the buffer is filled,  the save routine for this block is called. In your case, the os 8 times accesses the disk: turns on drive motor, reads the directory from disk (moves head to 0 track, reads data), searches for free blocks, updates file system tables, transfers data from buffer to disk, turns off drive motor etc. It cant be fast.
To save a complete file at once use CAS OUT DIRECT - it will be faster.

[ikonsgr]

Well,i've tried to increase the buffer size up to 8Kb ( .two_k_buffer defs 8192) , so, for a 16kb file, the sequence you described will happen only 2 times but still it needs exactly the same amount of time....     But, i think there is an "internal" 2K buffer used for disk I/O at specific address defined by firmware, the buffer you declare inside the code, is actually a different buffer placed somewhere in ram,and when this buffer is full,it's been copied to "internal" 2k disk buffer, maybe that can explain why there was no increase in speed using the much larger buffer of 8Kb.

Now, using a: save"file.scr",b,&c000,&4000  from basic, actual speed is about 2kb/sec. This most probable uses CAS_OUT_DIRECT you proposed, where no buffer is used and bytes are written directly from ram to disk, but unfortunately CAS_OUT_DIRECT has two major drawbacks:

1) It requires for the entire file to be placed in ram before writing to disk, so large files ~40-42kb (which was very common size for many cpc games), can't use this metohd, as they will take up all the available ram
2) It creates the header by itself, so any "weird" file without header or with non standard headers, can't be written either.On the other side, Byte by byte writing method, needs only a 2Kb buffer, and it creates an exact clone of the file, from first to last byte, so any file type (BAS,BIN,ASCII,custom,Headerless), of any size (even larger than 42kb), can be written to disk!

[andycadley]

Ultimately the disk system has to write out sectors, which involves locating suitable places to write to (by reading the catalog) and then writing out a block of data and then updating the catalog entry too. All of which takes time (much of it just spinning the disk). You can obviously do better if you take over some of the process yourself but a generalised solution (and particularly one originally designed around cassette like the firmware) is always going to pay a hefty price.

[ikonsgr]

Well, writing down a new and greatly improved "cas_out_char" routine, would be a bit difficult (if not impossible ), so i guess this is "as good as it gets".
Using track/sector writing is ofcourse much faster, but this goes better for copying hole dsk images something that is already done easily using cpc diskxp or usb floppy emulators .
I wanted to develop something new,like a direct FILE transfer method (using the serial port i designed) between amstrad<->amstrad or pc<->amstrad.
Anyway, there might be an alternative using "cas out direct" (which is about twice as fast) for smaller files (~35kb or smaller) with header, and use the slow "cas_out_car" routine for bigger and/or headerless files.

[Docent]

Well,i've tried to increase the buffer size up to 8Kb (.two_k_buffer defs 8192) so, for a 16kb file, the sequence you described will happen only 2 times but still it needs exactly the same amount of time....      But, i think there is an "internal" 2K buffer used for disk I/O at specific address defined by firmware, the buffer you declare inside the code, is actually a different buffer placed somewhere in ram,and when this buffer is full,it's been copied to "internal" 2k disk buffer, maybe that can explain why there was no increase in speed using the much larger buffer of 8Kb.

No, there is no internal buffer and the size of your buffer doesnt count - the pointer to it you provide in a call to CAS OUT OPEN is treated as a pointer to a 2kb buffer - its size is fixed in the ROM.

Now, using a: save"file.scr",b,&c000,&4000  from basic, actual speed is about 2kb/sec. This most probable uses CAS_OUT_DIRECT you proposed, where no buffer is used and bytes are written directly from ram to disk, but unfortunately CAS_OUT_DIRECT has two major drawbacks:

1) It requires for the entire file to be placed in ram before writing to disk, so large files ~40-42kb (which was very common size for many cpc games), can't use this metohd, as they will take up all the available ram

You can solve this problem by putting the saving code at the beginning of the stack area at #bf00 - there is 256 bytes available there and a pretty good chance that your code wont be overwritten if you fit your code in 100-200 bytes.  You can also put your saving code in screen area that is not used: at #c7d0-c7ff,#cfd0-cfff,#d7d0-d7ff,#dfd0-dfff,#e7d0-e7ff,#efd0-efff,#f7d0-f7ff,,#ffd0-ffff.

2) It creates the header by itself, so any "weird" file without header or with non standard headers, can't be written either. On the other side, Byte by byte writing method, needs only a 2Kb buffer, and it creates an exact clone of the file, from first to last byte, so any file type (BAS,BIN,ASCII,custom,Headerless), of any size (even larger than 42kb), can be written to disk!

You can manipulate the content of the header - CAS OUT OPEN returns in HL register a pointer to the header it created. You can set a number of fields in it like file type, length etc that you can receive from host. For example, on host you call CAS IN OPEN, transmit the header you got from it in HL, on target you call CAS OUT OPEN and modify the header to contain data you received from host and then transmit the content of the file, depending on its type and size (text files and bigger than 42 kb via CAS OUT CHAR other with CAS OUT DIRECT).
All files on disk (maybe except text files saved with CAS OUT CHAR) should have a header.
On stock 6128 the longest file you can load with CAS IN DIRECT without trashing system/basic variables is #a6bc ie 42584 bytes. You should be able to handle such file using trick from 1).

[ikonsgr]

Thank you very much for your reply and suggestions Docent, it really raised my mood about the hole matter! 
I really didn't think that i could use the hole screen memory for the assembly code, since for the pc->amstrad file transfers, everything will be controlled from a pc program, amstrad would only run a "slave" program that will execute commands from the pc, so there would be no user input and no need for showing anything on screen during the procedure, so no matters if you get any garbage on screen! :-)
About the second point you mention, according to firmware guide CAS OUT OPEN , doesn't create any header at all. You can also check this code where it clearly states that after CAS OUT OPEN routine is executed no header is automatically created:
;; open output file
ld b,end_filename-filename
ld hl,filename
ld de,two_k_buffer
call cas_out_open
;; at this point the file will be opened for output.
;; A header is *not* created or written to the output ;;
The automatic header creation is made by the CAS OUT DIRECT routine, where you need to put a couple of extra info (file type to Accumulator and execution address to BC registers) in order for the routin to automatically create the header for the file. I've tested many dozens of files so far, and found out that, at least with games, almost all files have headers, only ascii files created by application (like source code files from MAXAM) or some binary application files seemed to be headerless (e.g. checksum error). Unfortunately for such files, the only way to transfer them correctly is using the slow CAS OUT CHAR routine, as this doesn't create automatically any header, but as i said, headerless files seemed that are rarely used in games. Also, from 1500+ dsk game images checked, ~95% where extracted with normal files, and only the rest seem to use "weird" saving methods of direct sector writing to disk without using files (these usually don't give anything with "cat" and were executed using the well known run"disc" command)
Back to the "Drawing board" then, i hope soon will have some good news!;-)
Thanks again for your help!

[Docent]

Thank you very much for your reply and suggestions Docent, it really raised my mood about the hole matter! 
I really didn't think that i could use the hole screen memory for the assembly code, since for the pc->amstrad file transfers, everything will be controlled from a pc program, amstrad would only run a "slave" program that will execute commands from the pc, so there would be no user input and no need for showing anything on screen during the procedure, so no matters if you get any garbage on screen! :-)
About the second point you mention, according to firmware guide CAS OUT OPEN , doesn't create any header at all.

Do not believe in comments in some source Always verify such claims with the firmware guide. Here's the interesting part:
http://www.cpcwiki.eu/imgs/5/5d/S968se08.pdf - see paragraph 8.8:
"Before a file can be written to it must be opened (by calling CAS OUT OPEN). This sets up the filename (see 8. 10 below) and the rest of the header that will be written in each file block."

You can also check this code where it clearly states that after CAS OUT OPEN routine is executed no header is automatically created:
;; open output file
ld b,end_filename-filename
ld hl,filename
ld de,two_k_buffer
call cas_out_open
;; at this point the file will be opened for output.
;; A header is *not* created or written to the output ;;
The automatic header creation is made by the CAS OUT DIRECT routine, where you need to put a couple of extra info (file type to Accumulator and execution address to BC registers) in order for the routin to automatically create the header for the file.

This is only partially true - the header is not written to disk/casette by CAS OUT OPEN, but it is created by this function in the system area with some fields initialized. Other header fields can be initialized by the caller before calling CAS OUT DIRECT. Finally, some fields are set by CAS OUT DIRECT with provided parameters.
The easy way to find it out: put inc (hl) after call to CAS OUT OPEN and try to find the saved file on disk

I've tested many dozens of files so far, and found out that, at least with games, almost all files have headers, only ascii files created by application (like source code files from MAXAM) or some binary application files seemed to be headerless (e.g. checksum error). Unfortunately for such files, the only way to transfer them correctly is using the slow CAS OUT CHAR routine, as this doesn't create automatically any header, but as i said, headerless files seemed that are rarely used in games.

As I wrote before - only ascii or data files, saved by programs via CAS OUT CHAR don't have the header. For such files when opened, a fake header is constructed by amsdos. You can detect such files by testing  File type field of the header - if it is equal to unprotected ASCII type of version 1, then you can assume its a headerless file and copy it with CAS IN/OUT CHAR, otherwise use CAS IN/OUT DIRECT.

Also, from 1500+ dsk game images checked, ~95% where extracted with normal files, and only the rest seem to use "weird" saving methods of direct sector writing to disk without using files (these usually don't give anything with "cat" and were executed using the well known run"disc" command)

Such disks should always be duplicated by copying all sectors from host to target but the real problem is reliable detection of them... It probably would require reading and parsing dir entries directly from a disk.
There are also games that can be started using |cpm command - these can be detected by checking for sector #41 on track 0. If such sector exists and is not empty, its a big chance that it also needs to be duplicated sector by sector.
But to handle such games you'll need to implement reading/writing sectors - perhaps its better to just stick only to copying files.

Good luck with your project!

[arnoldemu]

I believe the reason CAS IN CHAR is so slow is that the disc motor is turned on/off frequently during reading.

[ikonsgr]

This is only partially true - the header is not written to disk/casette by CAS OUT OPEN, but it is created by this function in the system area with some fields initialized. Other header fields can be initialized by the caller before calling CAS OUT DIRECT. Finally, some fields are set by CAS OUT DIRECT with provided parameters.
The easy way to find it out: put inc (hl) after call to CAS OUT OPEN and try to find the saved file on disk

OK, but what really matters  is that ,using CAST OUT CHAR Instead of CAS OUT DIRECT,no header is actually written to the file.

Such disks should always be duplicated by copying all sectors from host to target but the real problem is reliable detection of them... It probably would require reading and parsing dir entries directly from a disk.
There are also games that can be started using |cpm command - these can be detected by checking for sector #41 on track 0. If such sector exists and is not empty, its a big chance that it also needs to be duplicated sector by sector.
But to handle such games you'll need to implement reading/writing sectors - perhaps its better to just stick only to copying files.
Good luck with your project!

Yes,but fortunately, the vast majority of games (~95%) were using normal files which can be transfered. Besides, the use of hole dsk images is already done pretty well and easy, either directly with hxc or gotek, or by transfer the dsk image to a real disk using cpcdiskxp. But DIRECT file transfer, between cpc<->pc , as far as i know is not.
If you are involved with amstrad more than a casual gamer (like i do ), it would be very nice and cool if you could transfer directly, for example, a BAS file (with a listing you just wrote on your amstrad) to your pc for backup reasons or/and testing with emulator, or if you have old 3" disks and want to use them (and have the exact retro feeling of the era ) you could directly transfer a game of 50-60-70kb (a typical  size of amstrad games) from your pc to your amstrad 3" disk drive, without needing to write first a hole 178kb dsk image to 3.5" disk with cpcdiskxp, or copying the dsk image to a usb stick for use with gotek/hxc  (which would also require to use the file manager in order to place the new dsk image to a new slot) and then copy to 3" disks...
Moreover, a "universal" serial interface (where you can hook up a serial cable,a bluetooth module or a wifi esp8266 module) like the one i design, could offer unlimited potential to what you can do with your amstrad! From Chat with your friends using amstrads, up to creating LAN/VLAN rts/rpg games where 2 or more players using different amstrad computers, play the same game....

[ikonsgr]

With a bit of patience and a LOT of percistence , i've finally managed to "customize" the "byte by byte" method to become MUCH faster:
http://www.cpcwiki.eu/forum/programming/how-to-readwrite-a-file-'byte-by-byte'-using-firmware-but-much-faster!/msg162099/#msg162099