News:

Printed Amstrad Addict magazine announced, check it out here!

Main Menu
avatar_cpcitor

Emulator feature: log CPC state changes, for postmortem debug.

Started by cpcitor, 11:46, 29 December 20

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

cpcitor

Hi!

Situation

Software in the middle of being written has bugs. Each of us who has written an ambitious prod has had a case when a nearly correct program has a bug, and that bug is hard to diagnose.
The worst that can happen is a program that works well for hours, and crash sometimes.

One of my program is in this state.

One day, I already diagnosed a similar problem. I figured out the problem was with an incorrect IX value. I logged IX by changing the source code of the emulator to just dump a text log of which instruction changed IX (which address, opcode, old value, new value). That was okay because the bug was quick to trigger. I could write simplest log, look at it, figure out I needed more information, add some more logging, and iterate until I found the location of the bug.

I have an idea: a generalization of that process. An emulator feature that would be very useful to diagnose a crash (or any other specific event, actually) in a long-running program.

Idea

The principle is that the emulator records state changes in some log format.

Here is just a sketch of what a log could mean (could be formatted totally differently):


On time 0x000123456, PC=0x01b9,  opcode xor b      changed register       A,      old value 0xf0, new value 0xcd
On time 0x000123457, PC=0x01ba,  opcode ld (hl),a  changed memory address 0xC234, old value 0xab, new value 0xcd


The "interesting" moment (e.g. moment the emulated program crashes the CPC, or triggers any event) could be automatically or even simply manually decided.

Then, the log file can be analyzed. Simplest would be a text-based readable log, that can be analyzed using any external tool.

Very powerful possibility: writing old and new values can allow backward playing of CPC state. When a bug occurs you see a consequence of something happening before. With such log, and a UI in the emulator to step backward, you can go back in time to see the exact upstream situation that yielded this consequence. Even "search backward in time for when register b got value 0x12".

But even a simple log file without a UI in emulator can be very useful.

Already existing ?

I looked at caprice32, cpcec, (also retrovirtualmachine and WinAPE description), nothing seems to do that.

Question

Does anyone know about an existing emulator that does this kind of CPC state log?
Had a CPC since 1985, currently software dev professional, including embedded systems.

I made in 2013 the first CPC cross-dev environment that auto-installs C compiler and tools: cpc-dev-tool-chain: a portable toolchain for C/ASM development targetting CPC, later forked into CPCTelera.

cpcitor

Floooh's emulator has a history of executed instructions available.

It's far from sufficient, just a hint at what can exist.

You can see it on https://floooh.github.io/tiny8bit/cpc-ui.html?type=cpc464 in menu "debug" choose "execution history".

See below, also:

[attach=1,msg196049]

Source code on https://github.com/floooh/chips-test and https://github.com/floooh/chips
Had a CPC since 1985, currently software dev professional, including embedded systems.

I made in 2013 the first CPC cross-dev environment that auto-installs C compiler and tools: cpc-dev-tool-chain: a portable toolchain for C/ASM development targetting CPC, later forked into CPCTelera.

Targhan

This is probably not exactly what you want, but since you're a developer, maybe you could use some C++ Z80 emulator out there, inject some data and check the states by yourself.

For AT2 test units, I embedded a Z80 emulator, run my compiled played and checked, frame by frame, if the output to the PSG matches the C++ player. Very useful and totally automatized!
You can check the emulator link from AT2 webpage. Including it to my own code was quite easy. Of course, you would have preferred a build-in solution, but...
Maybe you can create a simple monitor around that.
Targhan/Arkos

Arkos Tracker 2.0.1 now released! - Follow the news on Twitter!
Disark - A cross-platform Z80 disassembler/source converter
FDC Tool 1.1 - Read Amsdos files without the system

Imperial Mahjong
Orion Prime

cpcitor

Quote from: Targhan on 19:37, 29 December 20
This is probably not exactly what you want, but since you're a developer, maybe you could use some C++ Z80 emulator out there, inject some data and check the states by yourself.

For AT2 test units, I embedded a Z80 emulator, run my compiled played and checked, frame by frame, if the output to the PSG matches the C++ player. Very useful and totally automatized!
You can check the emulator link from AT2 webpage. Including it to my own code was quite easy. Of course, you would have preferred a build-in solution, but...
Maybe you can create a simple monitor around that.

Thanks Targhan for the idea.

"You can check the emulator link from AT2 webpage." -> Can't find it on http://www.julien-nevo.com/arkostracker/ and, none of the links in the menu seems to be an obvious candidate.

What you do fits your need very well, as AT2 players are standalone pure Z80 + AY 8912 based. You're right in testing in the condition closest to the "real thing".

In my case the "real thing" loads from disk, writes to screen, is interactive, so we're back to adjusting an existing emulator.

That said, I've had a look at Floooh's emulator design. It's very flexible and could be used for instrumentation. For example, unattended testing without the need of an actual graphical display during the test. For years, cpc-dev-tool-chain has included automated non-regression tests with caprice32, see for example https://github.com/cpcitor/cpc-dev-tool-chain/blob/master/tests/multiplication_benchmark_02_fast_constant_time_ab_aplusb_aminusb but it's a little hackish and the emulator actually assumes a graphical session, opens a window during the tests, can interfere with my keystrokes. Floooh's emulator would avoid all this.

Floooh's emulator design would lend itself very well to things like running any conversion program on the CPC, like converting ASCII text of a BASIC program into a binary actually saved by the firmware, convert any binary file to an audio file actually generated by the ROM's cassette firmware and whatnot, or even interaction with a CPC in a text console. Who'd want to interact with a CPC through a text-only SSH session?  ;D

Anyway, my interactive program test case needs a complete emulator, which would be useful to other people and cases. Additional non-regression tests (like a monkey-typing torture test for programs) would be fine with Floooh's emulator.
Had a CPC since 1985, currently software dev professional, including embedded systems.

I made in 2013 the first CPC cross-dev environment that auto-installs C compiler and tools: cpc-dev-tool-chain: a portable toolchain for C/ASM development targetting CPC, later forked into CPCTelera.

Targhan

Quote from: cpcitor on 20:55, 29 December 20"You can check the emulator link from AT2 webpage."
The credits page: http://www.julien-nevo.com/arkostracker/index.php/credits/

Well, the whole idea is "test unit". In AT2 tests, I don't test the PSG, but only what is SENT to the PSG.
Your bug is probably coming from the Z80, and more exactly, your code! As long as you feed the OUTs with the right values, everything is fine. So in my opinion, you must make sure that all the OUTs are send to the right chip with the right value. That's all that matters.

As for the loading, you could inject the data into the Z80 memory.
Targhan/Arkos

Arkos Tracker 2.0.1 now released! - Follow the news on Twitter!
Disark - A cross-platform Z80 disassembler/source converter
FDC Tool 1.1 - Read Amsdos files without the system

Imperial Mahjong
Orion Prime

mahlemiut

MAME has a trace command in its debugger - it records executed opcodes to a file, and it is possible to add register values or other data to the output.
- Barry Rodewald

norecess464

What @Targhan said. From what I experienced, for example cpcec from @cngsoft is a pretty hackable emulator and it would be relatively easy to implement such a log.

But first, and I may be potentially off topic here (sorry !), don't forget WinAPE emulator features memory breakpoints, cf. trigger when memory at specific location changes or has a predefined value.
My personal website: https://norecess.cpcscene.net
My current project is Sonic GX, a remake of Sonic the Hedgehog for the awesome Amstrad GX-4000 game console!

cpcitor

Quote from: norecess on 22:24, 29 December 20
What @Targhan said. From what I experienced, for example cpcec from @cngsoft is a pretty hackable emulator and it would be relatively easy to implement such a log.

Yes, cpcec is the one I favor: open-source, lightweight, faithful emulation, fast, advanced debugging features. (It would be even better if the debugger layout would benefit from the bigger definition of modern screens.)

Quote from: norecess on 22:24, 29 December 20
But first, and I may be potentially off topic here (sorry !), don't forget WinAPE emulator features memory breakpoints, cf. trigger when memory at specific location changes or has a predefined value.

Thanks @norecess for your suggestion.

The prod runs with firmware enabled, AT2 playing a tune on a interrupt handler installed by politely asking the firmware for 50Hz frequency. The handler saves regular and alternate registers then restores them. The player and tune are stored in central 32k to be safe from ROM switching. The game runs like a charm for hours, play, return to menu, return to game, switches songs, plays sound effects, huge area of screen moving, it works.

The fact is, the bug is rare. The observed effect of the bug is to reset the CPC. Observed a few times on cpcec, once on a real CPC.

For a start, I could put a traditional breakpoint at PC=0 just in case. It would still happen after the fact. And given how rare the bug is, I would prefer catch all interesting information.

Would a "memory breakpoint" be relevant in such a situation? What parameters would you suggest?

Thanks!
Had a CPC since 1985, currently software dev professional, including embedded systems.

I made in 2013 the first CPC cross-dev environment that auto-installs C compiler and tools: cpc-dev-tool-chain: a portable toolchain for C/ASM development targetting CPC, later forked into CPCTelera.

norecess464

QuoteYes, cpcec is the one I favor: open-source, lightweight, faithful emulation, fast, advanced debugging features. (It would be even better if the debugger layout would benefit from the bigger definition of modern screens.)

I made a custom version of cpcec, Linux-only, not really "advertised" yet but available here -- please note I don't provide any support for it. The addition provides a more usable debugger (no worries, @cngsoft is aware) and it's now my primary emulator to develop SonicGX.

My personal website: https://norecess.cpcscene.net
My current project is Sonic GX, a remake of Sonic the Hedgehog for the awesome Amstrad GX-4000 game console!

norecess464

QuoteThe handler saves regular and alternate registers then restores them
Double-check you are not using IX / IY. I may be wrong, but I think the firmware makes also use of those registers.
Also, are you making "safe" use of the stack at any time ? If not (like using SP to clear a buffer, or read data from somewhere etc), make sure to use DI/EI since Z80 interrupts could run firmware code at any time.

QuoteWould a "memory breakpoint" be relevant in such a situation? What parameters would you suggest?
I don't know if a memory breakpoint may be relevant for your situation. But this may give you ideas (eventually), like building a temporary array that you could continuously check with the debugger ?..
Honestly, those bugs are hard to chase.
If really you are blocked, try removing some block of code and see what happens -- maybe by proceeding like this you would be able to isolate the issue.
Or re-read your implementation, maybe something is writing data in memory at unexpected location..
My personal website: https://norecess.cpcscene.net
My current project is Sonic GX, a remake of Sonic the Hedgehog for the awesome Amstrad GX-4000 game console!

cpcitor

Quote from: norecess on 23:38, 29 December 20
Double-check you are not using IX / IY. I may be wrong, but I think the firmware makes also use of those registers.

Hmm, since you ask.

AFAIK the firmware is mostly written in 8080 code, and does not use IX or IY. I've read that firmware only needs to save AF' and B' (or C'). Saving all registers anyway is not costly, so I do it.

AT2 uses IX and IY and both register sets.

Because AT2 uses both register sets and swap them at places, I can't let the interrupt routine be interrupted, as is normal firmware practice. We would never now which of the register sets was active at interrupt time.
Interrupts can wait for a while, so I disable them for the whole handler.

So, I save all regular registers, including IX and IY. There's no alternate IX and IY so practically all registers are saved.

Here's the interrupt setup:

Quote
        ;; Read SOFT968
        ;; section 11.
        ;; section 12.
        ;; appendix 10.
        ;; Thanks Kevin Thacker for the scan+OCR job!

_hook_interrupt::

        ; Ask firmware to call at each video frame "play_one_music_frame".
        ld hl, #frame_flyback_event_block
        ld b, #0x81
        ld c, #0
        ld de, #on_frame_flyback
        ;; This routine enables interrupts.
        call FIRMWARE_KL_NEW_FRAME_FLY

        ret

on_frame_flyback::

        ; Arkos Tracker routine uses alternate Z80 register set,
        ; that the firmware expects unchanged.  So save the full Z80 state.

        ; The firmware documentation does not say so, but interrupts
        ; are enabled here in interrupt handler.
        ; This is done so that an interrupt handler can itself be interrupted.

        ; So far, so good, but this assumes that the code is ready to be interrupted.
        ; at any time, which implies not using the alternate register set.

        ; But Arkos Tracker playing routine uses both register sets!
        ; So, we are going to not only save both Z80 register sets,
        ; but also disable interruptions until we put them back.

        di

        push af
        push bc
        push de
        push hl
        push ix
        push iy

        exx
        ex af, af'

        push af
        push bc
        push de
        push hl

        call play_one_music_frame

        pop hl
        pop de
        pop bc
        pop af

        ex af,af'
        exx

        ld a, #1
        ld (interrupt_occured),a

        pop iy
        pop ix
        pop hl
        pop de
        pop bc
        pop af

        ei

        ret

It's ret not reti because it's a block called by firmware, not the Z80-level

Quote from: norecess on 23:38, 29 December 20
I don't know if a memory breakpoint may be relevant for your situation. But this may be give you ideas, like building a temporary array that you could continuously check with the debugger ?..

I don't understand the idea of the temporary array.

Quote from: norecess on 23:38, 29 December 20
Honestly, those bugs are hard to chase.
If really you are blocked, try removing some block of code and see what happens -- maybe by proceeding like this you would be able to isolate the issue.

Will continue investigating. Thank you for your attention.
Had a CPC since 1985, currently software dev professional, including embedded systems.

I made in 2013 the first CPC cross-dev environment that auto-installs C compiler and tools: cpc-dev-tool-chain: a portable toolchain for C/ASM development targetting CPC, later forked into CPCTelera.

Targhan

One idea would be to build your game without one element. For example, don't play the music. Then test it again and again. If it works fine then you've found the culprit ;).
Targhan/Arkos

Arkos Tracker 2.0.1 now released! - Follow the news on Twitter!
Disark - A cross-platform Z80 disassembler/source converter
FDC Tool 1.1 - Read Amsdos files without the system

Imperial Mahjong
Orion Prime

Targhan

Out of curiosity, why do you use system interruption when you could ditch the system, be faster and safer?
Targhan/Arkos

Arkos Tracker 2.0.1 now released! - Follow the news on Twitter!
Disark - A cross-platform Z80 disassembler/source converter
FDC Tool 1.1 - Read Amsdos files without the system

Imperial Mahjong
Orion Prime

cpcitor

Quote from: norecess on 23:21, 29 December 20
I made a custom version of cpcec, Linux-only, not really "advertised" yet but available here -- please note I don't provide any support for it. The addition provides a more usable debugger (no worries, @cngsoft is aware) and it's now my primary emulator to develop SonicGX.



Wow, a ncurses-based debugger is just an excellent idea!

I just ran it, will most certainly use it.

(Had a look at source code. Unusual practice of putting implementation in .h, works here because only one .c file includes the .h.)

Thanks for sharing! Big benefits ahead!
Had a CPC since 1985, currently software dev professional, including embedded systems.

I made in 2013 the first CPC cross-dev environment that auto-installs C compiler and tools: cpc-dev-tool-chain: a portable toolchain for C/ASM development targetting CPC, later forked into CPCTelera.

cpcitor

Quote from: mahlemiut on 22:05, 29 December 20
MAME has a trace command in its debugger - it records executed opcodes to a file, and it is possible to add register values or other data to the output.

Indeed https://docs.mamedev.org/debugger/execution.html#debugger-command-trace

Mame is really on the heavy side of things. No simple way to compile and run only one machine, plus compilation errors on things unrelated to CPC.

Will probably stay with lightweight solutions.
Had a CPC since 1985, currently software dev professional, including embedded systems.

I made in 2013 the first CPC cross-dev environment that auto-installs C compiler and tools: cpc-dev-tool-chain: a portable toolchain for C/ASM development targetting CPC, later forked into CPCTelera.

mahlemiut

You can do a single driver build of MAME:
make SOURCES=src/mame/drivers/amstrad.cpp
will produce a build with just the CPC/CPC+ drivers.
- Barry Rodewald

cpcitor

Quote from: mahlemiut on 12:20, 30 December 20
You can do a single driver build of MAME:
make SOURCES=src/mame/drivers/amstrad.cpp
will produce a build with just the CPC/CPC+ drivers.

Thanks!

After only 14 full minutes of compilation, 8 simultaneous process on a 10nth generation Intel with a SSD that blows 3 gigabytes per seconds, it finally generated a working binary. That's 3 times faster than the 43 minutes I waited before, yet somehow... heavyweight still.  :laugh:

Thanks to http://www.cpcwiki.eu/index.php/MAME for providing hints.

Thanks again @mahlemiut!
I can run the emulation now.
Had a CPC since 1985, currently software dev professional, including embedded systems.

I made in 2013 the first CPC cross-dev environment that auto-installs C compiler and tools: cpc-dev-tool-chain: a portable toolchain for C/ASM development targetting CPC, later forked into CPCTelera.

pelrun

I think it's about time I revisited that project to integrate a gdb stub into cpcec... stuff implementing my own debugger, I want the big gun! :D

cpcitor

Quote from: Targhan on 00:03, 30 December 20
Out of curiosity, why do you use system interruption when you could ditch the system, be faster and safer?

Out of curiosity, do you suggest that "ditching the system" is always faster and safer?  Depends on the scale and perspective IMHO.

In my latest released game, optimizing ASM routines yielded much higher speed benefit than what the whole firmware interrupt + keyboard scan + AT2 player could even consume.

Some might think I waste resources.  On the contrary, I like to master my productions.  I know how the firmware and its keyboard scan behave (e.g. debouncing, ticker option for callback location in ROM or RAM).  Exploring other options (implies more code) was on the top of the todo-list for some parts, like text and graphics on a custom-sized screen, not for interruption-driven features.  Delivering my latest prod on time was.

So, in my perspective it is faster and safer to use battle-tested tools.

I use custom solution instead of firmware every time the sum of benefits outweighs the sum of added costs: have to reproduce lost firmware features, more code, which comes with its constraints and transitive consequences.  Nice thing is, thanks to the firmware modularity you can reduce its footprint while keeping most benefits.

Hope this makes sense.  It totally makes sense in my overall perspective which is: master the machine.  The CPCs came with a nice firmware, much more well thought out than most machines of the era, do you suggest it should be thrown away by default?

Plus, knowing the firmware allows some nice loader tricks, like a big graphical splash screen which code fits in the first tape block, and more.
Had a CPC since 1985, currently software dev professional, including embedded systems.

I made in 2013 the first CPC cross-dev environment that auto-installs C compiler and tools: cpc-dev-tool-chain: a portable toolchain for C/ASM development targetting CPC, later forked into CPCTelera.

cpcitor

Quote from: pelrun on 04:42, 31 December 20
I think it's about time I revisited that project to integrate a gdb stub into cpcec... stuff implementing my own debugger, I want the big gun! :D

A gdbserver in CPC would be fun indeed. Ability to debug your code in any development environment, with C-level line-by-line step through, and ASM-level instruction-by-instruction step through viewing C and/or ASM looks interesting.
This assumes that you get proper debug information along with compiled code. Which means support from the compiler/assembler/linker.

I've never implemented a gdbserver myself, but I've used some in embedded systems and it's considered a relatively small job.

There are probably some subtle things to think about, like how to make gdb aware that you're stepping on ROM/RAM/etc code.  Either gdb already understand all necessary notions about bank swapping, or one might hack a solution like using different address ranges:
0xFFFF0000-0xFFFF3FFF lower ROM
0x00000000-0x00003FFF lower RAM
you get the idea...

That can be the start of a different conversation in this forum.

Had a CPC since 1985, currently software dev professional, including embedded systems.

I made in 2013 the first CPC cross-dev environment that auto-installs C compiler and tools: cpc-dev-tool-chain: a portable toolchain for C/ASM development targetting CPC, later forked into CPCTelera.

reidrac

What I have found very useful is a "rewind" feature.

When I have an unexpected crash, I can rewind until before the crash and then start the debugger until I see the problem.
Released The Return of Traxtor, Golden Tail, Magica, The Dawn of Kernel, Kitsune`s Curse, Brick Rick and Hyperdrive for the CPC.

If you like my games and want to show some appreciation, you can always buy me a coffee.

Targhan

Quote from: pelrun on 04:42, 31 December 20In my latest released game, optimizing ASM routines yielded much higher speed benefit than what the whole firmware interrupt + keyboard scan + AT2 player could even consume.

Of course, but this is not the same scale. There are already some read-to-use code that is much faster than the system's one, with less memory footprint, and by using them, you save memory and CPU time, and with all that combined, you maybe don't have to optimize the rest of your code like crazy.

Plus a system, even though reliable like the one we're lucky to have, is dependent to the state of the machine BEFORE you even run your own production. How many productions crashed because the developer didn't expect many ROMs to be used?

Anyway, I was just curious. but if you wanted to load Amsdos files without the system, I have a ready-to-use lib for that (see my signature).
Targhan/Arkos

Arkos Tracker 2.0.1 now released! - Follow the news on Twitter!
Disark - A cross-platform Z80 disassembler/source converter
FDC Tool 1.1 - Read Amsdos files without the system

Imperial Mahjong
Orion Prime

pelrun

Funnily enough, trace logs and "rewind" (usually called reverse debugging/execution in the literature) are about 90% identical. It really just depends on what you do with the trace once you have it  :D

cpcitor

Quote from: Targhan on 14:59, 31 December 20
Of course, but this is not the same scale. There are already some read-to-use code that is much faster than the system's one, with less memory footprint, and by using them, you save memory and CPU time, and with all that combined, you maybe don't have to optimize the rest of your code like crazy.

One day I may be writing a prod in a situation when loading from disc without the help of the firmware comes more or less naturally, then I'll seriously consider your FDC routines. I have downloaded FDCToolsV1_1.zip and will provide private feedback.

Quote from: Targhan on 14:59, 31 December 20
Plus a system, even though reliable like the one we're lucky to have, is dependent to the state of the machine BEFORE you even run your own production. How many productions crashed because the developer didn't expect many ROMs to be used?

Don't MC START PROGRAM and MC BOOT PROGRAM reset firmware, including disabling all ROMs and guaranteeing lots of free memory? run"myprod.bin calls such routines, while run"myloader.bas does not.

Quote from: Targhan on 14:59, 31 December 20
Anyway, I was just curious. but if you wanted to load Amsdos files without the system, I have a ready-to-use lib for that (see my signature).

I see that, too. Thanks.
Had a CPC since 1985, currently software dev professional, including embedded systems.

I made in 2013 the first CPC cross-dev environment that auto-installs C compiler and tools: cpc-dev-tool-chain: a portable toolchain for C/ASM development targetting CPC, later forked into CPCTelera.

cpcitor

Quote from: reidrac on 14:54, 31 December 20
What I have found very useful is a "rewind" feature.

When I have an unexpected crash, I can rewind until before the crash and then start the debugger until I see the problem.

Do you mean you know an existing rewind feature that works? Where?

That would avoid the need to build one!

One the MAME side, I see this:

#
# CORE STATE/PLAYBACK OPTIONS
#
-state               saved state to load
-autosave            automatically restore state on start and save on exit for supported systems
-rewind              enable rewind savestates
-rewind_capacity     rewind buffer size in megabytes
-playback            playback an input file
-record              record an input file
-record_timecode     record an input timecode file (requires -record option)
-exit_after_playback close the program at the end of playback
-mngwrite            optional filename to write a MNG movie of the current session
-aviwrite            optional filename to write an AVI movie of the current session
-wavwrite            optional filename to write a WAV file of the current session
-snapname            override of the default snapshot/movie naming; %g == gamename, %i == index
-snapsize            specify snapshot/movie resolution (<width>x<height>) or 'auto' to use minimal size
-snapview            snapshot/movie view - 'auto' for default, or 'native' for per-screen pixel-aspect views
-snapbilinear        specify if the snapshot/movie should have bilinear filtering applied
-statename           override of the default state subfolder naming; %g == gamename
-burnin              create burn-in snapshots for each screen


But this lets imagine gamer video with frame granularity rather than rewind log with microsecond accuracy.
I couldn't find details on the net and even on source code.

What do you think?
Had a CPC since 1985, currently software dev professional, including embedded systems.

I made in 2013 the first CPC cross-dev environment that auto-installs C compiler and tools: cpc-dev-tool-chain: a portable toolchain for C/ASM development targetting CPC, later forked into CPCTelera.

Powered by SMFPacks Menu Editor Mod