Image 3" floppy disks for emulator

[nivrig]

@ralferoo I'm not sure if my Atoms made it to an AF coverdisk - there was at least one other Amiga version of the game so it's possible you picked that one up. Spookily, one of the random disks I dumped above actually contained a CPC version of Atoms called Meltdown!

I kept my CPC stuff as I learned my lesson after what I did with my VIC-20 stuff - which was basically what you did to your CPC collection :/ 5 tapes of BASIC games I wrote, recorded over. AGGH!

[edit]
CPC "Meltdown":
http://robinnixon.com/articles/computingwta/1988-03/index.htm


Some Win 3.x versions of Atoms here:
http://members.chello.at/theodor.lauppert/games/atoms.htm

[SyX]

Could you upload one of this mfm dumps of standard DATA disks, girv?

It's for improving a mini-python script to convert them to DSK

Cholo, i have generated a .DSK from your MFM dump, it was very straightforward only cut the first 180KBs and insert directly the tracks in the DSK.

If we can standardize a "recipe" for creating CPC MFM dumps, we only need more dumps to create a MFM2DSK converter or a better DSK format that contemplates the protections in a more sane way.

[nivrig]

The i4 and resulting dsk files I was testing with are attached.


i4 format is only going to work for standard disks, I think, and even then you won't be able to tell which format the source was as the i4 dump file doesn't contain any sector or track numbers etc.


You're probably going to have better luck with the i2 format which contains the undecoded MFM data. I haven't dug into that one very much so I may be wrong on this, but I imagine (a) it contains all the missing format information and whatever else is on the tracks, (b) it won't detect weak bit protections. It'd be tricky enough to handle all cases, but I don't see why an i2->EDSK converter wouldn't be possible.


For the ultimate in preservation you really need to parse the STREAM format (or DRAFT when it's released) dumps and construct your DSK from that. That's what SPS do when making IPFs.


Also, doesn't EDSK already handle protected disks?

[Phi2x]

.

[SyX]

Thanks girv!!! 

i4 format is only going to work for standard disks, I think, and even then you won't be able to tell which format the source was as the i4 dump file doesn't contain any sector or track numbers etc.

Yes, of course, but it's perfect to make backups (or convert from DSK2MFM to write disks using Kryoflux) of not protected or self-made software. And the format for these disks are DATA or SYSTEM, which is not difficult to detect

Although how you said, perhaps it would be better to use the i2, at least while the Draft format is not published

Also, doesn't EDSK already handle protected disks?

As phi2x said EDSK has a few problems and maybe is better develop a more sensible format, instead of adding extensions to DSK.

[Cholo]

Success!

Nice one!! and thanks for the config file too  This really open up the Kryoflexs posibilities for quick data transfer to pc.

Cholo, i have generated a .DSK from your MFM dump, it was very straightforward only cut the first 180KBs and insert directly the tracks in the DSK.

Thanks! its good to know. If u need more test dumps let me know.

[SyX]

Thanks! its good to know. If u need more test dumps let me know.

Thanks!!!

I have finished a python script to convert these standard MFM dumps to DSK, needs python 3 (but if somebody wants a python 2 version, only has to ask ), the syntax is very easy "mfm2dsk.py file1.mfm file2.mfm ...", you can use wildcards, of course. And if you want to disable the horrible hack of autodetection of disk format, use the option -n to force the creation of DSK in DATA format or -n -s to force the creation of disk in SYSTEM format.

The script...

Code Select Expand

#!/usr/bin/env python
# -*- coding: utf-8 -*-

"""
MFM2DSK
(c) SyX, 2011
"""

import sys
import glob             # glob() expande los patrones de los ficheros en windows
import os               # path.exists()
from optparse import make_option, OptionParser

# Constantes
DSK_ID_EXT = b'\x45\x58\x54\x45\x4e\x44\x45\x44\x20\x43\x50\x43\x20\x44\x53\x4b' + \
        b'\x20\x46\x69\x6c\x65\x0d\x0a\x44\x69\x73\x6b\x2d\x49\x6e\x66\x6f\x0d\x0a'

DSK_AUTHOR = b'MFM2DSK by SyX'
TRACKS_PER_DISK = b'\x28'   # 40 pistas por disco
DSK_NUM_SIDES = b'\x01'

SECTORS_PER_TRACK = 9

DSK_FILL_BYTE = b'\xE5'

DISK_INFO_BLOCK = DSK_ID_EXT + DSK_AUTHOR + TRACKS_PER_DISK + DSK_NUM_SIDES + \
                 b'\x00' * 2 + b'\x13' * 40 + b'\x00' * (4 * 41)

TRACK_ID = b'\x54\x72\x61\x63\x6b\x2d\x49\x6e\x66\x6f\x0d\x0a'

LEN_SECTOR = 512
               
SECTOR_SORT =   [0, 5, 1, 6, 2, 7, 3, 8, 4]
#SECTOR_UNSORT = [0, 2, 4, 6, 8, 1, 3, 5, 7]

def crea_track_info_block(num_pista, system):
    """
    Devuelve una cadea con el Track Info Block correspondiente a la pista
    """
    # Añadimos la información de la pista   # offset    description         bytes
    tib_tmp = TRACK_ID                      # 00 - 0b   "Track-Info\r\n"           12
    tib_tmp += b'\x00' * 4                  # 0c - 0f   unused                      4
    tib_tmp += bytes((num_pista, ))         # 10        track number                1
    tib_tmp += b'\x00'                      # 11        side number                 1
    tib_tmp += b'\x01'                      # 12        data rate (0/SD-DD/HD/ED)   1
    tib_tmp += b'\x02'                      # 13        recording mode (0/FM/MFM)   1
    tib_tmp += b'\x02'                      # 14        sector size                 1
    tib_tmp += b'\x09'                      # 15        number of sectors           1
    tib_tmp += b'\x52'                      # 16        GAP#3 length                1
    tib_tmp += DSK_FILL_BYTE                # 17        filler byte                 1
    # Añadimos la lista de información de los sectores de dicha pista
    num_sector_tmp = (0x41 if system else 0xC1)
    last_sector = (0x45 if system else 0xC5)
    for i in range(SECTORS_PER_TRACK):
        tib_tmp += bytes((num_pista, ))     # 00        track (C in NEC765)         1
        tib_tmp += b'\x00'                  # 01        side (H in NEC765)          1
        tib_tmp += bytes((num_sector_tmp, ))# 02        sector ID (R in NEC765)     1
        num_sector_tmp += (+5 if num_sector_tmp < last_sector else -4)
        tib_tmp += b'\x02'                  # 03        sector size (N in NEC765)   1
        tib_tmp += b'\x00'                  # 04        FDC status register 1 (ST1 in NEC765) 1
        tib_tmp += b'\x00'                  # 05        FDC status register 2 (ST2 in NEC765) 1
        tib_tmp += b'\x00\x02'              # 06 - 07   actual data length in bytes 2
    # Añadimos el padding hasta llegar a los 256 bytes que ocupa el TIB
    tib_tmp += b'\x00' * 16 * 10
   
    return tib_tmp

# Procesa la línea de comandos   
def procesar_linea_comandos(linea_de_comandos):
    """
    Devuelve una tupla de dos elementos: (opciones, lista_de_ficheros).
    `linea_de_comandos` es una lista de argumentos, o `None` para ``sys.argv[1:]``.
    """
    if linea_de_comandos is None:
        linea_de_comandos = sys.argv[1:]

    version_programa = "%prog v0.1"
    uso_programa = "usage: %prog [options] file1 file2 ... fileX"
    descripcion_programa = "%prog convert from MFM to DSK."

    # definimos las opciones que soportaremos desde la línea de comandos
    lista_de_opciones = [
        make_option("-n", "--no_autodetect", action="store_false", dest="autodetection", default=True, help="Disable autodetection of disk format"),
        make_option("-s", "--system", action="store_true", dest="format_system", default=False, help="The MFM disk is in format SYSTEM")
    ]
       
    parser = OptionParser(usage=uso_programa, description=descripcion_programa,
        version=version_programa, option_list=lista_de_opciones)
   
    # obtenemos las opciones y la lista de ficheros suministradas al programa
    (opciones, lista_ficheros_tmp) = parser.parse_args(linea_de_comandos)

    # comprobamos el número de argumentos y verificamos los valores
    lista_ficheros = []
    for i in lista_ficheros_tmp:
        lista_ficheros = lista_ficheros + glob.glob(i)

    return opciones, lista_ficheros

# Función principal
def main(linea_de_comandos=None):
    """
    Función principal
    """
    # Obtenemos las opciones y argumentos suministrados al programa
    opciones, lista_ficheros = procesar_linea_comandos(linea_de_comandos)

    # Creamos un DSK a partir de cada fichero pasado como parámetro
    for mfm_disk in lista_ficheros:
        # Comprobamos que existe el fichero MFM
        if not(os.path.exists(mfm_disk)):
            print ("The file %s doesn't exist.", mfm_disk)
            continue

        # Lo cargamos
        print ("Opening file: " + mfm_disk)
        with open(mfm_disk,"rb") as fichero:
            fichero_en_sectores = fichero.read(180 * 1024) # Solo leemos los primeros 180 KBs
       
        # Añadimos el padding al fichero para hacerlo multiplo del tamaño de los sectores   
        fichero_en_sectores += (DSK_FILL_BYTE * (LEN_SECTOR - \
                                (len(fichero_en_sectores) % LEN_SECTOR)) \
                                if (len(fichero_en_sectores) % LEN_SECTOR) else b"")

        # Y lo partimos en trozos de 512 bytes
        fichero_en_sectores = [fichero_en_sectores [i * LEN_SECTOR: (i + 1) * LEN_SECTOR] \
                                for i in range(len(fichero_en_sectores) // LEN_SECTOR)]

        # Detectamos si el disco está en formato DATA ó SYSTEM (*** HACK :P ***)
        if opciones.autodetection:
            sector_de_sistema = DSK_FILL_BYTE * LEN_SECTOR
            opciones.format_system = True;
            for sector_tmp in fichero_en_sectores[0: SECTORS_PER_TRACK * 2]:
                if not sector_tmp.startswith(sector_de_sistema):
                    opciones.format_system = False
                    break

        # Indicamos el tipo de disco
        print("Format:", ("SYSTEM" if opciones.format_system else "DATA"))
       
        # Creamos un disco formateado
        disco = [[crea_track_info_block(num_pista, opciones.format_system), \
                [DSK_FILL_BYTE * LEN_SECTOR] * SECTORS_PER_TRACK] \
                for num_pista in range(ord(TRACKS_PER_DISK))]

        # A continuación lo vamos insertando en el disco
        num_pista = num_sector = sectores_totales = 0
        for sector_mfm in fichero_en_sectores:
            # Comprobamos si necesitamos pasar de pista
            if num_sector > 8:
                num_sector = 0
                num_pista += 1
            # Reemplazamos el sector existente por el sector con datos del fichero
            disco [num_pista] [1] [num_sector] = sector_mfm
            num_sector += 1
            sectores_totales += 1

        # Reordenamos los sectores de una pista y los precedemos por la TIB de esa pista
        for num_pista in range(ord(TRACKS_PER_DISK)):
            disco [num_pista] = [disco [num_pista] [0]] + \
                                [disco [num_pista] [1] [SECTOR_SORT [i]] \
                                for i in range(SECTORS_PER_TRACK)]

        # Creamos el DSK fusionando todas las pistas
        dsk_tmp = DISK_INFO_BLOCK
        for num_pista in range(ord(TRACKS_PER_DISK)):
            dsk_tmp += b"".join(disco [num_pista])

        # Y guardamos el disco
        print ("Saving file: " + mfm_disk + ".dsk")
        with open(mfm_disk + ".dsk","wb") as fichero:
            fichero.write(dsk_tmp)
        print()

    return 0    # EXIT_SUCCESS

if __name__ == "__main__":
    estado = main()
    sys.exit(estado)


[Cholo]

Phantastic! Gonna have to dig up a Python 3 installer 

Edit: 10 mins later and ive learn how to use Python 3.2.2 and execute the script  Works nicely .. even print to screen if its Data or System that is being converted.

[SyX]

I'm glad you like it and that everything went smooth, Cholo, if you need some help, only ask

Softpress people we need the DRAFT specification!!! 

OT: I use python mainly because is very easy to learn, real multiplatform (i use the same code in my home linux machine and my windows netbook), have a huge library and when i learned, i had the same "happy" feelings that when i had my CPC in the 80s

[Morn]

OT: I use python mainly because is very easy to learn, real multiplatform (i use the same code in my home linux machine and my windows netbook), have a huge library and when i learned, i had the same "happy" feelings that when i had my CPC in the 80s

Yes, firing up a Python interpreter or writing a Python script feels a lot like turning on a CPC and having the BASIC prompt appear. You just know that whatever you want to do with the machine will be achieved quickly, easily, and without much debugging effort. Therefore the happy feelings. 

[nivrig]

I've attached a short Java hack to do the same as Syx's script.


/**
* Convert KryoFlux "i4" format dump files to DSK format images.
*
* Usage:
* java KryoFluxDisk [options] [i4_dump_file] [i4_dump_file]...
*
* Output files are created as the same name as the input files
* except with ".dsk" extension.
*
* Options:
* -d create output files in DATA format (default)
* -s create output files in SYSTEM format
*
* Suggested KryoFlux command line to create suitable dump files:
* dtc -fdisk_i4_s0.bin -g0 -e39 -l8 -i4
*
* @author John Girvin
*/


Source is included - public domain, use as you will *looks at DevilMarkus*

[SyX]

Great Work girv!!!

Now that i'm more relaxed, i was going to convert my pyhon code to c++ for DevilMarkus, but you have saved me the effort and he will thank you

[nivrig]

Wee blog post about all this:
http://www.weedoorsbanging.com/archives/converting-kryoflux-dumps-to-amstrad-cpc-dsk.html


The source of the Java converter is on github:
https://gist.github.com/1310270

[Devilmarkus]

Thank you 2 very much for your help!
Sure, I can use this very much

[Devilmarkus]

Here's an example how I load the I4 examples into JavaCPC:

Code Select Expand

    int sectorsize = 512;

    public void loadKyroFluxDisk(String name, byte[] data) {
        int result = 0;
        for (int i = 0; i < sectorsize; i++) {
            result += data[i] & 0x0ff;
        }
        boolean datadisk = false;
        result /= sectorsize;
        result &= 0x0ffff;
        System.out.println("MFM checksum: " + result);
        if (result == 141) {
            datadisk = false;
            System.out.println("CPM plus System");
        } else if (result == 122) {
            datadisk = false;
            System.out.println("CPM 2.2 System");
        } else if (result == 0xe5) {
            datadisk = false;
            System.out.println("System without bootblock");
        } else {
            datadisk = true;
            System.out.println("DATA disk");
        }
        byte outFileData[] = new byte[194816];
        KryoFluxDisk.i4ToDsk(data, outFileData, datadisk);
        try {
            DSK_Load(name, outFileData);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

You can see that I detect the System format here.
This will not work with games / protected disks which have a bootsector, but is an attempt to find the accurate format.

Edit: Am I right, that the "test_i4_s0.mfm" is corrupt?
Only Er * Bert works...

[Devilmarkus]

Few i4 conversions I made from DSK files:
Containing CPM 2.2 system disk and all 4 CPM Plus disks

[Devilmarkus]

Little Addon for Girv's Java-Routine:

Code Select Expand

    static int[] SectorPos = {
        1, 6, 2, 7, 3, 8, 4, 9, 5
    };

    public static void DskToI4(byte[] data, String filename) {
        byte[] out = new byte[368640];
        int numberOfTracks = data[0x30] & 0xff;
        int numberOfSides = data[0x31] & 0xff;
        final byte[] trackSizes = new byte[256];
        System.arraycopy(data, 0x34, trackSizes, 0, numberOfTracks * numberOfSides);
        int offset = 512;
        int dskpos = 0;
        for (int track = 0; track < numberOfTracks; track++) {
            for (int side = 0; side < numberOfSides; side++) {
                try {
                    int trackLength = ((trackSizes[track * numberOfSides + side] & 0xff) * 0x100) - 256; // 256 bytes for each trackinfo
                    byte[] buffer = new byte[trackLength];
                    System.arraycopy(data, offset, buffer, 0, trackLength);
                    byte[][] bufsec = new byte[9][trackLength / 9];
                    for (int g = 0; g < 9; g++) {
                        System.arraycopy(buffer, g * trackLength / 9, bufsec[SectorPos[g] - 1], 0, bufsec[g].length);
                    }
                    for (int g = 0; g < 9; g++) {
                        System.arraycopy(bufsec[g], 0, buffer, g * trackLength / 9, bufsec[g].length);

                    }
                    System.arraycopy(buffer, 0, out, dskpos, buffer.length);
                    dskpos += trackLength;
                    offset += trackLength + 256; // 256 bytes for each trackinfo
                } catch (Exception e) {
                }
            }
        }
        filename = filename.replace(".dsk", ".mfm");
        if (!filename.endsWith(".mfm")) {
            filename += ".mfm";
        }
        File file = new File(filename);
        try {
            FileOutputStream fos = new FileOutputStream(file);
            fos.write(out);
            fos.close();
        } catch (Exception e) {
        }
    }

Not 100% accurate yet (no sector info used here!!!)

[dlfrsilver]

I picked up a small sample of EDSK files, dumped from originals, that I encountered problem with. You can find them on cpc-power.com:

   
• Billy la Banlieue
• Sapiens
• Lode Runner
• L'affaire Vera Cruz
• Skweek
• Bob Morane Chevalerie
• Moktar
• Les passagers du vent 2
• Prehistorik
• Baby Jo
• Livingstone 2
• Space Racer

It's normal, most of those disk have either bigger protection tracks or sector, for instance Livingstone 2 use a big sector (sectors with size higher than size 6 exists size 7,8,16 and even 32kb). The loriciels games use GAP protection, it maybe needs support from HxC.

If you take those EDSK and convert them to HFE, they won't work on a real CPC equipped with an HxC floppy emulator 
So we have had some debate whether this was the fault of the EDSK format (doesn't contain enough information about the disk), or bad dumps, or HFE limitations, or bugs in the HxC. It's not very clear where the problem is.(/quote]

It's not easy at all to spot where the problem is since the FDC emulation is not perfect. It would be as perfect as the real one, it would be easier to spot what is wrong.

For all I know, the HxC doesn't emulate weak sectors, which can be a problem with -some- of those titles.
On the other side, the EDSK format has some bizarro stuff in it, like the values of the flags of the FDC. That really shouldn't belong to a disk image format.

It should be done with multiple copies of those sectors, because weak sectors is a physical protection, not a tricky handmade one.

EDSK also allows tracks to have more than 6250bytes in it. But that violates physical laws. It's not possible to have that in real disks.
And it's a real problem because if you want to emulate the disk drive properly (rotation speed and stuff), it's not possible to have tracks larger than that.
So how an emulator author is supposed to deal with those EDSK that feature longer than real-life EDSK tracks?

this is a false information, and why many disks have not been supported until now. The max track size on a CPC track is 6305 bytes for a track with data.
Other thing, when the FDC reads an 8kb sector, it doesn't actually read 6144 bytes, but it fully reads 8192 bytes.

6250kb is a number which stands on no real FDC specs. If i want to write with a special mastering hardware a 32kb tracks just to piss you off, even on a CPC game,
like Obama said himself, "Yes We can !".

You see why i say that having a perfect FDC emulation must be done FIRST ? because most people have wrong beliefs about it, and hence emulation can go forward.

for instance, the coin-op hits compilation from usgold on disk use 6305 bytes PER TRACKS ! The dump we have actually made with Xexor 2.6 and CPDREAD is cut down of 160 bytes per track, which as a result make the games on the disk not working or even reset the CPC because program part are missing.

the only tool actually able to dump it is samdisk from Simon Owen, all the others even CPCdiskXP to my knowledge can't image those.

[Executioner]

It appears to me that the only Kyroflux format that is actually better than EDSK is the undocumented STREAM format. For that reason we should stick with EDSK for now which at least supports the majority of protections (and probably all the games listed that don't work could be supported by the EDSK format, just that the original readers didn't represent the image accurately).

[TFM]

Well, I actually think you are 100% right!

[dlfrsilver]

It appears to me that the only Kyroflux format that is actually better than EDSK is the undocumented STREAM format. For that reason we should stick with EDSK for now which at least supports the majority of protections (and probably all the games listed that don't work could be supported by the EDSK format, just that the original readers didn't represent the image accurately).

Yes that's true but the support of the kryoflux disk images (Since stream format is a RAW format, it needs human check to be processed then as IPF), require perfect FDC emulation. Look at the atari ST IPF support, they had to code a perfect WD1772 FDC controller emulator for support.

That's why i hope in the near future that we have AT LAST a perfect FDC emulation, to discard or see what need to be redumped in the EDSK we have.