Self-Modifying Code

When you read about programming in the 80s, often there is a discussion about how common assembly was.  You'll hear about all sorts of mystic tricks and tips that programmers back then used.  You might have also heard reference to "self-modifying code".

When I was younger, "self-modifying code" always sounded so advanced.  It was something that only the very brightest people did, and they surely must've done it to prevent people from disassembling their code.

The reality is much more mundane.  I'll show you a real example of self-modifying code from a piece of commercial software from the 80s.  I'll walk you through it, and explain why it was written this way.

The code in question is from an Apple IIgs game (the CPU is a 65816, which is a 16-bit variant of the classic 6502).

00/0e4e: a9 00 60    lda #$6000     ; A = $6000

00/0e51: 8d 6d 0e    sta $0e6d      ; mem[$0e6d] = A

...

00/0e65: e2 30       sep #$30       ; 8-bit registers

00/0e67: a2 00       ldx #$00       ; X = 0

00/0e69: bd 00 0a    lda $0a00, x   ; A = mem[$0a00 + X]

00/0e6c: 8d ff ff    sta $ffff      ; mem[$ffff] = A  !!!

00/0e6f: ee 6d 0e    inc $0e6d      ; mem[$0e6d]++

00/0e72: d0 03       bne $00/0e77   ; is result != 0?

00/0e74: ee 6e 0e    inc $0e6e      ; no, mem[$0e6e]++

00/0e77: e8          inx            ; X++

00/0e78: d0 ef       bne $00/0e69   ; is result != 0?

That's a chunk of real self-modifying code, let's go through it.   Notice the references to memory at addresses $0e6d and $0e6e.  If you look at the address labels on the side, you'll see those addresses point to the operand of the instruction I've marked with "!!!".  By modifying the values at those addresses, we cause the "sta" instruction to point to arbitrary 16-bit addresses.  That's the only part of this code that gets modified.  Mundane, right?

I put pseudocode over on the right so you can see what each instruction does even if you don't know 65816 assembly.  Basically the code does this:

uint8_t *ptr = &memAt6000;

...

   for (x = 0; x < 0x100; x++) {

     *ptr++ = memAtA00[x];

   }

The observant among you will wonder why we go through all this hassle when you could instead do: sta $6000, x.  (which is basically the same as ptr[x] = memAtA00[x])

There are several reasons.  One is the result isn't quite identical.  In the original, when the loop is finished, the pointer points to the end of the array, not the beginning. This is important in the original code because this is actually an inner loop that gets run multiple times by an outer loop, which leads us to two, because it gets run multiple times, the array grows beyond the bounds of an 8-bit index register.

Why not just use a variable elsewhere in memory to store the pointer then?  Surely this doesn't need to be self-modifying.

Well, actually it does.  It has to do with the addressing modes available, or not available as it were.

You see, the 65816 only allows indirect addressing on the direct page (known as the zero page on the 6502).  Absolute addressing doesn't offer any indirect modes.  Needless to say, the direct page gets a bit crowded, and programmers have just decided that its far more elegant to use self-modifying code to "fake" indirect absolute addressing.. which is why this technique is rampant on the Apple IIgs.

It's also less confusing in practice.  The direct page can move around, so you either keep track of it by hand (setting it to the appropriate place upon entering a routine, adding mental and real overhead) or you leave it in a single place the entire time (limiting you to less than 128 pointers for your entire program).  It becomes clear that this particular style of self-modifying code is an elegant solution to the problem.

This isn't the only use-case of self-modifying code, of course, but it is one of the most common, especially on machines with limited addressing modes.