Sounds like you’ve got a long and ugly road ahead of you. Here are some additional things to consider, based on years of experience in software product management (i.e., being in charge of finding ways to prevent people from doing what you’re trying to do).
There’s a good chance that a serial number that’s valid for a five-user license won’t be valid for a ten-user license. Depending on how the software was originally licensed, the format may be different for each “level” of licensing that the developer offered. For instance, individual licenses might have a particular string or range of strings in one position of the serial number, while multi-user licenses might have a different string or range of strings in that position. If the company offered the product in (for example) one-user, three-user, five-user, ten-user, fifty-user, and site license versions, the format for each might well be slightly different.
It’s also possible, even likely, that the serial number as encoded in the resource fork of the XTension is encrypted, meaning that unless you somehow figure out the encryption scheme as well, you’re going to get nowhere fast.
Also, you should have noticed by now in poking around with ResEdit that it displays different types of resources differently (‘icl8’ resources, for instance, which contain eight-bit color icons, are displayed in an bitmap icon editor). This is a convenient fiction, of course, at the level of the actual bytes contained in the code – it’s all hex at the lowest level. It sounds like what’s happening is that the resource you’ve identified as the likely source of the information you need is either a ‘code’ resource, which ResEdit displays as hex with little attempt to format beyond that. It used to be the case that Apple’s monthly Essentials, Tools, and Objects (ETO) developer’s CD contained some additional modules for ResEdit, including some that would allow you to more or less follow the program flow, but without knowing the 68000 instruction set pretty well, you’re not likely to be able to derive useful information from that. And in any case, it’s been years since I looked at Apple’s developer materials, so I have no idea whether such things are still available or not.
Probably the tool that’ll allow you to do as much as you’re going to be able is Quadrivio’s General Edit Lite. It’s a free text/hex editor that will display hex as ASCII and works on files (both resource and data forks) and even (God forbid) the contents of RAM. If you poke around a bit with it and find that it might be useful to you, you’d probably want to spring for the commercial version (~$175) to be able to finish the work.
Of course, you’ll want to practice safe hex: never work on your only copy of anything – it’s extremely easy to render a program completely inoperable by changing a single bit. Think twice (or three or four or five times) before making any changes if you don’t know what you’re doing. For God’s sake don’t open a live, important document in QXP with a hacked XTension loaded until you’ve satisfied yourself that it’s working correctly.
I have to say again that this is a more daunting project than I’d be willing to take on. Though I’m not a programmer, I’ve made extensive changes to the menus, text strings, layout of dialog boxes, and even to a minor extent the functionality of Mac programs using ResEdit, but you’re trying to do something that the developers of the product probably invested some time in ensuring that you wouldn’t be able to do. Even with extremely favorable winds, I’d put your odd of success at less than 5%, and the time required at upwards of six months of intense effort. If you make it, however, you’ll have learned a hell of a lot in the process.