Dear Mike, dear Sirs.
Thank you very much for the letter. The idea of runtime linking seems to be
quite natural, therefore last half a year couple of time I thought myself a
madman. I am glad to see your letter, it means if it be I am crazy I am not
alone :-)
Close to the matter. Let me summarize a little bit in order to bring some
clarity, I rather do it for myself :-)
So, AFAIU darwin GCC has a runtime linker. After you explanation to me
situation
looks like this:
1. during the compilation darwin's GCC adds 'nops' at the beginning of every
function.
2. When application started it is linked in the common way.
3. After certain change is applied application gets recompiled, BTW it
would be
interesting to know, for I believe only the changes needs to be compiled and
linked, so it would appear we have got a new module, with changes.
4. It is a question unto me how runtime linker knows its time, yet it loads
newly compiled module and AFAIU this module needs a common linkage also, in
order to handle the calls to the other functions
5. runtime linker mends 'nops' of the changed functions by means of
'jmp' to the
new functions.
6. That's it. We are done here.
I thought some kind of that, yet moved to matrix approach. Reason? The
clarity.
For instance you have many ... let me say 'revisions' of the same function. In
your case it is obvious how to handle jump to the newly compiled function, but
how to handle roll-back to some particular revision of this function?
How about
revisioning for file and for module? These thought restrained me from
doing any
assembler hacks into the system. The basic is to control, the
performance is at
the ... third. :-)
One more thing. My original purpose was to make possible to modify the
algorithms runtime, without recompilation. For this case we need interpreter.
In order to handle the calls to the interpreted function I am using approach
with proxy-functions. (BTW you may find this topic interesting:
http://docs.georgeshagov.com/twiki/tiki-index.php?page=Matrix+Linking+-+Proxing)
This approach BTW will work in your case, with 'nops'. But the management will
not be clean I suspect.
Performance.
Yes of cause it causes degradation of performance, especially
interpreting. I do
not really care about that by now, for instance I introduce so-called
stub-functions which delegate the call to the matrix, and that's it. It is
obvious that these entities might be easily avoided if the call to the matrix
will be handled instead to the call to the stub-function. Reason? Control. We
are able to make logging here, for instance. I have some ideas about
performance, I do not think this is a crucial issue.
C++.
Applying runtime linkage to C++ does not look to be a problem at all. I
said the
task is unsolvable due to interpreter. By now I know only two of them cint and
ch. Both of them are able to handle only C-calls, moreover there is no any
warranty that class signatures are similar for the binary code and
interpreter,
I am sure they are not :-)
And more over, Who needs C++? :-)
Synchronization and other challenges.
Mike, do you have a copyright on that phrase: "there are certain
realities when
doing this, and its important to understand the limitations and gotchas. " ?
:-)
About my initial approach. I completely reviewed it :-). it looks like this:
http://docs.georgeshagov.com/twiki/tiki-index.php?page=Matrix+Linking+-+The+shortest+way+to+go,
it would appear it is required to make an additional pass in order to
formalize
the matrix.
The only deal with the compiler is to handle the calls to the
functions, I mean
they are to be redelivered to the stubs.
Hmm....
It is a little bit longer I expected, sorry about that.
PS.
Dear Mike, I would be really interested in some questions regarded runtime
linker. I am going to download the sources, yet in that same time if it be
there is any additional dos about that, I would be really appreciate for the
link.
PPS
the article might be found here:
http://www.georgeshagov.com/matrixlinking/MatrixLinking/MatrixLinking.pdf ,
for
your convinience.
Sincerely yours, George.
Quoting Mike Stump <[EMAIL PROTECTED]>:
On Jan 10, 2006, at 4:32 AM, [EMAIL PROTECTED] wrote:
I have some kind of an idea, which I called 'matrix linking'. It
would appear
this is a new kind of linking.
Sorry, not very new. Microsoft has been doing it longer than we have
I believe.
On Jan 10, 2006, at 10:28 AM, Sean Callanan wrote:
In short, you are proposing that instead of linking an executable,
it be made into a bunch of shared libraries. The function calls
between these shared libraries be arbitrated by a "dispatcher"
which can dynamically reroute function calls.
There already exists a technique to do this if you're willing to
restart your app: incremental linking. Many IDEs support something
like this (it's called ZeroLink in Apple's Xcode, for example), and
even GNU ld has the -r option, which makes an object file
relocatable, making it quicker to link.
If you want to be able to do this without restarting, there are
some challenges:
1) Static variables in modules would present problems, as these
would have to somehow be reinitialized to their values in the old
module (for instance, if you have a static pointer to a malloc()ed
data structure).
-mfix-and-continue handles this case on darwin. See
TARGET_FIX_AND_CONTINUE in the compiler sources for the mods.
Roughly, we keep the old data and don't run the ctors for the new
data as I recall. The runtime linker handles most of the binding
issues, with little modification of the generated code.
3) You would have to figure out how to interpose on functions
called via function pointers.
We do this on darwin by reserving n bytes at the front of all
functions and put nops there, and then the loader patches those bytes
(COW) with a jump to the last one loaded.
4) In a multithreaded application, you'd have to make sure that
module replacement is atomic and you handle the case of replacing a
module at a time where a thread is executing a function in it.
:-)
Perhaps others on this list can come up with more issues to resolve.
I'd phrase it this way, there are certain realities when doing this,
and its important to understand the limitations and gotchas.
On Jan 11, 2006, at 5:16 AM, [EMAIL PROTECTED] wrote:
Interpreter is the weakest thing here, its communication with
compiled code -
especially. By now I am trying to modify GCC (cc1) in such way it would be
possible to apply matrix linking for .C modules only, C++ by now
looks to be
unsolvable task, that was the purpose of my writing, actually.
C++ can be done without too much work. :-) We do it on darwin. See
the codegen changes done by -mfix-and-continue. The are small,
minor and trivial. With it, we get into the same functionality
range as in Microsoft's compiler.
I need some kind of assistance with cc1 sources.
I'd recommend you copy in spirit the scheme used by darwin, trivial
compiler mods, and then just do up the changes to the runtime linker/
loader. Doing that, you don't need much of any help for the compiler.
I am trying to modify it in such way that during the compilation of
.C module on function description it would automatically produce
additional code (stub-function), and proxy function. These ones to
be compiled by some template, its compiled code (I thing the
assembler) is to be 'inserted' in the original module. The entire
calls to the function are to be redirected through the external
matrix.
Slow. We found there wasn't any advantage to making the code slow,
so we didn't. We found that we could get near to 100% speed of
normal code, the biggest slowdown being the 5 nops or so at the start
of each function, which, on modern machines, go very quickly.
What advantage do you think that making the code slow gives you over
the scheme used on darwin?
