So I have thought of several potential improvements to traditional STM, which bring it closer to optimistic locking, and wanted to hear your opinion:

1) Lock on race conditions.

Go ahead and execute the transaction, but if you detect a conflict, acquire a lock on the relevant object and execute the transaction again. Seems like a better decision, with a guaranteed worst execution time, than implementing some kind of throttling.

2) Non-transactional objects.

The idea is to have an STM method like so:

    void STMExecute<T>(T obj, Func<T, bool> transaction, Func<T, bool> failsafeTransaction)

STMExecute would keep track of the generation counter for obj, run the transaction lambda, and if it fails, or there is a conflict, acquire a lock on obj, and run failsafeTransaction.

For example, in a banking system, transaction would increment the account (obj)'s balance by 100, and failsafeTransaction would recompute the balance by going through the history of withdrawals and deposits for the account.

3) No side effect functions

STMExecute would be changed to

    void STMExecute<T>(ref T obj, Func<T, T> transaction)

A potential issue is that C# doesn't offer a way to enforce transactions have no side effects. This can be done by making immutable types, though.

4) What is a good benchmark?

What's a good, realistic way to test the performance of STM, compared to locks? The most obvious way - run several threads like so:

    while (true) {
        STMExecute(ref var, x => x + 1);
    }