Martin Fowler blogged about ErraticTestFail. I can not say the name is very descriptive :) but the problem is interesting. Such buggy tests arise, mostly, from static mocks or stubs shared by several test methods or cases. Now, ideally, your tests should be independent but there is a performance tradeoff - sometimes initalization takes too much time. The last thing you want  - your test suite to take too much time to run. That can lead to productivity degradation and the temptation of running test suite less often or (oh, God!) - not at all.

I think logic unit tests should be mocked enough not to need costly initializations, so if it is true - the problem would arise, mostly, in functional and integration tests.

JUnit allows to reload the run of each test. You would, usually, have it turned off for the performance reasons. But because of the ErraticTestFail possibility, it is  may be a good practice to:
1) Run daily tests on an integration server where JUnit reloads for each test run. Test may be fine-tuned to share setUp() and be faster but they must be written so that if each test is ran in a new classloader - they still work (even if slower).
2) When and if you notice something suspicious (like Martin\’s case) turn on reloading and see what happens, before you go on blaming your IDE, compiler or whatever is easier to blame :) \