Linux stability fix: v1.0.1 (Hoverboard)

After releasing the first stable version I have received lots of feedback from folks around the world. Most of you guys were positive about the release, except my good old pal: Mike Blackburn.

Probably there are not too many E-UAE users are trying to run it on LinuxPPC and one of the reasons is probably: there is no binary distribution. Yet, there are a handful of crazy people... One of them is Mike.

He was not happy about the final release, because it is not working at all on Linux. The situation was tricky, because it was working before, but not anymore after my (seemingly unrelated) changes regarding the removal of useless debug logging.

After a week of struggling, roughly 70 emails(!) he figured out what was wrong: cache flushing.

Thanks to him here is a new "release" for the few LinuxPPC users: v1.0.1.
It is not really a release, since there is no binary build available, but I made a remark in the release folder to get the sources from 1.0.1 tag and build it on LinuxPPC.

Nice work, Mike!

PPCJITBETA02 (The Beginning of a Beautiful Friendship)

We have just arrived to another exciting milestone on the long road: all the important instructions for the initial release are implemented under the JIT compiling.*

Lots of bugs were fixed, the emulator is much more stable now than the initial beta release.

Some new features are added too: I have merged the SAM440EP/Flex support (thanks to Soft3) and the CGX overlay for MorphOS (thanks to Thunder and Fab). See configuration documentation regarding how to set the overlay up.

As I already mentioned in the previous post: this beta release was delayed for a couple weeks due to a bug that slipped into the code base long time ago. It was discovered on Mac first, but I was able to reproduce it on MorphOS too. Took me a while to figure out what was going on, but it is fixed now.
This was a very tricky bug, it could be blamed for random crashes and endless loops also, not only on Mac, but on all supported platforms. It was triggered randomly based on the distance between the main application code and the code buffer in memory. (Thanks to Mike for discovering this right before I released the beta.)

I have spent a significant amount of time on figuring out how can I do the build for all supported platforms (AmigaOS4, MorphOS and MacOSX-PPC) using my environments. It wasn't easy, but finally I managed to do most of the release on my own.
As it seems MorphOS SDK does not support G5 yet, so I was not able to do the compiling by myself, but thanks to Fab the G5 executable is also available in the release package.
So, as of now users on all major supported platforms can grab the package and start using the right version.
(Sorry, Linux and BSD folks, you are still on your own.)

 

And the World trembled...

 ...or at least that tiny part which I am involved in when I am wearing my crazy latex suit with a huge letter "A" on my chest for my secret identity: the Amiga Software Developer.

After the first beta release forum posts, emails, news sites, blogs had risen in an enormous unmanageable thunderstorm, struck on me with insane amount of communication. (While the rest of the World barely noticed what have just happened.)
Finally I crawled through messages from every possible (and impossible) source and answered the questions to my best knowledge, accepted the good advices, kindly rejected some nonsense.

 

Aftermath

Since I received tons of feedback (good and bad), I inclined to draw some conclusions from the reaction to the very first beta release. Here is the summary for your benefit:

Some people don't understand how the JIT works and what is the exact purpose of it. All I can say is: please read the documentation... Some other (knowledgeable) folks stood up on the forums and educated the others, well done! I hope this helps, because I really don't have time to deal with it.

Many of the users have irrational expectations for how much the JIT compiling will speed up the emulator. (According to somebody: it supposed to be "ten times faster than the interpretive"... Err... Not likely. How did they come up with any number anyway?)

Well, the implementation is not finished yet, some of you guys don't really understand the concept of "beta release". Okay, I admit I was cheating a bit: technically the JIT compiler wasn't feature-complete when the first beta was done. Yet the remaining pieces were related to not too often used instructions anyway.
For the second beta the instructions are done*, yet there is clearly room for improvement regarding some bugs. Probably as soon as I will be able to fix up the optimization of the register- and flag-flow there will be a significant bump for the speed. (No, not "ten times" fold.)

It is hard to measure how much faster the programs are running and some lovely people baffled on this too. Since there is usually no obvious visual clue for the speedup and a 30%-50% increase in the processor speed is probably hard to notice while your favorite jump-and-run game is jumping and running.
Yet, you can feel that the whole emulation is more snappy than before probably even when you simply run Workbench. Except when it crashes. But even then: it crashes 30%-50% faster! :)

Too bad that some good souls are obsessed with their favourite game/program and keep saying that the JIT is worthless because it doesn't make any difference for that particular piece of software. As it seems this JIT compiling is not for you then.

There was one more interesting thing what I have noticed too late unfortunately: G5 support for MorphOS. Since I don't have a G5 machine I never considered that there is a need for that. But there were some murmur about the speed of the MorphOS version on G5 on some forums. No wonder: it needs a special version, which can be compiled from the sources for some time now. (Thanks to Tobias Netzel and to Fab for the special build.)
Probably the same applies to the PA Semi processor and the Amiga X1000, but I don't have that one either. (Donations? :)
Also the mysterious support for SAM440EP/Flex, what I have never heard of before. No wonder it was missed.

Fun fact from the Outer World: I tried to explain to my colleagues how I spent my Summer vacation. However, I am significantly older than almost any of them, so they were looking at me with confusion in their eyes mixed with a little pity. "Yea, my father loves fooling around with those old machines too!" - was one of the comments. Well put, Sir, well put.

 

Anyway...

To make you (some other geeks around the World) happy: here is the new beta...

https://sourceforge.net/projects/euaeppcjit/files/0.8.29-PPCJITBETA02

In case you stop reading here (or you already skipped the first cheesy part):
as always, please read the README for your comfort and safety. Thanks.

Since I bought an iBook for 50 NZD, now I can produce the MorphOS and the MacOSX versions too which were also included in this release together with the AmigaOS4 version. (And by buying a Mac I broke one of my principles: no Apple product crosses the door of my house. I hope you guys are content what you were doing...)

You can find the changes since the last beta in the README file, or in the changesets at the SourceForge repository from R67 down to R53.

 

Fragmentation

I must admit I have learned a lot in the past month about the sorry state of the E-UAE project. I didn't know what is the current situation of the various binary releases until I received some references to modified AmigaOS4 and MorphOS binary versions.
I guess this is the destiny of any abandoned open source project: lots of good people is trying to improve it, but nobody is standing up and takes over the maintenance of the project.

Well, I am of the same kind, as it seems. It was never my goal to take the ownership of the E-UAE project or fork it into a new iteration.
However, as soon as I released the first beta of the JIT compiled version the watching eye of the public turned to my little scared pet project and I received lot of questions about whether this-or-that particular fix from various developers were included or not. (Mostly not.)

To satisfy at least some part of the user base I tried to gather the various fixes from every corner of the Internet and applied them on the source code. This means no way new base source repository for the E-UAE project, but at least it will help whoever wants to grab the torch and probably it will be useful for you, dear user in the meanwhile in the form of the beta releases.

 

Progress indicator

As of now I switch from batch release strategy to immediate update. This means: I will commit each change one by one to the SourceForge repository as soon as the change is ready instead of buffering up lots of changes locally and commit them in a big changeset.
So, if you look for the repository changesets and the tickets then you can watch the progress of the project closely.

I also make use of the tickets in the completion of the various fixes and tasks:

https://sourceforge.net/p/euaeppcjit/tickets

I added milestones to the tickets, so you can get a feeling of the upcoming beta and the included changes, fixes.
Open tickets are defining the majority of the outstanding work. I am currently working on the accepted ticket, while pending tickets are already committed to the repository, but not released in binary form yet. Released tickets are the closed ones.

For PPCJITBET03 you can find the planned changes here:

https://sourceforge.net/p/euaeppcjit/tickets/milestone/PPCJITBETA03

There is also a milestone named "PARKED" which is a holding box for the various bugs and problems that are not considered for this project (yet).

 

Thanks

Finally, big thanks goes to: Thunder, kas1e, MickJT, Fab, Tobias Netzel and Mike Blackburn for helping me with lots of things regarding bug finding, fixing, platform support and constantly watching out for the updates on the repository.

I am still waiting for any (detailed) bug reports, just have a good read of the README file before you jump to your email client.

Footnote
*There is a fine print here: I was struggling with CMP2 instruction and finally I gave up after a couple days. The binary code for the instruction is bundled with CHK2 and I couldn't figure out how solve the exception handling for that. So, this instruction remains unimplemented for now, not a big deal luckily.

Stepping into the fourth year

And here we are again: the development of the PowerPC JIT compiling for E-UAE

passed the third year, stepping into the fourth.

Original image by OpenClipart

But what a year! Yes, finally we are getting very close to the Real Thing(tm). Even a beta version is available from the project, although probably it is not particularly useful to anybody who is not the adventurous type. There are many bugs to chase and also there is room to improve.

I know everybody is eagerly waiting for the second beta. It was prepared more than two weeks ago, the sources are released to SourceForge, the post is ready to launch...

But then suddenly Mike Blackburn came back to me (big thanks for that) and mentioned a show-stopper bug on Mac. In the meanwhile I confirmed that the same bug does exist on MorphOS too, so probably it is just a coincidence that the emulation works on AmigaOS4.

Right now I am trying to figure out what is going on, but for that I needed access to a Mac. So, it took me a while to set up everything and getting ready to debug this. After spending a couple days on chasing my own tail already I am no closer to the solution yet, but I promise I put all my free time into this.

Please relax and enjoy the beautiful Summer at the Southern Hemisphere and try not to freeze to death in the Winter at the North.

One small step for mankind, a giant leap for the project

I have no idea how did I manage to achieve this much in this update, but it is certainly a confident step forward. For this time the list is long and diversified:

• Implementation of Bcc.x addr, BCHG.B Dx,mem, BCHG.L Dx,Dy, BCLR.B Dx,mem, BCLR.L Dx,Dy, BRA.x abs, BSET.B Dx,mem, BSET.L Dx,Dy, BSR.x abs, BTST.B #imm,mem, BTST.B Dx,#imm, BTST.B Dx,mem, BTST.L Dx,Dy, CMP.x #imm,mem, CMP.x mem,Dy, CMP.x reg,Dy, CMPA.L reg,Ax, CMPA.W reg,Ax, CMPA.x mem,Ay, DBF.W Dx,addr, EOR.x #imm,mem, JMP.L abs, JMP.L mem, JSR.L abs, JSR.L mem, NEG.x mem, NOT.x mem, RTS, TAS.B Dx, TAS.B mem instructions.
• Cache invalidation fix for OSX 10.3.9 and below. (Thanks to Mike Blackburn again.)
• Fixed mask handling in BCHG.B Dx,mem instruction.
• Fixed missing register mapping in ASL.x #imm,Dy implementation.
• Fixed input dependency overwriting in certain memory-related allocation functions.
• Fixed dependency for destination memory pointer register in special memory reading.
• Fixed post address handler for condition code addressing modes, previously it might crash or call some random handler from the other addressing modes.
• Fixed instructions where temporary registers are allocated but not free'ed.
• Optimized masking for register to register bit instruction.
• Optimized the temporary register usage in helper_test_bit_register_register function.
• Optimized flag extraction in several shifting operation.
• Branch scheduling is more flexible: adding multiple interleaved branches is possible.
• Comment on missing implementation for an exception on loading odd address into PC.

A few highlights

First of all, let me brag around a little bit about the number of freshly implemented instructions. Right now 237 instructions are implemented out of 388, a solid 61% is done. (Previously the ratio was ~46%.)

More MacOSX versions are supported now, Mike fixed up the cache flushing a little bit and added the pre-10.4 versions too. Please read the included README file regarding the compiling instructions.

While I was working on the instructions I discovered a few bugs and glitches, which are now fixed in this release thus improving the overall stability.

I have also managed to optimize the compiled code for some instructions. Together with the implementation of some yet missing instructions the results for the Mandelbrot test (mandel_though_hw.kick.gz among the test kick files) improved a bit compared to the previous results:

Interpretive: 108 seconds (no change there...);
JIT compiled without optimization: 44 seconds (previously it was 52 seconds);
JIT compiled with optimization: 27 seconds (previously it was 32 seconds).

That was the time for the self-polishing and now back to work...

Mac and cheese... err... Linux

Big thanks to Mike Blackburn for some fixes for the Macintosh support and for implementing the Linux PPC support! Well done, Mike.

After this (and this) update Mac OSX 10.4 is supported too: the instruction cache flush needed a different implementation.

Also Linux PPC users can benefit from the PowerPC JIT.

The more the merrier.

Jinxed it (more Apples)

Speaking of apples. As it seems I managed to upset the Gods with the previous post somehow: they unleashed their wrath on me. Apple had released iOS6.

The funny part was: the developer pre-release was all hunky dory with our app, but then came the final release and things have changed dramatically overnight. Since I am working as a mobile app developer recently, I had to spend lot of time on it, even some weekend day(s).Okay, I stop grumbling, we are finally managed to take everything under control. It was stressful and annoying.

So, I guess you might guess why the recent update is so thin: when I finally managed to get home I was fully drained, I had no strength to look at one more line of code.
I have implemented three missing addressing mode which are included in this update. I also started to work on the complex addressing modes (also called 68020 addressing modes), which are... well... really complex. I am less than half way thru with them, but in the meanwhile I didn't want to hold back this:

G5 support

While I suffered in deep agony, luckily Tobias Netzel was busy again and managed to fix up E-UAE for the G5 (PowerPC 970) processors. He got rid of the mcrxr emulated unsupported instruction (see the chapter about mcrxr) for this processor type. He also done some optimizations regarding the microcoded instruction usage, but probably we need to do more about it once.

The situation was very much similar to the 68060 and some unsupported FPU instructions which were emulated by the OS. If you remember that time how much Oxypatcher increased the speed of some floating point calculation intensive apps you can understand why E-UAE was soo sluggishly slow on G5 before: almost half of the emulated instructions make use of the mcrxr instruction for emulating some arithmetic flags.
With the recent changes this PPC instruction is not used for anything if the emulator was compiled for G5 and the fix helps a lot on the interpretive emulation too.

Some benchmarks from Tobias using the Mandelbrot test (G5 - 2.1 GHz):

Interpretive emulation:

• using mcrxr: 5:02 secs
• without mcrxr: 59 secs

JIT without flag optimization:

• using mcrxr: 7:46 secs (yes, even slower than the interpretive...)
• without mcrxr: 27.5 secs

JIT with flag optimization:

• using mcrxr: 1:47 sec
• without mcrxr: 18 sec

Well done, Tobias! The PPC Mac users will be grateful again.
By the way, these changes might have affect on the Cell PPE and Xenon processors too. Any volunteering developers?

Bounty

Recently, I had a cautious look on the bounty page for the project and I was a bit shocked by the fact that Mr. William Morris donated 1000 EUR. That was half of the previously collected bounty. Very generous of you.

Maybe this post is not the best opportunity to thank you all for your support. I hope I fulfill the expectations rather sooner than later.

Apple from the Tree

I try to keep this post short. New update is available:

• MacOSX Darwin PowerPC support is implemented.
• Fixed address distance calculation for the PowerPC native relative branch instructions.
• Refactored the boolean values to use TRUE/FALSE preprocessor defines.

Big thanks goes to Tobias Netzel, who implemented the MacOSX support for the JIT compiling and helped me chasing down one more sneaky bug.

Some details on the bug: previously the negative relative branch calculation was completely wrong, which caused jumping to invalid addresses among certain situations and made the application crash.

END-OF-TRANSMISSION

Bug! *Splat*

Every developer knows the feeling when finally he/she finds a bug and slaps to the forehead while mumbling: "How on earth was this thing ever been working?..."

Well, it just happened to me, I have fixed a bug that stopped the ROM from booting. It was a rather stupid mistake (as usually); for the details check out the update.
In this other minor update I have fixed one more nuance with the wrong addresses in the dumped PowerPC code log.

Right now the emulation advances even further in the booting process than before, when it stops with this cryptic message:

Compiling error: instruction or addressing mode is not implemented, but marked as implemented: 0x323b

Unfortunately, this is true: this is a move instruction with complex addressing mode (68020), which one is not implemented yet.
Since the move instruction itself is marked as supported and all the addressing mode is listed in the descriptor it busts me big time and calls me a liar. Fair enough.

I promise that I implement all the missing addressing modes soon. Honest.

I was so excited that I tried to run the ROM without compiling the instructions but in this case I got back to the previous problem: the reboot loop. :(

At least one bug was squashed again.

In the meanwhile Tobias managed to port the JIT to PowerPC MacOSX. For the speed check out his comment. I hope he sends me the changes soon and I can add it into the main source repository.