Didactic Scrambled Egg Cipher Pre-Warmup

Well, I've been working on this challenge for a few days, but I'm wondering if there's a faster way to get the key than to go through all possibilities (256*256*256*256 is a bit too much for my computer to do in a reasonable amount of time). I think from some tests I did that I'm properly undoing all the other steps in the encryption, but I won't say how because that would give some of the challenge away. I mean, there must be some bits in the final answer that can be prematurely calculated with a single byte of the key, and you could probably somehow optimize on that.

Brute force worked for me. It doesn't take forever even on a slow machine with an interpreted language. You only have to detect failures fast.

Failures as in keys that are obviously wrong?

Yeah... I'll try that.

I'm wondering if there's actually a way to reverse the algorithm. The final AND mask which truncates it to 3 bytes is obviously lossy.. but there's many people smarter than me, so I do wonder if it's possible to reverse? I'd be fascinated at the way to reverse it, yet that would be spoiling the solution.

Well, if you don't just reverse the algorythm for decryption, but understand what it does - visualizing the bits affected by the steps helps - you'll see that there is a simple way not to have to test all 256^4 keys.