Actually abstract algebra was my favorite of the late level courses. It was just as abstract as everything else, but we started slow enough that I could brute force memorize things (abelian and groups and things), which I guess is the only way to learn that sort of material. Plus, with abstract algebra at least, I could see some real world uses. You can do rotations on a cube and understand that there are really only n transformations. Mostly I was just irked that I'd spent like 15 years of my life learning what I thought was math, only to be told that at best what I knew would be a hindrance. I hate memorizing things (I failed a lot of spelling tests in school because I felt like memorizing the spellings would be cheating), and that's really what you have to do when you deal with that stuff.
For instance, in AP physics I had to memorize very little. Most equations are just transformations of other equations, so if you understand the basic principles and a memorize a few key equations (or put them in your graphic calculator's memory (I don't consider this cheating, my calculator is my brain's external hard drive )), you can derive anything you need during the test. I think during one quiz on relativity I managed to derive E = m c^2 (though I think the question was on time dilation so this wasn't all that useful). But there's no deriving what an group means exactly. You can understand the identity principle, understand the idea of an operation, of inverses, of associativity, and yet be no closer to understanding how to answer a test question: "prove that natural numbers under addition form a group". It's just vocab word after vocab word. Physics was like that, too, but I had an easier time of that for some reason.
You could probably teach abstract algebra in elementary or middle school. There really isn't a lot of prereq knowledge required, beyond the ability to understand how proofs work. And there's nothing inherently difficult about the material beyond memorization, which is what kids that age do for school anyway. You could have applied math, which leads to calculus and engineering, and pure mathematics, which deals with proofs. Geometry like it's taught in early highschool would probably be a good intro course for pure math, since that's essentially what it is right now anyway. I would have received it a lot better if I was 10 years younger and it wasn't the last few classes before graduation.