It's a combination of the language and the code base itself. Basically, error handling in VB6 is terrible, and there's all sorts of weird performance overhead on things. It's also impossible to merge changes to forms created by the form creator, which is an important thing for a collaborative open source project. Everything it was meant to be, VB.NET does better.
The code base itself is sloppy spaghetti tied together with duct tape and hope. None of it is tested, so if you make one change, even seemingly innocuous, it might (and probably will) break everything. We could fix that by adding tests, but there's no good testing framework for VB6, and even if there were, just changing the code enough to make it testable would probably break things. There are global variables all over the place that manage the simulation state, and of course all sorts of random things will poke and prod at this state from everywhere and anywhere in the code.
The few times I've tried to do anything in the codebase have always been 10000% harder than they should be. It's a classic example of a codebase that grew larger than it was architected to grow.
That's why I hate it.