Hi Chris,
Short answer = you don't
have to remove the whole panel completely if you don't want to. You unscrew the door panel as IF you were going to lift it off, but there is enough space so that when the it then "hangs loose", you can lift enough of the bottom away to get at the stock speakers to replace them...without having to remove the door panel completely off. This isn't so much an issue on the fronts, but at the rear the windows have a small bit of triangular plastic at the top that means taking them off completely is quite difficult.
Also, the speakers are 16.5cm max - I chose 16.5cm coax's but the plastic "lip" around the circumference bought the total size to 17cm - so be aware that some 16.5cm speakers may physically be a bit larger. It had to be dremmeled off with mine in order to get the size back to 16.5 in order for them to fit the spacers.
I've done a brief guide with piccies
here for speaker replacement.