Does it matter if I replace 8 ohm speakers with 4 ohm ones?
Yes. If you are only replacing the speakers. Since the amp powering the speakers will output at 8ohms and the new speakers will be 4ohms, your ohm load will be high to low. This will cause the amp to work twice as hard and it will overheat, if not fail.