I don’t know about RSA, probably I should learn more about that but for the moment let me just say that I still believe what I wrote above to be correct as long as we stick to the “XOR Cipher” case.
The XOR Cipher is actually very simple and easy to understand. There are no prime numbers involved in any way. Probably easiest to think about it at the bit level: each bit in the key tells us whether or not the corresponding bit in the message should be flipped when encrypting/decrypting the message. That’s it.
This means it is theoretically impossible to crack such encryption if the key is never reused.
We could also use a Caesar cipher – that is also theoretically impossible to crack as long as we use a new key for each single character we encrypt. 
To illustrate the point, I have a single-character message here that I will now encrypt. I have used a Caesar cipher, the key is a number that I know but you don’t know (the number of places to move down the alphabet). Here is the encrypted message:
H
Now I challenge you to decrypt the message. Use all the quantum computers you want!