Stupid Encryption Idea

This is an incredibly gross oversimplification of the basic process as it stands now, to my knowledge, with no research, and little sleep.
I want to communicate with you.
My system and your system talk back and forth agreeing on two stupidly large prime numbers and their multiple or some other silliness.
Now if the number generator is compromised, without knowing anyone’s numbers it’s an open door.
As technology improves, brute forcing the numbers is easy.
If one of the two systems is compromised, the system is broken.

So here comes the (clever?). What if we only send a password once. Ever. Have it be one stupidly long password, and this password by rules associated with the password changes what hassles to each character. Like h moves up 5, and 3 is free. Once we have this password, we never use it. On each end of the communication, we individually choose whatever big number to mangle the password, we already know the rules and we know what the output looks like for the first block, which would be our stupid long password. So to decrypt, each party just brute forces. I’m sure someone already had done this, who knows, maybe that it’s the way it’s done.