Those are the basic arguments, however, Musk is correct, technology will speed up not slow down, as is the nature of technology.
Quantum processes will be understood and once that occurs, quantum computers will far exceed anything we could ever hope to simulate today.
A singularity will occur, like the one that started this simulation.
http://www.lgcnsblog.com/features/quantum-computers-a-step-above-your-average-computer/Pros and Cons of D-Wave
Pros
According to Google’s D-Wave benchmark in January of 2014, quantum computers show much higher speed in solving optimization problems compared to general PCs. Although there are reports that say its speed is sometimes slower than PCs, they seem to be faster than PCs, on average, to solve optimization problems involving data with regularity[2].
Cons
D-WAVE has three big cons.
First, despite being called the world’s first commercial quantum computer, it’s a shame that D-Wave is not considered a real quantum computer at the same time. This is because it’s designed to have an external computer read the processing results from the quantum CPU. Some may refer to it as just a “half-quantum” computer which consists of the regular workstation with a qubit CPU on the side[3].
Second, the CPU generates heat while operating, and the noise made while running the cooler to lower the temperature can create computing errors. The size of the computer is also quite large due to the big cooling unit on top to stabilize the temperature of absolute zero.
Finally, D-Wave is made based on the tunnel effect of quantum annealing unlike the formerly defined quantum computers. The tunnel effect here means the phenomenon where a particle stochastically tunnels through the energy barrier higher than its own potential energy. As a result, its computing speed is not overwhelmingly faster than in existing computers except for in particular calculating operations.