That is more a myth and makes no sense and is just the result of people benchmarking random nonsense. mitigations=off is 1-2% faster compiling open source software. On every not last gen silicon mitigations actually have a major impact on performance.