Pressure is only part of the story though. At altitude you are starting out with lower air density. Less molecules of air into the compressor, engine, and turbine...which is why turbo speeds are higher at altitude.
You need enough altitude margin in the compressor to deal with the change. If you are running near the top of the map at high boost / high turbo speed at sea level, and drive up to 5000 ft keeping your boost pressure set the same, there are a few things that can happen.
The turbo can go into overspeed and fail mechanically. Bad, mmmkay. But even if there's enough margin to stay away from that, you can get up into the area of poor efficiency at higher pressure ratios. The same effect happens here as when you find the boost limit of a given compressor at any altitude. The turbo spins faster but charge air temps increase so much that power plateaus. The whole system - engine, turbo, intercooler - is working much harder but power is only incrementally increasing, if at all.
Higher altitude will have a snowball effect on engine power output because there is much more energy required to make the same boost pressure. Exhaust backpressure goes up, VE goes down, charge air temps go up, knock threshold is reduced. Engine gets hotter and heavily loaded and makes less power.
Granted, if the turbo is matched to the engine and power goals with enough altitude margin at the beginning, then yes - you won't notice much of a difference. This is why modern VW turbo gas engines are so underrated at stock power levels: they have a ton of altitude margin and allegedly don't hardly de-rate at all. The GTI, Audi A4, etc. is what I'm referring to with the 2.0T and now 1.8T.
