Thanks for the clarification. Indeed, in the simulations you mentioned the results seem not to be different. Although I would remain open to the possibility that it is the model that distorts the analysis, here are some notes that might also be the case.
First thing I noticed is the use of semi-2d mesh. This is somewhat legal (in a physical sense), thought we are applying 3d models to a restrained system. Immediately it points to the 3-dimensional nature of turbulence. This kind of meshes are in fact used in OpenFOAM, but they need a specific definition on the mesh structure level. It is not available at the moment in SimScale online meshers (you could download the mesh, manually change the polyMesh/boundary file, re-upload the mesh and use 2D-empty boundary condition for a 2d simulation).
Next thing that drew my attention was the behavior of the flow field near the inlet (see figures below). We expect some unstable behavior near the inlet, but this seems quite strange. I would again attribute it to the semi-2d mesh. (this is an assumption)
Finally, lets consider the size of the system. The block of air is 500m high, and wind blows max with 18m/s. Now, our roughness changes it's height from 0.24m (Run 1) to 0.01m (Run 3). Although this is a significant absolute difference when you compare the two, if we compare these values to the size of the whole system, they are changes are not that big. (Here I have to make a disclaimer- I have not deeply investigated the roughness model, nor did I look for scientific references regarding the expected behavior of the system).
To sum up, having seen the differences in flow patterns in Simulation 1 (I am pretty sure I compared 2 simulations with same inlet velocity), I would still be hesitant to blame the model for lack of expected results.
First I would check the system's behavior on a 3d mesh (also with symmetry conditions, but given few dozens of cells in the Y direction), and by using bigger differences in roughness relative to geometry height (say 0.01m vs 1m?)
Hope this moves the study forward!