Traditional methods of generating vortex beams based on metasurfaces consist mainly in modulating propagation phase or geometric phase. Here, by introducing detour phase, we propose the construction of dual-polarized vortex beam generators in the form of metasurface and metagrating (MG). The phase is modulated through moving the position of meta-atoms instead of varying the geometrical parameters or rotating the unit cells. To use detour phase, two kinds of unit cells are designed to achieve specific diffraction order. Each unit can arbitrarily and independently adjust the operation frequency and diffraction angle of transverse electric (TE) and transverse magnetic (TM) polarizations. Two vortex beam generators are designed and fabricated with different topological charges carried by orthogonal polarizations. To demonstrate the ability to independently manipulate, two polarizations of the generator based on MG are designed in different frequency bands. Both the simulation and experimental results validate the proposed method, showing great potential for polarization division multiplexing in orbital angular momentum (OAM) communication systems.