I'm trying to find homography between two images. If I calculate homography based on only corresponding points on the ground then the transformation works well only for the points on the ground.

If I calculate homography based on only corresponding points on not on the ground then the transformation works well only for the points on that level.

How do I get homography that will work for any point on the image? Is it even possible ?