<aside> 📚
Paper
</aside>
<aside> <img src="https://prod-files-secure.s3.us-west-2.amazonaws.com/484e17ad-85eb-4d55-882f-74f0a5b963eb/e99242ca-56f5-40fe-bf37-705d2db26a0c/arxiv_logo.png" alt="https://prod-files-secure.s3.us-west-2.amazonaws.com/484e17ad-85eb-4d55-882f-74f0a5b963eb/e99242ca-56f5-40fe-bf37-705d2db26a0c/arxiv_logo.png" width="40px" />
arXiv
</aside>
<aside> <img src="https://prod-files-secure.s3.us-west-2.amazonaws.com/484e17ad-85eb-4d55-882f-74f0a5b963eb/59092ea3-5e4c-4a54-a20c-7e4272b5bbff/git_logo.png" alt="https://prod-files-secure.s3.us-west-2.amazonaws.com/484e17ad-85eb-4d55-882f-74f0a5b963eb/59092ea3-5e4c-4a54-a20c-7e4272b5bbff/git_logo.png" width="40px" />
Code
</aside>
The video playback might not work properly, so we uploaded a sample video on YouTube: https://www.youtube.com/watch?v=1c-62jlMh8M
https://www.youtube.com/watch?v=1c-62jlMh8M
Human image animation is a task which transfers a sequence of pose guidance to a character in an image. In this paper, we enhance the life-like experience that the animated human video offers by adding dynamic camera effects. However, simply employing off-the-shelf human image animation or novel view synthesis models in a cascaded manner results in artifacts. As such, we propose a method which jointly learns to perform both tasks. Specifically, our framework obtains an inpainted background image and isolates the human figure from animated frames. Most importantly, we propose a background depth reconstruction module, composed of a depth inpainter and a depth scaler, to obtain an aligned background depth. With the depth information, we convert the background and the foreground into 3D point clouds which are then rendered from target views. For the joint training, we suggest a T-shaped training method which resolves the time-consuming and costly process of collecting animated videos for each different view. In this paper, we prove the effectiveness of our background depth reconstruction module in preventing occlusions, and the superiority of our method over cascade approaches quantitatively and qualitatively.
