Faculty Profiles - RAO Anyi | The Hong Kong University of Science and Technology

Assistant Professor
Division of Arts and Machine Creativity
Division of Emerging Interdisciplinary Areas

Associate Director of Media Intelligence Research Center

Research Interest

AI for creativity

Computer vision

Human-computer interaction

Computer graphics

Film studies

Publications

2025 3

CineVision: An Interactive Pre-visualization Storyboard System for Director-Cinematographer Collaboration

Wei, Zheng; Wu, Hongtao; Zhang, Lvmin; Xu, Xian; Zheng, Yanfeng; Hui, Pan; Agrawala, Maneesh; Qu, Huamin; Rao, Anyi
Conference paper

Scaling In-the-Wild Training for Diffusion-based Illumination Harmonization and Editing by Imposing Consistent Light Transport

Zhang, Lvmin; Rao, Anyi; Agrawala, Maneesh
Conference paper

VideoRepainter: Keyframe-Guided Creative Video Inpainting

Guo, Yuwei; Yang, Ceyuan; Rao, Anyi; Meng, Chenlin; Bar-tal, Omer; Ding, Shuangrui; Agrawala, Maneesh; Lin, Dahua; Dai, Bo
Conference paper

2024 5

AnimatedDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning

Guo, Yuwei; Yang, Ceyuan; Rao, Anyi; Liang, Zhengyang; Wang, Yaohui; Qiao, Yu; Agrawala, Maneesh; Lin, Dahua; Dai, Bo
Conference paper

Cinematic Behavior Transfer via NeRF-based Differential Filming

Jiang, Xuekun; Rao, Anyi; Wang, Jingbo; Lin, Dahua; Dai, Bo
Conference paper

CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers

Proceedings of Machine Learning Research, v. 235, July 2024, p. 44960-44990
Shi, Dachuan; Tao, Chaofan; Rao, Anyi; Yang, Zhendong; Yuan, Chun; Wang, Jiaqi
Conference paper

ScriptViz: A Visualization Tool to Aid Scriptwriting based on a Large Movie Database

UIST '24: Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology / edited by Lining Yao, Mayank Goel, Alexandra Ion, Pedro Lopes. New York, United States : Association for Computing Machinery, 2024, p. 1-13, Article number 21
Rao, Anyi; Chou, Jean-peïc; Agrawala, Maneesh
Conference paper

SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), v. 15100 LNCS, October 2024, p. 330-348
Guo, Yuwei; Yang, Ceyuan; Rao, Anyi; Agrawala, Maneesh; Lin, Dahua; Dai, Bo
Conference paper

2023 7

A Coarse-to-Fine Framework for Automatic Video Unscreen

IEEE Transactions on Multimedia, v. 25, February 2022, article number 9709668, p. 2723-2733
Rao, Anyi; Xu, Linning; Li, Zhizhong; Huang, Qingqiu; Kuang, Zhanghui; Zhang, Wayne; Lin, Dahua
Article

Adding Conditional Control to Text-to-Image Diffusion Models

Proceedings of the IEEE International Conference on Computer Vision / IEEE. Piscataway, NJ : IEEE, 2023, p. 3813-3824, article number 10377881
Zhang, Lvmin; Rao, Anyi; Agrawala, Maneesh
Conference paper

Automatic Conversion of Music Videos into Lyrics Videos

UIST '23: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology / edited by Sean Follmer, Jeff Han, Jürgen Steimle, Nathalie Henry Riche. New York, NY : Association for Computing Machinery, 2023, p. 1-11, article number 13
Ma, Jiaju; Rao, Anyi; Wei, Li-Yi; Kazi, Rubaiat Habib; Shin, Valentina; Agrawala, Maneesh
Conference paper

Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Production

SIGGRAPH '23: ACM SIGGRAPH 2023 Posters / edited by Erik Brunvand, James Tompkin. New York, NY : Association for Computing Machinery, 2023, p. 1-2, article number 40
Rao, Anyi; Jiang, Xuekun; Guo, Yuwei; Xu, Linning; Yang, Lei; Jin, Libiao; Lin, Dahua; Dai, Bo
Conference paper

HireVAE: An Online and Adaptive Factor Model Based on Hierarchical and Regime-Switch VAE

Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, v. 2023-August, August 2023, p. 4903-4911
Wei, Zikai; Rao, Anyi; Dai, Bo; Lin, Dahua
Conference paper

Self-Supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences

Proceedings of the 37th AAAI Conference on Artificial Intelligence, AAAI 2023, v. 37, (3), June 2023, p. 3825-3833
Zhou, Yujie; Duan, Haodong; Rao, Anyi; Su, Bing; Wang, Jiaqi
Conference paper

Zero-shot Skeleton-based Action Recognition via Mutual Information Estimation and Maximization

MM '23: Proceedings of the 31st ACM International Conference on Multimedia / Association for Computing Machinery. New York, NY : Association for Computing Machinery, 2023, p. 5302-5310
Zhou, Yujie; Qiang, Wenwen; Rao, Anyi; Lin, Ning; Su, Bing; Wang, Jiaqi
Conference paper

2022 5

Jointly Learning the Attributes and Composition of Shots for Boundary Detection in Videos

IEEE Transactions on Multimedia, v. 24, June 2021, article number 9464668, p. 3049-3059
Jiang, Xuekun; Jin, Libiao; Rao, Anyi; Xu, Linning; Lin, Dahua
Article

AutoGPart: Intermediate Supervision Search for Generalizable 3D Part Segmentation

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition / IEEE. Piscataway, NJ : IEEE, 2022, p. 11614-11624, article number 9879431
Liu, Xueyi; Xu, Xiaomeng; Rao, Anyi; Gan, Chuang; Yi, Li
Conference paper

BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), v. 13692 LNCS, November 2022, p. 106-122
Xiangli, Yuanbo; Xu, Linning; Pan, Xingang; Zhao, Nanxuan; Rao, Anyi; Theobalt, Christian; Dai, Bo; Lin, Dahua
Conference paper

Shoot360: Normal View Video Creation from City Panorama Footage

SIGGRAPH '22: ACM SIGGRAPH 2022 Conference Proceedings / edited by Munkhtsetseg Nandigjav, Niloy J. Mitra, Aaron Hertzmann. New York, NY : Association for Computing Machinery, 2022, p. 1-9, article number 13
Rao, Anyi; Xu, Linning; Lin, Dahua
Conference paper

Temporal and Contextual Transformer for Multi-Camera Editing of TV Shows

Rao, Anyi; Jiang, Xuekun; Wang, Sichen; Guo, Yuwei; Wu, Xiaoyu; Lin, Dahua; Dai, Bo; Jin, Libiao
Conference paper

2021 1

BlockPlanner: City Block Generation with Vectorized Graph Representation

Proceedings of the IEEE International Conference on Computer Vision / IEEE. Piscataway, NJ : IEEE, 2021, p. 5057-5066, article number 9710289
Xu, Linning; Xiangli, Yuanbo; Rao, Anyi; Zhao, Nanxuan; Dai, Bo; Liu, Ziwei; Lin, Dahua
Conference paper

2020 4

A Local-to-Global Approach to Multi-Modal Movie Scene Segmentation

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition / IEEE. Piscataway, NJ : IEEE, 2020, p. 10143-10152, article number 9157529
Rao, Anyi; Xu, Linning; Xiong, Yu; Xu, Guodong; Huang, Qingqiu; Zhou, Bolei; Lin, Dahua
Conference paper

A Unified Framework for Shot Type Classification Based on Subject Centric Lens

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), v. 12356 LNCS, November 2020, p. 17-34
Rao, Anyi; Wang, Jiaze; Xu, Linning; Jiang, Xuekun; Huang, Qingqiu; Zhou, Bolei; Lin, Dahua
Conference paper

MovieNet: A Holistic Dataset for Movie Understanding

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), v. 12349 LNCS, October 2020, p. 709-727
Huang, Qingqiu; Xiong, Yu; Rao, Anyi; Wang, Jiaze; Lin, Dahua
Conference paper

Online Multi-modal Person Search in Videos

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), v. 12357 LNCS, October 2020, p. 174-190
Xia, Jiangyue; Rao, Anyi; Huang, Qingqiu; Xu, Linning; Wen, Jiangtao; Lin, Dahua
Conference paper

2018 1

HotFlip: White-Box Adversarial Examples for NLP

Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, v. 2, July 2018, p. 31-36
Ebrahimi, Javid; Rao, Anyi; Lowd, Daniel; Dou, Dejing
Conference paper

Conference paper 3

CineVision: An Interactive Pre-visualization Storyboard System for Director-Cinematographer Collaboration

Wei, Zheng; Wu, Hongtao; Zhang, Lvmin; Xu, Xian; Zheng, Yanfeng; Hui, Pan; Agrawala, Maneesh; Qu, Huamin; Rao, Anyi

Scaling In-the-Wild Training for Diffusion-based Illumination Harmonization and Editing by Imposing Consistent Light Transport

Zhang, Lvmin; Rao, Anyi; Agrawala, Maneesh

VideoRepainter: Keyframe-Guided Creative Video Inpainting

Guo, Yuwei; Yang, Ceyuan; Rao, Anyi; Meng, Chenlin; Bar-tal, Omer; Ding, Shuangrui; Agrawala, Maneesh; Lin, Dahua; Dai, Bo

Conference paper 5

AnimatedDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning

Guo, Yuwei; Yang, Ceyuan; Rao, Anyi; Liang, Zhengyang; Wang, Yaohui; Qiao, Yu; Agrawala, Maneesh; Lin, Dahua; Dai, Bo

Cinematic Behavior Transfer via NeRF-based Differential Filming

Jiang, Xuekun; Rao, Anyi; Wang, Jingbo; Lin, Dahua; Dai, Bo

CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers

Proceedings of Machine Learning Research, v. 235, July 2024, p. 44960-44990
Shi, Dachuan; Tao, Chaofan; Rao, Anyi; Yang, Zhendong; Yuan, Chun; Wang, Jiaqi

ScriptViz: A Visualization Tool to Aid Scriptwriting based on a Large Movie Database

UIST '24: Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology / edited by Lining Yao, Mayank Goel, Alexandra Ion, Pedro Lopes. New York, United States : Association for Computing Machinery, 2024, p. 1-13, Article number 21
Rao, Anyi; Chou, Jean-peïc; Agrawala, Maneesh

SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), v. 15100 LNCS, October 2024, p. 330-348
Guo, Yuwei; Yang, Ceyuan; Rao, Anyi; Agrawala, Maneesh; Lin, Dahua; Dai, Bo

Article 1

A Coarse-to-Fine Framework for Automatic Video Unscreen

IEEE Transactions on Multimedia, v. 25, February 2022, article number 9709668, p. 2723-2733
Rao, Anyi; Xu, Linning; Li, Zhizhong; Huang, Qingqiu; Kuang, Zhanghui; Zhang, Wayne; Lin, Dahua

Conference paper 6

Adding Conditional Control to Text-to-Image Diffusion Models

Proceedings of the IEEE International Conference on Computer Vision / IEEE. Piscataway, NJ : IEEE, 2023, p. 3813-3824, article number 10377881
Zhang, Lvmin; Rao, Anyi; Agrawala, Maneesh

Automatic Conversion of Music Videos into Lyrics Videos

UIST '23: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology / edited by Sean Follmer, Jeff Han, Jürgen Steimle, Nathalie Henry Riche. New York, NY : Association for Computing Machinery, 2023, p. 1-11, article number 13
Ma, Jiaju; Rao, Anyi; Wei, Li-Yi; Kazi, Rubaiat Habib; Shin, Valentina; Agrawala, Maneesh

Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Production

SIGGRAPH '23: ACM SIGGRAPH 2023 Posters / edited by Erik Brunvand, James Tompkin. New York, NY : Association for Computing Machinery, 2023, p. 1-2, article number 40
Rao, Anyi; Jiang, Xuekun; Guo, Yuwei; Xu, Linning; Yang, Lei; Jin, Libiao; Lin, Dahua; Dai, Bo

HireVAE: An Online and Adaptive Factor Model Based on Hierarchical and Regime-Switch VAE

Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, v. 2023-August, August 2023, p. 4903-4911
Wei, Zikai; Rao, Anyi; Dai, Bo; Lin, Dahua

Self-Supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences

Proceedings of the 37th AAAI Conference on Artificial Intelligence, AAAI 2023, v. 37, (3), June 2023, p. 3825-3833
Zhou, Yujie; Duan, Haodong; Rao, Anyi; Su, Bing; Wang, Jiaqi

Zero-shot Skeleton-based Action Recognition via Mutual Information Estimation and Maximization

MM '23: Proceedings of the 31st ACM International Conference on Multimedia / Association for Computing Machinery. New York, NY : Association for Computing Machinery, 2023, p. 5302-5310
Zhou, Yujie; Qiang, Wenwen; Rao, Anyi; Lin, Ning; Su, Bing; Wang, Jiaqi

Article 1

Jointly Learning the Attributes and Composition of Shots for Boundary Detection in Videos

IEEE Transactions on Multimedia, v. 24, June 2021, article number 9464668, p. 3049-3059
Jiang, Xuekun; Jin, Libiao; Rao, Anyi; Xu, Linning; Lin, Dahua

Conference paper 4

AutoGPart: Intermediate Supervision Search for Generalizable 3D Part Segmentation

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition / IEEE. Piscataway, NJ : IEEE, 2022, p. 11614-11624, article number 9879431
Liu, Xueyi; Xu, Xiaomeng; Rao, Anyi; Gan, Chuang; Yi, Li

BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), v. 13692 LNCS, November 2022, p. 106-122
Xiangli, Yuanbo; Xu, Linning; Pan, Xingang; Zhao, Nanxuan; Rao, Anyi; Theobalt, Christian; Dai, Bo; Lin, Dahua

Shoot360: Normal View Video Creation from City Panorama Footage

SIGGRAPH '22: ACM SIGGRAPH 2022 Conference Proceedings / edited by Munkhtsetseg Nandigjav, Niloy J. Mitra, Aaron Hertzmann. New York, NY : Association for Computing Machinery, 2022, p. 1-9, article number 13
Rao, Anyi; Xu, Linning; Lin, Dahua

Temporal and Contextual Transformer for Multi-Camera Editing of TV Shows

Rao, Anyi; Jiang, Xuekun; Wang, Sichen; Guo, Yuwei; Wu, Xiaoyu; Lin, Dahua; Dai, Bo; Jin, Libiao

Conference paper 1

BlockPlanner: City Block Generation with Vectorized Graph Representation

Proceedings of the IEEE International Conference on Computer Vision / IEEE. Piscataway, NJ : IEEE, 2021, p. 5057-5066, article number 9710289
Xu, Linning; Xiangli, Yuanbo; Rao, Anyi; Zhao, Nanxuan; Dai, Bo; Liu, Ziwei; Lin, Dahua

Conference paper 4

A Local-to-Global Approach to Multi-Modal Movie Scene Segmentation

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition / IEEE. Piscataway, NJ : IEEE, 2020, p. 10143-10152, article number 9157529
Rao, Anyi; Xu, Linning; Xiong, Yu; Xu, Guodong; Huang, Qingqiu; Zhou, Bolei; Lin, Dahua

A Unified Framework for Shot Type Classification Based on Subject Centric Lens

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), v. 12356 LNCS, November 2020, p. 17-34
Rao, Anyi; Wang, Jiaze; Xu, Linning; Jiang, Xuekun; Huang, Qingqiu; Zhou, Bolei; Lin, Dahua

MovieNet: A Holistic Dataset for Movie Understanding

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), v. 12349 LNCS, October 2020, p. 709-727
Huang, Qingqiu; Xiong, Yu; Rao, Anyi; Wang, Jiaze; Lin, Dahua

Online Multi-modal Person Search in Videos

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), v. 12357 LNCS, October 2020, p. 174-190
Xia, Jiangyue; Rao, Anyi; Huang, Qingqiu; Xu, Linning; Wen, Jiangtao; Lin, Dahua

Conference paper 1

HotFlip: White-Box Adversarial Examples for NLP

Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, v. 2, July 2018, p. 31-36
Ebrahimi, Javid; Rao, Anyi; Lowd, Daniel; Dou, Dejing

2018 1

HotFlip: White-Box Adversarial Examples for NLP

Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, v. 2, July 2018, p. 31-36
Ebrahimi, Javid; Rao, Anyi; Lowd, Daniel; Dou, Dejing
Conference paper

Teaching Assignment

AMCC5000	Creative Convergence: Foundations of Arts and Machine Creativity
AMCC5140	AI for Visual Arts and Creativity
AMCC6950A	Special Projects in Arts and Machine Creativity
EMIA6950F	Independent Study

EMIA6500K

Visual Computing for Visual Content Creation

EMIA6500H

AI for Visual Content Creation

No Teaching Assignments

Research Interest

Publications

2025 3

2024 5

2023 7

2022 5

2021 1

2020 4

2018 1

Conference paper 3

Conference paper 5

Article 1

Conference paper 6

Article 1

Conference paper 4

Conference paper 1

Conference paper 4

Conference paper 1

2018 1

Teaching Assignment

Research Postgraduate (RPG) Supervision

From January 2022 to December 2025 (As of 17 October 2025)

Current RPGs

Projects

From January 2023 to December 2025

Your browser is out of date!