無監督式領域適應 (UDA)
是轉移學習(Transfer Learning)的一個分支。UDA可以在僅有部分領域有標記資料時，將從此領域學習到的資訊應用在另一個沒有標記資料的領域上。這個技術在深度學習上有很大的用處，可以減少人工標記的成本。但是經由UDA方法訓練的模型參數量通常很大，並且在UDA情境下在目標領域的資料沒有標籤，導致一般的模型壓縮方法不可行或是效果不佳。在本論文中，我們提出一種基於估計領域間差異性(Domain Discrepancy Estimation)的方法實作UDA模型的壓縮。我們利用一種特殊的取樣方法，並在取樣的個體間計算在模型上的原始輸出的餘弦相似度，以此估計來源領域和目標領域間的差異性。這個估計值被利用於壓縮後模型最佳化的目標函數中，用以減少來源領域和目標領域間的差異性。我們將這個過程應用在一個迭代式的模型剪枝方法。在ImageCLEF-DA和Office-31資料集上，我們的方法比起既有的方法有更高的平均準確度，並且不需要存取原始模型而只依賴當前的模型。

Unsupervised domain adaptation (UDA) that transfers the knowledge learned from the domain containing well labeled data to the target domain with unlabeled data has received wide attention owing to the expense of data labeling. However, many UDA models are large, and the lack of target data labels makes most of pruning methods inapplicable. In this paper, we proposed a UDA compression algorithm, called DECUDA (Discrepancy Estimation for Compressing UDA models), which is based on a new domain discrepancy estimation method. DECUDA employs the cosine similarity of logits of instances from source domain and target domain, which are selected by a sampling technique, as an estimation of domain discrepancy. The estimated discrepancy is used in the loss function during the fine-tuning in an iterative pruning algorithms. Comparing to other methods which fetch extra information from full-size model, such as knowledge distillation based methods, DECUDA only relies on the current pruned model. Our method achieves higher average accuracy than other works on Office-31 and ImageCLEF-DA dataset.

中文摘要 1
Abstract 2
List of Figures 5
List of Tables 6
1 Introduction 7
2 Related Works 10
2.1 Unsupervised Domain Adaptation . . . . . . . . . . . . . . . . . . . . . 10
2.2 Deep Network Compression Methods . . . . . . . . . . . . . . . . . . . 11
2.3 Compression for UDA Models . . . . . . . . . . . . . . . . . . . . . . . 12
3 DECUDA Algorithm 14
3.1 Overview of DECUDA Algorithm . . . . . . . . . . . . . . . . . . . . . 14
3.1.1 Basic Architecture . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.1.2 Iterative Pruning Algorithm . . . . . . . . . . . . . . . . . . . . 16
3.1.3 Fine-Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2 Domain Discrepancy in UDA models . . . . . . . . . . . . . . . . . . . 17
3.2.1 Target Classification Loss and Domain Discrepancy . . . . . . . 17
3.2.2 Domain Discrepancy Estimation . . . . . . . . . . . . . . . . . . 19
3.2.3 Probabilistic Instance Sampling (PIS) . . . . . . . . . . . . . . . 19
3.2.4 Extra Fine-Tuning with Discrepancy Estimation . . . . . . . . . 21
4 Experiments 23
4.1 Experimental Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.3 Comparison with Other Methods . . . . . . . . . . . . . . . . . . . . . 25
4.4 Validation of Estimated Domain Discrepancy . . . . . . . . . . . . . . . 27
4.4.1 Ablation Study . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5 Conclusion and Future Work 31
References 32

[1] Rishi Bommasani et al. On the Opportunities and Risks of Foundation Models.
2021.
[2] Jiasi Chen and Xukan Ran. “Deep Learning With Edge Computing: A Review”.
Proc. IEEE 107.8 (2019), pp. 1655–1674.
[3] Mei Wang and Weihong Deng. “Deep visual domain adaptation: A survey”.
Neurocomputing 312 (2018), pp. 135–153.
[4] Shai Ben-David et al. “Analysis of Representations for Domain Adaptation”. Advances in Neural Information Processing Systems 19, Proceedings of the Twentieth Annual Conference on Neural Information Processing Systems, Vancouver,
British Columbia, Canada, December 4-7, 2006. Ed. by Bernhard Schölkopf,
John C. Platt, and Thomas Hofmann. MIT Press, 2006, pp. 137–144.
[5] Torsten Hoefler et al. “Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks”. J. Mach. Learn. Res. 22 (2021),
241:1–241:124.
[6] Lei Deng et al. “Model Compression and Hardware Acceleration for Neural
Networks: A Comprehensive Survey”. Proc. IEEE 108.4 (2020), pp. 485–532.
[7] Chaohui Yu et al. “Accelerating Deep Unsupervised Domain Adaptation with
Transfer Channel Pruning”. International Joint Conference on Neural Networks,
IJCNN 2019 Budapest, Hungary, July 14-19, 2019. IEEE, 2019, pp. 1–8.
[8] Le Thanh Nguyen-Meidine et al. “Joint Progressive Knowledge Distillation and
Unsupervised Domain Adaptation”. 2020 International Joint Conference on
Neural Networks, IJCNN 2020, Glasgow, United Kingdom, July 19-24, 2020.
IEEE, 2020, pp. 1–8.
[9] Le Thanh Nguyen-Meidine et al. “Knowledge distillation methods for efficient
unsupervised adaptation across multiple domains”. Image Vis. Comput. 108
(2021), p. 104096.
[10] Xiaoyu Feng et al. “ADMP: An Adversarial Double Masks Based Pruning
Framework For Unsupervised Cross-Domain Compression”. CoRR abs/2006.04127
(2020). arXiv: 2006.04127.
[11] Rang Meng et al. “Slimmable Domain Adaptation”. Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition (CVPR). June 2022,
pp. 7141–7150.
[12] Sinno Jialin Pan et al. “Domain Adaptation via Transfer Component Analysis”.
IEEE Trans. Neural Networks 22.2 (2011), pp. 199–210.
[13] Eric Tzeng et al. “Deep Domain Confusion: Maximizing for Domain Invariance”.
CoRR abs/1412.3474 (2014). arXiv: 1412.3474.
[14] Muhammad Ghifary, W. Bastiaan Kleijn, and Mengjie Zhang. “Domain Adaptive Neural Networks for Object Recognition”. PRICAI 2014: Trends in Artificial Intelligence - 13th Pacific Rim International Conference on Artificial
Intelligence, Gold Coast, QLD, Australia, December 1-5, 2014. Proceedings. Ed.
by Duc Nghia Pham and Seong-Bae Park. Vol. 8862. Lecture Notes in Computer
Science. Springer, 2014, pp. 898–904.
[15] Mingsheng Long et al. “Learning Transferable Features with Deep Adaptation Networks”. Proceedings of the 32nd International Conference on Machine
Learning, ICML 2015, Lille, France, 6-11 July 2015. Ed. by Francis R. Bach
and David M. Blei. Vol. 37. JMLR Workshop and Conference Proceedings.
JMLR.org, 2015, pp. 97–105.
[16] Mingsheng Long et al. “Unsupervised Domain Adaptation with Residual Transfer Networks”. Advances in Neural Information Processing Systems 29: Annual
Conference on Neural Information Processing Systems 2016, December 5-10,
2016, Barcelona, Spain. Ed. by Daniel D. Lee et al. 2016, pp. 136–144.
[17] Mingsheng Long et al. “Deep Transfer Learning with Joint Adaptation Networks”. Proceedings of the 34th International Conference on Machine Learning,
ICML 2017, Sydney, NSW, Australia, 6-11 August 2017. Ed. by Doina Precup
and Yee Whye Teh. Vol. 70. Proceedings of Machine Learning Research. PMLR,
2017, pp. 2208–2217.
[18] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. “Domain Adaptation for
Large-Scale Sentiment Classification: A Deep Learning Approach”. Proceedings
of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 - July 2, 2011. Ed. by Lise Getoor and Tobias
Scheffer. Omnipress, 2011, pp. 513–520.
[19] Muhammad Ghifary et al. “Deep Reconstruction-Classification Networks for
Unsupervised Domain Adaptation”. Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV. Ed. by Bastian Leibe et al. Vol. 9908. Lecture Notes in Computer
Science. Springer, 2016, pp. 597–613.
[20] Konstantinos Bousmalis et al. “Domain Separation Networks”. Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain. Ed. by
Daniel D. Lee et al. 2016, pp. 343–351.
[21] Yaroslav Ganin and Victor S. Lempitsky. “Unsupervised Domain Adaptation
by Backpropagation”. Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015. Ed. by Francis R.
Bach and David M. Blei. Vol. 37. JMLR Workshop and Conference Proceedings.
JMLR.org, 2015, pp. 1180–1189.
[22] Eric Tzeng et al. “Adversarial Discriminative Domain Adaptation”. 2017 IEEE
Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. IEEE Computer Society, 2017, pp. 2962–
2971.
[23] Fuzhen Zhuang et al. “Supervised Representation Learning: Transfer Learning
with Deep Autoencoders”. Proceedings of the Twenty-Fourth International Joint
Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina,
July 25-31, 2015. Ed. by Qiang Yang and Michael J. Wooldridge. AAAI Press,
2015, pp. 4119–4125.
[24] Hengyuan Hu et al. “Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures”. CoRR abs/1607.03250 (2016).
arXiv: 1607.03250.
[25] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. “ThiNet: A Filter Level Pruning
Method for Deep Neural Network Compression”. IEEE International Conference
on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. IEEE
Computer Society, 2017, pp. 5068–5076.
[26] Hao Li et al. “Pruning Filters for Efficient ConvNets”. 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26,
2017, Conference Track Proceedings. OpenReview.net, 2017.
[27] Wei Wen et al. “Learning Structured Sparsity in Deep Neural Networks”. Advances in Neural Information Processing Systems 29: Annual Conference on
Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona,
Spain. Ed. by Daniel D. Lee et al. 2016, pp. 2074–2082.
[28] Yihui He, Xiangyu Zhang, and Jian Sun. “Channel Pruning for Accelerating
Very Deep Neural Networks”. IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. IEEE Computer Society,
2017, pp. 1398–1406.
[29] Zhuang Liu et al. “Learning Efficient Convolutional Networks through Network
Slimming”. IEEE International Conference on Computer Vision, ICCV 2017,
Venice, Italy, October 22-29, 2017. IEEE Computer Society, 2017, pp. 2755–
2763.
[30] Jianbo Ye et al. “Rethinking the Smaller-Norm-Less-Informative Assumption
in Channel Pruning of Convolution Layers”. 6th International Conference on
Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May
3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[31] Pavlo Molchanov et al. “Pruning Convolutional Neural Networks for Resource
Efficient Inference”. 5th International Conference on Learning Representations,
ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
OpenReview.net, 2017.
[32] Pavlo Molchanov et al. “Importance Estimation for Neural Network Pruning”.
IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019,
Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE,
2019, pp. 11264–11272.
[33] Xiaojie Jin et al. “Training Skinny Deep Neural Networks with Iterative Hard
Thresholding Methods”. CoRR abs/1607.05423 (2016). arXiv: 1607.05423.
[34] Song Han et al. “DSD: Dense-Sparse-Dense Training for Deep Neural Networks”. 5th International Conference on Learning Representations, ICLR 2017,
Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[35] Jielei Wang et al. “Filter pruning with a feature map entropy importance criterion for convolution neural networks compressing”. Neurocomputing 461 (2021),
pp. 41–54.
[36] Ni Xiao and Lei Zhang. “Dynamic Weighted Learning for Unsupervised Domain
Adaptation”. IEEE Conference on Computer Vision and Pattern Recognition,
CVPR 2021, virtual, June 19-25, 2021. Computer Vision Foundation / IEEE,
2021, pp. 15242–15251.
[37] Christopher M. Bishop. Pattern recognition and machine learning, 5th Edition.
Information science and statistics. Springer, 2007. isbn: 9780387310732.
[38] Peter L. Bartlett et al. “Nearly-tight VC-dimension and Pseudodimension Bounds
for Piecewise Linear Neural Networks”. J. Mach. Learn. Res. 20 (2019), 63:1–
63:17.
[39] Han Zhao et al. “On Learning Invariant Representations for Domain Adaptation”. Proceedings of the 36th International Conference on Machine Learning,
ICML 2019, 9-15 June 2019, Long Beach, California, USA. Ed. by Kamalika
Chaudhuri and Ruslan Salakhutdinov. Vol. 97. Proceedings of Machine Learning
Research. PMLR, 2019, pp. 7523–7532.
[40] Shaoan Xie et al. “Learning Semantic Representations for Unsupervised Domain Adaptation”. Proceedings of the 35th International Conference on Machine
Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018.
Ed. by Jennifer G. Dy and Andreas Krause. Vol. 80. Proceedings of Machine
Learning Research. PMLR, 2018, pp. 5419–5428.
[41] Yang Zou et al. “Unsupervised Domain Adaptation for Semantic Segmentation
via Class-Balanced Self-training”. Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part
III. Ed. by Vittorio Ferrari et al. Vol. 11207. Lecture Notes in Computer Science.
Springer, 2018, pp. 297–313.
[42] S. P. Lloyd. “Least squares quantization in PCM”. IEEE Trans. Inform. Theory,
(1982), 28:129–137.
[43] Joonhyuk Kim et al. “Non-Probabilistic Cosine Similarity Loss for Few-Shot
Image Classification”. 31st British Machine Vision Conference 2020, BMVC
2020, Virtual Event, UK, September 7-10, 2020. BMVA Press, 2020.
[44] Kaiming He et al. “Deep Residual Learning for Image Recognition”. 2016 IEEE
Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las
Vegas, NV, USA, June 27-30, 2016. IEEE Computer Society, 2016, pp. 770–
778.
[45] Kate Saenko et al. “Adapting Visual Category Models to New Domains”. Computer Vision - ECCV 2010, 11th European Conference on Computer Vision,
Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV. Ed. by
Kostas Daniilidis, Petros Maragos, and Nikos Paragios. Vol. 6314. Lecture Notes
in Computer Science. Springer, 2010, pp. 213–226.
[46] Barbara Caputo et al. “ImageCLEF 2014: Overview and Analysis of the Results”. Information Access Evaluation. Multilinguality, Multimodality, and Interaction. Ed. by Evangelos Kanoulas et al. Cham: Springer International Publishing, 2014, pp. 192–211. isbn: 978-3-319-11382-1.
[47] Adam Paszke et al. “Automatic Differentiation in PyTorch”. NIPS 2017 Workshop on Autodiff. Long Beach, California, USA, 2017.
[48] Yunlong Zhang et al. “Hard class rectification for domain adaptation”. Knowl.
Based Syst. 222 (2021), p. 107011.
[49] Tobias Ringwald and Rainer Stiefelhagen. “Adaptiope: A Modern Benchmark
for Unsupervised Domain Adaptation”. IEEE Winter Conference on Applications of Computer Vision, WACV 2021, Waikoloa, HI, USA, January 3-8, 2021.
IEEE, 2021, pp. 101–110.