Human-centered Sensing and Computing: Research Progress and Prospects

doi:10.3981/j.issn.2097-0781.2023.01.009

Abstract

Abstract:

The development of the internet of things (IoT), edge computing, mobile internet, and artificial intelligence (AI) has greatly enriched human-centered sensing and computing scenarios. Human-centered sensing and computing regard people as the subject or participant of sense and consciously or unconsciously combine human intelligence in the sensing and computing process, which can thus improve model adaptability and protect humans from the risks of AI. This paper discusses the concept of human-centered sensing and computing and summarizes the corresponding research works and typical applications from different aspects. Then, the paper points out a series of major challenges and research problems and finally proposes prospects for human-centered sensing and computing research on human-machine collaborating ways and ethics.

Key words: human-centered sensing and computing, intelligent sensing, crowd sensing

摘要：

物联网、边缘计算、移动互联网和人工智能等技术的发展，极大丰富了“以人为中心”的感知与计算场景。人为中心感知计算将人作为感知的目标主体或参与者，在感知计算过程中有意识或无意识地贡献人类智慧，提高了模型适应能力和避免人工智能危害人类。文章对人为中心感知计算的概念进行了探讨，从不同角度总结了相关研究工作与典型应用，提出了若干重要挑战与主要研究问题。最后，对人为中心感知计算在人机协作方式、伦理等研究方向进行了展望。

关键词: 人为中心感知计算, 智能感知, 群智感知

YU Zhiwen, LI Qingyang, WANG Qianru, GUO Bin. Human-centered Sensing and Computing: Research Progress and Prospects[J]. Science and Technology Foresight, 2023, 2(1): 118-131.

於志文, 李青洋, 王倩茹, 郭斌. 人为中心感知计算研究进展及展望[J]. 前瞻科技, 2023, 2(1): 118-131.

Figures/Tables 1

References 94

[1]	McGregor S. Preventing repeated real world AI failures by cataloging incidents: The AI incident database[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(17): 15458-15463. DOI URL
[2]	Shneiderman B. Design lessons from AI’s two grand goals: Human emulation and useful applications[J]. IEEE Transactions on Technology and Society, 2020, 1(2): 73-82. DOI URL
[3]	Quinn A J, Bederson B B. Human computation: A survey and taxonomy of a growing field[C]// Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York: ACM Press, 2011: 1403-1412.
[4]	叶宁, 王汝传. 无线传感器网络数据融合模型研究[J]. 计算机科学, 2006, 33(6): 58-60.
[5]	Tseng C M. Operating styles, working time and daily driving distance in relation to a taxi driver’s speeding offenses in Taiwan[J]. Accident Analysis and Prevention, 2013, 52: 1-8. DOI URL
[6]	Greaves S P, Ellison A B. Personality, risk aversion and speeding: An empirical investigation[J]. Accident Analysis and Prevention, 2011, 43(5): 1828-1836. DOI URL
[7]	Wang W, Liu A X, Shahzad M. Gait recognition using WiFi signals[C]// Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. New York: ACM Press, 2016: 363-373.
[8]	Dang X, Cao Y, Hao Z, et al. WiGId: Indoor group identification with CSI-based random forest[J]. Sensors, 2020, 20(16), doi: 10.3390/s20164607. DOI
[9]	Xu W, Yu Z W, Wang Z, et al. AcousticID: Gait-based human identification using acoustic signal[J]. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2019, 3(3): 1-25.
[10]	Jiang W W, Luo J Y. Graph neural network for traffic forecasting: A survey[J]. Expert Systems With Applications, 2022, 207, doi: 10.1016/j.eswa.2022.117921. DOI
[11]	Zhang Y, Bocquet M, Mallet V, et al. Real-time air quality forecasting, part I: History, techniques, and current status[J]. Atmospheric Environment, 2012, 60: 632-655. DOI URL
[12]	Graham E A, Lam Y, Yuen E M. Forest understory soil temperatures and heat flux calculated using a Fourier model and scaled using a digital camera[J]. Agricultural and Forest Meteorology, 2010, 150(4): 640-649. DOI URL
[13]	Shilton K, Burke J, Estrin D, et al. Designing the personal data stream: Enabling participatory privacy in mobile personal sensing[C/OL]. [2023-01-13]. https://terpconnect.umd.edu/-kshilton/pdf/ShiltonEtAlTPRCpreprint.pdf.
[14]	Zhang H W, Naik V. Data transport control in wireless sensor networks[M]// Misra S, Woungang I, Misra S. Guide to Wireless Sensor Networks. London: Springer, 2009: 239-259.
[15]	Lindgren A, Doria A, Schelén O. Probabilistic routing in intermittently connected networks[J]. ACM SIGMOBILE Mobile Computing and Communications Review, 2003, 7(3): 19-20. DOI URL
[16]	Ye F, Qian Y, Hu R Q. Smart service-aware wireless mixed-area networks[J]. IEEE Network, 2019, 33(1): 84-91. DOI
[17]	Wang W, Liu X, Yao Y, et al. CRF: Coexistent routing and flooding using WiFi packets in heterogeneous IoT networks[C]// Proceedings of the IEEE Conference on Computer Communications. Piscataway: IEEE Press, 2019: 19-27.
[18]	Burke J A, Estrin D, Hansen M, et al. Participatory sensing[C/OL]. [2023-01-13]. https://people.eecs.berkeley.edu/-sylvia/papers/estrin-participatorysens.pdf.
[19]	刘云浩. 群智感知计算[J]. 中国计算机学会通讯, 2012, 8(10): 38-41.
[20]	Xiao Z, Lim H B, Ponnambalam L. Participatory sensing for smart cities: A case study on transport trip quality measurement[J]. IEEE Transactions on Industrial Informatics, 2017, 13(2): 759-770. DOI URL
[21]	Bao X, Choudhury R R. MoVi: Mobile phone based video highlights via collaborative sensing[C]// Proceedings of the 8th International Conference on Mobile Systems, Applications, And Services. New York: ACM Press, 2010: 357-370.
[22]	Marcus A, Bernstein M S, Badar O, et al. Twitinfo: Aggregating and visualizing microblogs for event exploration[C]// Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York: ACM Press, 2011: 227-236.
[23]	Chen H H, Guo B, Yu Z W, et al. A generic framework for constraint-driven data selection in mobile crowd photographing[J]. IEEE Internet of Things Journal, 2017, 4(1): 284-296.
[24]	Cao L B. Non-IIDness learning in behavioral and social data[J]. The Computer Journal, 2014, 57(9): 1358-1370. DOI URL
[25]	Xu J, Wei W, Cao L B. Copula-based high dimensional cross-market dependence modeling[C]// Proceedings of the 2017 IEEE International Conference on Data Science and Advanced Analytics. Piscataway: IEEE Press, 2018: 734-743.
[26]	Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate[DB/OL]. arXiv preprint: 1409.0473, 2014.
[27]	Pan S J, Yang Q. A survey on transfer learning[J]. IEEE Transactions on Knowledge and Data Engineering, 2010, 22(10): 1345-1359. DOI URL
[28]	Wang L, Geng X, Ma X, et al. Ridesharing car detection by transfer learning[J]. Artificial Intelligence, 2019, 273: 1-18. DOI
[29]	Yao H X, Liu Y D, Wei Y, et al. Learning from multiple cities: A meta-learning approach for spatial-temporal prediction[C]// Proceedings of the World Wide Web Conference. New York: ACM Press, 2019: 2181-2191.
[30]	Luo T, Das S K, Tan H P, et al. Incentive mechanism design for crowdsourcing: An all-pay auction approach[J]. ACM Transactions on Intelligent Systems and Technology, 2016, 7(3): 1-26.
[31]	Dimitriou T, Krontiris I. Privacy-respecting auctions and rewarding mechanisms in mobile crowd-sensing applications[J]. Journal of Network and Computer Applications, 2017, 100: 24-34. DOI URL
[32]	Yang D, Fang X, Xue G. Truthful incentive mechanisms for k-anonymity location privacy[C]// Proceedings of the 2013 IEEE Conference on Computer Communications. Piscataway: IEEE Press, 2013: 2994-3002.
[33]	Gao L, Hou F, Huang J W. Providing long-term participation incentive in participatory sensing[C]// Proceedings of the 2015 IEEE Conference on Computer Communications. Piscataway: IEEE Press, 2015: 2803-2811.
[34]	Feng Z N, Zhu Y M, Zhang Q, et al. TRAC: Truthful auction for location-aware collaborative sensing in mobile crowdsourcing[C]// Proceedings of the 2014 IEEE Conference on Computer Communications. Piscataway: IEEE Press, 2014: 1231-1239.
[35]	Dissanayake I, Mehta N, Palvia P, et al. Competition matters! Self-efficacy, effort, and performance in crowdsourcing teams[J]. Information & Management, 2019, 56(8): 103158. DOI URL
[36]	Brito J, Vieira V, Duran A. Towards a framework for gamification design on crowdsourcing systems: The G. A.M.E. approach[C]// Proceedings of the 2015 12th International Conference on Information Technology-New Generations. Piscataway: IEEE Press, 2015: 445-450.
[37]	Jaimes L G, Vergara-Laurens I J, Raij A. A survey of incentive techniques for mobile crowd sensing[J]. IEEE Internet of Things Journal, 2015, 2(5): 370-380. DOI URL
[38]	Hoc J M. Towards a cognitive approach to human-machine cooperation in dynamic situations[J]. International Journal of Human-computer Studies, 2001, 54(4): 509-540. DOI URL
[39]	Chen H H, Guo B, Yu Z W, et al. Toward real-time and cooperative mobile visual sensing and sharing[C]// Proceedings of the 35th Annual IEEE International Conference on Computer Communications. Piscataway: IEEE Press, 2016, doi: 10.1109/INFOCOM.2016.7524481. DOI
[40]	Abad A, Nabi M, Moschitti A. Autonomous crowdsourcing through human-machine collaborative learning[C]// Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM Press, 2017: 873-876.
[41]	Biagioni J, Gerlich T, Merrifield T, et al. EasyTracker:Automatic transit tracking, mapping, and arrival time prediction using smartphones[C]// Proceedings of the 9th ACM Conference on Embedded Networked Sensor Systems. New York: ACM Press, 2011: 68-81.
[42]	Guo B, Ouyang Y, Zhang C, et al. CrowdStory: Fine-grained event storyline generation by fusion of multi-modal crowdsourced data[J]. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2017, 1(3): 1-19.
[43]	Settles B. Active learning[J]. Synthesis Lectures on Artificial Intelligence and Machine Learning, 2012, 6(1): 1-114.
[44]	Liu Y Z, Li Z, Zhou C, et al. Generative adversarial active learning for unsupervised outlier detection[J]. IEEE Transactions on Knowledge and Data Engineering, 2020, 32(8): 1517-1528.
[45]	Yang J, Drake T, Damianou A, et al. Leveraging crowdsourcing data for deep active learning an application: Learning intents in alexa[C]// Proceedings of the 2018 World Wide Web Conference. New York: ACM Press, 2018: 23-32.
[46]	Ostheimer J, Chowdhury S, Iqbal S. An alliance of humans and machines for machine learning: Hybrid intelligent systems and their design principles[J]. Technology in Society, 2021, 66, doi: 10.1016/j.techsoc.2021.101647. DOI
[47]	Schydlo P, Rakovic M, Jamone L, et al. Anticipation in human-robot cooperation: A recurrent neural network approach for multiple action sequences prediction[C]// Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA). New York:ACM Press, 2018: 1-6.
[48]	Cai C J, Reif E, Hegde N, et al. Human-centered tools for coping with imperfect algorithms during medical decision-making[C]// Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. New York: ACM Press, 2019, doi: 10.1145/3290605.3300234. DOI
[49]	Li X C, Xie K, Wang X, et al. Online internet anomaly detection with high accuracy: A fast tensor factorization solution[C]// Proceedings of the 2019 IEEE Conference on Computer Communications. Piscataway: IEEE, 2019: 1900-1908.
[50]	Li Q Y, Yu Z W, Yao L N, et al. RLTIR: Activity-based interactive person identification via reinforcement learning tree[J]. IEEE Internet of Things Journal, 2022, 9(6): 4464-4475. DOI URL
[51]	Nushi B, Kamar E, Horvitz E. Towards accountable AI: Hybrid human-machine analyses for characterizing system failure[C]// Chen Y, Kazai G.Proceedings of the Sixth AAAI Conference on Human Computation and Crowdsourcing. Palo Alto: AAAI Press, 2018: 126-135.
[52]	Wang Y F, Yu Z W, Yang F, et al. Video pedestrian re-identification based on human-machine collaboration[C]// Sun Y, Liu D, Liao H, et al. Proceedings of the CCF Conference on Computer Supported Cooperative Work and Social Computing. Singapore: Springer, 2021: 410-423.
[53]	Yang F, Yu Z W, Chen L M, et al. Human-machine cooperative video anomaly detection[J]. Proceedings of the ACM on Human-Computer Interaction, 2020, 4(CSCW3), doi: 10.1145/3434183. DOI
[54]	Dürr F, Skvortsov P, Rothermel K. Position sharing for location privacy in non-trusted systems[C]// Proceedings of the 2011 IEEE International Conference on Pervasive Computing and Communications (PerCom). Piscataway:IEEE Press, 2011: 189-196.
[55]	Boukoros S, Humbert M, Katzenbeisser S, et al. On (the lack of) location privacy in crowdsourcing applications[DB/OL]. arXiv preprint: 1901.04923, 2019.
[56]	Krontiris I, Dimitriou T. Privacy-respecting discovery of data providers in crowd-sensing applications[C]// Proceedings of the 2013 IEEE International Conference on Distributed Computing in Sensor Systems. Piscataway: IEEE Press, 2013: 249-257.
[57]	Dingledine R, Mathewson N, Syverson P. Tor: The second-generation onion router[R]. Washington, D. C.: Naval Research Laboratory, 2004.
[58]	Christin D, Roßkopf C, Hollick M, et al. IncogniSense: An anonymity-preserving reputation framework for participatory sensing applications[J]. Pervasive and Mobile Computing, 2013, 9(3): 353-371. DOI URL
[59]	Wu S, Wang X L, Wang S, et al. K-anonymity for crowdsourcing database[J]. IEEE Transactions on Knowledge and Data Engineering, 2014, 26(9): 2207-2221. DOI URL
[60]	Shin M, Cornelius C, Peebles D, et al. AnonySense: A system for anonymous opportunistic sensing[J]. Pervasive and Mobile Computing, 2011, 7(1): 16-30. DOI URL
[61]	Zhang L C, Wang X M, Lu J L, et al. An efficient privacy preserving data aggregation approach for mobile sensing[J]. Security and Communication Networks, 2016, 9(16): 3844-3853. DOI URL
[62]	Sahai A, Waters B. Fuzzy identity-based encryption[C]// Proceedings of the 24th Annual International Conference on Theory and Applications of Cryptographic Techniques. New York: ACM Press, 2005: 457-473.
[63]	Hale M, Egerstedt M. Constrained multi-agent optimization with differentially private states[DB/OL]. ArXiv preprint: 1507.04371, 2015.
[64]	Han S, Topcu U, Pappas G J. Differentially private distributed constrained optimization[J]. IEEE Transactions on Automatic Control, 2017, 62(1): 50-64. DOI URL
[65]	Salah K, Rehman M H U, Nizamuddin N, et al. Blockchain for AI: Review and open research challenges[J]. IEEE Access, 2019, 7: 10127-10149. DOI
[66]	Konečný J, Mcmahan H B, Yu F X, et al. Federated learning: Strategies for improving communication efficiency[DB/OL]. arXiv preprint: 1610.05492, 2016.
[67]	Salem A, Zhang Y, Humbert M, et al. ML-leaks: Model and data independent membership inference attacks and defenses on machine learning models[C]// Proceedings of the 2019 Network and Distributed System Security Symposium. Reston: Internet Society, 2019: 24-27.
[68]	Beaulieu-Jones B K, Wu Z S, Williams C, et al. Privacy-preserving generative deep neural networks support clinical data sharing[J]. bioRxiv, 2018, doi: 10.1101/159756. DOI
[69]	Juuti M, Szyller S, Marchal S, et al. PRADA: Protecting against DNN model stealing attacks[C]// Proceedings of the 2019 IEEE European Symposium on Security and Privacy (Euro S&P). Piscataway:IEEE Press, 2019: 512-527.
[70]	Bonawitz K, Ivanov V, Kreuter B, et al. Practical secure aggregation for privacy-preserving machine learning[C]// Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. New York: ACM Press, 2017: 1175-1191.
[71]	Wei K, Li J, Ding M, et al. Federated learning with differential privacy: Algorithms and performance analysis[J]. IEEE Transactions on Information Forensics and Security. Piscataway: IEEE Press, 2020: 3454-3469.
[72]	Li Z, Hubchak M, Zhu Y. Deep leakage from gradients in multiple-label medical image classification[C]// Proceedings of the 2021 IEEE 9th International Conference on Healthcare Informatics (ICHI). Piscataway:IEEE Press, 2021: 447-448.
[73]	Geng X, Li Y G, Wang L Y, et al. Spatiotemporal multi-graph convolution network for ride-hailing demand forecasting[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33(1): 3656-3663. DOI URL
[74]	Huang W H, Song G J, Hong H K, et al. Deep architecture for traffic flow prediction: Deep belief networks with multitask learning[J]. IEEE Transactions on Intelligent Transportation Systems, 2014, 15(5): 2191-2201. DOI URL
[75]	Ye J C, Sun L L, Du B W, et al. Co-prediction of multiple transportation demands based on deep spatio-temporal neural network[C]// Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. New York: ACM Press, 2019: 305-313.
[76]	Yu B, Yin H T, Zhu Z X. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting[C]// Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence. California: International Joint Conferences on Artificial Intelligence Organization, 2018: 3634-3640.
[77]	Uddin M Y S, Wang H Y, Saremi F, et al. PhotoNet: A similarity-aware picture delivery service for situation awareness[C]// Proceedings of the 2011 IEEE 32nd Real-time Systems Symposium. Piscataway: IEEE Press, 2012: 317-326.
[78]	Wu Y B, Wang Y, Hu W J, et al. Resource-aware photo crowdsourcing through disruption tolerant networks[C]// 2016 IEEE 36th International Conference on Distributed Computing Systems (ICDCS). Piscataway:IEEE Press, 2016: 374-383.
[79]	van Leuken R H, Garcia L, Olivares X, et al. Visual diversification of image search results[C]// Proceedings of the 18th International Conference on World Wide Web. New York: ACM Press, 2009: 341-350.
[80]	Zheng Y Q, Shen G B, Li L Q, et al. Travi-navi: Self-deployable indoor navigation system[J]. ACM Transactions on Networking, 2017, 25(5): 2655-2669.
[81]	Koutsopoulos I. Optimal incentive-driven design of participatory sensing systems[C]// Proceedings of the 2013 IEEE Conference on Computer Communications. Piscataway: IEEE Press, 2013: 1402-1410.
[82]	Sutton R S. Generalization in reinforcement learning: Successful examples using sparse coarse coding[C]// Advances in Neural Information Processing Systems 8, NIPS. Cambridge: MIT Press, 1995: 1038-1044.
[83]	Mnih V, Kavukcuoglu K, Silver D, et al. Playing atari with deep reinforcement learning[DB/OL]. arXiv preprint: 1312.5602, 2013.
[84]	Silver D, Lever G, Heess N, et al. Deterministic policy gradient algorithms[C]// Proceedings of the 31st International Conference on Machine Learning, PMLR. New York: ACM Press, 2014, 32(1): 387-395.
[85]	Barto A G, Sutton R S, Anderson C W. Neuronlike adaptive elements that can solve difficult learning control problems[J]. IEEE Transactions on Systems, Man, and Cybernetics, 1983, SMC-13(5): 834-846. DOI URL
[86]	Mallya A, Davis D, Lazebnik S. Piggyback: Adapting a single network to multiple tasks by learning to mask weights[C]// Proceedings of the 15th European Conference on Computer Vision, ECCV 2018. New York: ACM Press, 2018: 72-88.
[87]	Fernando C, Banarse D, Blundell C, et al. PathNet: Evolution channels gradient descent in super neural networks[DB/OL]. arXiv preprint: 1701.08734, 2017.
[88]	Xu J, Zhu Z X. Reinforced continual learning[DB/OL]. arXiv preprint: 1805. 12369, 2018.
[89]	Park G M, Yoo S M, Kim J H. Convolutional neural network with developmental memory for continual learning[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 32(6): 2691-2705. DOI URL
[90]	Huang K L, Kanhere S S, Hu W. Are you contributing trustworthy data? The case for a reputation system in participatory sensing[C]// Proceedings of the 13th ACM International Conference on Modeling, Analysis, and Simulation of Wireless and Mobile Systems. New York: ACM Press, 2010: 14-22.
[91]	Kang J W, Xiong Z H, Niyato D, et al. Reliable federated learning for mobile networks[J]. IEEE Wireless Communications, 2020, 27(2): 72-80.
[92]	Miao C L, Jiang W J, Su L, et al. Cloud-enabled privacy-preserving truth discovery in crowd sensing systems[C]// Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems. New York: ACM Press, 2015: 183-196.
[93]	Meng C S, Jiang W J, Li Y L, et al. Truth discovery on crowd sensing of correlated entities[C]// Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems. New York: ACM Press, 2015: 169-182.
[94]	Lee J D, See K A. Trust in automation: Designing for appropriate reliance[J]. Human Factors, 2004, 46(1): 50-80. DOI PMID