Echo chambers are social phenomena that amplify agreement and suppress opposing views in social media which may lead to fragmentation and polarization of the user population. In prior research, echo chambers have mainly been modeled as a result of social information diffusion. While most scientific work has framed echo chambers as a result of epistemic imbalances between polarized communities, we argue that members of echo chambers often actively discredit outside sources to maintain coherent world views. We therefore argue that two different types of echo chambers occur in social media contexts: Epistemic echo chambers create information gaps mainly through their structure whereas ideological echo chambers systematically exclude counter-attitudinal information. Diversifying recommendations by simply widening the scope of topics and viewpoints covered to counteract the echo chamber effect may be ineffective in such contexts. To investigate the characteristics of this dual echo chamber view and to assess the depolarizing effects of diversified recommendations, we apply an agent-based modeling approach. We rely on knowledge graph embedding techniques not only to generate recommendations, but also to show how to utilize logical graph queries in embedding spaces to diversify recommendations aimed at challenging polarization in online discussions. The results of our evaluation indicate that counteracting the two different types of echo chambers requires fundamentally different diversification strategies.

There is significant interest lately in using synthetic data and simulation infrastructures for various types of recommender systems research. However, there are not currently any clear best practices around how best to apply these methods. We proposed a workshop to bring together researchers and practitioners interested in simulating recommender systems and their data to discuss the state of the art of such research and the pressing open methodological questions. The workshop resulted in a report authored by the participants that documents currently-known best practices on which the group has consensus and lays out an agenda for further research over the next 3–5 years to fill in places where we currently lack the information needed to make methodological recommendations.

In this study, we introduce the payload optimization method for federated recommender systems (FRS). In federated learning (FL), the global model payload that is moved between the server and users depends on the number of items to recommend. The model payload grows when there is an increasing number of items. This becomes challenging for FRS if it is running in production mode. To tackle the payload challenge, we formulated a multi-arm bandit solution that selected part of the global model and transmitted it to all users. The selection process was guided by a novel reward function suitable for FL systems. So far as we are aware, this is the first optimization method that seeks to address item dependent payloads. The method was evaluated using three benchmark recommendation datasets. The empirical validation confirmed that the proposed method outperforms the simpler methods that do not benefit from the bandits for the purpose of item selection. In addition, we have demonstrated the usefulness of our proposed method by rigorously evaluating the effects of a payload reduction on the recommendation performance degradation. Our method achieved up to a 90% reduction in model payload, yielding only a ∼ 4% - 8% loss in the recommendation performance for highly sparse datasets.

Online personalized advertising is often very effective in identifying relevant audiences for each piece of content, which has led to its widespread adoption. In today’s internet, however, these advertising systems are used not only to market products, but also consequential life opportunities such as employment or housing, as well as socially important political messaging. This has led to increasing concerns about the presence of algorithmic bias and possible discrimination in these important domains — with results showing problematic biases along gender, race, and political affiliation, even when the advertiser might have targeted broadly. A growing body of work focuses on measuring and characterizing these biases, as well as finding ways to mitigate these effects and building responsible systems. However, these results often emerge from different scientific communities and are often disconnected in the literature. In this paper, I attempt at bridging the gap between isolated efforts to either measure these biases, or to mitigate them. I discuss how the need to measure bias in advertising, and the efforts to mitigate it, despite being distant in the literature, are complementary problems that need to center their methodolgy around user studies. This paper presents a research agenda that focuses on the need for user-centric measurements of bias, by collecting real ads from users, and using surveys to understand user perceptions for these ads. My approach also calls for incorporating user sentiments into the mitigation efforts, by constraining optimization on user values that emerge from surveys. Finally, I also emphasize the need for involving users in the evaluation of responsible advertising systems; efforts to mitigate bias eventually need to be contextualized in terms of benefits to users instead of simple performance tradeoffs. My focus on the users is motivated by the fact that they are stakeholders in personalized advertising, vulnerable at the hand of algorithmic bias and harm, and therefore crucial in both efforts to measure and mitigate these effects.

We propose AC-CRS, a novel conversational recommendation system based on reinforcement learning that better models user interaction compared to prior work. Interactive recommender systems expect an initial request from a user and then iterate by asking questions or recommending potential matching items, continuing until some stopping criterion is achieved. Unlike most existing works that stop as soon as an item is recommended, we model the more realistic expectation that the interaction will continue if the item is not appropriate. Using this process, AC-CRS is able to support a more flexible conversation with users. Unlike existing models, AC-CRS is able to estimate a value for each question in the conversation to make sure that questions asked by the agent are relevant to the target item (i.e., user needs). We also model the possibility that the system could suggest more than one item in a given turn, allowing it to take advantage of screen space if it is present. AC-CRS also better accommodates the massive space of items that a real-world recommender system must handle. Experiments on real-world user purchasing data show the effectiveness of our model in terms of standard evaluation measures such as NDCG.

Building recommendation systems for enterprise software has many unique challenges that are different from consumer-facing systems. When applied to different organizations, the data used to power those recommendation systems vary substantially in both quality and quantity due to differences in their operational practices, marketing strategies, and targeted audiences. At Salesforce, as a cloud provider of such a system with data across many different organizations, naturally, it makes sense to pool data from different organizations to build a model that combines all values from different brands. However, multiple issues like how do we make sure a model trained with pooled data can still capture customer specific characteristics, how do we design the system to handle those data responsibly and ethically, i.e., respecting contractual agreements with our clients, legal and compliance requirements, and the privacy of all the consumers. In this proposal, We present a framework that not only utilizes enriched user-level data across organizations, but also boosts business-specific characteristics in generating personal recommendations. We will also walk through key privacy considerations when designing such a system.

Motives or goals are recognized in psychology literature as the most fundamental drive that explains and predicts why people do what they do, including when they browse the web. Although providing enormous value, these higher-ordered goals are often unobserved, and little is known about how to leverage such goals to assist people’s browsing activities. This paper proposes to take a new approach to address this problem, which is fulfilled through a novel neural framework, Goal-directed Web Browsing (GoWeB). We adopt a psychologically-sound taxonomy of higher-ordered goals and learn to build their representations in a structure-preserving manner. Then we incorporate the resulting representations for enhancing the experiences of common activities people perform on the web. Experiments on large-scale data from Microsoft Edge web browser show that GoWeB significantly outperforms competitive baselines for in-session web page recommendation, re-visitation classification, and goal-based web page grouping. A follow-up analysis further characterizes how the variety of human motives can affect the difference observed in human behavioral patterns.

Providing suitable recommendations is of vital importance to improve the user satisfaction of music recommender systems. Here, users often listen to the same track repeatedly and appreciate recommendations of the same song multiple times. Thus, accounting for users’ relistening behavior is critical for music recommender systems. In this paper, we describe a psychology-informed approach to model and predict music relistening behavior that is inspired by studies in music psychology, which relate music preferences to human memory. We adopt a well-established psychological theory of human cognition that models the operations of human memory, i.e., Adaptive Control of Thought—Rational (ACT-R). In contrast to prior work, which uses only the base-level component of ACT-R, we utilize five components of ACT-R, i.e., base-level, spreading, partial matching, valuation, and noise, to investigate the effect of five factors on music relistening behavior: (i) recency and frequency of prior exposure to tracks, (ii) co-occurrence of tracks, (iii) the similarity between tracks, (iv) familiarity with tracks, and (v) randomness in behavior. On a dataset of 1.7 million listening events from Last.fm, we evaluate the performance of our approach by sequentially predicting the next track(s) in user sessions. We find that recency and frequency of prior exposure to tracks is an effective predictor of relistening behavior. Besides, considering the co-occurrence of tracks and familiarity with tracks further improves performance in terms of R-precision. We hope that our work inspires future research on the merits of considering cognitive aspects of memory retrieval to model and predict complex user behavior.

In the last few years, a renewed interest of the research community on conversational recommender systems (CRSs) is emerging. This is probably due to the great diffusion of Digital Assistants (DAs) such as Amazon Alexa, Siri, or Google Assistant that are revolutionizing the way users interact with machines. DAs allow users to execute a wide range of actions through an interaction mostly based on natural language messages. However, although DAs are able to complete tasks such as sending texts, making phone calls, or playing songs, they are still at an early stage on offering recommendation capabilities by using the conversational paradigm. In addition, we have been witnessing the advent of more and more precise and powerful recommendation algorithms and techniques able to effectively assess users’ tastes and predict information that would probably be of interest to them. Most of these approaches rely on the collaborative paradigm (often exploiting machine learning techniques) and do not take into account the huge amount of knowledge, both structured and non-structured ones, describing the domain of interest of the recommendation engine. Although very effective in predicting relevant items, collaborative approaches miss some very interesting features that go beyond the accuracy of results and move in the direction of providing novel and diverse results as well as generating an explanation for the recommended items. Furthermore, this side information becomes crucial when a conversational interaction is implemented, in particular for the preference elicitation, explanation, and critiquing steps.

Modern online services continuously generate data at very fast rates. This continuous flow of data encompasses content – e.g. posts, news, products, comments –, but also user feedback – e.g. ratings, views, reads, clicks –, together with context data – user device, spacial or temporal data, user task or activity, weather. This can be overwhelming for systems and algorithms designed to train in batches, given the continuous and potentially fast change of content, context and user preferences or intents. Therefore, it is important to investigate online methods able to transparently adapt to the inherent dynamics of online services. Incremental models that learn from data streams are gaining attention in the recommender systems community, given their natural ability to deal with the continuous flows of data generated in dynamic, complex environments. User modeling and personalization can particularly benefit from algorithms capable of maintaining models incrementally and online. The objective of this workshop is to foster contributions and bring together a growing community of researchers and practitioners interested in online, adaptive approaches to user modeling, recommendation and personalization, and their implications regarding multiple dimensions, such as evaluation, reproducibility, privacy and explainability.

Retail services in corporate international banks are often constrained by the compliance and infrastructure specificities of the countries in which they operate. In addition, the business model and customers interactions through a banking app differ greatly from other major retail services in sectors like digital streaming or e-commerce platforms. We introduce ING’s Bank retail recommender system, elaborating on how we account for the global vs. local requirements, and how can we benefit from such an approach for our model selection as well as products serving.

A dialog-based interactive recommendation task is where users can express natural-language feedback when interacting with the recommender system. However, the users’ feedback, which takes the form of natural-language critiques about the recommendation at each iteration, can only allow the recommender system to obtain a partial portrayal of the users’ preferences. Indeed, such partial observations of the users’ preferences from their natural-language feedback make it challenging to correctly track the users’ preferences over time, which can result in poor recommendation performances and a less effective satisfaction of the users’ information needs when in presence of limited iterations. Reinforcement learning, in the form of a partially observable Markov decision process (POMDP), can simulate the interactions between a partially observable environment (i.e. a user) and an agent (i.e. a recommender system). To alleviate such a partial observation issue, we propose a novel dialog-based recommendation model, the Estimator-Generator-Evaluator (EGE) model, with Q-learning for POMDP, to effectively incorporate the users’ preferences over time. Specifically, we leverage an Estimator to track and estimate users’ preferences, a Generator to match the estimated preferences with the candidate items to rank the next recommendations, and an Evaluator to judge the quality of the estimated preferences considering the users’ historical feedback. Following previous work, we train our EGE model by using a user simulator which itself is trained to describe the differences between the target users’ preferences and the recommended items in natural language. Thorough and extensive experiments conducted on two recommendation datasets – addressing images of fashion products (namely dresses and shoes) – demonstrate that our proposed EGE model yields significant improvements in comparison to the existing state-of-the-art baseline models.

Podcasts have continued to experience rapid growth in both cultural relevance as well as research attention. Coming off the success of the first PodRecs Workshop for Podcast Recommendations at RecSys in 2020, as well as to build upon the research datasets and prior work released in the last year, the second PodRecs Workshop for Podcast Recommendations was held at RecSys 2021 to further develop the community of researchers and practitioners interested in the recommendation of podcasts.

In many domains, user preferences are similar locally within like-minded subgroups of users, but typically differ globally between those subgroups. Local recommendation models were shown to substantially improve top-K recommendation performance in such settings. However, existing local models do not scale to large-scale datasets with an increasing number of subgroups and do not support inductive recommendations for users not appearing in the training set. Key reasons for this are that subgroup detection and recommendation get implemented as separate steps in the model or that local models are explicitly instantiated for each subgroup. In this paper, we propose an End-to-end Local Factor Model (Elfm) which overcomes these limitations by combining both steps and incorporating local structures through an inductive bias. Our model can be optimized end-to-end and supports incremental inference, does not require a full separate model for each subgroup, and has overall small memory and computational costs for incorporating local structures. Empirical results show that our method substantially improves recommendation performance on large-scale datasets with millions of users and items with considerably smaller model size. Our user study also shows that our approach produces coherent item subgroups which could aid in the generation of explainable recommendations.

Adding confidence estimates to predicted ratings has been shown to positively influence the quality of the recommendations provided by a recommender system. While confidence over single point predictions of ratings and preferences has been widely studied in literature, limited effort has been put in exploring the benefits provided by user-level confidence indices. In this work we exploit a recently introduced user-level confidence index, called eigenvalue confidence index, in order to provide maximum confidence recommendations for item-based recommender systems. We firstly derive a closed form solution to calculate the index, then we propose a new recommendation methodology for item-based models, called eigenvalue perturbation, founded on the strongly positive correlation between the index value and the accuracy of the recommendations. We show and discuss the accuracy results obtained with a comprehensive set of experiments over several datasets and using different item-based models, empirically proving that applying the new technique we are able to outperform the original recommendation models in most of the experimental configurations.

Recently, recommendation systems have been proven to be susceptible to malicious perturbations of the model weights. To overcome this vulnerability, Adversarial Regularization emerged as one of the most effective solutions. Interestingly, the technique not only robustifies the model, but also significantly increases its accuracy. To date, unfortunately, the effect of Adversarial Regularization beyond-accuracy evaluation dimensions is unknown. This paper sheds light on these aspects and investigates how Adversarial Regularization impacts the amplification of popularity bias, and the deterioration of novelty and coverage of the recommendation list. The results highlight that, with imbalanced data distribution, Adversarial Regularization amplifies the popularity bias. Moreover, the empirical validation on five datasets confirms that it degrades the diversity and novelty of the generated recommendation. Code and data are available at https://github.com/sisinflab/The-Idiosyncratic-Effects-of-Adversarial-Training.

Within decentralized organizations, the local demand for recommender systems to support business processes grows. The diversity in data sources and infrastructure challenges central engineering teams. Achieving a high delivery velocity without technical debt requires a scalable approach in the development and operations of recommender systems. At the HEINEKEN Company, we execute a machine learning operations method with five best practices: pipeline automation, data availability, exchangeable artifacts, observability, and policy-based security. Creating a culture of self-service, automation, and collaboration to scale recommender systems for decentralization. We demonstrate a practical use case of a self-service ML workspace deployment and a recommender system, that scale faster to subsidiaries and with less technical debt. This enables HEINEKEN to globally support applications that generate insights with local business impact.

In the recommender systems literature, it has been shown that, in addition to improving system effectiveness, explaining recommendations may increase user satisfaction, trust, persuasion and loyalty. In general, explanations focus on the filtering algorithms or the users and items involved in the generation of recommendations. However, on certain domains that are rich on user-generated textual content, it would be valuable to provide justifications of recommendations according to arguments that are explicit, underlying or related with the data used by the systems, e.g., the reasons for customers’ opinions in reviews of e-commerce sites, and the requests and claims in citizens’ proposals and debates of e-participation platforms. In this context, there is a need and challenging task to automatically extract and exploit the arguments given for and against evaluated items. We thus advocate to focus not only on user preferences and item features, but also on associated arguments. In other words, we propose to not only consider what is said about items, but also why it is said. Hence, arguments would not only be part of the recommendation explanations, but could also be used by the recommendation algorithms themselves. To this end, in this thesis, we propose to use argument mining techniques and tools that allow retrieving and relating argumentative information from textual content, and investigate recommendation methods that exploit that information before, during and after their filtering processes.

Graph neural networks (GNNs) have recently gained significant momentum in the recommendation community, demonstrating state-of-the-art performance in top-k recommendation and next-item recommendation. Despite promising results on GNN-based recommendation and search, most of the current GNN research remains essentially concentrated on more traditional tasks such as classification or regression. The GReS workshop on Graph Neural Networks for Recommendation and Search is then a first endeavor to bridge the gap between the RecSys and GNN communities, and promote recommendation and search problems amongst GNN practitioners.

Booking.com is the world’s leading online travel platform where users make many decisions supported by our recommendations, such as destinations, travel dates, facilities, etc. This leads to a complex User Interface (UI) containing many widgets of different relevance for different users. We address the problem of constructing an optimal UI, a non-trivial problem, mainly due to user preferences evolving over time and multiple independent teams collaboratively building the UI. Our goal is to provide a personalized User Experience (UX) which adapts to changes in the environment and ensures governable, collaborative product development. The solution relies on a Multi Armed Bandits (MAB) framework currently allowing product teams to collaborate on the construction of UIs and serving millions of users every day. We present examples of our solution and lessons learned during their implementation.

Newsletters have (re-) emerged as a powerful tool for publishers to engage with their readers directly and more effectively. Despite the diversity in their audiences, publishers’ newsletters remain largely a one-size-fits-all offering, which is suboptimal. In this paper, we present NU:BRIEF, a web application for publishers that enables them to personalize their newsletters without harvesting personal data. Personalized newsletters build a habit and become a great conversion tool for publishers, providing an alternative readers-generated revenue model to a declining ad/clickbait-centered business model. Demo: https://demo.nubrief.com/md03PaAJSwXMegL5BbKpQlArK3elb3hDUglcHodx4gE=/ Explainer video: https://www.youtube.com/watch?v=AUZGuyPJYH4

We investigate whether model extraction can be used to ‘steal’ the weights of sequential recommender systems, and the potential threats posed to victims of such attacks. This type of risk has attracted attention in image and text classification, but to our knowledge not in recommender systems. We argue that sequential recommender systems are subject to unique vulnerabilities due to the specific autoregressive regimes used to train them. Unlike many existing recommender attackers, which assume the dataset used to train the victim model is exposed to attackers, we consider a data-free setting, where training data are not accessible. Under this setting, we propose an API-based model extraction method via limited-budget synthetic data generation and knowledge distillation. We investigate state-of-the-art models for sequential recommendation and show their vulnerability under model extraction and downstream attacks. We perform attacks in two stages. (1) Model extraction: given different types of synthetic data and their labels retrieved from a black-box recommender, we extract the black-box model to a white-box model via distillation. (2) Downstream attacks: we attack the black-box model with adversarial samples generated by the white-box recommender. Experiments show the effectiveness of our data-free model extraction and downstream attacks on sequential recommenders in both profile pollution and data poisoning settings.

A key distinguishing feature of conversational recommender systems over traditional recommender systems is their ability to elicit user preferences using natural language. Currently, the predominant approach to preference elicitation is to ask questions directly about items or item attributes. These strategies do not perform well in cases where the user does not have sufficient knowledge of the target domain to answer such questions. Conversely, in a shopping setting, talking about the planned use of items does not present any difficulties, even for those that are new to a domain. In this paper, we propose a novel approach to preference elicitation by asking implicit questions based on item usage. Our approach consists of two main steps. First, we identify the sentences from a large review corpus that contain information about item usage. Then, we generate implicit preference elicitation questions from those sentences using a neural text-to-text model. The main contributions of this work also include a multi-stage data annotation protocol using crowdsourcing for collecting high-quality labeled training data for the neural model. We show that out approach is effective in selecting review sentences and transforming them to elicitation questions, even with limited training data.

In the past few years we observed a renewed interest in conversational recommender systems (CRS) that interact with users in natural language. Most recent research efforts use neural models trained on recorded recommendation dialogs between humans, supporting an end-to-end learning process. Given the user’s utterances in a dialog, these systems aim to generate appropriate responses in natural language based on the learned models. An alternative to such language generation approaches is to retrieve and possibly adapt suitable sentences from the recorded dialogs. Approaches of this latter type are explored only to a lesser extent in the current literature. In this work, we revisit the potential value of retrieval-based approaches to conversational recommendation. To that purpose, we compare two recent deep learning models for response generation with a retrieval-based method that determines a set of response candidates using a nearest-neighbor technique and heuristically reranks them. We adopt a user-centric evaluation approach, where study participants (N=60) rated the responses of the three compared systems. We could reproduce the claimed improvement of one of the deep learning methods over the other. However, the retrieval-based system outperformed both language generation based approaches in terms of the perceived quality of the system responses. Overall, our study suggests that retrieval-based approaches should be considered as an alternative or complement to modern language generation-based approaches.

As the adoption and diversity of cyber threat intelligence solutions continue to grow, questions about their effectiveness, particularly in regards to the quality of the data they provide, remain unanswered. Several studies have highlighted data quality issues as one of the most common barriers to effective threat intelligence sharing. Nevertheless, research and practice lack a common understanding of the expected quality of threat intelligence. To investigate these issues, our research utilised a systematic literature review followed by a modified Delphi study that involved 30 threat intelligence experts in Europe. We identify a set of threat intelligence quality dimensions along with revised definitions for threat data, information and intelligence.

Mobile malware are malicious programs that target mobile devices. They are an increasing problem, as seen in the rise of detected mobile malware samples per year. The number of active smartphone users is expected to grow, stressing the importance of research on the detection of mobile malware. Detection methods for mobile malware exist but are still limited. In this paper, we propose dynamic malware-detection methods that use device information such as the CPU usage, battery usage, and memory usage for the detection of 10 subtypes of Mobile Trojans on the Android Operating System (OS). We use a real-life sensor dataset containing device and malware data from 47 users for a year (2016) to create multiple mobile malware detection methods. We examine which features, i.e. aspects, of a device, are most important to monitor to detect (subtypes of) Mobile Trojans. The focus of this paper is on dynamic hardware features. Using these dynamic features we apply the following machine learning classifiers: Random Forest, K-Nearest Neighbour, and AdaBoost.

Battery-free sensing devices harvest energy from their surrounding environment to perform sensing, computation, and communication. This enables previously impossible applications in the Internet-of-Things. A core challenge for these devices is maintaining usefulness despite erratic, random or irregular energy availability; which causes inconsistent execution, loss of service and power failures. Adapting execution (degrading or upgrading) seems promising as a way to stave off power failures, meet deadlines, or increase throughput. However, because of constrained resources and limited local information, it is a challenge to decide when would be the best time to adapt, and how exactly to adapt execution. In this paper, we systematically explore the fundamental mechanisms of energy-aware adaptation, and propose heuristic adaptation as a method for modulating the performance of tasks to enable higher sensor coverage, completion rates, or throughput, depending on the application. We build a task based adaptive runtime system for intermittently powered sensors embodying this concept. We complement this runtime with a user facing simulator that enables programmers to conceptualize the tradeoffs they make when choosing what tasks to adapt, and how, relative to real world energy harvesting environment traces. While we target battery-free, intermittently powered sensors, we see general application to all energy harvesting devices. We explore heuristic adaptation with varied energy harvesting modalities and diverse applications: machine learning, activity recognition, and greenhouse monitoring, and find that the adaptive version of our ML app performs up to 46% more classifications with only a 5% drop in accuracy; the activity recognition app captures 76% more classifications with only nominal down-sampling; and find that heuristic adaptation leads to higher throughput versus non-adaptive in all cases.

For an online delivery platform, accurate physical locations of merchants are essential for delivery scheduling. It is challenging to maintain tens of thousands of merchant locations accurately because of potential errors introduced by merchants for profits (e.g., potential fraud). In practice, a platform periodically sends a dedicated crew to survey limited locations due to high workforce costs, leaving many potential location errors. In this paper, we design and implement ALWAES, a system that automatically identifies and corrects location errors based on fundamental tradeoffs of five measurement strategies from manual, physical, and virtual data collection infrastructures for online delivery platforms. ALWAES explores delivery data already collected by platform infrastructures to measure the travel time of couriers between merchants and verify all merchants' locations by cross-validation automatically. We explore tradeoffs between performance and cost of different measurement approaches. By comparing with the manually-collected ground truth, the experimental results show that ALWAES outperforms three other baselines by 32.2%, 41.8%, and 47.2%, respectively. More importantly, ALWAES saves 3,846 hours of the delivery time of 35,005 orders in a month and finds new erroneous locations that initially were not in the ground truth but are verified by our field study later, accounting for 3% of all merchants with erroneous locations.

We present DualRing, a novel ring-form input device that can capture the state and movement of the user's hand and fingers. With two IMU rings attached to the user's thumb and index finger, DualRing can sense not only the absolute hand gesture relative to the ground but also the relative pose and movement among hand segments. To enable natural thumb-to-finger interaction, we develop a high-frequency AC circuit for on-body contact detection. Based on the sensing information of DualRing, we outline the interaction space and divide it into three sub-spaces: within-hand interaction, hand-to-surface interaction, and hand-to-object interaction. By analyzing the accuracy and performance of our system, we demonstrate the informational advantage of DualRing in sensing comprehensive hand gestures compared with single-ring-based solutions. Through the user study, we discovered the interaction space enabled by DualRing is favored by users for its usability, efficiency, and novelty.

Contact-less liquid identification via wireless sensing has diverse potential applications in our daily life, such as identifying alcohol content in liquids, distinguishing spoiled and fresh milk, and even detecting water contamination. Recent works have verified the feasibility of utilizing mmWave radar to perform coarse-grained material identification, e.g., discriminating liquid and carpet. However, they do not fully exploit the sensing limits of mmWave in terms of fine-grained material classification. In this paper, we propose FG-LiquID, an accurate and robust system for fine-grained liquid identification. To achieve the desired fine granularity, FG-LiquID first focuses on the small but informative region of the mmWave spectrum, so as to extract the most discriminative features of liquids. Then we design a novel neural network, which uncovers and leverages the hidden signal patterns across multiple antennas on mmWave sensors. In this way, FG-LiquID learns to calibrate signals and finally eliminate the adverse effect of location interference caused by minor displacement/rotation of the liquid container, which ensures robust identification towards daily usage scenarios. Extensive experimental results using a custom-build prototype demonstrate that FG-LiquID can accurately distinguish 30 different liquids with an average accuracy of 97%, under 5 different scenarios. More importantly, it can discriminate quite similar liquids, such as liquors with the difference of only 1% alcohol concentration by volume.

For the past several decades, machine learning has played an important role in sports science with regard to player performance and result prediction. However, it is still challenging to quantify team-level game performance because there is no strong ground truth. Thus, a team cannot receive feedback in a standardized way. The aim of this study was twofold. First, we designed a metric called LAX-Score to quantify a collegiate lacrosse team's athletic performance. Next, we explored the relationship between our proposed metric and practice sensing features for performance enhancement. To derive the metric, we utilized feature selection and weighted regression. Then, the proposed metric was statistically validated on over 700 games from the last three seasons of NCAA Division I women's lacrosse. We also explored our biometric sensing dataset obtained from a collegiate team's athletes over the course of a season. We then identified the practice features that are most correlated with high-performance games. Our results indicate that LAX-Score provides insight into athletic performance beyond wins and losses. Moreover, though COVID-19 has stalled implementation, the collegiate team studied applied our feature outcomes to their practices, and the initial results look promising with regard to better performance.

Continuous and unobtrusive monitoring of facial expressions holds tremendous potential to enable compelling applications in a multitude of domains ranging from healthcare and education to interactive systems. Traditional, vision-based facial expression recognition (FER) methods, however, are vulnerable to external factors like occlusion and lighting, while also raising privacy concerns coupled with the impractical requirement of positioning the camera in front of the user at all times. To bridge this gap, we propose ExpressEar, a novel FER system that repurposes commercial earables augmented with inertial sensors to capture fine-grained facial muscle movements. Following the Facial Action Coding System (FACS), which encodes every possible expression in terms of constituent facial movements called Action Units (AUs), ExpressEar identifies facial expressions at the atomic level. We conducted a user study (N=12) to evaluate the performance of our approach and found that ExpressEar can detect and distinguish between 32 Facial AUs (including 2 variants of asymmetric AUs), with an average accuracy of 89.9% for any given user. We further quantify the performance across different mobile scenarios in presence of additional face-related activities. Our results demonstrate ExpressEar's applicability in the real world and open up research opportunities to advance its practical adoption.

The US CDC has recognized moist-heat as one of the most effective and accessible methods of decontaminating N95 masks for reuse in response to the persistent N95 mask shortages caused by the COVID-19 pandemic. However, it is challenging to reliably deploy this technique in healthcare settings due to a lack of smart technologies capable of ensuring proper decontamination conditions of hundreds of masks simultaneously. To tackle these challenges, we developed an open-source wireless sensor platform---VeriMask1 ---that facilitates per-mask verification of the moist-heat decontamination process. VeriMask is capable of monitoring hundreds of masks simultaneously in commercially available heating systems and provides a novel throughput-maximization functionality to help operators optimize the decontamination settings. We evaluate VeriMask in laboratory and real-scenario clinical settings and find that it effectively detects decontamination failures and operator errors in multiple settings and increases the mask decontamination throughput. Our easy-to-use, low-power, low-cost, scalable platform integrates with existing hospital protocols and equipment, and can be broadly deployed in under-resourced facilities to protect front-line healthcare workers by lowering their risk of infection from reused N95 masks. We also memorialize the design challenges, guidelines, and lessons learned from developing and deploying VeriMask during the COVID-19 Pandemic. Our hope is that by reflecting and reporting on this design experience, technologists and front-line health workers will be better prepared to collaborate for future pandemics, regarding mask decontamination, but also other forms of crisis tech.

The fall detection system is of critical importance in protecting elders through promptly discovering fall accidents to provide immediate medical assistance, potentially saving elders' lives. This paper aims to develop a novel and lightweight fall detection system by relying solely on a home audio device via inaudible acoustic sensing, to recognize fall occurrences for wide home deployment. In particular, we program the audio device to let its speaker emit 20kHz continuous wave, while utilizing a microphone to record reflected signals for capturing the Doppler shift caused by the fall. Considering interferences from different factors, we first develop a set of solutions for their removal to get clean spectrograms and then apply the power burst curve to locate the time points at which human motions happen. A set of effective features is then extracted from the spectrograms for representing the fall patterns, distinguishable from normal activities. We further apply the Singular Value Decomposition (SVD) and K-mean algorithms to reduce the data feature dimensions and to cluster the data, respectively, before input them to a Hidden Markov Model for training and classification. In the end, our system is implemented and deployed in various environments for evaluation. The experimental results demonstrate that our system can achieve superior performance for detecting fall accidents and is robust to environment changes, i.e., transferable to other environments after training in one environment.

Timely forecasting the urban anomaly events in advance is of great importance to the city management and planning. However, anomaly event prediction is highly challenging due to the sparseness of data, geographic heterogeneity (e.g., complex spatial correlation, skewed spatial distribution of anomaly events and crowd flows), and the dynamic temporal dependencies. In this study, we propose M-STAP, a novel Multi-head Spatio-Temporal Attention Prediction approach to address the problem of multi-region urban anomaly event prediction. Specifically, M-STAP considers the problem from three main aspects: (1) extracting the spatial characteristics of the anomaly events in different regions, and the spatial correlations between anomaly events and crowd flows; (2) modeling the impacts of crowd flow dynamic of the most relevant regions in each time step on the anomaly events; and (3) employing attention mechanism to analyze the varying impacts of the historical anomaly events on the predicted data. We have conducted extensive experimental studies on the crowd flows and anomaly events data of New York City, Melbourne and Chicago. Our proposed model shows higher accuracy (41.91% improvement on average) in predicting multi-region anomaly events compared with the state-of-the-arts.

In-app privacy notices can help smartphone users make informed privacy decisions. However, they are rarely used in real-world apps, since developers often lack the knowledge, time, and resources to design and implement them well. We present Honeysuckle, a programming tool that helps Android developers build in-app privacy notices using an annotation-based code generation approach facilitated by an IDE plugin, a build system plugin, and a library. We conducted a within-subjects study with 12 Android developers to evaluate Honeysuckle. Each participant was asked to implement privacy notices for two popular open-source apps using the Honeysuckle library as a baseline as well as the annotation-based approach. Our results show that the annotation-based approach helps developers accomplish the task faster with significantly lower cognitive load. Developers preferred the annotation-based approach over the library approach because it was much easier to learn and use and allowed developers to achieve various types of privacy notices using a unified code format, which can enhance code readability and benefit team collaboration.

This study investigated how staff working at a Danish nursing home experienced, perceived, and used circadian lighting for two years after its installation. The purpose of the installed circadian lighting was to improve the staff and residents' health and comfort. The paper is based on an action research methodology that used interviews, observations, and a questionnaire to investigate 42 staff members' perceived visual comfort, satisfaction with, and perceptions of the usefulness of the circadian lighting. The findings revealed that circadian light was perceived as satisfactory by the staff and was perceived as a more adequate light for work than the existing lighting system. Being able to adjust the lighting was perceived as important by staff for maintaining visibility, setting the lighting depending on the activities, and meeting residents' needs. This paper demonstrates the value of applying mixed methods when analyzing subjective assessment of light and visual comfort. We present an alternative card sorting method for studying perceptions of a 24-hour lighting application. Finally, the study demonstrates the value of evaluating the lighting with end-users after two years in use to improve future lighting installations and to adjust the current installation.

Counting mosquitoes in the wild is a crucial capability for monitoring, prediction, and control of vector-borne diseases. Current approaches are mainly manual, where specially designed mosquito traps or ovitraps are placed in areas of interest and recovered the next day. The counting itself is performed in an entomological laboratory, where individual mosquitoes are classified into species and counted. This process is costly, slow and inefficient. At the same time, mosquito counting is most relevant in tropical and sub-tropical countries, where mosquitoes spread deadly diseases like malaria, yellow fever and dengue fever. Many countries in these regions have relatively weak public health systems and so cannot support large-scale vector counting efforts. In this paper, we present a system architecture and a prototype to count mosquitoes in the wild with an Internet of Things approach. A sensor board is developed to gather audio data, and models are developed to detect, classify, and count mosquito species. Here, we present our prototype and an extensive background study of classifying mosquitoes based on sound recordings and some preliminary results and discussion.

Participatory budgeting (PB) is already well established in Scotland in the form of community led grant-making yet has recently transformed from a grass-roots activity to a mainstream process or embedded 'policy instrument'. An integral part of this turn is the use of the Consul digital platform as the primary means of citizen participation. Using a mixed method approach, this ongoing research paper explores how each of the 32 local authorities that make up Scotland utilise the Consul platform to engage their citizens in the PB process and how they then make sense of citizens' contributions. In particular, we focus on whether natural language processing (NLP) tools can facilitate both citizen engagement, and the processes by which citizens' contributions are analysed and translated into policies.

Agriculture including aquaculture has been changing through multiple technological transformations in recent years. The Internet of Things (IoT) and Artificial Intelligence (AI) are providing remarkable technological innovations on fish farming. In this research, we present an automated IoT and AI-based system to improve fish farming. The proposed system uses multiple sensors to measure in real-time water quality chemical parameters such as: temperature, pH, turbidity, electrical conductivity, total dissolved solids, etc., from the fish pond and send them on a cloud database to allow fish farmers to access them in realtime with their devices (mobile phone, PC, tablets). The system contains three web applications which fish farmers can use. The first web application enables farmers with realtime visualizations of sensors data, issues alerts and remote pumps controls. Fish farmers can use the second web application for fish disease detection and to receive suggestions for diseases' care. This would help to classify two fish diseases which are: Epizootic Ulcerative Syndrome(EUS), and Ichthyophthirus(Ich). The third web application is a digital community platform for knowledge sharing, capacity building, market opportunities and collaboration among fish farmers. Our system can help reduce human efforts, reinforce capacity building, increase fish production and market opportunities for fish farmers.

Fatigue is a common symptom in various diseases, including multiple sclerosis (MS). The current standard method to assess fatigue is through questionnaires, which has several shortcomings; questionnaires are subjective, prone to recall bias, and potentially confounded by other symptoms like stress and depression. Thus, there is an unmet medical need to develop objective and reliable methods to evaluate fatigue. Our study seeks to develop an objective and ubiquitous monitoring tool for assessing fatigue. Leveraging a smartphone-based rapid tapping task, we conducted a two-week in-the-wild study with 35 MS patients. We explore the association between tapping derived metrics and perceived fatigue assessed with two standard clinical scales: fatigue severity scale (FSS) and fatigue scale for motor and cognitive function (FSMC). Our novel smartphone-based fatigue metric, mean tapping frequency, objectively ranks perceived fatigue with a mean AUCROC = .76, CI = [.71, .81] according to the FSMC, and a mean AUCROC = .81, CI = [.76, .86] according to the FSS. These results demonstrate that our approach is feasible and valid in uncontrolled environments. In this work, we provide a promising tool for objective fatigue monitoring to be used in clinical trials and routine medical care.

Text entry on a smartwatch is challenging due to its small form factor. Handwriting recognition using the built-in sensors of the watch (motion sensors, microphones, etc.) provides an efficient and natural solution to deal with this issue. However, prior works mainly focus on individual letter recognition rather than word recognition. Therefore, they need users to pause between adjacent letters for segmentation, which is counter-intuitive and significantly decreases the input speed. In this paper, we present 'Write, Attend and Spell' (WriteAS), a word-level text-entry system which enables free-style handwriting recognition using the motion signals of the smartwatch. First, we design a multimodal convolutional neural network (CNN) to abstract motion features across modalities. After that, a stacked dilated convolutional network with an encoder-decoder network is applied to get around letter segmentation and output words in an end-to-end way. More importantly, we leverage a multi-task sequence learning method to enable handwriting recognition in a streaming way. We construct the first sequence-to-sequence handwriting dataset using smartwatch. WriteAS can yield 9.3% character error rate (CER) on 250 words for new users and 3.8% CER for words unseen in the training set. In addition, WriteAS can handle various writing conditions very well. Given the promising performance, we envision that WriteAS can be a fast and accurate input tool for smartwatch.

This paper presents WR-Hand, a wearable-based system tracking 3D hand pose of 14 hand skeleton points over time using Electromyography (EMG) and gyroscope sensor data from commercial armband. This system provides a significant leap in wearable sensing and enables new application potentials in medical care, human-computer interaction, etc. A challenge is the armband EMG sensors inevitably collect mixed EMG signals from multiple forearm muscles because of the fixed sensor positions on the device, while prior bio-medical models for hand pose tracking are built on isolated EMG signal inputs from isolated forearm spots for different muscles. In this paper, we leverage the recent success of neural networks to enhance the existing bio-medical model using the armband's EMG data and visualize our design to understand why our solution is effective. Moreover, we propose solutions to place the constructed hand pose reliably in a global coordinate system, and address two practical issues by providing a general plug-and-play version for new users without training and compensating for the position difference in how users wear their armbands. We implement a prototype using different commercial armbands, which is lightweight to execute on user's phone in real-time. Extensive evaluation shows the efficacy of the WR-Hand design.

Smart speakers, such as Google Home and Amazon Echo, have become popular. They execute user voice commands via their built-in functionalities together with various third-party voice-controlled applications, called skills. Malicious skills have brought significant threats to users in terms of security and privacy. As a countermeasure, only skills passing the strict vetting process can be released onto markets. However, malicious skills have been reported to exist on markets, indicating that the vetting process can be bypassed. This paper aims to demystify the vetting process of skills on main markets to discover weaknesses and protect markets better. To probe the vetting process, we carefully design numerous skills, perform the Turing test, a test for machine intelligence, to determine whether humans or machines perform vetting, and leverage natural language processing techniques to analyze their behaviors. Based on our comprehensive experiments, we gain a good understanding of the vetting process (e.g., machine or human testers and skill exploration strategies) and discover some weaknesses. In this paper, we design three types of attacks to verify our results and prove an attacker can embed sensitive behaviors in skills and bypass the strict vetting process. Accordingly, we also propose countermeasures to these attacks and weaknesses.

A myriad of IoT applications, ranging from tracking assets in hospitals, logistics, and construction industries to indoor tracking in large indoor spaces, demand centimeter-accurate localization that is robust to blockages from hands, furniture, or other occlusions in the environment. With this need, in the recent past, Ultra Wide Band (UWB) based localization and tracking has become popular. Its popularity is driven by its proposed high bandwidth and protocol specifically designed for localization of specialized "tags". This high bandwidth of UWB provides a fine resolution of the time-of-travel of the signal that can be translated to the location of the tag with centimeter-grade accuracy in a controlled environment. Unfortunately, we find that high latency and high-power consumption of these time-of-travel methods are the major culprits which prevent such a system from deploying multiple tags in the environment. Thus, we developed ULoc, a scalable, low-power, and cm-accurate UWB localization and tracking system. In ULoc, we custom build a multi-antenna UWB anchor that enables azimuth and polar angle of arrival (henceforth shortened to '3D-AoA') measurements, with just the reception of a single packet from the tag. By combining multiple UWB anchors, ULoc can localize the tag in 3D space. The single-packet location estimation reduces the latency of the entire system by at least 3×, as compared with state of art multi-packet UWB localization protocols, making UWB based localization scalable. ULoc's design also reduces the power consumption per location estimate at the tag by 9×, as compared to state-of-art time-of-travel algorithms. We further develop a novel 3D-AoA based 3D localization that shows a stationary localization accuracy of 3.6 cm which is 1.8× better than the state-of-the-art two-way ranging (TWR) systems. We further developed a temporal tracking system that achieves a tracking accuracy of 10 cm in mobile conditions which is 4.3× better than the state-of-the-art TWR systems.

This paper presents a non-invasive design, namely RF-ray, to recognize the shape and material of an object simultaneously. RF-ray puts the object approximate to an RFID tag array, and explores the propagation effect as well as coupling effect between RFIDs and the object for sensing. In contrast to prior proposals, RF-ray is capable to recognize unseen objects, including unseen shape-material pairs and unseen materials within a certain container. To make it real, RF-ray introduces a sensing capability enhancement module and leverages a two-branch neural network for shape profiling and material identification respectively. Furthermore, we incorporate a Zero-Shot Learning based embedding module that incorporates the well-learned linguistic features to generalize RF-ray to recognize unseen materials. We build a prototype of RF-ray using commodity RFID devices. Comprehensive real-world experiments demonstrate our system can achieve high object recognition performance.

The rising of online social platforms makes large volumes of data about social relationships and interactions available to the research community. In the varied ecosystem of techno-social platforms, blockchain-based online social networks - BOSNs - are gaining momentum since the underlying blockchain offers data validation, data storage, and data decentralization. As data sources, BOSNs provide high-resolution temporal data about the evolution of the social network and on the interactions of users with the platform services. In this study, we focus on a few temporal characteristics, by analyzing the dynamics of the link creation process and the claiming of rewards in the BOSN Steemit. We model blockchain data as a temporal directed network from which we extract the time series characterizing link creation and reward claims. Adopting a user-centric approach, we evaluate the heterogeneity of the time series through the inter-event time distribution, the burstiness, the bursty train size distribution, and the fitting of inter-event times by power law models. The outcomes of the analysis highlight that the above processes show bursty traits typical of human dynamics. However, the two aspects present a few differences concerning the types of models describing their behavior and the time scale of their bursty nature. To sum up, the creation of new relationships and the reward claim dynamics ask for specific models able to reproduce their general bursty traits but taking into account their specificities and relations with other services and mechanisms offered by BOSN platforms.

The advancements in terms of networking, image resolution, computer vision (CV) and mobile cloud computing performances are transforming Mobile Augmented Reality (MAR) into a technology which may be put to good use in a variety of everyday contexts. To make this point, we here consider an artisanal craft rooted back into the past, key locksmithing, and show how today MAR capabilities may simplify and ameliorate the performances of such ancient trade. To this aim, we introduce the requirements posed by such craft and present a MAR-based workflow which may be implemented to support and speed its execution. This could also impact the everyday lives of ordinary citizens, since it pose the bases of remote locksmithing activities.