This paper studies the creation of textual descriptions of user activities and interactions on smartphones. Our approach of referring to encrypted mobile traffic exceeds traditional smartphone activity classification methods in terms of model scalability and output readability. The paper addresses two obstacles to the realization of this idea: the semantic gap between traffic features and smartphone activity captions, and the lack of textually annotated traffic data. To overcome these challenges, we introduce a novel smartphone activity captioning system, called T2T (Traffic-to-Text). T2T consists of a flow feature encoder that converts low-level traffic characteristics into meaningful latent features and a caption decoder to yield readable transcripts of smartphone activities. In addition, T2T achieves the automatic textual annotation of mobile traffic by feeding synchronized screen capture videos into the Qwen-VL-Max vision-language model, and proposing multi-stage losses for effective cross-model training. We evaluate T2T on 40,000 traffic-description pairs collected in two real-world environments, involving 8 smartphone users and 20 mobile apps. T2T achieves a BLEU-4 score of 5
Due to the openness of the wireless medium, smartphone users are susceptible to user privacy attacks, where user privacy information is inferred from encrypted Wi-Fi wireless traffic. Existing attacks are limited to recognizing mobile apps and their actions and cannot infer the smartphone user identity, a fundamental part of user privacy. To overcome this limitation, we propose U-Print, a novel attack system that can passively recognize smartphone apps, actions, and users from over-the-air MAC-layer frames. We observe that smartphone users usually prefer different add-on apps and in-app actions, yielding different changing patterns in Wi-Fi traffic. U-Print first extracts multi-level traffic features and exploits customized temporal convolutional networks to recognize smartphone apps and actions, thus producing users' behavior sequences. Then, it leverages the silhouette coefficient method to determine the number of users and applies the k-means clustering to profile and identify smartphone users. We implement U-Print using a laptop with a Kali dual-band wireless network card and evaluate it in three real-world environments. U-Print achieves an overall accuracy of 98.4% and an F1 s
Excessive use of smartphones is a worldwide known issue. In this study, we proposed a notification-based intervention approach to reduce smartphone overuse without making the user feel any annoyance or irritation. Most of the work in this field tried to reduce smartphone overuse by making smartphone use more difficult for the user. In our user study (n = 109), we found that 19.3% of the participants are unwilling to use any usage-limiting application because a) they do not want their smartphone activities to get restricted or b) those applications are annoying. Following that, we devised a hypothesis to minimize smartphone usage among undergraduates. Finally, we designed a prototype for Android, "App Usage Monitor," and conducted a 3-week experiment through which we found proof of concept for our hypothesis. In our prototype, we combined techniques such as nudge and visualization to increase self-awareness among the user by leveraging notifications.
Smartphones sensors are now commonly used by a worldwide audience thanks to their availability, high connectivity, and versatility. Here, we present a methodology to use a collection of smartphones, namely a fleet, as a distributed network of time-synchronized mechanical sensors. We first present the mechanical tests we develop to evaluate the smartphone sensor accuracy. We then describe how to use efficiently a distributed network of smartphones as autonomous sensors. We use a combination of an Android application hosted on each phone (Gobannos), and a server application (Phonefleet) on a controlling host to perform the tasks in parallel remotely. We implement in particular a time synchronization protocol based on UDP communication. We achieved an accuracy of the smartphone clock synchronisation of 60 microseconds. Using two test cases in realistic outdoor conditions, we eventually prove the reliability of a smartphone fleet to measure mechanical wave measurements in field conditions.
The quality of social interaction is crucial for psychological and physiological health. Previous research shows that smartphones can negatively impact face-to-face social interactions. Many HCI studies have addressed this by limiting smartphone use during social interactions. While these studies show a decrease in smartphone use, restrictive approaches have their drawbacks. Users need high levels of self-regulation to follow them, and they may cause unintended effects like withdrawal symptoms. Given the impact of smartphones on social interactions, both positive and negative, new solutions are needed to reduce the negative effects of excessive smartphone use without resorting to restrictive methods. This thesis aims to explore smartphone use behavior in the context of social interactions and relationships using various data collection techniques to understand how this behavior hinders and supports social interactions. We began with in situ observations and focus group sessions. Based on insights from these steps, we developed two research prototypes to improve social interactions without restricting smartphone use. We gathered user feedback, reactions, and concerns about these pro
Excessive smartphone use is now widely considered a personal and societal problem. It is recognized by application and smartphone makers, who provide tools to track the amount of use, set limits, or block certain services at predefined times. These tools, while powerful, may require significant cognitive effort to operate: configuration parameters need to be set, and captured statistics need to be analyzed. To offer a complementary solution, we propose a radically different approach. We employ the keyboard of a smartphone as an output device. With each press of a key, the user is given a high-level, qualitative, color-encoded estimate of the amount of recent smartphone use. The technique, dubbed the informative keyboard, is a case of implicit interaction: the user's intention is to enter text but, while typing, they receive the feedback. In the paper, we elaborate the concept, identify design decisions, describe our implementation, present the outcome of a questionnaire-based evaluation, and point to some other applications of the informative keyboard.
While text entry is an essential and frequent task in Augmented Reality (AR) applications, devising an efficient and easy-to-use text entry method for AR remains an open challenge. This research presents STAR, a smartphone-analogous AR text entry technique that leverages a user's familiarity with smartphone two-thumb typing. With STAR, a user performs thumb typing on a virtual QWERTY keyboard that is overlain on the skin of their hands. During an evaluation study of STAR, participants achieved a mean typing speed of 21.9 WPM (i.e., 56% of their smartphone typing speed), and a mean error rate of 0.3% after 30 minutes of practice. We further analyze the major factors implicated in the performance gap between STAR and smartphone typing, and discuss ways this gap could be narrowed.
This paper examines inequalities in the usage of smartphone technology based on five samples of smartphone owners collected in Germany and Austria between 2016 and 2020. We identify six distinct types of smartphone users by conducting latent class analyses that classify individuals based on their frequency of smartphone use, self-rated smartphone skills, and activities carried out on their smartphone. The results show that the smartphone usage types differ significantly by sociodemographic and smartphone-related characteristics: The types reflecting more frequent and diverse smartphone use are younger, have higher levels of educational attainment, and are more likely to use an iPhone. Overall, the composition of the latent classes and their characteristics are robust across samples and time.
Smartphones and smartphone apps have undergone an explosive growth in the past decade. However, smartphone battery technology hasn't been able to keep pace with the rapid growth of the capacity and the functionality of smartphones and apps. As a result, battery has always been a bottleneck of a user's daily experience of smartphones. An accurate estimation of the remaining battery life could tremendously help the user to schedule their activities and use their smartphones more efficiently. Existing studies on battery life prediction have been primitive due to the lack of real-world smartphone usage data at scale. This paper presents a novel method that uses the state-of-the-art machine learning models for battery life prediction, based on comprehensive and real-time usage traces collected from smartphones. The proposed method is the first that identifies and addresses the severe data missing problem in this context, using a principled statistical metric called the concordance index. The method is evaluated using a dataset collected from 51 users for 21 months, which covers comprehensive and fine-grained smartphone usage traces including system status, sensor indicators, system even
Vibration feedback is common in everyday devices, from virtual reality systems to smartphones. However, cognitive and physical activities may impede our ability to sense vibrations from devices. In this study, we develop and characterize a smartphone platform to investigate how a shape-memory task (cognitive activity) and walking (physical activity) impair human perception of smartphone vibrations. We measured how Apple's Core Haptics Framework parameters can be used for haptics research, namely how hapticIntensity modulates amplitudes of 230 Hz vibrations. A 23-person user study found that physical (p<0.001) and cognitive (p=0.004) activity increase vibration perception thresholds. Cognitive activity also increases vibration response time (p<0.001). This work also introduces a smartphone platform that can be used for out-of-lab vibration perception testing. Researchers can use our smartphone platform and results to design better haptic devices for diverse, unique populations.
Smartphone technology is more and more becoming the predominant communication tool for people across the world. People use their smartphones to keep their contact data, to browse the internet, to exchange messages, to keep notes, carry their personal files and documents, etc. Users while browsing are also capable of shopping online, thus provoking a need to type their credit card numbers and security codes. As the smartphones are becoming widespread so do the security threats and vulnerabilities facing this technology. Recent news and articles indicate huge increase in malware and viruses for operating systems employed on smartphones (primarily Android and iOS). Major limitations of smartphone technology are its processing power and its scarce energy source since smartphones rely on battery usage. Since smartphones are devices which change their network location as the user moves between different places, intrusion detection systems for smartphone technology are most often classified as IDSs designed for mobile ad-hoc networks. The aim of this research is to give a brief overview of IDS technology, give an overview of major machine learning and pattern recognition algorithms used i
Smartphones and robots can have an adversarial or a symbiotic relationship because they strive to serve overlapping customer needs. While smartphones are prevalent, humanoid robots are not. Even though considerable public and private resources are being invested in developing and commercializing humanoid robots, progress seems stalled and no humanoid robot can be said to be successful with consumers. A part from the obvious engineering differences between humanoids and smartphones, other economic factors influence this situation. On one hand, the product cycle of robots is slower than smartphones. This makes robot computing hardware, (as it with automobile's infotainment systems), perennially outdated when side-by-side to a smartphone. On the other hand, the incentives to develop Apps are high for smartphones and they are comparatively low for robot platforms. Here, we point to how smartphones could be used to lower hardware cost and foster robot app development.
Smartphones are equipped with sensors such as accelerometers, gyroscope, and GPS in one cost-effective device with an acceptable level of accuracy. There have been some research studies carried out in terms of using smartphones to measure the pavement roughness. However, a little attention has been paid to investigate the validity of the measured pavement roughness by smartphones via other subjective methods such as the user opinion. This paper aims at calculating the pavement roughness data with a smartphone using its embedded sensors and investigating its correlation with a user opinion about the ride quality. In addition, the applicability of using smartphones to assess the pavement surface distresses is examined. Furthermore, to validate the smartphone sensor outputs objectively, the Road Surface Profiler is applied. Finally, a good roughness model is developed which demonstrates an acceptable level of correlation between the pavement roughness measured by smartphones and the ride quality rated by users.
With the rapid growth of sensor technology, smartphone sensing has become an effective approach to improve the quality of smartphone applications. However, due to time-varying wireless channels and lack of incentives for the users to participate, the quality and quantity of the data uploaded by the smartphone users are not always satisfying. In this paper, we consider a smartphone sensing system in which a platform publicizes multiple tasks, and the smartphone users choose a set of tasks to participate in. In the traditional non-cooperative approach with incentives, each smartphone user gets rewards from the platform as an independent individual and the limit of the wireless channel resources is often omitted. To tackle this problem, we introduce a novel cooperative approach with an overlapping coalition formation game (OCF-game) model, in which the smartphone users can cooperate with each other to form the overlapping coalitions for different sensing tasks. We also utilize a centralized case to describe the upper bound of the system sensing performance. Simulation results show that the cooperative approach achieves a better performance than the non-cooperative one in various situa
Authentication of smartphone users is important because a lot of sensitive data is stored in the smartphone and the smartphone is also used to access various cloud data and services. However, smartphones are easily stolen or co-opted by an attacker. Beyond the initial login, it is highly desirable to re-authenticate end-users who are continuing to access security-critical services and data. Hence, this paper proposes a novel authentication system for implicit, continuous authentication of the smartphone user based on behavioral characteristics, by leveraging the sensors already ubiquitously built into smartphones. We propose novel context-based authentication models to differentiate the legitimate smartphone owner versus other users. We systematically show how to achieve high authentication accuracy with different design alternatives in sensor and feature selection, machine learning techniques, context detection and multiple devices. Our system can achieve excellent authentication performance with 98.1% accuracy with negligible system overhead and less than 2.4% battery consumption.
Geometric optical distortion is a significant contributor to the astrometric error budget in large telescopes using adaptive optics. To increase astrometric precision, optical distortion calibration is necessary. We investigate using smartphone OLED screens as astrometric calibrators. Smartphones are low cost, have stable illumination, and can be quickly reconfigured to probe different spatial frequencies of an optical system's geometric distortion. In this work, we characterize the astrometric accuracy of a Samsung S20 smartphone, with a view towards providing large format, flexible astrometric calibrators for the next generation of astronomical instruments. We find the placement error of the pixels to be 189 nm +/- 15 nm RMS. At this level of error, milliarcsecond astrometric accuracy can be obtained on modern astronomical instruments.
The widespread use of smartphones gives rise to new security and privacy concerns. Smartphone thefts account for the largest percentage of thefts in recent crime statistics. Using a victim's smartphone, the attacker can launch impersonation attacks, which threaten the security of the victim and other users in the network. Our threat model includes the attacker taking over the phone after the user has logged on with his password or pin. Our goal is to design a mechanism for smartphones to better authenticate the current user, continuously and implicitly, and raise alerts when necessary. In this paper, we propose a multi-sensors-based system to achieve continuous and implicit authentication for smartphone users. The system continuously learns the owner's behavior patterns and environment characteristics, and then authenticates the current user without interrupting user-smartphone interactions. Our method can adaptively update a user's model considering the temporal change of user's patterns. Experimental results show that our method is efficient, requiring less than 10 seconds to train the model and 20 seconds to detect the abnormal user, while achieving high accuracy (more than 90%)
Smartphones have been shipped with multiple wireless network interfaces in order to meet their diverse communication and networking demands. However, as smartphones increasingly rely on wireless network connections to realize more functions, the demand of energy has been significantly increased, which has become the limit for people to explore smartphones' real power. In this paper, we first review typical smartphone computing systems, energy consumption of smartphone, and state-of-the-art techniques of energy saving for smartphones. Then we propose a location-assisted Wi-Fi discovery scheme, which discovers the nearest Wi-Fi network access points (APs) by using the user's location information. This allows the user to switch to the Wi-Fi interface in an intelligent manner when he/she arrives at the nearest Wi-Fi network AP. Thus we can meet the user's bandwidth needs and provide the best connectivity. Additionally, it avoids the long periods in idle state and greatly reduces the number of unnecessary Wi-Fi scans on the mobile device. Our experiments and simulations demonstrate that our scheme effectively saves energy for smartphones integrated with Wi-Fi and cellular interfaces.
The outbreak of COVID-19 exposed the inadequacy of our technical tools for home health surveillance, and recent studies have shown the potential of smartphones as a universal optical microscopic imaging platform for such applications. However, most of them use laboratory-grade optomechanical components and transmitted illuminations to ensure focus tuning capability and imaging quality, which keeps the cost of the equipment high. Here we propose an ultra-low-cost solution for smartphone microscopy. To realize focus tunability, we designed a seesaw-like structure capable of converting large displacements on one side into small displacements on the other (reduced to ~9.1%), which leverages the intrinsic flexibility of 3D printing materials. We achieved a focus-tuning accuracy of ~5 micron, which is 40 times higher than the machining accuracy of the 3D-printed lens holder itself. For microscopic imaging, we use an off-the-shelf smartphone camera lens as the objective and the built-in flashlight as the illumination. To compensate for the resulting image quality degradation, we developed a learning-based image enhancement method. We use the CycleGAN architecture to establish the mapping
Artificial Intelligence has now taken centre stage in the smartphone industry owing to the need of bringing all processing close to the user and addressing privacy concerns. Convolution Neural Networks (CNNs), which are used by several AI applications, are highly resource and computation intensive. Although new generation smartphones come with AI-enabled chips, minimal memory and energy utilisation is essential as many applications are run concurrently on a smartphone. In light of this, optimising the workload on the smartphone by offloading a part of the processing to a cloud server is an important direction of research. In this paper, we analyse the feasibility of splitting CNNs between smartphones and cloud server by formulating a multi-objective optimisation problem that optimises the end-to-end latency, memory utilisation, and energy consumption. We design SmartSplit, a Genetic Algorithm with decision analysis based approach to solve the optimisation problem. Our experiments run with multiple CNN models show that splitting a CNN between a smartphone and a cloud server is feasible. The proposed approach, SmartSplit fares better when compared to other state-of-the-art approaches