LokaPulseTM
Determining your heart rate from a selfie video.
This page is dedicated to the journey of LokaPulseTM,
from idea to testing to deployment and improvements.
We dive deep into backend and frontend technologies, challenges and eureka moments. For people interested in the big ideas and key takeaways, look to the IDEA and TODAY sections, as well as any highlighted areas.
We dive deep into backend and frontend technologies, challenges and eureka moments. For people interested in the big ideas and key takeaways, look to the IDEA and TODAY sections, as well as any highlighted areas.
IDEA
Mobile is just scratching the surface of machine learning capabilities. LokaPulseTM
was designed to showcase the possibilities of ML and Flutter by giving users an Apple
and Android app that would measure their heart rate in an innovative and beautiful way.
With a simple 15-second video, LokaPulseTM can extract a user’s BPM information, categorize the data, and compare it to previous videos, giving anyone with a smartphone the ability to see and track their heart rate.
The idea for LokaPulseTM started when a Loka developer stumbled upon a method called Eulerian Video Magnification. “Super Seeing”, as it’s also known, amplifies and visualizes temporal variations in videos that are difficult or even impossible to see with the naked eyes. These visual variations include motion and color.
As Loka developers and the team behind LokaPulseTM, we wanted to see for ourselves how this method worked. We implemented the algorithm from scratch and applied it to several different videos. The results were inspiring. Even the most subtle variations were amplified beyond what we imagined.
Though the potential applications for this technology were endless, what really caught our attention was the color magnification. The idea of extracting “invisible” biosignals from a video was exciting to say the least. And we knew it could be used as a heart rate calculator.
This is the moment that sparked the big question, what would happen if everyone with a phone could measure their heart rate with a selfie video? What else could this unlock for people’s daily health - and for life sciences? And so, the idea of developing LokaPulseTM was born.
JOURNEY
Going from idea to fully-functioning app is never short on challenges. The question is, when will the challenges emerge and how will you respond.
Our first step was to do an in-depth analysis of the state of the art (SOTA) and available datasets. After we explored all kinds of possible bio signals that can be extracted from video, we decided to start with heart rate estimation.
The research into extracting a heart rate signal from video led us to a technology called remote photoplethysmography (rPPG) which relies on the reflection of light on the subjects skin.
This technology is the most prevalent in SOTA and seemed to benefit from the increased variations provided by Eulerian color Magnification. However, as we dove deeper into SOTA methods and comparative studies, we noticed a troubling pattern: The improvements provided by Eulerian Magnification often didn’t outweigh its limitations.
The improvements provided by Eulerian Magnification often didn’t outweigh its limitations.
Here’s what we encountered as we explored SOTA methods and tools:
- Increased variations caused by “noise”, e.g. motion and light conditions.
- Performance seemed highly dependent on parameter tuning reducing the system’s robustness for use in the wild.
- The tools we came across relied on extremely outdated library versions
- The tools had serious bugs that prevented efficient use.
- The tools they simply had very few methods implemented
To overcome these obstacles, we developed a pipeline architecture based on these tools where we could plug and play at different stages to compare performances of the different methods and processing steps. For the comparisons, this pipeline architecture could be used in a benchmarking tool that allowed automated comparisons for different parameters.
Benchmarking different methods and parameters required labelled data with which to compare the predictions. We evaluated commonly used datasets focusing on their similarity to our use case and on subject diversity. We started testing with a dataset that would be our ideal scenario and from there moved to videos captured in real situations.
Through experiments and research, we concluded that SOTA methods tended to require fine tuning for each specific dataset – and most tended to increase the pre and post processing complexity by adding extra stages while relying on the same core principles. Since we required near real time results as we iterated, we leaned toward simpler approaches, even if they were not the most recent and best performing methods.
Here’s the flow of the solution we chose
- Image rescale and crop on the app.
- Use a Deep Learning model to detect and extract user’s face.
- Extract the PPG signal.
- Use Welch’s method to obtain the power spectrum and calculate the heart rate.
As we moved to real world application, we faced three critical obstacles:
- Variations in camera quality between phones (e.g. resolutions and frame rate), which prevent video compression after recording.
- Lighting conditions - the most persistent and stubborn problem.
- And the biggest limitation of the app: Performance deteriorates sharply for darker skin types
The issue of skin tone stems from using rPPG to extract the heart rate. Since rPPG relies on the difference between reflected and absorbed light, skin types that absorb more light provide a signal with lower Signal to Noise Ratio (SNR) damaging performance, similarly to what happens under poor lighting conditions.
This isn’t a new problem. Products like Oxymeters, which became more prevalent during the pandemic, rely on PPG signals and are charged with the same issues. This continues to be a challenge to LokaPulseTM and other programs using PPG signals will need to face and overcome.
FRAMEWORK
At the beginning of any software project, there is always a big question: Which mobile framework/language is better for my use case – and is it good enough for the long term? For LokaPulseTM, we decided that we wanted a multi-platform framework and narrowed it down to two options: Flutter or React Native.
Both are really good options for our use case, offering multi-platform support and large communities. Ultimately Flutter was the right choice based on the following criteria:
Support: thanks to the Flutter canvas approach, we don't need to worry about outdated APIs or integrations with a specific platform. Flutter provides a white canvas and puts us in charge of rendering each pixel on the screen.
Performance: 60fps (or even 120 fps) is a really attractive feature for smooth user experiences.
Custom design: We could create our own designs, so we didn't need to reuse the platform specific UI components.
Time-to-market: Flutter is known for lightning fast development.
For us, building LokaPulseTM became as much of a way to showcase the capabilities of machine learning as it was to highlight the outstanding performance of Flutter.
FRONTEND
At its core, LokaPulseTM is a Flutter app that measures a users heart rate via video recording of their face. The application provides a way to record a video, process the video, and present all the data connected with previous measurements.
To improve the performance and user experience, the front end is capable of “cropping” the video to only send the required information, the white square that contains the face. With this we ensure that we are not using extra data of your carrier plan and also the Face detection and heart rate algorithms run faster than using the original video.
To improve the performance and user experience, the front end is capable of “cropping” the video to only send the required information, the white square that contains the face. With this we ensure that we are not using extra data of your carrier plan and also the Face detection and heart rate algorithms run faster than using the original video.
Tagging
LokaPulseTM uses a custom ffmpeg command for tagging. After recording a video, users are prompted to select a tag, which defines the state that they were in while recording the video.
Tags can be used to track whether a user was stressed, relaxed, exhausted, excited, etc., so users can better understand their results and trends over weeks and months. LokaPulseTM also gives users the option to manually enter a comment, or even use speech-to-text, for better explanation of the moment when the measurement was taken.
LokaPulseTM uses a custom ffmpeg command for tagging. After recording a video, users are prompted to select a tag, which defines the state that they were in while recording the video.
Tags can be used to track whether a user was stressed, relaxed, exhausted, excited, etc., so users can better understand their results and trends over weeks and months. LokaPulseTM also gives users the option to manually enter a comment, or even use speech-to-text, for better explanation of the moment when the measurement was taken.
When you’re running, your heart rate will be elevated. LokaPulseTM allows users to tag these measurements and separate them from categories like ‘chill time’.
Speech-to-text
Speech-to-text offered our team some challenges. The package that we used worked differently on Apple and Google platforms. Little tweaks and workarounds were necessary. For example, LokaPulseTM would stop working after a few moments of silence on Android phones. So after 30 seconds of silence, we made it so the speech-to-text would stop and re-engage.
Speech-to-text offered our team some challenges. The package that we used worked differently on Apple and Google platforms. Little tweaks and workarounds were necessary. For example, LokaPulseTM would stop working after a few moments of silence on Android phones. So after 30 seconds of silence, we made it so the speech-to-text would stop and re-engage.
Anomalies
We built a way to detect anomalies into LokaPulseTM. If a measurement result is not in the range of 60 to 120 bpm, the application gives the user a notification that the heart rate is not in their normal range or inaccurate due shaking or poor lighting conditions.
We built a way to detect anomalies into LokaPulseTM. If a measurement result is not in the range of 60 to 120 bpm, the application gives the user a notification that the heart rate is not in their normal range or inaccurate due shaking or poor lighting conditions.
Seeing & Managing Results
Users can find their data in a list organized by date, tag, and heart rate. Pressing on any measurement gives users the option to see, edit and delete past notes.
Users can find their data in a list organized by date, tag, and heart rate. Pressing on any measurement gives users the option to see, edit and delete past notes.
Visualization
For a better user experience, we created a separate screen where data is presented as a bar chart for the current week and month, with multiple tag selection capabilities. Users can also see their average, maximum and minimum measurements.
For a better user experience, we created a separate screen where data is presented as a bar chart for the current week and month, with multiple tag selection capabilities. Users can also see their average, maximum and minimum measurements.
Implementation
LokaPulseTM is coded on Flutter 2.0, which implements null safety. For the state management of the application, we have two providers: The first is used for authentication. It allows the app to respond to a user based on whether they’re logged in or not. This provider is based on the Amazon Cognito. The second is the results provider, which saves the data that is being fetched from the backend and being modified through user interactions.
LokaPulseTM is coded on Flutter 2.0, which implements null safety. For the state management of the application, we have two providers: The first is used for authentication. It allows the app to respond to a user based on whether they’re logged in or not. This provider is based on the Amazon Cognito. The second is the results provider, which saves the data that is being fetched from the backend and being modified through user interactions.
Logic & Widget
When developing LokaPulseTM, we had in mind optimization and reusability. With Flutter you can extract all your widgets, functions and screens in a different directory, this will make it easier to import, test and reuse all that code keeping the code clean and separate between business logic, utilities and screens.
When developing LokaPulseTM, we had in mind optimization and reusability. With Flutter you can extract all your widgets, functions and screens in a different directory, this will make it easier to import, test and reuse all that code keeping the code clean and separate between business logic, utilities and screens.
Testing
LokaPulseTM is built to look great and respond intuitively on phones and tablets. So for widget testing, we called the same tests on two different resolutions – one is for a phone and the other on a tablet. Mocking the files was an easy task thanks to the mockito flutter package. This package was essential to our testing. Not only did it help us understand how the app would look on phones and tablets, it also helped us fix rendering issues that might occur on some devices.
LokaPulseTM is built to look great and respond intuitively on phones and tablets. So for widget testing, we called the same tests on two different resolutions – one is for a phone and the other on a tablet. Mocking the files was an easy task thanks to the mockito flutter package. This package was essential to our testing. Not only did it help us understand how the app would look on phones and tablets, it also helped us fix rendering issues that might occur on some devices.
We used all of the phones and tablets available to us, including holding weekly internal testing sessions at Loka where volunteers would test on their own devices. This helped us spot and fix problems that came up on a greater number of phones and tablets. Additionally Loka PulseTM has Firebase Crashlytics, so if the app crashed on someone’s device or if exceptions were spotted, we could easily track the issue and where it occurred.
UX
Leveraging Flutter capabilities of render at an outstanding 60fps (or 120fps if your device is capable), we decide to include animations, transitions and sounds to enrich que user experience and increase the probabilities of reusability. The hero animation is probably the most engaging, as it animates widgets between screens.
Leveraging Flutter capabilities of render at an outstanding 60fps (or 120fps if your device is capable), we decide to include animations, transitions and sounds to enrich que user experience and increase the probabilities of reusability. The hero animation is probably the most engaging, as it animates widgets between screens.
BACKEND
Serverless for more savings
We chose to implement a serverless architecture for LokaPulseTM. Serverless architectures offer a number of advantages over traditional cloud-based or server-centric infrastructure, which aligned well with the goals for Loka PulseTM.
We chose to implement a serverless architecture for LokaPulseTM. Serverless architectures offer a number of advantages over traditional cloud-based or server-centric infrastructure, which aligned well with the goals for Loka PulseTM.
- Serverless is managed by the cloud provider, making it easier to quickly build a proof of concept project.
- In traditional architectures, the developer has to provision and pay in advance for server capacity needed – regardless of whether it gets used or not.
- With serverless, you only pay for what you use, which helps drive costs down in many applications.
- Serverless scales automatically as the number of users and the usage increases.
Backend flow
- The mobile app gets a signed url that allows it to upload a video to an S3 bucket.
- When a video gets uploaded to the S3 bucket, it triggers the invocation of the Heart Rate Calculation Lambda function, which computes the heart rate for that video and saves the result in a DynamoDB table
- The mobile app can then get the results from this table, and associate tags and notes with each video.
- The authentication of the users is managed by AWS Cognito.
- AWS QuickSight is used to create interactive dashboards that give us powerful BI data, such as number of users, number of uploaded videos and high level statistics about the uploaded videos.
- AWS Backup automatically backups data from DynamoDB
TODAY
LokaPulseTM was selected as a showcase application at Flutter Engage 2021. Run by Google, Flutter Engage focuses on the biggest product announcements and most exciting technical demos. LokaPulseTM is now part of the Google Favorites Flutter Showcase apps, which is a great achievement for Loka and our Flutter team.
All that said, we are just scratching the surface of what’s possible with heart rate calculation, Eulerian Magnification, machine learning, and expressing it all visually with frameworks like Flutter. In terms of biosignals, the next logical steps are working on extracting the heart rate variability, the breathing rate and the “holy grail”, blood pressure.
LokaPulseTM at Flutter Engage. Next year we hope to be center stage.
With all these in place, we could provide users a very complete analysis of their current health condition. However, before moving to extracting more signals, we need to improve the current application.
Our ongoing work is focused on tackling the main sources of noise, like camera stabilization and background extraction, and the limitations of rPPG, including the need for uniform lighting conditions and the degraded performance on users with darker skin colors. Our team is excited to evolve LokaPulseTM and continue to show how this technology can improve people’s everyday lives.
Reach out to learn more or partner on innovative products.
Contact Us >
APPENDIX
An ongoing addendum of applications for LokaPulseTM technology.
Detect Deepfakes
According to researchers and publications like Forbes and Independent, Deepfakes are the most dangerous crime of the future. The good news is that all these artificially generated videos leave behind some signature on the PPG signal. Using our technology, we could better identify deepfakes, decrease the spread of fake news, and help protect online communities.
According to researchers and publications like Forbes and Independent, Deepfakes are the most dangerous crime of the future. The good news is that all these artificially generated videos leave behind some signature on the PPG signal. Using our technology, we could better identify deepfakes, decrease the spread of fake news, and help protect online communities.
Silicon Valley Office 
San Francisco Office
