This page is dedicated to the journey of LokaPulseTM,
from idea to testing to deployment and improvements.
We
dive deep into backend
and frontend technologies, challenges and eureka moments. For people
interested in the big
ideas and key takeaways, look to the IDEA and TODAY sections, as well as
any highlighted areas.
IDEA
Mobile is just scratching the surface of machine learning capabilities.
LokaPulseTM
was designed to showcase the possibilities of ML and Flutter by giving
users an Apple
and Android app that would measure their heart rate in an innovative and
beautiful way.
With a simple 15-second video,
LokaPulseTM can extract a user’s BPM
information, categorize the data, and compare it to previous videos,
giving anyone with a smartphone the ability to see and track their heart
rate.
The idea for LokaPulse
TM started when a
Loka developer stumbled upon a
method called
Eulerian Video Magnification. “Super
Seeing”, as it’s also known, amplifies and visualizes temporal
variations in videos that are difficult or even impossible to see with
the naked eyes. These visual variations include motion and color.
As Loka developers and the team behind LokaPulseTM, we wanted to see for ourselves how
this method worked. We implemented the algorithm from scratch and
applied it to several different videos. The
results were inspiring. Even the most subtle variations were amplified
beyond what we imagined.
Though the potential applications for this technology were endless, what
really caught our attention was the color magnification. The idea of
extracting “invisible” biosignals from a video was exciting to say the
least. And we knew it could be used as a heart rate calculator.
“Invisible” bio signals being visualized through Eulerian Video
Magnification
This is the moment that sparked the big question, what would happen if
everyone with a phone could measure their heart rate with a selfie
video? What else could this unlock for people’s daily health - and for
life sciences? And so, the idea of developing LokaPulseTM was born.
JOURNEY
Going from idea to fully-functioning app is never short on challenges.
The question is, when will the challenges emerge and how will you
respond.
Our first step was to do an in-depth analysis of the state of the art
(SOTA) and available datasets. After we explored all kinds of possible
bio signals that can be extracted from video, we decided to start with
heart rate estimation.
The research into extracting a heart rate signal from video led us to a
technology called
remote photoplethysmography (rPPG) which
relies on the reflection of light on the subjects skin.
This technology is the most prevalent in SOTA and seemed to benefit from
the increased variations provided by Eulerian color Magnification.
However, as we dove deeper into SOTA methods and comparative studies, we
noticed a troubling pattern: The improvements provided by Eulerian
Magnification often didn’t outweigh its limitations.
The improvements provided by Eulerian Magnification often didn’t
outweigh its limitations.
Here’s what we encountered as we explored SOTA methods
and tools:
- Increased variations caused by “noise”, e.g. motion
and light conditions.
- Performance seemed highly dependent on parameter
tuning reducing the system’s robustness for use in the wild.
- The tools we came across relied on extremely outdated
library versions
- The tools had serious bugs that prevented efficient
use.
- The tools they simply had very few methods
implemented
To overcome these obstacles, we developed a
pipeline architecture based on these tools where we could plug
and play at different stages to compare performances of the different
methods and processing steps. For the comparisons, this pipeline
architecture could be used in a benchmarking tool that allowed automated
comparisons for different parameters.
Benchmarking different methods and parameters required labelled data
with which to compare the predictions. We evaluated commonly used
datasets focusing on their similarity to our use case and on subject
diversity. We started testing with a dataset that would be our ideal
scenario and from there moved to videos captured
in real situations.
Through experiments and research, we concluded that SOTA methods tended
to require fine tuning for each specific dataset – and most tended to
increase the pre and post processing complexity by adding extra stages
while relying on the same core principles. Since
we required near real time results as we iterated, we leaned toward
simpler approaches, even if they were not the most recent and
best performing methods.
Here’s the flow of the solution we chose
- Image rescale and crop on the app.
- Use a Deep Learning model to detect and extract
user’s face.
- Extract the PPG signal.
- Use Welch’s method to obtain the power
spectrum and calculate the heart rate.
As we moved to real world application, we
faced three critical obstacles:
- Variations in camera quality between phones (e.g.
resolutions and frame rate), which prevent video compression after
recording.
- Lighting conditions - the most persistent and
stubborn problem.
- And the biggest limitation of the app: Performance deteriorates sharply for darker skin
types
Darker skin tones offer less distinction between reflected and absorbed
light.
The issue of skin tone stems from using rPPG to
extract the heart rate. Since rPPG relies on the difference between
reflected and absorbed light, skin types that absorb more light provide
a signal with lower Signal to Noise Ratio (SNR) damaging performance,
similarly to what happens under poor lighting conditions.
This isn’t a new problem. Products like
Oxymeters, which became more prevalent
during the pandemic, rely on PPG signals and are charged with the same
issues. This continues to be a challenge to LokaPulse
TM and other programs using PPG signals
will need to face and overcome.
FRAMEWORK
At the beginning of any software project, there is always a big
question: Which mobile framework/language is better for my use case –
and is it good enough for the long term? For LokaPulseTM, we decided that we wanted a
multi-platform framework and narrowed it down to two options: Flutter or
React Native.
Both are really good options for our use case, offering multi-platform
support and large communities. Ultimately Flutter was the right choice
based on the following criteria:
Support: thanks to the Flutter canvas
approach, we don't need to worry about outdated APIs or integrations
with a specific platform. Flutter provides a white canvas and puts us in
charge of rendering each pixel on the screen.
Performance: 60fps (or even 120 fps) is a
really attractive feature for smooth user experiences.
Custom design: We could create our own
designs, so we didn't need to reuse the platform specific UI components.
Time-to-market: Flutter is known for
lightning fast development.
For us, building LokaPulseTM became as much of a way to showcase
the capabilities of machine learning as it was to highlight the
outstanding performance of Flutter.
FRONTEND
At its core, LokaPulseTM is a Flutter
app that measures a users heart rate via video recording of their face.
The application provides a way to record a video, process the video, and
present all the data connected with previous measurements.
To improve the performance and user experience, the front end is capable
of “cropping” the video to only send the required information, the white
square that contains the face. With this we ensure that we are not using
extra data of your carrier plan and also the Face detection and heart
rate algorithms run faster than using the original video.
Tagging
LokaPulseTM uses a custom ffmpeg
command for tagging. After recording a video, users are prompted to
select a tag, which defines the state that they were in while recording
the video.
Tags can be used to track whether a user was stressed, relaxed,
exhausted, excited, etc., so users can better understand their results
and trends over weeks and months. LokaPulseTM also gives users the option to
manually enter a comment, or even use speech-to-text, for
better explanation of the moment when the measurement was taken.
When you’re running, your heart rate will be elevated. LokaPulseTM allows users to tag these measurements
and separate them from categories like ‘chill time’.
Speech-to-text
Speech-to-text offered our team some challenges. The package that we
used worked differently on Apple and Google platforms. Little tweaks and
workarounds were necessary. For example, LokaPulseTM would stop working after a few moments
of silence on Android phones. So after 30 seconds of silence, we made it
so the speech-to-text would stop and re-engage.
Anomalies
We built a way to detect anomalies into LokaPulseTM. If a measurement result is not in the
range of 60 to 120 bpm, the application gives the user a notification
that the heart rate is not in their normal range or inaccurate due
shaking or poor lighting conditions.
Seeing & Managing Results
Users can find their data in a list organized by date, tag, and heart
rate. Pressing on any measurement gives users the option to see, edit
and delete past notes.
Visualization
For a better user experience, we created a separate screen where data is
presented as a bar chart for the current week and month, with multiple
tag selection capabilities. Users can also see their average, maximum
and minimum measurements.
Implementation
LokaPulseTM is coded on Flutter 2.0,
which implements null safety. For the state management of the
application, we have two providers: The first is used for
authentication. It allows the app to respond to a user based on whether
they’re logged in or not. This provider is based on the Amazon Cognito.
The second is the results provider, which saves the data that is being
fetched from the backend and being modified through user interactions.
Logic & Widget
When developing LokaPulseTM, we had in
mind optimization and reusability. With Flutter you can extract all your
widgets, functions and screens in a different directory, this will make
it easier to import, test and reuse all that code keeping the code clean
and separate between business logic, utilities and screens.
Testing
LokaPulse
TM is built to look great and
respond intuitively on phones and tablets. So for widget testing, we
called the same tests on two different resolutions – one is for a phone
and the other on a tablet. Mocking the files was an easy task thanks to
the
mockito flutter package. This package was
essential to our testing. Not only did it help us understand how the app
would look on phones and tablets, it also helped us fix rendering issues
that might occur on some devices.
We used all of the phones and tablets available to us, including holding
weekly internal testing sessions at Loka where volunteers would test on
their own devices. This helped us spot and fix problems that came up on
a greater number of phones and tablets. Additionally Loka PulseTM has Firebase Crashlytics, so if the app
crashed on someone’s device or if exceptions were spotted, we could
easily track the issue and where it occurred.
UX
Leveraging Flutter capabilities of render at an outstanding 60fps (or
120fps if your device is capable), we decide to include animations,
transitions and sounds to enrich que user experience and increase the
probabilities of reusability. The hero animation is probably the most
engaging, as it animates widgets between screens.
BACKEND
Serverless for more savings
We chose to implement a serverless architecture for LokaPulseTM. Serverless architectures offer a
number of advantages over traditional cloud-based or server-centric
infrastructure, which aligned well with the goals for Loka PulseTM.
- Serverless is managed by the cloud provider, making
it easier to quickly build a proof of concept project.
- In traditional architectures, the developer has to
provision and pay in advance for server capacity needed – regardless
of whether it gets used or not.
- With serverless, you only pay for what you use, which
helps drive costs down in many applications.
- Serverless scales automatically as the number of
users and the usage increases.
Backend flow
- The mobile app gets a signed url that allows it to
upload a video to an S3 bucket.
- When a video gets uploaded to the S3 bucket, it
triggers the invocation of the Heart Rate Calculation Lambda function,
which computes the heart rate for that video and saves the result in
a DynamoDB table
- The mobile app can then get the results from this
table, and associate tags and notes with each video.
- The authentication of the users is managed by AWS
Cognito.
- AWS QuickSight is used to create
interactive dashboards that give us powerful BI data, such as number
of users, number of uploaded videos and high level statistics about
the uploaded videos.
- AWS Backup automatically backups data
from DynamoDB
TODAY
LokaPulse
TM was selected as a showcase
application at
Flutter Engage 2021. Run by Google, Flutter
Engage focuses on the biggest product announcements and most exciting
technical demos. LokaPulse
TM is now
part of the
Google Favorites Flutter Showcase apps,
which is a great achievement for Loka and our Flutter team.
All that said, we are just scratching the
surface of what’s possible with heart rate calculation,
Eulerian Magnification, machine learning, and expressing it all visually
with frameworks like Flutter. In terms of biosignals, the next logical
steps are working on extracting the heart rate variability, the
breathing rate and the “holy grail”, blood
pressure.
LokaPulseTM at Flutter Engage. Next
year we hope to be center stage.
With all these in place, we could provide users
a very complete analysis of their current health condition. However,
before moving to extracting more signals, we need to improve the current
application.
Our ongoing work is focused on tackling the main
sources of noise, like camera stabilization and background extraction,
and the limitations of rPPG, including the need for uniform lighting
conditions and the degraded performance on users with darker skin
colors. Our team is excited to evolve LokaPulseTM and continue to show how this
technology can improve people’s everyday lives.
APPENDIX
An ongoing addendum of applications for LokaPulseTM technology.
Detect Deepfakes
According to researchers and publications like
Forbes and Independent,
Deepfakes are the most dangerous crime of the
future. The good news is that all these artificially
generated videos leave behind some signature on the PPG signal. Using
our technology, we could better identify deepfakes, decrease the spread
of fake news, and help protect online communities.