SeeFit

SeeFit teaches a user how to exercise in the right way. Sometimes, we may be unsure whether we’re exercising our bodies correctly. To solve this problem, our virtual coach will give feedback based its detection of a user’s body movement. If a user moves correctly, the coach will give positive feedback, so that the user gradually learns the correct movements. SeeFit also makes exercise more entertaining by providing auditory and visual inputs based on the user’s body movement. Ultimately, we want to add a multiplayer section that allows interaction among different users, with the movements of each user generating sound, and the sounds combining to create music.

Team

Hyejun Youn
Brian Li - software engineer (Yale Univ, CS student)
Hannah Han - data scientist (Brown Univ, CS student)
David Wang - software engineer, designer (CMU, CS student)

seefit

My Role - Project Leader, UIUX Designer, Engineer

Woked on research, ideation, prototyping, front-end coding using HTML and CSS, javascript, video production using After Effects, 3d animation

Tools

Figma, Mixamo, Unity p5js, openCV After Effects javascript

Concept Background

I saw people with disabilities going to a service center to move and exercise their bodies. They became more energized and engaged with the exercise as they danced with one another while listening to pop music. We thought we could create a website that teaches users how to exercise, detects their body movements, gives feedback about their movement, and generates sounds corresponding to their movements.

Initial Goals

I wanted to create an application that makes exercise more entertaining by providing auditory and visual inputs based on the user’s body movement. we wanted to create a project that is more playful and entertaining, rather than having a linear outcome of users and computers.

Sketch

Research

First, I started researching some of the problems people with disabilities - especially Down Syndrome face - and the reasons that they might attend a service center. Especially in Asian countries, disabled people are not often seen in public, because many of them stay in their homes, suffering from social isolation.

Down syndrome (DS) is a risk factor for the development of depression. Biological events that may adversely affect brain developmen social withdrawal and compulsive behavior (routinized behavior) are associated features in depressed adults with Down syndrome.

Both the thyroid and a lower metabolic rate contribute to people with Down syndrome being overweight. This means that children with Down syndrome burn fewer calories overall compared to a typical child and need to need to exercise more to burn off the same number of calories.

A behavioral presentation as a form of communication is particularly salient for individuals with developmental disabilities who are unlikely to express distress verbally or to be able to use abstract thinking skills to connect experiences with thoughts and emotions.

Youth with DS had higher rates of overweight and obesity than youths without DS. Likely determinants of obesity included increased leptin, decreased resting energy expenditure, comorbidities, unfavorable diet, and low physical activity levels.

Problems

We first tried to use openCV to create a stick-figure superimposed on a human body to track movements. It was our first time using OpenCV, and we ran into many issues with different functions and programs, such as Caffe. We ultimately had to switch to a combination of OpenCV and Tensorflow. This was an arduous process because we had to learn everything from scratch. Fortunately, I have experience with p5js and processing, and also knew how to use Kinect for body tracking. I also found a more recent, simpler version using p5js that can track human body movement without connecting to kinect.

How We Built It

We built our platform using an open-source library called Openpose; specifically, we used MLI instead of COCO. We also edited some of the functionality to better fit our purposes. For example, we took out the hand and finger calculations and only focused on the major joints to improve computational speed, since we were mainly focusing on large-scale general body movements during a workout.

Since I learned javascript from my processing course at RISD, I briefly showed the other students the basic command. They quickly were able to figure out tracking point for each body part. For squat, we wrote when ankles are lower than hips, it plays sound.