AI Motion Sensing System Report

1. Project Overview

This project aims to develop an AI-powered motion training platform using standard webcam input and real-time pose recognition. It allows users to perform sports training without any wearables, utilizing computer vision techniques for posture tracking, movement analysis, and feedback generation.

2. Core Technology Stack

Deep Learning Framework

PyTorch and Ultralytics YOLO form our foundational AI platform.

Pose Detection

YOLO11n-pose model tracks 17 COCO keypoints for precise human movement.

Ball Detection

A custom-trained YOLO model (0108ballbest.pt) ensures accurate ball recognition.

Real-time Processing

OpenCV manages high-speed video stream analysis and visualization.

3. Directory Structure

sport/
├── .gitignore
├── .idea\
│   ├── .gitignore
│   ├── app - algorithm - master.iml
│   ├── inspectionProfiles\
│   │   ├── Project_Default.xml
│   │   └── profiles_settings.xml
│   ├── misc.xml
│   ├── modules.xml
│   └── vcs.xml
├── .venv\
├── README.md
├── ball_chart.png
├── configs\
│   ├── base_pt.yaml
│   ├── base_torchscript.yaml
│   ├── base_yolo.yaml
│   └── bytetrack.yaml
├── gravity_chart.png
├── handleFrame.py
├── heatmap.png
├── model\
│   └── model_tools.py
├── realtime_main.py
├── realtime_main_optimized.py
├── requirements.txt
├── run.sh
├── src\
│   ├── ActionJudge.py
│   ├── BallProtectorChallenge.py
│   ├── BaseInterface.py
│   ├── ContactChallenge.py
│   ├── DeepSquat.py
│   ├── DribbleCounter.py
│   ├── DribbleFreeCounter.py
│   ├── DribbleLowTypeCounter.py
│   ├── DribbleTwoBallCounter.py
│   ├── DribbleTypeCounter.py
│   ├── HandEyeContactChallenge.py
│   ├── JumpingJack.py
│   ├── MobileDribbleChallenge.py
│   ├── Report.py
│   ├── SidebySideJumpChallenge.py
│   ├── Skeleton.py
│   ├── SlideCounter.py
│   ├── StartPositionJudge.py
│   └── Tester.py
├── test.sh
├── test_all.py
├── test_main.py
├── tools\
│   ├── __init__.py
│   ├── generate_point_func.py
│   ├── img_tools.py
│   ├── output_obj.py
│   └── utils.py
└── weights\
    ├── 0108ballbest.pt
    ├── 0108ballbest.torchscript
    ├── 0108ballbest_wrapped_torchscript_model.ptl
    ├── ballbest.pt
    ├── ballbest.torchscript
    ├── ballbest_wrapped_torchscript_model.ptl
    ├── yolo11n - pose.onnx
    ├── yolo11n - pose.pt
    ├── yolo11n - pose.torchscript
    └── yolo11n - pose_wrapped_torchscript_model.pt

4. User Interface Implementation

The user interface is implemented using the tkinter library in realtime_main_optimized.py. It categorizes and displays different types of training programs, such as dribbling, fitness, and agility training.

ealtime_main_optimized.py
# ... existing code ...
# Tkinter UI implementation code here
# ... existing code ...

5. Algorithm Analysis

5.1 BaseInterface (Core Class)

The BaseInterface class serves as a unified interface for all sports training classes, ensuring consistent method calls across different training modules.

class BaseInterface:
    def __init__(self):
        self.api_info = {
            "status": 0,
            "count": 0,
            "score": 0,
            "report_info": None,
            "action_metric_info": None,
            "start_position_info": None
        }
        self.action_judge = ActionJudge(self.fps, metric_type=2)

    def run(self, frame_id, frame, end=False):
        # Main processing pipeline
        res_pose, res_boxes = self.run_model(frame_id, frame, detect_type="all")
        # Process results and generate output
        self.process(frame_id, sks=sks, balls=ball_boxes)
        self.report_info = self.reporter.process(frame_id, sks, ball_boxes)
        self.action_metric_info = self.action_judge.action_decision(frame_id, sks, ball_boxes)
        return self.make_success_outobj(frame_id, annotated_frame, sks, ball_boxes)

Key Features:

• Base class for all training modules
• Handles model initialization and frame processing
• Manages reporting and action judgment
• Provides common interface for all derived classes

5.2 ActionJudge

The ActionJudge class is responsible for evaluating the standardization of user actions. It uses metrics such as center of gravity, hand height, and hand width to make judgments.

class ActionJudge:
    def __init__(self, fps: int = 25, metric_type: int = 2):
        # Initializes with frame rate and metric type
        self.gravity_heights = deque(maxlen=int(self.fps) * self.metric_time_tap)
        # Various message attributes for feedback
        self.gravity_message = ""
        self.hands_height_message = ""
        # Threshold ratios for different body parts
        self.threshold_height_ratio = 0.7
        self.threshold_hands_x_ratio = 1.5
        self.threshold_feet_x_ratio = 1

    def action_decision(self, frame_id: int, sks: list, balls: list):
        # Main decision function that calls specific metric functions
        if self.metric_type == 0 or self.metric_type == 2:
            self.metric_center_gravity(sk)
            self.metric_hands_dist(sk)
            self.metric_feet_dist(sk)
        if self.metric_type == 1 or self.metric_type == 2:
            self.metric_center_gravity(sk)
            self.metric_ball_height(sk, ball)
        return self.report_message()

Key Features:

• Evaluates basketball movements and provides feedback
• Tracks center of gravity over time (2-second window)
• Checks hand/feet positions relative to shoulders
• Provides specific feedback messages for each metric
• Supports both ball and no-ball scenarios

5.3 Training Modules

DribbleFreeCounter.py

class DribbleFreeCounter(BaseInterface):
    def __init__(self, fps, video_width, video_height):
        self.api_info = {
            "v_count": 0,  # V-dribble count
            "vertical_count": 0,  # Vertical dribble count
            'total_count':0,  # Total dribble count
        }
        self.ball_trajectory = []

    def judge_dribble_type(self, trajectory):
        # Classifies dribble type based on movement pattern
        if max_horizontal_disp > self.v_movement_threshold:
            return "V"
        elif max_vertical_disp > self.vertical_movement_threshold:
            return "Vertical"
        return "Unknown"

Key Features:

• Tracks dribble patterns (V-shaped vs vertical)
• Analyzes ball trajectory
• Counts different dribble types separately

DeepSquat.py

class DeepSquat(BaseInterface):
    def __init__(self, fps, width, height):
        self.states = ("init", "decline", "rise", "loss")
        self.api_info = {
            "pose_state": "init",
            "speed": 0.0,
        }

    def process(self, frame_id, sks=None, balls=None):
        # Tracks squat depth and form
        avg_hip_y = (left_hip[1] + right_hip[1]) / 2
        avg_knee_y = (left_knee[1] + right_knee[1]) / 2
        dist_hip_knee = abs(avg_hip_y - avg_knee_y)
        # Updates state and counts

Key Features:

• Monitors squat depth (hip-knee distance)
• Tracks movement speed
• Provides form feedback

BallProtectorChallenge.py

class BallProtectorChallenge(BaseInterface):
    def __init__(self, fps, video_width, video_height):
        self.api_info = {
            "count": 0,  # Dribble count
            "punish_count": 0,  # Protection failures
            "all_score": 0,  # Total score
        }

    def process(self, frame_id, sks=None, balls=None):
        # Generates attacking points to protect against
        if self.ball_point_bool:
            start_point, end_point, dict_ball_point = generate_moving_point_for_ball(...)
        # Checks if ball was protected
        ball_dist = euclidean_distance(ball, self.api_info['ball_point'])
        if ball_dist < self.api_info['ball_scale']:
            curr_score = -10  # Penalty for failed protection

Key Features:

• Simulates defensive scenarios
• Tracks successful protections vs failures
• Generates dynamic challenge points

Key Technical Aspects

1. Pose Estimation:

   • Uses YOLO-based models for body and ball detection
   • Tracks 17 keypoints (COCO format) for body pose

2. Movement Analysis:

   • Center of gravity tracking
   • Limb position relative to body proportions
   • Ball trajectory analysis

3. Feedback System:

   • Real-time feedback messages
   • Multiple metrics for comprehensive evaluation
   • Visual indicators on video output

4. Challenge Modules:

   • Specialized classes for different drills
   • Configurable difficulty parameters
   • Score tracking and performance metrics

Usage Example

# Initialize a training module
trainer = DribbleFreeCounter(fps=30, video_width=640, video_height=480)
trainer.init_module(det_model, pose_model)

# Process video frames
for frame in video_frames:
    output = trainer.run(frame_id, frame)
    # Get feedback messages
    feedback = output.data.api_info["action_metric_info"]

6. Technical Challenges and Solutions

Real-time Performance Bottleneck

Problem: Multiple models degrade processing speed.

Solution: Model quantization via TorchScript, asynchronous processing, and result reuse.

Detection Accuracy

Problem: False positives/negatives in complex scenarios.

Solution: Multi-frame fusion, dynamic confidence thresholds, and targeted data augmentation.

We overcome significant technical hurdles. Our solutions ensure robust performance and high detection accuracy.

7. Conclusion & Next Steps

• Key Takeaways: Our system offers adaptive, precise, and real-time motion detection.

• Future Enhancements: Integrate more complex challenges and expand sports variety.

• Investment Opportunities: Explore partnerships for market expansion and R&D.

The AI motion sensing system is ready for real-world integration. We invite collaboration to drive further innovation and market penetration.