MotionRender: A simple Python implementation of video motion visualization for 3D motion capture data

Implementation

The version of the MotionRender library described here was developed on a Python 3.X scientific tool stack environment. It has dependencies on the pandas and matplotlib python libraries. In addtion, for back end rendering, the ffmpeg tool (free and open source) or similar tool is needed.

We have 15 joint positions in the example subject motion captured data sets used in developing the library. Each joint position is captured approximately 30 times per second by the Kinect sensor programmed with the NiTE 2.0 skeleton tracking library software. So while the library was generated specifically with 15 3D motion capture points, the general format of the input data needed to render a 3D animation is

timeStamp, joint0X, joint0Y, joint0Z,
       joint1X, joint1Y, joint1Z,
 …,
 jointNX, jointNY, jointNZ

as a standard comma separated value (csv) data file, where $N$ can be specified when rendering a motion capture animation of how many joint/position points are in the input. The only other information needed by the library is a joint connection graph of the points if we want to visualize the relationship of the rendered points to one another while being animated. Again using the Kinect 15 joint position data, we specify a joint graph for the software like this:

# the relationship of the joint points to assigned numeric position.
# the numeric id corresponds to the expected position in the input file
joint_names = [
  ‘Head’, ‘Neck’, ‘Torso’,         # 0 1 2
  ‘LeftShoulder’, ‘RightShoulder’, # 3 4
  ‘LeftElbow’, ‘RightElbow’,     # 5 6
  ‘LeftHand’, ‘RightHand’,      # 7 8
  ‘LeftHip’, ‘RightHip’,       # 9 10
  ‘LeftKnee’, ‘RightKnee’,       # 11 12
  ‘LeftFoot’, ‘RightFoot’      # 13 14
]
# joint position graph for visualizing relationship between joints
joint_graph = [
  (0, 1), (1, 3), (1, 4),        # head to neck, neck to shoulders
  (3, 5), (4, 6),             # shoulders to elbows
  (5, 7), (6, 8),             # elbows to hands
  (3, 2), (4, 2), (9, 2), (10, 2),  # shoulders and hips to torso
  (3, 9), (4, 10), (9, 10),       # shoulders to hips, connect hips
  (9, 11), (10, 12),           # hips to knees
  (11, 13), (12, 14),           # knees to feet
]

The user specifies the joint connection graph as a second input to the library for operation. The symbolic names of the joints must match the names in the time stamped motion capture time series of points. For example, the full Kinect connection graph of the 15 captured joints is specified as:

head neck
neck leftShoulder
neck rightShoulder
leftShoulder leftElbow
rightShoulder rightElbow
leftElbow leftHand
rightElbow rightHand
leftShoulder torso
rightShoulder torso
leftHip torso
rightHip torso
leftShoulder leftHip
rightShoulder rightHip
leftHip rightHip
leftHip leftKnee
rightHip rightKnee
leftKnee leftFoot
rightKnee rightFoot

Operation

MotionRender requires matplotlib and pandas libraries for its operation.

$ sudo pip install matplotlib pandas

In addition the ffmpeg library is used as a back end frame renderer for creating video files.

$ sudo apt install ffmpeg

Install the MotionRender library from standard PyPi python library.

$ pip install motionrender

The library expects a standard csv file of values, where the first column is a time stamp. Subsequent columns are the data for motion capture point 0 (x, y, z), motion capture point 1, etc. So given 15 motion capture points, this library expects and array with 46 columns, where the first column is a time stamp, and the subsequent columns are the 3D positions for all points captured. This could be loaded from a csv file, or generated from some other source.

The second parameter corresponds to the joint graph shown above. This is expected to be a space separated file of edge relations defined between the joint position names. The joint position names for the time series capture data should match the joint position names in the joint graph. The library currently contains to main API functions that may be used. The render_frame() member funtction of the main MotionRender class renders single frames from the motion capture data. While the render_animation() function renders video files. Actually the default behavior of both of these methods is to return figure and animation object instances respectively. However, the user can supply a file name to these API calls and the figure/video will then be saved in the specified file. Following is a quick example of using the library interactively to read in motion capture data, and render it as figures and videos.

$ pip install motionrender
Collecting motionrender
  Downloading motionrender-1.0.0-py3-none-any.whl (18 kB)
Installing collected packages: motionrender
Successfully installed motionrender-1.0.0

$ python
Python 3.9.7 (default, Sep 16 2021, 13:09:58)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from motionrender import MotionRender
>>> mr = MotionRender("data/standing-subject.csv", "data/standing-joint-graph.csv")
>>> fig = mr.render_frame(1636576712852000, figure_name="standing-subject.png")
>>> anim = mr.render_animation(movie_name="standing-subject.mov") processing frame: 0 processing frame: 500

Additional files are included in the main directory of the example project, that includes the data file use cases shown here. The test files contain additional examples of configuring rendering options of the library. You can invoke them in the example project as:

$ python test-plot.py
$ python test-render.py

In addition there is also a jupyter notebook present in the example project named test.ipynb that shows further how the library can be used interactively.