Python 3 contains an http.server module that allows you to serve files within a given working folder over HTTP.
$ cd ~/Downloads/videos
$ python3 -m http.server
The above makes an HTTP endpoint available to any other machine on your local network. Any files listed can be downloaded with a regular web browser. Below is an example response from the above server.
$ curl -s http://192.168.0.21:8000/ | head -n15
<!DOCTYPE HTML>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href="20220918_110038.mp4">20220918_110038.mp4</a></li>
<li><a href="20220918_155052.mp4">20220918_155052.mp4</a></li>
<li><a href="20220918_155130.mp4">20220918_155130.mp4</a></li>
<li><a href="20220918_155229.mp4">20220918_155229.mp4</a></li>
<li><a href="20220918_161913.mp4">20220918_161913.mp4</a></li>
It's very handy for copying files between desktops and laptops with mass storage and mobile devices on a local network. It works via HTTP so there is no need for special apps on your phone or exotic protocols. A regular web browser can download the files on its own.
But if you want to download a large video, the whole file needs to be downloaded before you can skip through and watch from any arbitrary point. To add to this, there are no thumbnails, just filenames listed for each video.
In this post, I'm going to create a Python script that will allow you to see animated thumbnails of your videos served from your desktop or laptop to any of your mobile or other devices on your own network. You will be able to start streaming the videos as soon as you hit play and timeline skipping is supported.
Below is an example screenshot from my phone showing two videos on my desktop from my holiday in Athens, Greece last year.
Installing Prerequisites
I've installed Homebrew on my 2020 MacBook Pro and I'll use it to install Python, Caddy Server, Bento and FFmpeg.
$ brew install \
caddy \
bento4 \
ffmpeg \
virtualenv
If you're running Ubuntu 20 LTS, use the following installation commands.
$ sudo apt update
$ sudo apt install -y \
debian-keyring \
debian-archive-keyring \
apt-transport-https
$ curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' \
| sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
$ curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' \
| sudo tee /etc/apt/sources.list.d/caddy-stable.list
$ sudo apt update
$ sudo apt install \
build-essential \
caddy \
cmake \
ffmpeg \
python3-pip \
python3-virtualenv
Bento4 will need to be compiled from source.
$ git clone https://github.com/axiomatic-systems/Bento4.git
$ cd Bento4/
$ mkdir -p cmakebuild
$ cd cmakebuild/
$ cmake -DCMAKE_BUILD_TYPE=Release ..
$ make -j $(nproc)
$ sudo make install
If you're running Ubuntu for Windows, use the following installation commands.
$ sudo apt install -y \
debian-keyring \
debian-archive-keyring \
apt-transport-https
$ curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' \
| sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
$ curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' \
| sudo tee /etc/apt/sources.list.d/caddy-stable.list
$ sudo apt update
$ sudo apt install \
build-essential \
caddy \
cmake \
ffmpeg \
python3.8-venv
$ git clone https://github.com/axiomatic-systems/Bento4.git
$ cd Bento4/
$ mkdir -p cmakebuild
$ cd cmakebuild/
$ cmake -DCMAKE_BUILD_TYPE=Release ..
$ make -j $(nproc)
$ sudo make install
Then, regardless of operating system, run the following to set up Python's virtual environment.
$ virtualenv ~/.hls
$ source ~/.hls/bin/activate
$ python3 -m pip install -U \
rich \
shpyx \
typer
HTTP Live Streaming
HTTP Live Streaming (HLS) was developed by Apple and released back in 2009. It works by breaking up videos into chunks and presenting a playlist that explains at which point in a timeline a given chunk's video is for. HLS has become extremely popular for streaming video online. The youtube-dl project mentions HLS in the code for 266 of the 803 sites it can download videos from.
The Bento4 toolkit is a popular open source project that can generate the files needed for HLS. The project's scope goes far beyond HLS yet only contains ~50K lines of C++ code. Its lead developer Gilles Boccon-Gibod works on Augmented Reality at Google.
Bento4's mp42hls utility can take a source video and generate two metadata files and a stream.ts video file. This file is in H.264 format with either AAC, MP3 or EC-3 used for audio and either an MPEG-2 or MPEG-4 Part 14 container. I've found stream.ts files to be ~10% larger than the MP4 source files my Samsung Galaxy S22 Ultra produces.
The first metadata file created is an iframes.m3u8 playlist file. This is typically a few KB in size.
$ head iframes.m3u8
#EXTM3U
#EXT-X-VERSION:4
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-I-FRAMES-ONLY
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-TARGETDURATION:0
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:0.500000,
#EXT-X-BYTERANGE:31208@376
stream.ts
The second metadata file is stream.m3u8 and contains the segments explaining which byte offsets represent each point in the stream.ts file's timeline. This metadata file is typically a few KBs in size as well.
$ cat stream.m3u8
#EXTM3U
#EXT-X-VERSION:4
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-TARGETDURATION:6
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:6.000000,
#EXT-X-BYTERANGE:1095476@0
stream.ts
#EXTINF:6.366667,
#EXT-X-BYTERANGE:1785060@1095476
stream.ts
#EXTINF:6.033333,
#EXT-X-BYTERANGE:1377476@2880536
stream.ts
#EXTINF:6.233333,
#EXT-X-BYTERANGE:1157892@4258012
stream.ts
#EXTINF:6.366667,
#EXT-X-BYTERANGE:1713432@5415904
stream.ts
#EXTINF:6.266667,
#EXT-X-BYTERANGE:1746520@7129336
stream.ts
#EXTINF:6.366667,
#EXT-X-BYTERANGE:2081160@8875856
stream.ts
#EXTINF:6.100000,
#EXT-X-BYTERANGE:2061608@10957016
stream.ts
#EXTINF:6.000000,
#EXT-X-BYTERANGE:1397780@13018624
stream.ts
#EXTINF:3.766667,
#EXT-X-BYTERANGE:563248@14416404
stream.ts
#EXT-X-ENDLIST
Generating Streams & Thumbnails
I've built a Python script which will locate any MP4 files in the folder it's executed in, generate HLS streams of their contents, produce video thumbnails showing clips from their source videos and an index.html file that Caddy Server can use by default to display video players for each file when the index.html file is served over HTTP.
The script below can be downloaded from GitHub.
I'm using Typer to generate the command line interface as well as provide basic input validation. Below is the help screen it generated.
$ python ~/hls.py --help
Usage: hls.py [OPTIONS]
╭─ Options ────────────────────────────────────────────────────────╮
│ --install-complet… [bash|zsh|fish|po Install completion │
│ wershell|pwsh] for the specified │
│ shell. │
│ [default: None] │
│ --show-completion [bash|zsh|fish|po Show completion for │
│ wershell|pwsh] the specified │
│ shell, to copy it │
│ or customize the │
│ installation. │
│ [default: None] │
│ --help Show this message │
│ and exit. │
╰──────────────────────────────────────────────────────────────────╯
╭─ Alternative Binaries ───────────────────────────────────────────╮
│ --ffmpeg TEXT [default: ffmpeg] │
│ --ffprobe TEXT [default: ffprobe] │
│ --mp42hls TEXT [default: mp42hls] │
╰──────────────────────────────────────────────────────────────────╯
╭─ Video Quality ──────────────────────────────────────────────────╮
│ --rate INTEGER [default: 4000] │
│ --bufsize INTEGER [default: 8000] │
╰──────────────────────────────────────────────────────────────────╯
╭─ Thumbnail Settings ─────────────────────────────────────────────╮
│ --thumb-fps INTEGER [default: 10] │
│ --thumb-height INTEGER [default: 360] │
│ --thumb-n-seconds INTEGER [default: 145] │
│ --thumb-secs-per-clip INTEGER [default: 1] │
│ --thumb-qscale INTEGER [default: 50] │
│ --thumb-compression-le… INTEGER [default: 3] │
╰──────────────────────────────────────────────────────────────────╯
$ vi ~/hls.py
Below are the imports for this script.
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import json
from glob import glob
from os import chdir, getcwd
from os.path import splitext
from pathlib import Path
from shutil import move
from tempfile import TemporaryDirectory
from urllib.parse import quote
from rich.progress import Progress, track
from shpyx import run as execute
import typer
app = typer.Typer(rich_markup_mode='rich')
This script will use FFmpeg to determine the length of each video it's working on. These are used for the (albeit rough) conversion progress calculation. I'm using Rich to display progress bars with the estimated time of completion.
def get_length(filename: str, ffprobe: str):
cmd = '%(ffprobe)s ' \
'-v error ' \
'-hide_banner ' \
'-print_format json ' \
'-show_streams ' \
'"%(filename)s"'
return float(json.loads(execute(cmd % {'filename': filename,
'ffprobe': ffprobe}).stdout)
['streams']
[0]
.get('duration', 0.0))
This script uses the base filename of any video for the resulting stream and metadata files. The following is a filename extension removal helper used in this script.
remove_ext = lambda filename: splitext(filename)[0]
By default, a one-second clip every 145 seconds throughout the video will be collected and merged together to create a video thumbnail. The thumbnail will be in WebP format. This format was designed by Google and released in 2010. I found these files to be 10x smaller than using GIF while supporting a much wider array of colours and superior image quality.
def build_thumbnail(filename: str,
ffmpeg: str,
fps: int,
height: int,
n_seconds: int,
secs_per_clip: int,
qscale: int,
compression_level: int):
filename = remove_ext(filename)
cmd = '%(ffmpeg)s ' \
'-loglevel error ' \
'-hide_banner ' \
'-y '\
'-i "%(filename)s.mp4" ' \
'-vf "fps=%(fps)d,' \
'scale=-2:%(height)d:flags=lanczos,' \
'select=\'lt(mod(t,%(n_seconds)d),%(secs_per_clip)d)\',' \
'setpts=N/FRAME_RATE/TB" ' \
'-loop 0 ' \
'-qscale %(qscale)d ' \
'-compression_level %(compression_level)d ' \
'"%(filename)s.webp"'
if not Path(filename + '.webp').is_file():
execute(cmd % {'filename': filename,
'ffmpeg': ffmpeg,
'fps': fps,
'height': height,
'n_seconds': n_seconds,
'secs_per_clip': secs_per_clip,
'qscale': qscale,
'compression_level': compression_level})
FFmpeg will convert source video into H.264 with a default bitrate of 4000 kb/s. This video will then be fed into Bento4's mp42hls utility to produce the HLS stream and its metadata files.
def ffmpeg_convert(filename: str,
ffmpeg: str,
mp42hls: str,
rate: int,
bufsize: int):
cmd = '%(ffmpeg)s ' \
'-i "%(filename)s" ' \
'-loglevel error ' \
'-hide_banner ' \
'-y ' \
'-c:v libx264 ' \
'-preset ultrafast ' \
'-b:v %(rate)dk ' \
'-maxrate %(rate)dk ' \
'-bufsize %(bufsize)dk ' \
'-vf scale=-1:720 ' \
'out.mp4'
cmd2 = '%s out.mp4 --output-single-file' % mp42hls
start_folder = getcwd()
with TemporaryDirectory() as folder:
chdir(folder)
execute(cmd % {'filename': start_folder + '/' + filename,
'ffmpeg': ffmpeg,
'rate': rate,
'bufsize': bufsize})
execute(cmd2)
filename = remove_ext(filename)
iframes_m3u8 = open('iframes.m3u8', 'r')\
.read()\
.replace('stream.ts', '%s.ts' % filename)
stream_m3u8 = open('stream.m3u8', 'r')\
.read()\
.replace('stream.ts', '%s.ts' % filename)
chdir(start_folder)
move("%s/stream.ts" % folder, "./%s.ts" % filename)
open('%s.iframes.m3u8' % filename, 'w').write(iframes_m3u8)
open('%s.stream.m3u8' % filename, 'w').write(stream_m3u8)
The script generates an index.html file which contains video tags pointing to each video's HLS stream as well as their WebP thumbnail. The WebP thumbnails don't contain any sound and will play as soon as the page has loaded.
def generate_index():
# Generate an index.html listing all .m3u8 files
# and their .webp thumbnails.
vid_html = '''
<video controls
preload="auto"
video
poster="%(filename)s.webp"
data-setup='{"fluid":true,
"controls": true,
"autoplay": false,
"preload": "auto"}'
width="100%%">
<source src="%(filename)s.stream.m3u8"
type="application/x-mpegURL">
</video>
'''
open('index.html', 'w')\
.write('<hr/>'.join([vid_html % {'filename': quote(remove_ext(x))}
for x in sorted(glob('*.ts'))]))
The following sets up the CLI, finds every MP4 file in the current working directory, collects each video's length and generates each video's HLS and thumbnail files.
alt_bin = {'rich_help_panel': 'Alternative Binaries'}
vid_qual = {'rich_help_panel': 'Video Quality'}
thumb_set = {'rich_help_panel': 'Thumbnail Settings'}
job_track = {'rich_help_panel': 'Job Tracking'}
@app.command()
def build(ffmpeg: str = typer.Option('ffmpeg', **alt_bin),
ffprobe: str = typer.Option('ffprobe', **alt_bin),
mp42hls: str = typer.Option('mp42hls', **alt_bin),
rate: int = typer.Option(4000, **vid_qual),
bufsize: int = typer.Option(8000, **vid_qual),
thumb_fps: int = typer.Option(10, **thumb_set),
thumb_height: int = typer.Option(360, **thumb_set),
thumb_n_seconds: int = typer.Option(145, **thumb_set),
thumb_secs_per_clip: int = typer.Option(1, **thumb_set),
thumb_qscale: int = typer.Option(50, **thumb_set),
thumb_compression_level: int = typer.Option(3, **thumb_set)):
# Find all MP4 files in folder without a corresponding .ts file.
# This will skip videos already generated.
workload = set([remove_ext(x)
for x in glob('*.mp4')]) - \
set([remove_ext(x)
for x in glob('*.ts')])
workload = set([x + '.mp4' for x in workload])
workload2 = {}
# Get length of videos.
for filename in track(workload,
description='Fetching video lengths...'):
workload2[filename] = get_length(filename, ffprobe)
total_duration = sum(workload2.values())
# Generate .ts and .m3u8 files starting with the shortest videos first
with Progress() as progress:
task = progress.add_task('Converting videos to HLS...',
total=total_duration)
for filename, duration in sorted(workload2.items(),
key=lambda x: x[1]):
build_thumbnail(filename=filename,
ffmpeg=ffmpeg,
fps=thumb_fps,
height=thumb_height,
n_seconds=thumb_n_seconds,
secs_per_clip=thumb_secs_per_clip,
qscale=thumb_qscale,
compression_level=thumb_compression_level)
ffmpeg_convert(filename, ffmpeg, mp42hls, rate, bufsize)
generate_index()
progress.update(task, advance=duration)
if __name__ == "__main__":
app()
The above can be run in any folder containing MP4 videos. The output generated will also live in the same folder. Below is an example execution and the CLI's output.
$ cd ~/Downloads/videos
$ python3 ~/hls.py
Fetching video lengths... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Converting videos to HLS... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
The following shows the index.html file, two source MP4 files, their HLS stream containers and metadata files and thumbnails generated by the above script.
$ ls -lh
1.1K ... Athens Fall 2022.iframes.m3u8
53M ... Athens Fall 2022.mp4
1.1K ... Athens Fall 2022.stream.m3u8
58M ... Athens Fall 2022.ts
237K ... Athens Fall 2022.webp
584B ... Turtles in Athens.iframes.m3u8
140M ... Turtles in Athens.mp4
570B ... Turtles in Athens.stream.m3u8
24M ... Turtles in Athens.ts
129K ... Turtles in Athens.webp
935B ... index.html
Serving HLS over HTTPS
The following will launch Caddy Server's File Server on TCP port 8000. I discuss Caddy Server in greater detail in my File Sharing with Caddy & MinIO blog post.
$ caddy file-server --listen :8000
Below is a screenshot from Brave running on my Samsung Galaxy S22 Ultra. It's connected via Wi-Fi to my Desktop PC. Both videos can be played at the same time or independently.