Remote Storage Optimization - Part 1

I started this project because videos were taking up too much space on my phone. It was frustrating to constantly run out of storage, especially when I wanted to capture new memories. I realized I needed a solution that would work not just for my phone, but for all my devices. My goal was to optimize storage on my PC by reducing video file sizes using HEVC encoding with a quality setting of -qp28. I also wanted to compress PDFs to their optimal sizes. To make this system work across all my devices, I decided to use Syncthing. This way, I could process files from my mobile phone using either my main PC or a server. This setup would allow me to keep all my videos and documents while freeing up valuable storage space. It’s a practical solution for anyone dealing with storage issues across multiple devices.

This tutorial introduces a powerful bash function that automates the process of compressing videos to the HEVC (High Efficiency Video Coding) format using NVIDIA’s NVENC hardware encoding.

Why HEVC Compression?

HEVC, also known as H.265, offers superior compression efficiency compared to older codecs like H.264. It can reduce file sizes by up to 50% while maintaining similar quality, making it ideal for storage optimization and efficient streaming.

What This Script Covers

This bash function, named nvenchevc, accomplishes the following:

Identifies video files in various formats within the current directory and its subdirectories.
Checks if each video is already in HEVC format.
For non-HEVC videos, it uses FFmpeg with NVENC to compress them to HEVC.
Preserves audio and subtitle tracks during compression.
Replaces the original file with the compressed version upon successful conversion.

The Script

Here’s the complete nvenchevc function:

nvenchevc() {
    # Define video formats
    VIDEO_FORMATS="mp4|avi|mkv|mov|wmv|flv|webm|m4v|mpg|mpeg|m2v|3gp|3g2|mxf|ts|mts|m2ts|vob|ogv|drc|gifv|mng|qt|yuv|rm|rmvb|asf|amv|m4p|f4v|f4p|f4a|f4b"

    # Function to check if a file is HEVC encoded
    is_hevc() {
        ffprobe -v error -select_streams v:0 -show_entries stream=codec_name -of default=noprint_wrappers=1:nokey=1 "$1" | grep -q "hevc"
    }

    # Function to process a single file
    process_file() {
        local input_file="$1"
        local output_file="${input_file%.*}_temp.mp4"

        if [ ! -f "$input_file" ]; then
            echo "Error: File not found: $input_file"
            return 1
        fi

        if is_hevc "$input_file"; then
            echo "Skipping $input_file (already HEVC)"
        else
            echo "Processing $input_file"
            stdbuf -oL ffmpeg -nostdin -i "$input_file" -c:v hevc_nvenc -qp 28 -c:a copy -c:s copy "$output_file" 2>&1 | stdbuf -oL tr '\r' '\n' | tail -n 1
            if [ ${PIPESTATUS[0]} -eq 0 ]; then
                mv "$output_file" "$input_file"
                echo "Completed processing $input_file"
            else
                echo "Error processing $input_file"
                rm -f "$output_file"
            fi
        fi
    }

    # Main script
    find . -type f -regextype posix-extended -regex ".*\.($VIDEO_FORMATS)$" -print0 | while IFS= read -r -d '' file; do
        process_file "$file"
    done
}

How to Use

Add this function to your .zshrc or .bashrc file.
Reload your shell configuration or start a new terminal session.
Navigate to the directory containing your videos.
Run the function by typing nvenchevc and pressing Enter.

The function will process all compatible video files in the current directory and its subdirectories, compressing them to HEVC format using NVENC.

This script provides an efficient way to optimize your video storage, leveraging the power of NVIDIA’s hardware encoding for faster compression. It’s particularly useful for bulk processing of large video collections, helping you save significant storage space while maintaining video quality.

Future Enhancements

Edge Case Handling
- Implement robust error handling for unusual file formats or corrupted videos
Quality-Based Processing
- Develop a system to assess video quality and process only when reduction in size is significant
PDF Optimization
- Create a bulk compression tool for PDFs, maintaining readability while reducing file size
Automated File Processing
- Set up a file watcher to automatically process new videos and PDFs as they’re added
Remote Workflow Integration
- Configure Syncthing to enable seamless file synchronization and processing across devices

Chirag Bharambe