I’ve posted a bunch about LBS (my NAS) before because it’s such a core part of the way I backup and maintain data¹; from important documents and photos to old work and hobbies. But space is always a cost, especially with regular offsite backups. As an example of my data hoarding I’ve got about 78GB of photos, some of which date back to 2001. Thankfully the photos from that far back were taken on a digital camera that saved them to floppy disk but the proliferation of amazingly powerful smartphone cameras makes it so easy to take heaps of incredibly high resolution snaps with no remorse or regret.

Millennial-Core Social Media

I take a lot of silly photos throughout my day on Snapchat, sending them back and forth with friends to keep abreast of our going ons, espeically for those of us who live in geographically isolated locales. I usually save a copy of the photo I take so I’ve got my own record of my adventures, hobbies, or interesting things I want to investigate later. During a recent backup of my phone to LBS I noticed that each Snapchat photo was a JPEG about 0.5MB - 2MB in size, which isn’t massive by itself but made me curious about how much space this was taking up in my archives.

I use a Pixel 4a with a 12.2MP rear camera and an 8MP rear camera. That rear camera gives me a 4032x3024 photo and when using the standard Google Android camera app those pictures are between 1MB - 4MB. Meanwhile, the saved photos from snapchat are 4000x1957 or about 7.8MP. Since Snapchat is designed to take photos or videos quickly and the focus is on ephemeral pics that disappear rather than photos to be saved/posted/printed there’s not nearly as much in the way of optimisation as Google provides based on its research (such as HDR+). Thus the photos are often a bit blurry, a bit crunchy, a bit unfocused, etc. I still want a copy of these moments, but it won’t be one I plan to print and frame.

How Much Snapchat Do I Have?

Okay, so I’ve been using Snapchat daily for nearly a decade, I probably have a few photos. Let’s find out by searching my pictures directory:

du -a . | grep -E "Snapchat.*(jpeg|jpg)" | wc -l

This gives me 18,702 photos. Alright, that’s a lot. But I’ve had older and shittier phones in the past so maybe they’re not taking that much space?

du -a . | grep -E "Snapchat.*(jpeg|jpg)" | awk '{ sum += $1 } END{ print sum}'

Ooof, 16,868,926KB or 16.8GB. Out of the 78GB of photos I have that’s 21% which are stored really inefficiently for the quality of picture. Maybe we can compress them further with an aim to retain perceptual quality while saving disk space at home and money in online backups?

JPEG Re-Compression

ffmpeg is an amazingly powerful command line tool for media manipulation whose full syntax can only be recalled by the most arcane and powerful of computer mages. Thankfully there’s a bunch of one-liners scattered about the internet so I can crib the fragments I need.

Here’s one of those example one-liners to recompress a JPEG:

ffmpeg -i input.jpg -q:v 20 output.jpg

The -q:v 20 let’s me pick a quality to encode the JPEG file at, with the parameter being a number between 1 and 30 on a scale that should approximate best quality through to best compression.

I went through my photos and picked out a representative sample of Snapchat photos over the years, trying to get a gamut of:

• Different devices,
• Rear vs front camera,
• Landscape to portrait to macro,
• Lighting conditions,
• Busy vs empty.

I tested a variety of quality parameters against each sample pic, with the aim to find a single “magic” value that provided a balance of quality and compression for all images with the view to run a batch convert of all my Snapchat images. The review of the photos was entirely subjective based on my own perception of the before and after photos when zoomed in and zoomed on both a computer and smartphone. In the end, I settled on 10 as the magic number for the following reasons:

• Compression between 80% and 90%.
• No perceptive quality degradation while zoomed out or “fit to page”.
• Imperceptible JPEG artifacts at any “normal” zoom levels.
• Minimal JPEG artifacts at “hey i want to see the time on the reflection of that clock in the window in the background” zoom levels.
• The JPEG compression actually smoothed out some shot noise in low light shots and made them more visually appealling.

Higher values could give better compression but with with quickly diminshing returns (80% - 90% vs 90% - 95%) and were likely to show a lot more crunchy visual artifacts in regular viewing scenarios.

Let’s Do It

While I originally planned to do all the conversions on my TrueNAS box inside a jail, there were a few issues with the package management that would require an OS upgrade. That’s not a task for today so I decided to do it on my Windows PC and just take the hit of reading over the network. Another key requirement was to keep the original creation date from the old files (when the photo was taken) and set the “last modify” to the same so that the timestamps on the compressed files would match the original, allowing for easier viewing/sorting. I ended up putting together the following Powershell script:

Get-ChildItem -Path '\\nas.home\pool-data\pictures' -File -Recurse | ForEach-Object {
    $newname = [System.IO.Path]::GetFileNameWithoutExtension($_.FullName);
    $fullname = $_.FullName
    $newdirectory = $_.DirectoryName.Substring(39);
    $created = $_.CreationTime;
    $extn = [IO.Path]::GetExtension($_.FullName)
    if ($newname -like "*Snapchat-*" -And ($extn -eq ".jpg" -Or $extn -eq ".jpeg")) {	
        if(!(Test-Path -Path ".\compressed\$newdirectory"))
        {
            New-Item -Name ".\compressed\$newdirectory" -ItemType Directory
            Write-Host "New folder created successfully!" -f Green
        }
        ffmpeg -y -i $_.FullName -q:v 10 ".\compressed\$newdirectory\$newname.jpg"
        Get-ChildItem  ".\compressed\$newdirectory\$newname.jpg" | % {$_.LastWriteTime = $created}
        Get-ChildItem  ".\compressed\$newdirectory\$newname.jpg" | % {$_.CreationTime = $created}
    }
};

This would recursively go through all the folders for my pictures on LBS, finding only saved Snapchats and capturing the appropriate metadata. It would then compress them and write the output to a file on my local PC and set the metadata on the new versions. I didn’t want to immediately overwrite the images so I could do some more spot checks on quality and size changes and then back up the originals on another folder (that doesn’t have offsite/offline backup).

I let the script run while I ducked out to the corner store and came back to it completed. All my spot checks matched my discoveries about qualities before, the files had the correct metadata, and I was ready to migrate them to the NAS.

The size of the compressed files ended up being 3.2GB, roughly 80% compression over the whole collection and I’ve now got a script I can run whenever I back up my phone.