Invalid Characters: Uploading 700GB to OneDrive using Chrome on Mac OS

New Contributor

Hi everyone, I'm Francesco from Italy and I'm trying to upload my entire HDD to OneDrive Business (1 TB of free space). 
I'm actually a Mac user (the HDD belongs to an old Macintosh), and that's the reason why I have to stick with a browser upload. Technically speaking I could also just drag my entire HDD to OneDrive Business Folder, but this would cause the HDD being copied to the SSD (where the OS is installed and where the OneDrive folder is).

Summing up:

HDD: 700 GB --> I want to upload the entire disk to OneDrive
HDD: belongs to a Mac --> I can only upload the content via browser (Chrome in that case)

Unfortunately some files in the HDD have "invalid characters" not accepted by OneDrive. I've just discovered that OneDrive has a lots of limitation when it comes to uploading files:

 

" * : < > ? / \ | [OneDrive Standard]
~ " # % & * : < > ? / \ { | } [OneDrive Server]

Empty Files (0 kb)

Empty Folders (0 kb)

File Named with a "Double Space" (i.e. "Family  Portrait")

 

And so on...
I hope you can understand that I can't deal with "manual renaming" each file and that's why I looked for a "Shell Script" to run under MacOS in order to batch rename everything.

 

This is the script I found:

 

#!/usr/bin/env bash

# The following characters are considered impermissible in a basename:
#
# - Left Square Bracket: [
# - Right Square Bracket: ]
# - Left Parenthesis: (
# - Reverse Solidus: \
# - Colon: :
# - Quotation Mark "
# - Single Quotation Mark '
# - Asterisk *
# - Question Mark ?
# - Less-than Sign <
# - Greater-than Sign >
# - Vertical Line |
# - Plus Sign +
# - Space Character
# - UnderScore _
#
# 1. Sed is utilized for character replacement therefore characters listed
# in the bracket expression [...] must be escaped as necessary.
# 2. Any forward slashes `/` in the basename are substituted by default with
# a Colon `:` at the shell level - so it's unnecessary to search for them.
#
declare -r IMPERMISSIBLE_CHARS="[][()\\:\"'*?<>|+_ ]"
declare -r REPLACEMENT_STRING="-"

# Obtain the POSIX path of each selected item in the `Finder`. Input must
# passed to this script via a preceding `Get Selected Finder Items` action
# in an Automator Services workflow.
declare selected_items=("$@")

declare -a sorted_paths
declare -a numbered_paths

# Prefix the POSIX path depth level to itself to aid sorting.
for ((i = 0; i < "${#selected_items[@]}"; i++)); do
numbered_paths+=("$(echo "${selected_items[$i]}" | \
awk -F "/" '{ print NF-1, $0 }')")
done

# Sort each POSIX path in an array by descending order of its depth.
# This ensures deeper paths are renamed before shallower paths.
IFS=$'\n' read -rd '' -a sorted_paths <<< \
"$(printf "%s\\n" "${numbered_paths[@]}" | sort -rn )"


# Logic to perform replacement of impermissible characters in a path basename.
# @param: {Array} - POSIX paths sorted by depth in descending order.
renameBaseName() {
local paths=("$@") new_basename new_path

for path in "${paths[@]}"; do

# Remove numerical prefix from each $path.
path="$(sed -E "s/^[0-9]+ //" <<< "$path")"

# Replaces impermissible characters in path basename
# and subsitutes uppercase characters with lowercase.
new_basename="$(sed "s/$IMPERMISSIBLE_CHARS/$REPLACEMENT_STRING/g" <<< \
"$(basename "${path}")" | tr "[:upper:]" "[:lower:]")"

# Concatenate original dirname and new basename to form new path.
new_path="$(dirname "${path}")"/"$new_basename"

# Only rename the item when:
# - New path does not already exist to prevent data loss.
# - New basename length is less than or equal to 255 characters.
if ! [ -e "$new_path" ] && [[ ${#new_basename} -le 255 ]]; then
mv -n "$path" "$new_path"
fi
done
}

renameBaseName "${sorted_paths[@]}"

 

The shell seems to be working fine, but there are other issues:

- It's deleting (replacing with a space) also some valid characters that I don't want to remove


- As soon as a file is renamed it looses also each capital letter


- I can't apply the "script" to my HDD folder, but I have to expand each folder and subfolder and apply to single items

 

- The script can be applied to maximum 1.000 items per time

 

Is there a workaround?
I just wanted to backup my old HDD to OneDrive. That's it.

3 Replies

@Bacchinif 
Why not use a tool with a trial license?
There are tools that offer a trial license that enables you to upload the data set. These tools typically cater for illegal characters, skip macOS ds_store, provide error handling, ...

There are several tools but support for macOS is limited. See collab365. SLIM Companion Explorer is a tool that also supports macOS.

Paul | SLIM Applications

Dear @Paul de Jong thanks for your prompt response.

 

I wasn't aware of that services, thanks for your good advice. Do you think is it worth giving them a try? Of course, I don't want to pay for the service and I think Microsoft itself should be able to handle the issues like Google Drive and Dropbox usually do. 

 

@Bacchinif It is not an actual service. The trial version of the tool can be used for 8 days so transferring the 700 GB is possible within that time frame (I know because I am affliated with the vendor).  The positive side is that there are no costs.

The other options to upload such a large amount of data from a mac is limited. Using PowerShell is the other main option. I am not aware how well this works on mac OS (e.g. you may run into problems with the illegal characters)

 

Paul