Intro

My Client needed a way to migrate from Bitbucket server to GitHub cloud,

Migrating from Bitbucket Server to GitHub Cloud can be a challenging task, especially when dealing with large repositories and sensitive data. Each migration is unique, requiring a tailored plan to ensure a smooth transition. With careful planning and the right tools, this process can be managed effectively.

This article explores a strategy that I personally used to successfully migrate repositories.

Migrating Large Repos Securely

1. Planning the Migration

• Assess the current repository structure and size.
• Identify reasons for migration (e.g., scalability, performance, organizational changes).
• Plan the timeline.

2. Choosing the Right Migration Strategy

• Clone and Push: Directly clone the repository and push it to the new location.
• Incremental Migration: Migrate in smaller chunks to minimize risk and downtime.
• Splitting Repositories: Separate large repositories into smaller, more manageable ones.

3. Tools for Migration

• Git clone –mirror: For exact replication including all branches and tags.
• Git bundle: Creates a single file containing the repository’s history.
• Third-party Tools: Platforms like GitHub, GitLab, and Bitbucket often provide migration tools.

In this article we will use git clone —-mirror and git-filter-repo, and give an example of pushing a big repo with a lot of commits.

for files above 100 mb you can use git lfs , in this article i’m using git-filter-repo and deleting all files that are larger than 50mb.

git-filter-repo provides a robust solution for cleaning up repositories and removing sensitive data effectively.

Examples:

1. removing sensitive data from a repository.
2. filter blobs sizes.
3. rename all files
4. replace text and much more.

warning – git-filter-repo rewrite your history.

#!/bin/bash
clone_url="your_git_repo" organization="rotemdeveleap" new_repo_name="new_repo"
#creating git repository with github cli.
gh repo create $organization/$new_repo_name --private 
git clone --mirror $clone_url
cd <cloned repo>
git-filter-repo --strip-blobs-bigger-than 50M
git remote add origin 'https://github.com/$organization/$new_repo_name.git'

Now you can push your repo.

But what happens if you need to push in parts?

you can push your commits with steps (this example is for the master branch only.)

#!/bin/bash />main_branch="master"
# Push commits in steps of 1000 to GitHub
step_commits=$(git log --oneline --reverse refs/heads/$main_branch | awk 'NR % 1000 == 0')
echo "$step_commits" | while read commit message; do
git push origin "$commit:refs/heads/$main_branch" done
#push last 1000 commits
step_commits=$(git log --oneline --reverse refs/heads/$main_branch | awk 'NR % 1 == 0' | tail -n 1000)
echo "$step_commits" | while read commit message; do
git push origin "$commit:refs/heads/$main_branch" done

Bonus : If you want to push specific branches like releases :

#!/bin/bash
git for-each-ref --format='%(refname:short)' 'refs/heads' | grep -E '^release/.*' | xargs -I{} git push origin {}
git push origin --tags

Summary:

During the process, I gained several valuable insights and skills:

Understanding the Importance of good Planning,

How to choose the right migration strategy using strategy Selection and risk Management, to utilize the right Tool, and how to handle large files and sensitive data

Overall, this migration project was an excellent opportunity to enhance my skills in repository management, automation, and the use of advanced Git tools. These lessons will be invaluable for future projects involving repository migration and management.