Introduction
In today's rapidly evolving digital landscape, data is the lifeblood of any application or service. Developers and businesses often find themselves needing to adapt and scale their databases to meet changing requirements. One common scenario is the migration from Firebase Firestore, a popular NoSQL database, to Azure Cosmos DB, Microsoft's globally distributed, multi-model database service. This migration can be driven by various factors, such as performance optimization, scalability needs, compliance requirements, or a shift in the cloud ecosystem.
In this comprehensive guide, we will take you on a journey through the process of migrating your data from Firebase Firestore to Azure Cosmos DB. We will begin by exploring the fundamental differences and structures of these two databases. Understanding the intricacies of both Firestore and Cosmos DB is crucial for a successful migration, as it will enable you to make informed decisions and minimize potential pitfalls along the way.
What does the blog entail?
- Brief overview of what is Firebase Firestore and Azure cosmos db.
- Configure the Firebase Firestore database for export.
- Export the Firebase Firestore data to a JSON file.
- Transform/modify Firebase Firestore JSON object.
- Provision Azure Cosmos DB.
- Use the Azure Cosmos DB Data Migration Tool to import the JSON file into Azure Cosmos DB.
What is Firebase Firestore?
Firestore Overview
Firebase Firestore is a NoSQL, document-oriented database that is part of the Firebase platform, a cloud-based mobile and web application development platform by Google. Firestore is designed for real-time data synchronization, scalability, and seamless integration with other Firebase services.
Firestore Data Model
Firestore stores data in collections, which are analogous to tables in traditional relational databases. Each collection contains a set of documents, and documents are the individual data units in Firestore. Documents are represented in JSON format, making it easy to work with structured data.
Firestore's data model includes collections, documents, and fields:
- Collections: Collections are containers for documents and are often used to group related data. Collections are dynamic, meaning you can create them on the fly as you add documents.
- Documents: Documents are individual records within collections. Each document has a unique identifier and contains a set of key-value pairs.
- Fields: Fields are the key-value pairs within a document. Firestore supports various data types, including strings, numbers, Booleans, arrays, and nested objects.
Firestore's flexible data model and real-time capabilities make it a popular choice for building modern web and mobile applications.
What is Azure Cosmos DB?
Azure Cosmos DB Overview
Azure Cosmos DB is a fully managed NoSQL and relational database for modern app development. It is a globally distributed, multi-model database service provided by Microsoft Azure. It is designed for high availability, low latency, and seamless scaling across multiple regions. Cosmos DB supports multiple data models, including document, key-value, graph, column-family, and table, making it suitable for a wide range of applications.
Azure Cosmos DB APIs
Azure Cosmos DB offers multiple database APIs, which include NoSQL, MongoDB, PostgreSQL Cassandra, Gremlin, and Table. These APIs allow your applications to treat Azure Cosmos DB as if it were various other database technologies, without the overhead of management, and scaling approaches. Azure Cosmos DB helps you to use the ecosystems, tools, and skills you already have for data modelling and querying with its various APIs.
For this blog, we are going to focus on API FOR NOSQL since it’s the most compatible API with our scenario.
Cosmos DB Data Model
Azure Cosmos DB's data model is rooted in its support for multiple data models, but for the purpose of this migration, we will focus on the document data model, which is most like Firestore.
- Databases: In Cosmos DB, databases are top-level containers for your data. Each database can contain multiple collections.
- Collections: Collections are containers for documents, similar to Firestore collections. Collections in Cosmos DB can have different partition keys, enabling efficient distribution of data.
- Documents: Documents in Cosmos DB are similar to Firestore documents. They are JSON objects with unique identifiers and are stored within collections.
- Partitioning: Cosmos DB uses a partitioning mechanism to distribute data across physical partitions for scalability. Choosing the right partition key is crucial for optimal performance.
- Cosmos DB's multi-region replication and global distribution capabilities make it a powerful choice for applications with worldwide user bases.
Our Project Structure
Firebase Firestore Database
We are going to focus on a task management system database. This is the database structure we are going to use.
We have two collections tasksCollection and usersCollections. Each has three documents with ids like task1-id, task2-id, task3-id, user1-id, user2-id, user3-id.
Each document has various fields comprised of key and value pairs.
Requirements to get started.
- Azure subscription, if you’re a student redeem Azure for students else create a free account.
- VS Code.
- Firebase account.
- Node.js installed.
Configure the Firebase Firestore database for export.
- Assuming you have logged into the Firebase Firestore database, and you have populated the data above on your database. You will have the below database structure. TasksCollection with 3 documents and usersCollection with 3 documents.
- Click project overview, select project settings.
- Click service accounts and then new private key.
- Clicking that button will generate this JSON file. The name may differ but what matters is the content of that file. It contains credentials which should be kept secret. We are going to use them to access our database.
Rename the Json file to appConfig.json and open that folder in VS Code. Then add this command on the terminal ` npx -p node-firestore-import-export firestore-export -a appConfig.json -b firebaseExportedDBFile.json`. This will use appConfig.json data to export our database to firebaseExportedDBFile.json.
Export the Firebase Firestore data to a JSON file.
- If you get this prompt, just accept.
- Wait till all collections are exported in the file.
- After opening the file, we have all our data into two collections.
Transform/modify Firebase Firestore JSON object.
- We need to make the ids to be part of Json object and make the collections elements be an array of objects.
- We need to transform our collections i.e. task collection into an array of objects.
- Clone this repo, using
git clone https://github.com/kelcho-spense/Firebase_to_Cosmos_DB_JSON_Object_Parser.git
Move firebaseExportedDBFile.json inside the folder we cloned. Make sure the file matches the filename we are importing. The function below uses loops and node.js fs module to remove and restructure our and export all the collections inside the original firebaseExportedDBFile.json into collection files.
Run
cd Firebase_to_Cosmos_DB_JSON_Object_Parser
on the terminal to change directory, then run
yarn or npm install
to install the node packages. Then to start the node.js local server
yarn start or npm start
- This will produce two collections.
Provision Azure Cosmos DB.
Search for Azure Cosmos DB and select the first option.
- Click create.
- Click create Azure Cosmos DB for NoSQL.
- Create a new resource group and account name, the click next.
- Leave this as default or enable these options for geo back up.
- Select all networks and TLS 1.0 (this will work with the current tool we will use to import data to cosmosdb)
- Let’s go with the defaults or select your appropriate options for periodic back up policy.
- Select service-management key.
- Click create to provision our services.
- Go to resource.
- Click data explorer and create new database.
- Provide database id and click okey.
- Hover the three dots next to our DB name and click on new container.
- Provide container id as our collection name and partition key for the tasksCollection.
- Do the same for the usersCollection.
Use the Azure Cosmos DB Data Migration Tool to import the JSON file into Azure Cosmos DB.
- Visit this link, download the zip file and extract the files.
- After extracting run the dtui.exe file. Using this tool am going to import my two collection files.
- Click add files, select tasksCollection.json. Click next.
- Back to the portal, click keys and click the eye icon on primary connection string then copy this secret string.
- Back to Azure Cosmos DB data migration tool, paste the primary connection string in the connection string. Add Database=taskManagementDB and click verify to test coonection. If connection is okey add the collectionName and the partition key which you want to map our JSON data. Click next.
- Click next.
- Confirm everything is okey and click next.
- When successful you should see the number of transferred documents.
- Let’s confirm this in the portal. Go to data explorer, click TasksCollection, items then id and you will see the details of each document(JSON object).
- To see all data, click on new SQL Query icon then click execute query button. All the records(documents) will be visible.
- Repeat the above process with usersCollection.
- And there you have it, we have successfully imported both collections from firebase.
NB: you can as well import the original JSON file we exported from Firestore DB, but I just wanted us to simulate the file structure in firebase.
Check these resources for more information.
- Welcome to Azure Cosmos DB
- Choose an API in Azure Cosmos DB
- Azure Cosmos DB pricing
- Data migration tool