Migrating MongoDB GUIDs from Legacy to Standard Format (MongoDB v2 to v3)

Introduction

When upgrading from MongoDB v2 to v3, one significant change is the GUID representation format. MongoDB v2 used UuidLegacy (CSharpLegacy) format while MongoDB v3 uses UuidStandard format by default. This causes compatibility issues when attempting to read legacy data with newer MongoDB driver versions.

This guide provides a comprehensive approach to migrating your existing MongoDB data from legacy GUID format to the standard format used in MongoDB v3 and later versions.

Prerequisites

Step 1: Understanding the Migration Process

The migration process includes:

  1. Backup: Creating a complete backup of your MongoDB databases
  2. Collection Migration: For each collection in each database:
    • Reading documents with legacy GUID format
    • Converting GUIDs to standard format
    • Saving to a temporary collection
    • Renaming collections to maintain data
  3. Configuration Update: Updating application settings to use standard GUID representation

Step 2: Setting Up the Migration Tools

Add the following classes to your project:

MigrateGuidCommand Class

This command class provides a simple way to execute the migration:

            
using Microsoft.Extensions.Logging;
using System;
using System.Threading.Tasks;
using Volo.Abp.DependencyInjection;

namespace Solann.Software.Upgrade
{
    public class MigrateGuidCommand : ITransientDependency
    {
        private readonly MongoDbGuidMigrationService _migrationService;
        private readonly ILogger<MigrateGuidCommand> _logger;

        public MigrateGuidCommand(
            MongoDbGuidMigrationService migrationService,
            ILogger<MigrateGuidCommand> logger)
        {
            _migrationService = migrationService;
            _logger = logger;
        }

        public string Name => "migrate-guid";

        public string Description => "Migrate GUID format from Legacy to Standard in MongoDB";

        public async Task ExecuteAsync()
        {
            _logger.LogInformation("Starting MongoDB GUID format migration...");

            try
            {
                await _migrationService.ExecuteMigrationAsync();
                _logger.LogInformation("MongoDB GUID format migration completed successfully!");
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error during MongoDB GUID migration");
                throw;
            }
        }
    }
}

MongoDbGuidMigrationService Class

This service handles the actual migration work:

            
using MongoDB.Bson;
using MongoDB.Bson.Serialization.Serializers;
using MongoDB.Bson.Serialization;
using MongoDB.Driver;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Volo.Abp.DependencyInjection;
using Microsoft.Extensions.Configuration;
using System.IO;
using System.Diagnostics;

namespace Solann.Software.Upgrade
{
    public class MongoDbGuidMigrationService : ITransientDependency
    {
        private readonly IMongoDatabase _defaultDatabase;
        private readonly IMongoDatabase _quickDatabase;
        private readonly IConfiguration _configuration;

        public MongoDbGuidMigrationService(
           IMongoClient mongoClient,
           IConfiguration configuration)
        {
            _configuration = configuration;

            // Get database names from connection strings
            var defaultDbName = GetDatabaseNameFromConnectionString(
                configuration.GetConnectionString("Default") ?? "Software1");

            var quickDbName = GetDatabaseNameFromConnectionString(
                configuration.GetConnectionString("Quick") ?? "Software1Quick");

            _defaultDatabase = mongoClient.GetDatabase(defaultDbName);
            _quickDatabase = mongoClient.GetDatabase(quickDbName);
        }

        // Extract database name from connection string
        private string GetDatabaseNameFromConnectionString(string connectionString)
        {
            try
            {
                if (connectionString.Contains("/"))
                {
                    return connectionString.Split('/').Last().Split('?').First();
                }
            }
            catch
            {
                // Ignore errors and use default value
            }
            return "Software1"; // Default value
        }

        // Step 1: Backup all databases
        public async Task BackupDatabasesAsync()
        {
            try
            {
                // Create backup directory
                string backupFolderName = DateTime.Now.ToString("yyyyMMdd_HHmmss");
                string backupBasePath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "MongoBackups");
                string backupPath = Path.Combine(backupBasePath, backupFolderName);

                // Ensure directories exist
                if (!Directory.Exists(backupBasePath))
                {
                    Directory.CreateDirectory(backupBasePath);
                }

                if (!Directory.Exists(backupPath))
                {
                    Directory.CreateDirectory(backupPath);
                }

                Console.WriteLine($"Using backup directory: {backupPath}");

                // Databases to backup
                var databases = new List<(string ConnectionString, string DatabaseName)>
                {
                    (GetConnectionStringWithoutDatabase("Default"), _defaultDatabase.DatabaseNamespace.DatabaseName),
                    (GetConnectionStringWithoutDatabase("Quick"), _quickDatabase.DatabaseNamespace.DatabaseName)
                };

                foreach (var (connectionString, databaseName) in databases)
                {
                    Console.WriteLine($"Starting backup for database {databaseName}...");
                    Console.WriteLine($"Connection string: {connectionString}");

                    try
                    {
                        // Create directory for this database
                        string dbBackupPath = Path.Combine(backupPath, databaseName);
                        Directory.CreateDirectory(dbBackupPath);

                        // Get mongodump path
                        string mongoDumpPath = FindMongoDumpPath();
                        Console.WriteLine($"Using mongodump at: {mongoDumpPath}");

                        // Setup backup process
                        var psi = new ProcessStartInfo
                        {
                            FileName = mongoDumpPath,
                            Arguments = $"--uri=\"{connectionString}\" --db={databaseName} --out=\"{dbBackupPath}\"",
                            RedirectStandardOutput = true,
                            RedirectStandardError = true,
                            UseShellExecute = false,
                            CreateNoWindow = true
                        };

                        Console.WriteLine($"Executing: {psi.FileName} {psi.Arguments}");

                        // Create process
                        var backupProcess = new Process { StartInfo = psi };

                        // Buffers for output
                        var outputBuilder = new StringBuilder();
                        var errorBuilder = new StringBuilder();

                        // Handle output events
                        backupProcess.OutputDataReceived += (sender, args) =>
                        {
                            if (!string.IsNullOrEmpty(args.Data))
                            {
                                outputBuilder.AppendLine(args.Data);
                                Console.WriteLine($"[mongodump] {args.Data}");
                            }
                        };

                        backupProcess.ErrorDataReceived += (sender, args) =>
                        {
                            if (!string.IsNullOrEmpty(args.Data))
                            {
                                errorBuilder.AppendLine(args.Data);
                                Console.WriteLine($"[mongodump ERROR] {args.Data}");
                            }
                        };

                        // Start process
                        backupProcess.Start();
                        backupProcess.BeginOutputReadLine();
                        backupProcess.BeginErrorReadLine();

                        // Set timeout to 10 minutes
                        bool completed = backupProcess.WaitForExit(10 * 60 * 1000);

                        if (!completed)
                        {
                            Console.WriteLine($"WARNING: Process mongodump for {databaseName} timed out, forcing termination.");
                            try
                            {
                                backupProcess.Kill(true);
                            }
                            catch (Exception killEx)
                            {
                                Console.WriteLine($"Error terminating process: {killEx.Message}");
                            }
                        }

                        // Get exit code
                        int exitCode = completed ? backupProcess.ExitCode : -1;

                        // Handle results
                        if (exitCode == 0)
                        {
                            Console.WriteLine($"Successfully backed up database {databaseName}. Data saved at: {dbBackupPath}");
                        }
                        else
                        {
                            Console.WriteLine($"Failed to backup database {databaseName}. Exit code: {exitCode}");
                            Console.WriteLine($"Error output: {errorBuilder}");
                        }
                    }
                    catch (Exception dbEx)
                    {
                        Console.WriteLine($"Error backing up database {databaseName}: {dbEx.Message}");
                        Console.WriteLine($"Stack trace: {dbEx.StackTrace}");
                        // Continue with next database
                    }
                }

                Console.WriteLine($"All database backup process completed. Data saved at: {backupPath}");
            }
            catch (Exception ex)
            {
                Console.WriteLine($"Error during backup process: {ex.Message}");
                Console.WriteLine($"Stack trace: {ex.StackTrace}");
                throw;
            }
        }

        // Extract connection string without database
        private string GetConnectionStringWithoutDatabase(string connectionStringName)
        {
            string connectionString = _configuration.GetConnectionString(connectionStringName);

            if (string.IsNullOrEmpty(connectionString))
            {
                return "mongodb://localhost:27017";
            }

            if (connectionString.Contains("/?"))
            {
                // With query parameters
                return connectionString.Split('/').Take(connectionString.Split('/').Length - 1).Aggregate((a, b) => a + "/" + b);
            }
            else if (connectionString.Contains('/'))
            {
                // Without query parameters
                string[] parts = connectionString.Split('/');
                if (parts.Length > 3) // mongodb://host:port/database
                {
                    return string.Join("/", parts.Take(parts.Length - 1));
                }
            }

            return connectionString;
        }

        // Find mongodump path
        private string FindMongoDumpPath()
        {
            string[] possiblePaths = new[]
            {
                @"C:\Program Files\MongoDB\Server\6.0\bin\mongodump.exe",
                @"C:\Program Files\MongoDB\Server\5.0\bin\mongodump.exe",
                @"C:\Program Files\MongoDB\Server\4.4\bin\mongodump.exe",
                @"C:\Program Files\MongoDB\Tools\100\bin\mongodump.exe",
                @"C:\Program Files\MongoDB\Database Tools\mongodump.exe",
                @"mongodump.exe", // For Windows PATH
                @"mongodump"      // For Linux/macOS
            };

            foreach (var path in possiblePaths)
            {
                if (path == "mongodump.exe" || path == "mongodump" || File.Exists(path))
                {
                    return path;
                }
            }

            // Use default if not found
            return Environment.OSVersion.Platform == PlatformID.Win32NT ? "mongodump.exe" : "mongodump";
        }

        // Migrate a single collection
        public async Task MigrateCollectionAsync<T>(IMongoDatabase database, string collectionName)
        {
            Console.WriteLine($"Starting migration for collection: {collectionName}");

            try
            {
                // 1. Read all documents from source collection
                var collection = database.GetCollection<BsonDocument>(collectionName);
                var documents = await collection.Find(FilterDefinition<BsonDocument>.Empty).ToListAsync();
                Console.WriteLine($"Found {documents.Count} documents in {collectionName}");

                if (documents.Count > 0)
                {
                    // 2. Create temporary collection
                    string tempCollectionName = $"{collectionName}_temp";
                    try
                    {
                        await database.DropCollectionAsync(tempCollectionName);
                    }
                    catch
                    {
                        // Ignore if collection doesn't exist
                    }

                    // 3. Get temporary collection
                    var tempCollection = database.GetCollection<BsonDocument>(tempCollectionName);

                    // 4. Process and convert each document
                    var convertedDocuments = new List<BsonDocument>();
                    foreach (var doc in documents)
                    {
                        // Create a copy for processing
                        var convertedDoc = doc.ToBsonDocument();
                        ConvertGuidFields(convertedDoc);
                        convertedDocuments.Add(convertedDoc);
                    }

                    // 5. Save converted documents
                    if (convertedDocuments.Count > 0)
                    {
                        await tempCollection.InsertManyAsync(convertedDocuments);
                    }

                    // 6. Rename collections
                    try
                    {
                        await database.DropCollectionAsync($"{collectionName}_old");
                    }
                    catch
                    {
                        // Ignore if collection doesn't exist
                    }

                    await database.RenameCollectionAsync(collectionName, $"{collectionName}_old");
                    await database.RenameCollectionAsync(tempCollectionName, collectionName);
                }

                Console.WriteLine($"Completed migration for collection: {collectionName}");
            }
            catch (Exception ex)
            {
                Console.WriteLine($"Error migrating collection {collectionName}: {ex.Message}");
                throw;
            }
        }

        // Convert all GUID fields from legacy to standard
        private void ConvertGuidFields(BsonDocument doc)
        {
            // Process _id field if it's a GUID
            if (doc.Contains("_id") && doc["_id"].IsBsonBinaryData)
            {
                var binaryData = doc["_id"].AsBsonBinaryData;
                if (binaryData.SubType == BsonBinarySubType.UuidLegacy)
                {
                    // Get GUID from legacy binary data
                    var guid = binaryData.ToGuid(GuidRepresentation.CSharpLegacy);
                    // Create new BsonBinaryData with Standard format
                    doc["_id"] = new BsonBinaryData(guid, GuidRepresentation.Standard);
                }
            }

            // Process child fields
            foreach (var key in doc.Names.ToList())
            {
                if (doc[key].IsBsonDocument)
                {
                    ConvertGuidFields(doc[key].AsBsonDocument);
                }
                else if (doc[key].IsBsonBinaryData)
                {
                    var binaryData = doc[key].AsBsonBinaryData;
                    if (binaryData.SubType == BsonBinarySubType.UuidLegacy)
                    {
                        var guid = binaryData.ToGuid(GuidRepresentation.CSharpLegacy);
                        doc[key] = new BsonBinaryData(guid, GuidRepresentation.Standard);
                    }
                }
                else if (doc[key].IsBsonArray)
                {
                    var array = doc[key].AsBsonArray;
                    for (int i = 0; i < array.Count; i++)
                    {
                        if (array[i].IsBsonDocument)
                        {
                            ConvertGuidFields(array[i].AsBsonDocument);
                        }
                        else if (array[i].IsBsonBinaryData)
                        {
                            var binaryData = array[i].AsBsonBinaryData;
                            if (binaryData.SubType == BsonBinarySubType.UuidLegacy)
                            {
                                var guid = binaryData.ToGuid(GuidRepresentation.CSharpLegacy);
                                array[i] = new BsonBinaryData(guid, GuidRepresentation.Standard);
                            }
                        }
                    }
                }
            }
        }

        // Main migration method
        public async Task ExecuteMigrationAsync()
        {
            try
            {
                // 1. Backup data
                await BackupDatabasesAsync();

                // 2. Migrate Default DB collections
                Console.WriteLine("Starting migration for Default database");
                var defaultCollections = await _defaultDatabase.ListCollectionNamesAsync();
                foreach (var collection in await defaultCollections.ToListAsync())
                {
                    if (!collection.StartsWith("system.") && !collection.EndsWith("_old"))
                    {
                        await MigrateCollectionAsync<BsonDocument>(_defaultDatabase, collection);
                    }
                }

                // 3. Migrate Quick DB collections
                Console.WriteLine("Starting migration for Quick database");
                var quickCollections = await _quickDatabase.ListCollectionNamesAsync();
                foreach (var collection in await quickCollections.ToListAsync())
                {
                    if (!collection.StartsWith("system.") && !collection.EndsWith("_old"))
                    {
                        await MigrateCollectionAsync<BsonDocument>(_quickDatabase, collection);
                    }
                }

                // 4. Update application configuration
                Console.WriteLine("Migration complete. Application configuration needs to be updated to use Standard GUID format.");
            }
            catch (Exception ex)
            {
                Console.WriteLine($"Error during migration process: {ex.Message}");
                Console.WriteLine($"Stack trace: {ex.StackTrace}");
                throw;
            }
        }
    }
}

Step 3: Integrating with Your Project

To integrate this migration tool with your project:

  1. Add the migration classes shown above to your project
  2. Configure the DbMigratorHostedService to execute the migration: 
                        
public class DbMigratorHostedService : IHostedService
{
    private readonly IHostApplicationLifetime _hostApplicationLifetime;
    private readonly IConfiguration _configuration;

    public DbMigratorHostedService(IHostApplicationLifetime hostApplicationLifetime, IConfiguration configuration)
    {
        _hostApplicationLifetime = hostApplicationLifetime;
        _configuration = configuration;
    }

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        using (var application = await AbpApplicationFactory.CreateAsync<SoftwareDbMigratorModule>(options =>
        {
           options.Services.ReplaceConfiguration(_configuration);
           options.UseAutofac();
           options.Services.AddLogging(c => c.AddSerilog());
           options.AddDataMigrationEnvironment();
        }))
        {
            await application.InitializeAsync();

            // Check if GUID migration should run
            var shouldMigrateGuids = _configuration.GetValue<bool>("Migration:MigrateGuids", false);

            // Check command line arguments
            var commandLineArgs = Environment.GetCommandLineArgs();
            if (commandLineArgs != null && (commandLineArgs.Contains("--migrate-guids") || commandLineArgs.Contains("-mg")))
            {
                shouldMigrateGuids = true;
            }

            if (shouldMigrateGuids)
            {
                Log.Information("Starting MongoDB GUID format migration...");

                // Get MigrateGuidCommand service and execute
                var migrateGuidCommand = application.ServiceProvider.GetRequiredService<MigrateGuidCommand>();
                await migrateGuidCommand.ExecuteAsync();

                Log.Information("MongoDB GUID format migration completed.");
            }
            else
            {
                // Run normal migrations if needed
                await application
                    .ServiceProvider
                    .GetRequiredService<SoftwareDbMigrationService>()
                    .MigrateAsync();
            }

            await application.ShutdownAsync();
            _hostApplicationLifetime.StopApplication();
        }
    }

    public Task StopAsync(CancellationToken cancellationToken)
    {
        return Task.CompletedTask;
    }
}
  3. Update your module to register the migration services: 
                        
public override void ConfigureServices(ServiceConfigurationContext context)
{
    // Existing configuration...

    // Register MongoDB migration services
    context.Services.AddTransient<MongoDbGuidMigrationService>();
    context.Services.AddTransient<MigrateGuidCommand>();

    // Register IMongoClient for dependency injection
    context.Services.AddSingleton<IMongoClient>(sp => {
        var configuration = sp.GetRequiredService<IConfiguration>();
        var connectionString = configuration.GetConnectionString("Default");
        return new MongoClient(connectionString);
    });
}

Step 4: Running the Migration

To run the migration, you have several options:

Option 1: Configure in appsettings.json

            
{
    "Migration": {
        "MigrateGuids": true
    },
    "ConnectionStrings": {
        "Default": "mongodb://localhost:27017/Software1",
        "Quick": "mongodb://localhost:27017/Software1Quick"
    }
}

Option 2: Run with command-line argument

            
dotnet run --migrate-guids

Important Considerations

  1. Always backup your data before running the migration
  2. Test in a non-production environment first
  3. Plan for downtime as this process modifies your entire database
  4. Monitor the migration process carefully
  5. Have a rollback plan in case of issues
  6. Keep the _old collections until you've verified everything works

Conclusion

Migrating from MongoDB v2's legacy GUID format to MongoDB v3's standard format requires careful planning and execution. The approach described in this guide provides a reliable way to convert your data while maintaining data integrity and relationships between collections.

Author's Note: This article documents a technical challenge I encountered while developing SEO tools based on the ABP Framework. The migration process outlined here was successfully implemented in production at Solann Software, where we specialize in custom business software solutions and digital marketing tools. Feel free to visit our site for more technical insights or to learn about our services.