Best practices for safely performing schema changes in Azure Database for MySQL
Azure Database for MySQL - Flexible Server is built on the open-source MySQL database engine, and the service supports MySQL 8.0 and newer versions. This means that users can take advantage of the flexibility and advanced capabilities of MySQL’s latest features while benefitting from a fully managed database service. While newer versions and features can provide a lot of value, the recent issues identified with MySQL versions 8.0+ makes it important to be aware of potential risks that can occur during certain operations, particularly if you are making online schema changes. Issues with data loss and duplicate keys with Online DDL Online Data Definition Language (DDL) operations are a powerful feature in MySQL, enabling schema changes like ALTER TABLE or OPTIMIZE TABLE with minimal impact on table availability. These operations are designed to reduce downtime by allowing concurrent reads and writes during schema modifications, making them an essential tool for managing active databases efficiently. However, a recent post on the Percona blog, Who Ate My MySQL Table Rows? highlights critical risks associated with MySQL 8.0.x versions after 8.0.27 and all versions beyond 8.4.y. Specifically, the open-source INPLACE algorithm, commonly used for online schema changes, can lead to data loss and duplicate key errors under certain conditions. These issues arise from constraints in the INPLACE algorithm, particularly during ALTER TABLE and OPTIMIZE TABLE operations, exposing vulnerabilities that compromise data integrity and system reliability. These risks are called out in the following bug reports: Bug #115511: Data loss during online ALTER operations with concurrent DML Bug #115608: Duplicate key errors caused by online ALTER operations Documented issues related to the INPLACE algorithm (used for online DDL) can cause: Data Loss: Rows may be accidentally deleted or become inaccessible. Duplicate Keys: Indexes can end up with duplicate entries, leading to data consistency issues and potential replication errors. Problems arise when INPLACE operations, such as ALTER TABLE or OPTIMIZE TABLE, run concurrently with: DML operations (INSERT, UPDATE, DELETE): Modifications to table data during the rebuild. A purge activity: Background cleanup operations for old row versions in InnoDB. These scenarios can lead to anomalies resulting from race conditions and incomplete synchronization between concurrent activities. Impact on Azure Database for MySQL - Flexible Server Customers For Azure Database for MySQL Flexible Server customers using MySQL 8.0+ and all versions after 8.4.y, this issue is particularly critical as it affects: Data Integrity: During schema changes such as ALTER TABLE or OPTIMIZE TABLE run using the INPLACE algorithm, data rows may be lost or duplicated if these operations run concurrently with a DML activity (e.g., INSERT, UPDATE, or DELETE) or background purge tasks. This can compromise the accuracy and reliability of the database, potentially leading to incorrect query results or the loss of critical business data. Replication Instability: Duplicate keys or missing rows can interrupt replication processes, which rely on a consistent data stream across the primary and replica servers. These issues can arise when there are concurrent insertions into the table during schema changes, leading to data inconsistencies between the primary and replicas. Such inconsistencies may result in replication lag, errors, or even a complete breakdown of high-availability setups, requiring manual intervention to restore synchronization. Operational Downtime: Resolving these issues often involves manually syncing data or restoring backups. These recovery efforts can be time-consuming and disruptive, leading to extended downtime for applications and potential business impact. Recommendations for safe schema changes on Azure Database for MySQL flexible servers To minimize the risks of data loss and duplicate keys while making schema changes, follow these best practices: Set old_alter_table=ON to Default to COPY Algorithm Enable the server parameter old_alter_table system variable so that ALTER TABLE operations without a specified ALGORITHM default to using the COPY algorithm instead of INPLACE. This reduces the risk for users who do not explicitly specify the ALGORITHM in their commands. Learn more on how configure server parameters in Azure Database for MySQL. Avoid using ALGORITHM=INPLACE Do not explicitly use ALGORITHM=INPLACE for ALTER TABLE commands, as it increases the risk of data loss or duplicate keys. Back up your data before schema changes Always perform a full on-demand backup of your server before executing schema changes. This precaution ensures data recoverability in case of unexpected issues. Learn more on how to take full on-demand backups for your server. Avoid Concurrent DML during schema changes Schedule schema changes like ALTER TABLE and OPTIMIZE TABLE during application maintenance windows when no concurrent writes activities occur. This minimizes race conditions and synchronization conflicts. Use External Tools for Safer Online Schema Changes Consider using external tools like pt-online-schema-change to modify table definitions without blocking concurrent changes. These tools enable you to make schema changes with minimal impact on availability and performance. Learn more about pt-online-schema-change. Disclaimer: The pt-online-schema-change tool is not managed or supported by Microsoft; use it at your discretion. Mitigation plans To address these risks, we’re actively working to integrate the necessary fixes to ensure a more robust and reliable experience for our customers. New Servers Fully Secured by End of February 2025 All new Azure Database for MySQL Flexible Server instances created after 1 st March 2025, will include the latest fixes, ensuring that schema changes are safeguarded against data loss and duplicate key risks. Rollout for Existing Servers For existing servers, we will roll out patches during upcoming maintenance windows by end of Q1 of Calendar Year 2025 We recommend monitoring your Azure portal for scheduled maintenance windows and Release notes for announcements about critical updates and patches. Priority updates available upon request If you require an urgent update outside of the scheduled maintenance windows, you can contact Azure Support. Provide the necessary server details and an appropriate maintenance window, and our team will work with you to prioritize the patching process. Note that priority patching will be available by February 2025. We recommend monitoring Release notes for announcements about critical updates and patches. Conclusion Safely managing schema changes on MySQL servers requires understanding the risks associated with online DDL operations, such as potential data loss and duplicate keys. To help safeguard data integrity and maintain server stability, implement best practices, for example enabling the COPY algorithm, using offline operations if feasible, or scheduling changes during low activity periods. Fixes are expected by the end of February 2025, and new Azure Database for MySQL flexible servers will be fully protected against these bugs. We will apply updates to existing servers during maintenance windows in Q1 2025. Following the recommendations above will help ensure that you can confidently make schema changes while preserving the reliability and performance of your server.
Modification loss in excel files hosted on OneDrive
On December 4th I discovered losses of modifications in some files. When opening an Excel file I often work with, I was suprised since I was sure I added some lines into it a few days ago. I looked at version history and found a strange behaviour on December 4th: several versions on 29th November 15:06 with the same size. One of the 3 versions of 29th November 15:06 seem to be the latest in terms of modifications, even the size was similar. So I manually recovered this one. I did the same on a 2nd excel file to confirm the issue. Other files may be impacted. How can that happen? Could it be a ntp issue leading OneDrive not to know which version is the latest? I opened a case at MS support, but they closed the case w/o resolution. They even didn't try to understand the issue, just sending a TN how to restore a version of a file. But first I need to identify all the impacted files, then to find the correct version. I can't do these actions manually on hundred or maybe thousands of files. How could I proceed? Do u think it is related to Excel or OneDrive or to sthg else?
Unexpected data loss for extensions, local storage, etc. in Edge Beta
I haven't found a similar report, and this is very unusual for me. A few days ago, after I opened Edge, I noticed that all extension icons became default (blank), but others looked fine. Soon more things became default (like asking for default browser), but history, bookmarks and tabs were kept. Eventually I noticed that all installed extensions' files were lost (the list was kept, but all "corrupted"), and the extension's data was also gone, including data that wasn't backed up. I noticed that folders like "User Data.CHROME_DELETE" appeared, which contained a lot of files, but it didn't seem to contain extension files and data. I have never switched Edge versions to other branches. Today, Edge doesn't start (hangs, seem more than 1 hour) for me, had to kill the process and restart, then I noticed that this was happening again. Many extensions are kept at this time, but some are "corrupted" again. It appears that all extensions data is lost again, the history is a few days ago, and was synced to retrieve some. Cookies is lost. At this time, in AppData\Local\Microsoft\Edge\User Data\Snapshots, I see "103.0.1264.71" (07-28) & "103.0.1264.77" (07-31) folders, contains backups of some important files. But I don't think this includes the extensions storage that I value most (including things like OneTab). Maybe something related for config: * In my config, the AppData\Local\Microsoft\Edge Beta\User Data is a symlink to AppData\Local\Microsoft\Edge\User Data, I don't remember the exact reason, but it was at peace in the previous few dozens of months. * C drive is occasionally less well-off, but I don't think this should cause data loss. * A few days ago, I moved out the Group Policy - Computers - CacheDir, and added the Group Policy - Current User - CacheDir (i.e. https://docs.microsoft.com/DeployEdge/microsoft-edge-policies#diskcachedir). It points to the same directory on another HDD. * Launch Enhanced Options and Windows Update, maybe related, possibly it starts Edge stable and causes some data migrations, and a big failure for backups. 104.0.1293.35 in Edge Beta's edge://version for now.To Do changes for on-premises mailbox users
Since I don't see this having been announced in the To Do blog or in the Microsoft 365 Message Center, I'm sharing this here. ------------------------------------------------------------ https://support.microsoft.com/en-gb/office/changes-to-microsoft-to-do-access-for-on-premises-mailbox-accounts-f0cb4131-c8c0-4c80-a4df-b75482f62be1?ui=en-US Changes to Microsoft To Do access for on-premises mailbox accounts Starting January 22nd, 2021, anyone who uses To Do or the Tasks app in Teams with an on-premises mailbox will only be able to sign in to To Do on the web. You won’t be able to sign in to To Do on Android, iOS, Mac, or Windows. In To Do on the web, you’ll be able to view your lists and tasks, but anything you edit or add won’t be saved and will be lost once you sign out. Starting February 22nd, 2021, anyone with an on-premises mailbox won’t be able to sign in to To Do on any platform. Make sure to back up your To Do data before then by printing your lists or saving them as PDFs. Once your organization has moved your mailbox to the cloud, you’ll be able to sign in to To Do and manually reenter the information you saved. How moving mailboxes from on-premises to Exchange Online affects your To Do data Microsoft To Do needs an Exchange Online mailbox to store and sync your tasks, which means on-premises mailboxes need to be moved to Exchange Online. If you use To Do or the Tasks app in Teams with an on-premises mailbox, you’ll lose your To Do data when your on-premises mailbox is moved to Exchange Online. To prepare for that, make sure to either print your lists or save them as PDFs so you can manually recreate them in To Do. For IT admins at companies with on-premises mailboxes When you tell employees you’ll be migrating their on-premises mailboxes to Exchange Online, remind anyone who uses Microsoft To Do to print their lists or save them as PDFs. People who use the Tasks app in Teams will need to download To Do so they can print or save their lists. Once you've migrated their mailboxes to Exchange Online, employees will need to manually reenter their saved list information in To Do. For other employees at companies with on-premises mailboxes If you use Microsoft To Do or the Tasks app in Teams, your To Do lists and tasks may be lost when your organization migrates its mailboxes from on-premises to Exchange Online. As a precaution, print your To Do lists or save them as PDFs. Once your mailbox has been migrated to Exchange Online, you can manually reenter the information you saved. For more info, talk to your IT admin. ------------------------------------------------------------ I work for a Microsoft partner, am currently working on an EXO migration engagement with a customer, and a VIP user just lost all of their To Do Lists/Tasks upon being migrated to EXO. I hadn't seen the KB article previously, and don't remember seeing anything come through in my Message Center notifications. When I search the M365 Message Center today, I see nothing about this.
