Latest update prevents edits on cloud files

Discussion of the macOS applications DiveLogDT and Dive Log Manager
akouris
Posts: 6
Joined: Sat Nov 24, 2012 11:48 pm

Latest update prevents edits on cloud files

Post by akouris »

I updated today to the latest DiveLogDT version.

After so many years of using the software, the developer decided to prevent users from using any cloud service to work on their documents/databases.

This is 2021. Instead of the developer finding a way to finally embrace automatic online syncing, and cloud services, the developer is choosing a backwards move - one that can only create additional mess: if you want to make any change, you have to manually copy the file on a local drive, make the changes, and then remember to manually sync that file to the cloud service by either manually copying the new file, or "exporting" it.

What a mess! What an awful, user-hostile decision.

I can understand that the decision was made because some of the users ended up messing their files.

But first of all this is their mistake, and if the developer wants to actually help, they should develop a cloud syncing solution, not lock everyone else's ability to use cloud services to sync their files.

At least they should give that option to the users, to choose whether they want to continue using cloud files.
support
Posts: 742
Joined: Wed Mar 31, 2010 4:35 pm

Re: Latest update prevents edits on cloud files

Post by support »

Hi,

Thank you for your feedback on the new version of DiveLogDT. We certainly appreciate hearing about user's needs and experience so that we have the opportunity to improve our products. Let me also say that we understand your point and they are certainly valid. In the short term, we will look a solution that will allow you to continue to use DiveLogDT in the way that you have been using it.

I would like your indulgence to explain what is going on behind the scenes here since this is a public forum and some people might become misinformed from your post. I want to make sure that other readers understand the risks that storing your logbook database file in a "cloud synchronized" folder presents.

Cloud storage systems like iCloud Drive, Dropbox, Google Drive, OneDrive, etc. behave very differently from local file systems or even network file systems (like AFS, NFS or SMB). In the case of cloud file systems you are interacting with a local copy of the file that is periodically synchronized with the provider's servers. This synchronization takes place behind the scenes and can happen at unexpected times. For example, let's say that you are interacting with a file on your iPhone that is not connected to the internet that results in changes to that file. Then a few days later your iPhone comes connected to the internet. At that time (or some short time later) it will push the new version of the file up to the servers. Then, a short time after that, if your Mac is connected to the internet, that file will replace the local copy of that file on your Mac. This is of course not an issue if you haven't wanted access to that file on your Mac in the meantime. This is a very simplified view of what happens (and the exact behavior is very different for each of the cloud service providers). DiveLogDT keeps your logbook in a database file (it uses a database system called Sqlite). As far as the cloud service provider is concerned the logbook database is just a file. However, the underlying Sqlite system has a fairly complex relationship with this file (including creating and updating temporary files from time to time as it does transactions in the database). It is admittedly a rare occurrence, but it is possible for the cloud system to synchronize the logbook database with the sever in the middle of a transaction. If this occurs, and the database is not synchronized again after the transaction is completed but before it is used on a different system (keep in mind that this might not be due to an obvious action on the user's part like opening the file because of the asynchronous nature of cloud systems) the database file can become corrupted. This is the situation has occurred for multiple users who were using their logbook directly in a cloud synchronized folder and is the issue that the new feature of DiveLogDT was designed to avoid.

The concern is that users might not be aware of this possibility, and in fact probably are not and don't really need to be, and the resulting corruption can be unrecoverable without a backup. Further, it is a situation that can be very hard to avoid because of the nature of cloud systems. By the way, database files are not the only type of file that can be affected by this situation. For example, the virtual disk used by virtual machines can also run afoul of cloud servers, as well as other types of databases like Apples Core Data which a large number of App's use.

DiveLogDT, Dive Log on iOS and Diving Log 6.0 on Windows synchronize with each other via cloud services by ensuring that the logbook database is never opened on the cloud synchronized file system directly, but rather is always copied to a local location before being acted on. Then, once the synchronization operation is complete, the logbook file is copied back to the cloud synchronized folder in a safe way. On iOS this is easy to enforce since there is no way to specify the logbook database location that Dive Log uses. However, on the Mac it is easy to open a logbook file from any location. The "Cloud" item under the SYNCHRONIZE panel in the sidebar of DiveLogDT is meant to be used to synchronize between your open logbook file and the selected cloud service file. This is initiated manually (more on why in a minute) and ensures the same safe access to the cloud version of the logbook file that Dive Log on iOS ensures with it's "Import", "Export" and "Synchronize Dive Log Logbook" functions on the Synchronize tab.

Our recommendation for using cloud synchronization between Dive Log on iOS, DiveLogDT on Mac and Diving Log 6.0 on Windows, is to have your active logbook file stored on a local file system (as a side note, a network file system like AFS, SMB or NFS is also safe as they implement "file system" semantics) and then use the provided interfaces in the App's to synchronize with a cloud service version of the logbook. We do not recommend manually copying the active logbook database file (using Finder for example) to and from the cloud synchronized folder as that provides no assurance that the file is not open by another application or DiveLogDT itself.

Your point about the age of dynamic synchronization between devices that we have all become use to is certainly valid. There are a number of reasons that we do not currently support dynamic automatic synchronization (like Contacts or Notes for example). The biggest reason is that our products (particularly Dive Log on iOS) are very often used where internet access is not available or where it is prohibitively expensive (for example on a live aboard dive boat). We use a manually initiated synchronization via the cloud to ensure that you are in control of when the synchronization occurs. We spend a lot of our time with little to no internet access and find that we are constantly having to "clean up" issues with various applications that do do automatic synchronization via the "cloud" (Apple's own Numbers app and Notes apps are ones I regularly have issues with). The other reason that we currently don't have a dynamic synchronization solution is because we support synchronization with multiple applications on multiple platforms (for the technically inclined, this means that we can't use Apple only or Windows only technologies that can help with cloud based database updates). Of course there are solutions that are possible to implement that could address these cross platform limitations but we are constrained by our development resources and the resources available to the developers that synchronize with our products. Logbook software is not a big market so this is very much a personal passion project for all independent logbook software makers.

As I mentioned at the beginning, we will look at re-enabling the type of use that you have been using up until now. With a solid internet connection and carefully ensuring exclusive access to the logbook file stored in the cloud you can probably avoid the corruption issues that other users have experienced. That said, I would recommend that you regularly take a backup of your logbook file - use the File->Backup option in DiveLogDT or by exporting a copy to a cloud service using the features in Dive Log and DiveLogDT so that you can get back to a known state if the worst were to occur.

Again, thank you for your perspective on this change to DiveLogDT. It was certainly not indented as a user-hostile action but rather as a way to protect unsuspecting users from unintended data loss. Your experience with and reaction to this decision is valuable to us and we appreciate you sharing it.

Regards,
Greg and Janice
akouris
Posts: 6
Joined: Sat Nov 24, 2012 11:48 pm

Re: Latest update prevents edits on cloud files

Post by akouris »

Dear Greg and Janice,

First of all thank you for the prompt reply.

I should take the opportunity to thank you for the software - I am a very satisfied paid user on both iOS + Mac.

I am anxiously waiting for the update which will bring previous functionality back.

I have been using the software for many years, having logged over 1.000 dives in it.

I understand how cloud syncing works (and the dangers associated), and I also appreciate that some users might not understand that, and they might leave DT sessions open on different computers, and they might end up meshing their files.

Even for them, I think that the choice you made is the wrong one. Trying to protect the database integrity is creating an even bigger problem, one that will end up pushing them to search for alternarive software: today it was the first day that instead of just opening my computer to log a dive, I had to start looking at files and make sure that I am opening the correct "version".

In an age where most users are starting to not even understand what a "file" is, you are asking them to manually manage versioning.

This is a backwards move, and maybe you should implement somekind of internal save/backup from DT instead of the current situation.
support
Posts: 742
Joined: Wed Mar 31, 2010 4:35 pm

Re: Latest update prevents edits on cloud files

Post by support »

Thank you for your additional feedback and for your patience as we work through the options.

Again, for others that may be reading this thread, I'd like to make one point clear. If you always open the same logbook file (DiveLogDT will open the last used logbook file each time you start) and you use the "Cloud" option under the SYNCHRONIZE item in the sidebar and "Two-Way Sync" with any cloud stored logbook, you do not need to worry about "versions". The database is designed to support complex synchronization across time by keeping careful accounting information. For example, if you edited a dive on your iPhone and then edited the dive site associated with that dive on your Mac then synced your Mac with the "cloud" file and later synced your iPhone with the cloud file then synced your Mac again, both devices would have the correct version of that dive. In fact, iCloud Drive can cause "conflict" versions of files depending on when the files manage to migrate through the internet. DiveLogDT and Dive Log manage these conflict files automatically (using the aforementioned accounting information) to ensure that all the updates are all applied in the correct order. If you have ever opened a Numbers or Pages document that says there is a "conflict" and asks you which file would you like to keep, then you have encountered this conflict mechanism. However, Numbers/Pages has no way to automatically address these conflicts so it asks you to decide. The mechanism used in DiveLogDT/Dive Log is designed to handle this without user intervention freeing users from worrying about file version management.

We are looking at a number of different options (including ones along the lines you suggest). Thank you again for sharing your thoughts on this issue, they are very helpful.

Cheers,
Greg
Jason12154
Posts: 2
Joined: Wed Jul 21, 2021 4:54 pm

Re: Latest update prevents edits on cloud files

Post by Jason12154 »

I just got back from a week long dive trip only to find out that my DiveLog DT is now a read only file with no option so sync or update. This is very inconvenient since my dive computer has to be connected to my computer to be down loaded because it does not have blue tooth capability. A quick google search led me to this thread, I was wondering if there is any update on restoring the previous capabilities of this app? Having to search through files on my computer and trying to guess if I have the right one is not very practical, I might as well just go back to using a paper dive log. I'm not a "big tech guy" by any standards, but isn't the whole point of any app or program to be able to do what you want by simply opening it up and not have to search through files on your computer? Up until now this app worked great on my Mac in combination with the Dive Log app on my iPhone, I'm hoping there is a fix coming soon!
support
Posts: 742
Joined: Wed Mar 31, 2010 4:35 pm

Re: Latest update prevents edits on cloud files

Post by support »

As noted in the thread above, the intention of the change in the recent DiveLogDT update was to *protect* users from corrupting their logbook file. As you indicated, it is true that sometimes it is difficult to remember where the logbook file is stored on your Mac's hard drive. So the intention was to let users know that the logbook file they have just opened is in a "dangerous" place and that it should be moved somewhere else where it is not susceptible to corruption. If it is read only, then yes, you can not update it by Editing or Syncing or Downloading new dives from a dive computer.

It's also worthwhile to note that this "corruption" problem can exist for other Applications besides DiveLogDT that use database files in non-hidden folders.

In terms of work flow, the idea was that you could have your original logbook open, and then open (read only) the version of that logbook in the Cloud that is used for syncing to other devices, and take a look at it (without accidentally changing it) and review it, and then re-open your original local logbook. At that point, you would SYNCHRONIZE with that identified Cloud verison of the logbook.

The original post here is saying that they know the file is in a "dangerous" (ie: Cloud backed) location, but that they want the option to use it there anyway. And we are thinking about allowing that option. If users are good about having a backup of their logbook file that they can revert to should something happen to the it while the Cloud is syncing automatically in the background, then "overriding" the read only part is a valid option.

In the short term, if you move your logbook file out of the Cloud Drive and back onto your local hard drive, then it will no longer be read only. Generally, anywhere in your Documents folder is fine. If you have opted-in to the Apple "store all my files in the Cloud" service, then you can move your logbook file to your "Home Directory". This is typically "/Users/<username>" on Macintosh HD. Feel free to contact us if you need help doing this.

If you have DiveLogDT open, the name of your logbook file that you created is shown in the lower left portion of the window and has an extension of ".sql". If you want to know what folder it's located in, you can use the DiveLogDT "File -> Open" menu item and it will show you the folder it's currently located in. So moving forward, you would "File -> New" a new logbook file in a local location, and then use the SYNCHRONIZE "Cloud" panel and select "Import/Restore Logbook FROM Cloud" to get your Cloud based logbook into the new local one you just created.

Please feel free to contact us if you need any help with this process on your Mac. We are considering making a change for the next release of DiveLogDT in this area and so feedback like this is helpful.

Regards,
Jason12154
Posts: 2
Joined: Wed Jul 21, 2021 4:54 pm

Re: Latest update prevents edits on cloud files

Post by Jason12154 »

Thank you for the information. Apparently my documents are going to iCloud. I was able to move the .sql file onto my Mac and it seems to be working fine.
Jayip54
Posts: 3
Joined: Tue Jul 27, 2021 11:47 am

Re: Latest update prevents edits on cloud files

Post by Jayip54 »

I have also been a paid user for many years and have always had it pointed to the log file on my mac that syncs with dropbox. I have never had an issue with the files getting corrupted and this update is very frustrating. Why this was not fully advised with the update I do not know and why not an option to shut it down or not. I do not find the explanation acceptable as as I said, the file is pointed to the one that is physically on my laptop, in the dropbox folder so it syncs with the cloud. And the dropbox files are all physically on the laptop, not in the cloud. Please fix this.

Jay
akouris
Posts: 6
Joined: Sat Nov 24, 2012 11:48 pm

Re: Latest update prevents edits on cloud files

Post by akouris »

As I originally raised the issue, I am very happy to see that the latest update available for download restores cloud syncing functionality.

At this point I would like to thank Greg and Janice for providing us with a great software, continuous development, and keeping their ears open to what their users say.

It would be a great time to leave a positive review on AppStore!
Jayip54
Posts: 3
Joined: Tue Jul 27, 2021 11:47 am

Re: Latest update prevents edits on cloud files

Post by Jayip54 »

I agree - glad to see the update and how quickly it came. Thanks

Jay
Post Reply