Database Encryption

[matteosecli]

Hello, I'm having troubles with the database after the last Mendeley update. My current situation is

• Mendeley 1.19.1 (latest version) on my personal laptop (MacOS)
• Mendeley 1.17.11 on my office workstation (Linux)

They are both synced to each other; however, Menotexport can read the database on the Linux box but not on my MacOS laptop. The error I get is the following:

# <Menotexport>: Failed to recoganize the given database file.
file is not a database

I'm pretty sure the problem is due to the fact that Mendeley has recently decided to encrypt the user database "to comply with GDPR & keep your data safe [...]" (which I think it's totally bull**it & it's not what GDPR says). I've found a comment on the issue & some instructions to obtain a decrypted database here, but I was not able to test them on my updated Mendeley install on MacOS because it seems that the MacOS version doesn't have a debug mode.

I'm reporting this as a bug even if it's actually Mendeley's fault; however, I think it could be useful to keep track of the issue.

Thanks again for your work! 🙂

[Xunius]

Hi matteosecli,

Many thanks for the report! It's a big thing, if I can't figure out a way to deal with the encryption then this tool will be dead. I'll put a note in the readme.

[williamgunn]

Hey @Xunius, we're really sorry we broke your tool with the encryption! We want to get it working again and provide a way to access the local database that we can officially support so that tools will keep working as we make changes. We're currently collecting use cases and setting up some one-on-ones with the development team. If you're interested, please email me.

[maphouse]

has this been solved? As a mendeley user (version 1.19.2), can i still use this tool? Would appreciate a definite answer before cloning. Thanks @Xunius !

[tuxflo]

I'm also interested since in version 1.19.3 there is no option to export highlights at all (only sticky notes). I just thought I found a great tool to do the research work for my thesis...

[matteosecli]

@maphouse @tuxflo I think you can still download and use either Mendeley 1.18 (Stable) or Mendeley 1.18.1 (Development) as a workaround. I suggest to install the old version on another machine or on a virtual machine in order to avoid potential conflicts with the newer version.

Here are the links (directly from Mendeley's servers) of these old versions:

Mendeley 1.18 (Stable)

• Ubuntu [32 bit]: https://desktop-download.mendeley.com/download/apt/pool/main/m/mendeleydesktop/mendeleydesktop_1.18-stable_i386.deb
• Ubuntu [64 bit]: https://desktop-download.mendeley.com/download/apt/pool/main/m/mendeleydesktop/mendeleydesktop_1.18-stable_amd64.deb
• Linux [32 bit]: https://desktop-download.mendeley.com/download/linux/mendeleydesktop-1.18-linux-i486.tar.bz2
• Linux [64 bit]: https://desktop-download.mendeley.com/download/linux/mendeleydesktop-1.18-linux-x86_64.tar.bz2
• Windows: https://desktop-download.mendeley.com/download/Mendeley-Desktop-1.18-win32.exe
• MacOS: https://desktop-download.mendeley.com/download/Mendeley-Desktop-1.18-OSX-Universal.dmg

Mendeley 1.18.1 (Development)

• Ubuntu [32 bit]: https://desktop-download.mendeley.com/download/apt/pool/main/m/mendeleydesktop/mendeleydesktop_1.18.1-stable_i386.deb
• Ubuntu [64 bit]: https://desktop-download.mendeley.com/download/apt/pool/main/m/mendeleydesktop/mendeleydesktop_1.18.1-stable_amd64.deb
• Linux [32 bit]: https://desktop-download.mendeley.com/download/linux/mendeleydesktop-1.18.1-linux-i486.tar.bz2
• Linux [64 bit]: https://desktop-download.mendeley.com/download/linux/mendeleydesktop-1.18.1-linux-x86_64.tar.bz2
• Windows: https://desktop-download.mendeley.com/download/Mendeley-Desktop-1.18.1-win32.exe
• MacOS: https://desktop-download.mendeley.com/download/Mendeley-Desktop-1.18.1-OSX-Universal.dmg

[tuxflo]

Thanks for the quick reply. I just installed the Windows Version 1.18.1 but it seems that there is no export feature as well.... maybe I'd just look for some alternative to Mendeley although I really like the Annotation & Sync feature... and it seems to be the only "integrated" tool that runs on Linux as well.

[matteosecli]

@tuxflo have you tried to open the PDFs one-by-one and then select "File -> Export PDF with Annotations"? I have Mendeley 1.19.2 and this feature is still present, though Mendeley does not have bulk-exports and the annotations are exported in a non-standard way -- that's also why Menotexport was made.

Anyway I fully understand your point: it's one of the best reference managers out there and I also need a multi-platform solution, too bad they keep "closing" it more and more.

[tuxflo]

You are totally right: it is possible to do that when the document is open in the internal PDF Viewer (I already edited my comment but for some reason it wasn't saved)

Currently I'm facing troubles with installing your tool: do I have to install some python dependencies first? And do I have to clone the repo or is conda doing that as well (sorry I never heard of conda before...)

[matteosecli]

@tuxflo this tool was written by @Xunius (not by myself), so it's actually better to wait for his answer for this sort of stuff! 😃

[tuxflo]

OK Nevermind, I just skipped the conda stuff and did a regular install and now everything is just fine!
I'm using 18.1, it syncs and the (editable) bulk annotation works. Thanks so much for your help @matteosecli and @Xunius for creating this awesome little tool 👍

[matteosecli]

Glad that it works! 😄

[Xunius]

@tuxflo Glad that this little thing helps.

@matteosecli Thanks for helping out. I was away the previous week. And I just noticed that the PDFs we are exporting don't preserve metadata (some pieces of info the original PDF creator inserts into the file, like title, author, and even doi). For ordinary user this won't be noticeable at all, but for others it may be a pity to lose them as it's much easier to recognize a document from its metadata than parsing the file. So I'm planning to add this.
Also I'm "secretly" making a rip off a mendeley (working name "elemendy" XD). Of cause it won't be as powerful but it will be totally transparent. I would let you know if you are interested, and when it gets somewhere (currently it's still preprepre alpha).

[tuxflo]

@Xunius Yes I'm interested! I just had the same idea (build a open source alternative to mendeley) yesterday.
Currently I don't have time to do some coding myself (writing my thesis, thats why I came in touch with mendeley) but here are just some thoughts I had:

• I would not take care of syncing and use external existing tools for that (nextcloud, gdrive, dropbox...)
• It would be a huge improvement if there would be touch support in the pdf viewer for scrolling and for annotating (Currently there is no pdf viewer for linux that can handle annotations on touch devices)
• Highlights made with other tools should be imported and visible/editable in the tool (one huge disadvantage of Mendeley in my opinion)
• Auto save feature would be nice

If you are still working on the project after march next year I'd be happy to help to get this thing done :)
I have some knowledge in different programming languages, especially in C++ and Qt (but python is good as well)

[matteosecli]

Hi @Xunius, sorry for the very long absence! I thought I would have had less work after October, but at this point I think it was just a sci-fi dream and the work kept increasing and increasing...😅

Anyway I really like your idea! I was trying to look into that by myself a couple of years ago, but I really don't know anything about user interfaces (I'm a command-line guy) so I quickly gave up. But I'd be very interested if you let me know when you have something, at least I can help you testing or (if I have time) give some contributions to the backend part.

About the metadata problem...do you already have some hint about the cause and (possibly) the solution? Unfortunately I don't know so much about the PDF library you're using here. Otherwise one could always resort to a brute-force thing like: copy the meta before adding annotations and then paste them back.

[Xunius]

Hi @tuxflo @matteosecli
Glad to hear positive feedbacks from both of you (I just realized that we probably have a Mendeley guy in this thread, but that's ok I guess).

I can totally appreciate that we all have our own priorities, my own progress has been slowed down a lot recently.

I agree with tuxflo that I'm not going to implement any syncing and will let the user use whatever to their liking. Auto-save is on my plan as well. In terms of pdf viewer, I have little idea how difficult it would be to embed a viewer into this, my plan (at least for the 1st stage) is to let the user fire up an external viewer, and move onto an embedded one should the 1st stage goes smoothly.

A few other things on my list:

• I'm using PyQt5 to build the GUI, currently have got the main frame set up, with the basic structure similar to Mendeley's, plus some minor tweakings in the right hand side tab panel. The top menus, option dialogues etc are still empty now.
• For database I'm using sqlite, same as Mendeley as well. The reading part is done. I didn't use Qt's sql interface, but stored the data into dictionaries and displayed them using QAbstractView. Probably won't be as efficient. The saving part hasn't started yet.
• For Bib files I'm planning to use bibtexparser, and doi querying habanero. No idea for RIS yet.
• For duplicate checking I'm using fuzzywuzzy.
• For fetching metadata from PDFs (a real nice feature of Mendeley), I'm still relying on pdfminer and PyPDF2, this is when I noticed that preserving metadata would be beneficial because it's so much easier. For those that don't have metadata, I'm testing parsing only the 1st page and trying to guess the title, authors etc.. But I kind of doubt this would be a good option, because unless I could make it real accurate, it might be better to leave a field empty than filling with wrong info. For some PDFs pdfminer seems to take quite a long time to parse even the 1st page, which is another discouraging factor.
• Another nice feature of Mendeley is full text search, which I really want to have. I've heard about "xapian", but no idea how easy it would be to implement.
• I'm still collecting ideas. For instance I thought about inserting a note pad somewhere so you can jog down some quick thoughts that's not associated with any document like currently in Mendeley's note tab. But then I thought the user might have his favorite text editor at hand so it doesn't make too much sense to have such a thing, unless it can be made into a really good one.

That' about it, still a whole lot to be done. If you get any idea please let me know. Winter holiday is coming so I should be able to make some process soon.