discursivescript

[discursivescript]

Layman’s Guide to Mass-Exporting Comments from Dreamwidth and LiveJournal

The documentation for this particular feature is piecemeal and scattered over various FAQs, wikis, dev comms, Github repos, etc. No one source quite covers all the bases (basic usage, different URLs for LJ and DW, community export). So here's a writeup of what I’ve pieced together.

Exporting comments, rather than just posts, isn't a "regular user" feature of either site. It’s part of the XML-RPC API intended for programmers, site-to-site communication such as Dreamwidth's importer, and client programs (anyone else remember Semagic?). So following these instructions will get you a giant pile of XML that isn't very pretty or user-friendly. However, it is fairly readable in Firefox and Chrome, which will automatically indent and color-code it for you.

1. Go to LJ or DW and log into the account whose comments you want, if you aren't logged in already. For communities, log in as one of the mods. There's no way to mass-fetch comments from a journal you don't control--you'll just have to set a crawler to slurp them up from flat view.

2. Go to https://www.livejournal.com/export_comments.bml?get=comment_body&startid=0 or http://www.dreamwidth.org/export_comments?get=comment_body&startid=0

3. For communities, add &authas=communityname to the end of the URL.

4. Save as an XML file.

5. This will get you comment texts and timestamps, but only cryptic IDs for accounts and posts. For a map of IDs to usernames, change get=comment_body to get=comment_meta.

6. The comments come back 1000 at a time, so to get the next page of comments, change startid=0 to startid=1000. comment_meta comes back 10,000 at a time.

7. Rinse and repeat until you run out of comments.

8. To back up posts as well, check out LJ's journal export FAQ, which also has instructions on editing the XML file for display. There is probably a way to make the jitemid from the comments XML match up with the ID of the parent post it represents, but I haven't dug that deep yet.

9. Congratulations, you now have a giant pile of (relatively straightforward) XML to work with. Search for a utility that can do what you want with it, or find a friend who's learning to code and tell them you've got a project for them.

Sources: LiveJournal "Exporting Comments" API documentation Dreamwidth's code for importing comments from LJ (see do_authed_comment_fetch) dw_dev post about exporting community comments For developers who want a non-browser-dependent way to authenticate and use the comments endpoint, LJ's XML-RPC Client/Server Protocol documentation has you covered.