bigbearlabs

[bigbearlabs]

Recovering data from an unmountable Time Machine image

Last week, I noticed that I hadn't moved over my iTunes folder when I migrated from an iMac with 1TB storage to my current MacBook Pro with only 512GB storage, about 18 months ago.

I couldn't recall whether I had deliberately chosen to temporarily rely on the music uploaded to iCloud, or if I had simply postponed it as a lower-priority task with a few things to figure out (e.g. where would be the best location for it?)

In any case, I wanted to play my music offline again without depending on Apple Music, so reached out to the Time Machine backup of my previous iMac to restore ~/Music/iTunes.

And that's how I fell into the rabbit hole that took me a week to climb out of.

The Time Machine .sparsebundle, all 1.1TB faithfully copied onto my Synology NAS, did not mount. As usual, the immediate error message was not helpful in understanding why it's failing, or what I could try. Trying to mount in the command line with very conservative options didn't particularly help either:

user@picklerick> hdiutil attach -nomount -noverify -noautofsck -readonly /Volumes/backup-hfs/tunip.sparsebundle hdiutil: attach failed - image not recognised user@picklerick>

Well, this is a surprise, I thought. After all, I made quite a bit effort to get Time Machine working well over the network over a long period of time, with the expectation that the payoff would be that I have backups.

Feeling backstabbed at that blind faith, and with mild regrets of not having thought of some means to ensure the the validity of the backups before deciding to fully rely on them, I took a deep breath, then looked into the contents of the .sparsebundle directory to eyeball-diff it with another Time Machine backup of my wife's MacBook Air.

One obvious difference was the absence in the unmountable image of files Info.plist and Info.bckup. Perhaps I had not fully rsync'ed the Time Machine backup after decommissioning my previous Mac?

So I opened up Info.plist from the 'reference' sparse bundle to see if fudging the contents of the missing file could make the .sparsebundle mountable. I first tried using the same values, which resulted in hdiutil now yielding a block device, implying it was now convinced that the sparse bundle was a disk image, albeit without the partitions.

user@picklerick> hdiutil attach -nomount -noverify -noautofsck -readonly /Volumes/backup-hfs/tunip.sparsebundle /dev/disk6

This was in contrast to what the hdiutil output when I attached my reference sparse bundle:

user@picklerick> hdiutil attach -nomount -noverify -noautofsck -readonly /Volumes/backup-hfs/old-and-accessible-time-machine-backup.sparsebundle /dev/disk3 GUID_partition_scheme /dev/disk3s1 EFI /dev/disk3s2 Apple_HFS

But at this point, I felt a little optimistic. Perhaps I could recreate the partition map for the disk presented by the sparse bundle, and regain access to the partition with the files still intact. I just needed to grab the directory for my music, which I recall was ~100GB in size. There'd be no guarantees that the extent of the corruption of the image wouldn't affect data I needed, but it's not like I needed the entire backup recovered, so perhaps I can go down this rabbit hole a little deeper...

This trade-off analysis was concluded when I recalled that I had started that music collection back in the days when iTunes was SoundJam MP... I decided the historical value outweighed what felt like a fair chance that a lot of time and effort would be expended for an eventual dead end.

A few evenings later, I was indeed able to copy over ~/Music/iTunes Music. I'll skip a lot of details during the trial-and-error phase -- pity as it is though, that the pain I went through is not fully conveyed -- and document the steps that yielded the final result, in the hopes that some will find it useful in their own data recovery rabbit hole.

Set the size property of the .sparsebundle's Info.plist to a value observed from another Time Machine backup image that was roughly the same effective size -- this turns out to 5000981078016 (5TB). This was different from my initial reference sparse bundle which was set to 2TB. It would appear Time Machine would incrementally increase the sparse bundle's 'theoretical' size as the effective data footprint increases.

Run diskutil info on the reference sparse bundle's block device and testdisk on the corrupt sparse bundle to determine what partition map to re-create.

The most crucial part of recreating the missing partition information was to work out the offset and length values of the HFS partition that would have held all the backup data. Some googling returned two invaluable tools to examine the corrupt backup for lost filesystems -- testdisk and hfsprescue.

testdisk's partition scan confusingly returned many, many HFS partitions. Was this a symptom of corruption? I found some documentation from hfsprescue that suggested the authoritative entry is the first one, so decided it's not necessarily a bad sign. I examined my reference sparse bundle's virtual disk and found it encouraging that the HFS on its partition map had the same offset and size as the one testdisk found on my corrupt disk:

TestDisk 7.0, Data Recovery Utility, April 2015 Christophe GRENIER http://www.cgsecurity.org Disk /dev/disk3 - 5000 GB / 4657 GiB - 9767541168 sectors (RO) Current partition structure: Partition Start End Size in sectors 1 P EFI System 40 409639 409600 [EFI System Partition] 2 P Mac HFS 409640 9767278983 9766869344 [disk image]

A relatively new gotcha here was that as of Mojave, Time Machine backups became protected by the security mechanism that requires an app to to be added to the 'Full Disk Access' list in the Security & Privacy preference pane. This means you must add your terminal app -- I normally use iTerm but for this job chose to go with Terminal.app -- to that list before using testdisk or other tools with block devices, in order not to get weird errors. Pretty much all the steps below require this set-up.

Commit the partition map to a recovery copy of the corrupt bundle.

So it was time to try to write this partition map to a copy of the corrupt disk, and see if the HFS became mountable. I was initially misled by a few articles online to use pdisk to write the partition map, and it didn't help either that testdisk gave me the same advice when I had mistakenly chosen the mac partition scheme. After I recalled that hdiutil had already told me the bundle needed to have a GUID partition scheme and selected 'EFI GPT' as the partition table type, I was able to use testdisk to set up and write the partition data.

Attach the recovery copy, and confirm hdiutil reports 2 filesystems on the virtual disk.

user@picklerick> hdiutil attach -nomount -noverify -noautofsck -readonly /Volumes/backup-hfs/tunip.sparsebundle /dev/disk6 GUID_partition_scheme /dev/disk6s1 EFI /dev/disk6s2 Apple_HFS

Hmm, this is looking pretty good...

Mount the data partition.

user@picklerick> diskutil mount /dev/disk6s2 Volume Time Machine Backups on /dev/disk6s2 mounted

Oh wow, really?

Access the files.

user@picklerick> ls /Volumes/Time\ Machine\ Backups/Backups.backupdb/tunip/Latest/Macintosh\ HD/Users/user/Music Audio Music Apps GarageBand iTunes user@picklerick> rsync -av /Volumes/Time\ Machine\ Backups/Backups.backupdb/tunip/Latest/Macintosh\ HD/Users/user/Music /Volumes/disk2 ...

Guess how it felt to see that rsync output spill over the terminal window?

Very sweet.

My takeaways from this exercise:

When a filesystem doesn't mount, it's not necessarily the case that the files are not accessible. If there is a way to recover the partition map, you may be able to regain access to the filesystem. How easy this is would depend on many circumstances.

When copying large amounts of file data around, I should consider a means to verify the copy was really successful. A quick first idea that comes to mind is to cmopute and compare checksums; I'll have to look into the details on the next situation that requires this step.

Don't rely on a time machine backup set as your exclusive means to restore data. Lose access to that backup set for whatever reason, lose access to your data. I'd never given it much thought previously, but it seems network Time Machine set-ups are particularly vulnerable, since that .sparsebundle directory had better mount properly, else the entire backup set is inaccessible.

[bigbearlabs]

Keeping shell history contextually relevant

When working with a Unix command line, it’s common to interactively recall commands in the history, to avoid repeated typing with a frequently used command, or to correct or revise a command already issued.

In a typical shell session, I would cd to a directory that contains source files, then run a command that I would have already issued in that directory, e.g. build the project. Much less frequently, I would stitch together a command to e.g. ack the source for some keyword, then end up having to tweak that command a few times to get the result I want. Such commands are rarely useful outside the context of that project directory, so I thought it would be useful to have the shell’s interactive history scoped to the working directory.

Based on some suggestions on this thread, I put together a very small zsh plugin which saves the history in a file .history in the current directory, and have found it useful for my shell sessions. I have even started creating new directories for some sysadmin-flavoured shell sessions, such as setting up a new linux box, so I can capture a contextually focused history of commands issued either in 1 shell session, or in successive sessions.

I’ve also found other shell plugins after writing dir-scoped-history with very similar goals; a lot of them had substantially more functionality than my plugin, but browsing through the code of all the projects made me feel they were generally too much shell scripting code to bring into my .zshrc. Here they are for posterity.

https://github.com/jimhester/per-directory-history https://github.com/tymm/zsh-directory-history https://github.com/ericfreese/zsh-cwd-history

[bigbearlabs]

The Too Many Tabs problem (0)

Here’s a Product Hunt thread on the problem of ending up with too many browser tabs on one’s computer: I don't want to have 3,000 tabs open anymore. How do you all do it?

It’s a problem I had myself, that got me started in 2011 to work on a product to automate the organisation of digital artefacts, and got me to release the shell of what I had developed —the chassis of the car I originally wanted to make, after discovering the engine I needed was beyond my budget to build — as a separate app.

For better or for worse, I don’t think there’s another problem that I had obsessed over more in my life.

In fact, it’s still a chunky part of the new Mac app I’m working on. And since I had learned a few things about this problem since my initial fascination with it, I hope I can share findings and my personal conclusions (or interim summaries) in the future.

[bigbearlabs]

QuickCursor: use favourite editor features

QuickCursor is a nifty utility for macOS for people who work with text a lot. It lets you assign a hot key to invoke your favourite text editor with a text selection in, say, a web page, so you can use all the text editing tricks you’ve burned into your muscle memory. The save and close the editor window, and the text selection in the web page will be updated accordingly.

I have Command+Option+E assigned to Sublime Text 3, so in situations where I need to write something long, I will Command+A (select text), Command+Opton+E (edit in Sublime), then edit as necessary, then Command+S (save), Command+W (close) and voila, text editing mini-mission accomplished.

QuickCursor was discontinued as a product on the Mac App Store. The QuickCursor GitHub repo cites incompatibility with the MAS requirement for Sandboxing, but since the code has been open-sourced you can build your own copy.

If you, like me, purchased a copy on the MAS prior to it being discontinued, you can still use it by taking the following steps:

Go to System Preferences > Security & Privacy > Privacy > Accessibility, and add QuickCursor.app to the list to give it accessibility permissions.

Manually add the bundle ID of your text editor since the app is unable to find all compatible text editors on your system.

[bigbearlabs]

Fuck the epic

Ah, the liberating feeling I realise I had often forgotten when I tell myself to just fuck the epic — and get on with a short summary instead. Because the epic might never get written, but a something short, even if it may have a bunch of shortcomings, almost always feels tempting to finish and publish.

[bigbearlabs]

PR

Bloody hidden. Or obscured, maybe. That's where we are, at the moment. But come to think of it, it seems PR effort seems to be a common thread with all successful indies. Must lay more bricks here.

[bigbearlabs]

Choosing a keyboard for the iPad Pro

Since I decided to use the iPad pro in the living room, replacing my 3rd gen 'The new' iPad. It has to: - be living-room compatible, at least for writing, but ideally also for programming. - needs some way to accommodate the Pencil, since I will want to spend some time with both options to understand how usage will shake out. - can't be too heavy -- contrary to every other reviewer, I feel the iPad pro is alreay heavy without any case. More details to come as I get more info.

[bigbearlabs]

Seems pretty fatal if classification UI doesn't consult a user ontology of some kind.

How to best capture such a thing is yet another massive challenge. Previous considerations include mining information silos the user provides, eg personal folder schemes or a selection when the user signs up.

[bigbearlabs]

Enabling comments turned out to be a headache, but found a great theme. Now tweaking...

[bigbearlabs]

One unsolved challenge

Andreas Koller captures a concise example of how colour and its communicative feature has been inappropriately used — specifically for this case, ignored — by the designers. (I dare speculate, it was probably by the ‘designers’; i.e. the situation is a product of there being none.)

For a lot of you reading this right now, this is just another #fail; the mistake is so obvious that it practically shouts out at you, and you’d just chalk it up on your list of ‘another small and silly software design mistake’ encounters and move on. For a lot of others, it’s subtle perhaps, but once someone points it out, you see it, and you see it clearly.

But for a few of you, this case may be bewildering. “What’s the problem?”, you may be asking. I know this will be the reaction of a few people, no matter how incredible, because of the following true story.

I experienced a strange series of events on the topic of colour choice, and perhaps more specifically about incorrectly choosing red, on a project for an iOS institutional financial services app. A colleague, who shall not be named, changed the information-oriented app’s colour scheme from white-on-black — modelled from the Bloomberg app’s design circa 2010, as a starting point prior to there being anyone responsible for design — to red-on-black.

Yes, you read correctly. Red-on-black. So, in addition to the mistake of picking as the base text colour, the special colour red, which, in the context of textual communications is conventionally reserved for stuff requiring extra attention, typically something negative or requiring caution, another mistake was layered; degraded legibility due to a combination of colours that yields very poor contrast.

This, I thought, was a simple and obvious error; someone checked in a typo or something. Except it was not. The problem was that the poor guy who made the change, intentionally as a means to ‘improve design’, did not at all get that the resultant situation was undesirable. When asked why he chose to change the main type’s colour from white to red, he replied — with a whiff of pride! — that he had read from the corporate branding / design guidelines that the colours used for branding of the company was red, black and white. That’s why, he said, red on black makes for a better designed app. Now, he continued, our app looks less like the Bloomberg app.

After a few seconds of interpreting non-verbal signals made me conclude this wasn’t a practical joke, it dawned on me that I had just received my first ominous indication that the situation was nothing like my first assessment.

I attempted to explain how red text has a kind of special usage, which sets expectations of users who come across red text, which then makes the choice of employing it a bit more unusual than other colours. The colleague didn’t quite understand.

'But the guidelines say red is one of the three colours we can use!'

'Did it say you can use it for text?'

'No, but it's not going to say, you can use this colour for this and that for that…'

'Ok, but black and white are also in the guidelines. Why choose red? Red is a special colour.'

'White on black is like Bloomberg! We can't do that. Now our app is very unique.'

'……' (in head: unique it is, dear colleague!)

Realising that the explanation not making sense to him was a combination of my lack of articulation on design matters, and the very absence of that sensibility that made him to make the change in the first place without self-restraint, I then tried to explain that besides red being a ‘special’ colour, the legibility of the information in the app had drastically decreased.

'Really? I can read everything just fine!'

I was lost for words at this point.

So strong and confident was my colleague’s response that I was mildly relieved to receive confirmation I wasn’t going nuts here. In a team chat shortly after, another colleague in the US put it more bluntly: ‘What the f*** happened? This is ugly.’ And not long after, the colleague who made the change went into ‘defensive mode’.

What then happened was both a bit humorous and a bit tragic. On the team conference call, the guy suggested that we ‘vote’ for or against the change. He then cast the first vote — ‘I like this change. What about the others?’ Though I can’t read minds, there were enough non-verbal details at this point that made it clear to me that, when he suggested voting, he actually did not think the change would be voted against.

How many alarm bells are ringing on your radar of failing project signals right now?

To sum up: without the project having a designer of any kind — i.e. no graphic designer, no UI designer, most certainly no UX designer — the developer who felt that the app’s design should be improved over the quick-and-dirty copy of the Bloomberg app, concluded it would be a good idea to improve the design by making the text red, because red is the prominent colour in the corporate branding / design guidelines. Far from being able to recognise the problem with his idea, he was confident enough that merits of the change would speak for themselves, that he then made the change without discussing it with anyone.

As humans, we make mistakes all the time. And if they are all so obvious that they jump out, well, that would mean we’d all be aware of the mistakes we make; clearly, this is not the case. I can cite a lot of my own mistakes that, although they may seem so obvious now it’s almost embarrassing to think I’d made them, were not so obvious at the time. So in general, I find it’s difficult to presume that people notice things that I’d assume are ‘common sense’ kind of things; different versions of myself don’t even stand up to that assumption.

But in this case, the gravity of the situation was such that I really had to think. Here was a situation where the app we were working on had changed from ‘draft design’ status, to a ‘draft design’ status where you could barely read anything because the text was in red on a black background, because there was one person who thought it was a better design choice. I mean, red text! And the person who made the change did so thinking it would be an improvement of the design, and was completely unaware of the problem he created.

Thankfully, voting results reflected what I thought was the inevitable conclusion based on, well, what I thought was either ‘common sense’ or ‘intrinsic deductions close to the sensory level’, and the case closed. I then lobbied to get a designer on board, and I like to think that this small incident gave me some ammo to succeed on that.

But, what kept lingering in my mind was the absurdity of the series of events, and how relatively ineffective I was at dealing with it as a member of the team, as testified by the frustrations mounting inside of me.

App developer wants to improve design of the app.

He thinks a way to to do this is to apply colours mentioned in the corporate design guidelines.

He then chooses a colour from one of the colours for branding, but one which is not appropriate for body text, entirely unaware of the inappropriateness.

He then makes this change on his own initiative, resulting in an outcome that was received by the team in a seriously negative way.

Explaining why the choice is a bad one goes nowhere.

The team’s negative reaction to the change does not hint the guy in the right direction.

We end up with a vote-casting situation regarding the design error.

In particular, I found it remarkable that what I — and I’m pretty confident, many others — would deem such an obvious design error, was actually quite hard to communicate and get understood; the communication effort failed, which led us to the voting situation. It was puzzling; it didn’t seem to be a case of easy things being easy, and difficult things being difficult. Nonetheless, it had all happened before my very own eyes.

So, perhaps it’s the case that HSBC’s short message in red happened in a similar context.

I learned quite a few things by reflecting on this case but ultimately, it’s still on my ‘unsolved challenges’ list. because I was seriously dissatisfied with my effectiveness with which I could deal with the situation.

Sure, perhaps communication never succeeds unless the recipient is somehow ready to receive the message in some way. But I couldn’t help but think that this was the most poignant case of the feeling I have every now and then, when I think something’s not right, but I’m not able to explain why, in a way that’s clear and does justice to the proportions of the details; in a way that even a child can understand. It made me feel that I needed to up my game in communications.

I may have made some progress since then, but I’m still working on that one.

Meanwhile, I advise all software teams without designers to take more care before choosing to make something red, particularly with text. You may make your colleagues think about a past incident many times over.

[bigbearlabs]

A story of a forgetful bear watcher

Fancy a bug report in the form of a RubyMotion app featuring a bear and a cameraman? Of course you do!

https://gist.github.com/sohocoke/10333185

Sorry my git commit hooks are broken so the formatting is not the best.

[bigbearlabs]

Updated WebBuddy 2 alpha

All your web history, graphically visualised and grouped into stacks.

http://alpha.webbuddyapp.com

[bigbearlabs]

Hack on WebBuddy plug-ins

Happy new year everyone.

I’m pleased to release the early access code for WebBuddy plug-ins. Hack on the filtering, eval or find plug-ins for WebBuddy, or write your own plug-in to extend WebBuddy, in cutting-edge HTML5, (S)CSS and CoffeeScript.

Very sparse documentation at the moment. Code changes daily but pushes are rarer, due to clean-up not taking enough priority. You are encouraged to refactor!

GitHub repository: https://github.com/bigbearlabs/webbuddy-plugins

[bigbearlabs]

WebBuddy alpha campaign now starting.

After a few maintenance releases on the [store](http://appstore.com/mac/webbuddy), we're open for [alpha testers for the next major release](http://alpha.webbuddyapp.com).

[bigbearlabs]

WebBuddy 1.1.9 released

Get WebBuddy 1.1.9!

Ported from MacRuby to RubyMotion, this version features compatibility with OS X Mavericks, and should launch very fast. Like, orders of magnitude faster than 1.1.8.

See if you can find the easter egg.

As always, your feedback is most welcome!

[bigbearlabs]

OneHour Advanced settings

2-finger drag for longer than 300 pixels (points really) will reveal an advanced panel that I use to explore what would be the best parameters for e.g. visual aspects of OneHour. I doubt many people will find this by themselves nor need it, but it may be fun for those of you who want to play around with the timer's looks.

If you'd like to discuss things you want to play with here, please visit the support forum.

[bigbearlabs]

Finally resolving the recent typography fiasco as we're transitioning to Tumblr.