Recovering a file git never committed

A few days ago a colleague came to me with a file that was gone. It was part of a product feature that had already been built twice, each time taking a couple of months, and nobody wanted to build it a third time.

The product was built with ASP.NET Web Forms and kept in git. In Web Forms each page is mainly a pair of files: an .aspx file with the page’s markup, and a .cs “code-behind” file with the C# that runs on the server. Visual Studio usually adds a third, generated .designer.cs file. The missing file was ImageCropping.aspx.cs, the code-behind for an image-cropping page in the product’s media upload section.

My colleague had been working on it for a while, and it had never been committed. git log had nothing to show for it, and neither did the reflog.

We got it back anyway, because at some point they had run git add on it, and git keeps what you stage.

In short: git add writes a copy of the file into .git/objects the moment you stage it. A git reset --hard, for example, removes a newly added file from your working tree, but the copy stays in the object database, unreferenced, until garbage collection removes it. git fsck --unreachable lists these copies. Search them for a string you know was in the file, then copy the right one out of .git/lost-found.

Git has always made me curious

Every time I’ve tried to learn git properly, or to build a piece of it myself, I’ve come away with more questions than I started with. That curiosity is the reason I knew where to look when my colleague’s file disappeared.

I’ve had a GitHub account since 2013, when my first repositories were coursework from online courses. For a long time after that, git was just how things got online. This blog started as a Jekyll site on GitHub Pages. In 2017 I rebuilt it with Hugo and the cocoa-eh theme, and in 2018 moved to the jane theme. From the Hugo rebuild on, it ran on two git repositories: one for the content, and one for the HTML that Hugo generated from it, which GitHub Pages served. Every post meant building the site and pushing it.

That setup kept making me ask how far I could push git. In 2018 I wanted to post from my phone, so I installed git on it (it was rooted). That let me commit new posts, but not build the site, which still needed Hugo on a computer. I thought about a server that would rebuild and push the site whenever a commit landed, then found Forestry, a web-based CMS that let me edit posts from a browser. I wrote that up in a static blog with a web-based CMS.

In 2019 I set out to learn git properly. That summer, on my third try at the dgplug summer training, a session took us through git basics and commit messages, and I wrote up what I read about git log. Knowing the commands wasn’t enough; I wanted to know how they worked. So I tried to write my own git by following the Write Yourself a Git tutorial. My first attempt stalled, and I started over in a short series of posts, building it step by step in Python. I got as far as init, which creates the .git folder and the .git/objects directory inside it.

That directory is where the file came back from. Everything git stores goes into .git/objects, compressed and named by the hash of its contents, and git rarely deletes anything from it the moment you lose it.

What git keeps, and what it doesn’t

Here is the whole recovery in one loop: what git add writes, what a hard reset removes, and how git fsck gets the file back.

Before running anything, work out which case you’re in. They need different tools.

If a file was only ever saved to disk, git never had a copy of it, but your editor might. VS Code’s Timeline view and JetBrains’ Local History both keep their own snapshots of saved files. Not every editor does, and the ones that do keep them for a limited time, so find out what yours keeps before you need it.

Git also cleans up after a while. Garbage collection runs on its own now and then, after commits and fetches, and when it runs it removes unreachable loose objects older than two weeks by default. Reflog entries last 90 days, or 30 days for commits that are no longer reachable. So don’t run git gc while you’re recovering, and start soon.

The commands, in bash and PowerShell

Every block below comes in two versions:

• bash: Git Bash on Windows, and the terminal on macOS and Linux. The commands also work in zsh, the default shell on macOS.
• PowerShell: Windows PowerShell 5.1, and PowerShell 7 (pwsh) on any OS.

Two PowerShell problems to know about first.

1. Quote anything with braces. PowerShell reads {1} in HEAD@{1} as a script block, so git receives HEAD@ and stops with fatal: ambiguous argument 'HEAD@'. Write 'HEAD@{1}' in single quotes. Single quotes work in bash and zsh too, so it’s easiest to always use them.
2. Don’t restore files with >. In Windows PowerShell 5.1, git cat-file -p <hash> > file re-encodes the output: by default the file comes out as UTF-16, other settings add a byte-order mark, and binary files are corrupted either way. PowerShell 7.4 and later keep the bytes intact. Copy from .git/lost-found instead, as shown below. Git writes those files itself, so you get exactly the bytes it stored.

1. Make sure it was never committed

Search every branch and the reflog for the path:

1

git log --all --reflog --oneline -- "path/to/ImageCropping.aspx.cs"

This one is the same in PowerShell. In our case it printed nothing: the file had never been in a commit. If it prints commits, skip to section 6 or section 7.

2. Find the staged copies

git fsck --unreachable --no-reflogs lists every object that nothing points to. --no-reflogs matters here: without it, anything the reflog still mentions counts as reachable and is left out. Filter for blobs, which hold file contents, then keep the ones that contain a string you know was in your file. A class name works well.

bash:

1
2
3

git fsck --unreachable --no-reflogs 2>/dev/null | awk '$2=="blob"{print $3}' | while read b; do
  git cat-file -p "$b" | grep -qi "ImageCropping" && echo "$b $(git cat-file -s "$b") bytes"
done

PowerShell:

1
2
3
4
5
6

git fsck --unreachable --no-reflogs 2>$null | ForEach-Object {
  if ($_ -match '^unreachable blob (\w+)') {
    $b = $Matches[1]
    if ((git cat-file -p $b) -match 'ImageCropping') { "$b  $(git cat-file -s $b) bytes" }
  }
}

We got seven matches:

1
2
3
4
5
6
7

3f9a2c1d7e4b8a6f0c5d2e9b1a7f4c3e8d6b0a21  8225 bytes
b81e04c7a39f2d6e5c0b7a1f8e3d9c2b4a6f0e57  2883072 bytes
9c4e7a2b1f8d3c6e0a5b9f2d7e1c4a8b3f6d0e92  14003 bytes
e27d5f0a8c1b4e9d3f6a2c7b0e5d8f1a4c9b3e66  1515520 bytes
5d1f8b3e6a0c9d2f7b4e1a8c5f3d0b6e9a2c7f14  16463 bytes
a6c3e9f1d4b7a0e2c8f5d1b9e6a3c0f7d2b8e4f9  318566 bytes
7e0b4d9a2f6c1e8b5d3a0f7c4e9b2d6a1f8c5e30  2316 bytes

There are seven because a class name doesn’t appear in just one file. The three large blobs, 300 KB to 2.8 MB, are far too big to be one page’s source; compiled output such as a DLL contains class names as strings too. A source file like this one is a few kilobytes, which left four candidates.

Several candidates is normal anyway. Every git add stores a new blob, so if you staged a file three times while working on it, you’ll find three versions.

3. Work out which file each candidate is

A Web Forms page can leave up to three files behind, and all of them mention ImageCropping:

• ImageCropping.aspx, the markup, which starts with <%@ Page.
• ImageCropping.aspx.cs, the code-behind, which contains partial class ImageCropping.
• ImageCropping.aspx.designer.cs, which Visual Studio generates. It also contains partial class ImageCropping, but it starts with an <auto-generated> comment.

Print the start of each candidate to see which one it is:

bash:

1
2
3
4

for b in 3f9a2c1 9c4e7a2 5d1f8b3 7e0b4d9; do
  echo "===== $b"
  git cat-file -p "$b" | head -n 12
done

PowerShell:

1
2
3
4

"3f9a2c1", "9c4e7a2", "5d1f8b3", "7e0b4d9" | ForEach-Object {
  "===== $_"
  git cat-file -p $_ | Select-Object -First 12
}

Git accepts a short hash wherever it accepts a full one, as long as it’s unique.

4. Save every candidate and compare them

Don’t pick one yet. Save all of them to a folder outside the repository, each named by its hash and the kind of file it is, and list when each was staged.

The staging time comes from the blob itself. Until git packs it, a blob is a file under .git/objects, named by its hash: the first two characters are the folder and the rest are the filename. Its modified time is when that content was last staged, so the newest version of each file is usually the one you want.

bash:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

mkdir -p ../recovered
for b in 3f9a2c1 9c4e7a2 5d1f8b3 7e0b4d9; do
  ext=unknown
  git cat-file -p "$b" | grep -q "<%@ Page" && ext=aspx
  git cat-file -p "$b" | grep -q "partial class ImageCropping" && ext=aspx.cs
  git cat-file -p "$b" | grep -q "<auto-generated>" && ext=aspx.designer.cs
  git cat-file -p "$b" > "../recovered/$b.$ext"
  full=$(git rev-parse "$b")
  echo "$b  $ext  $(git cat-file -s "$b") bytes  $(ls -l ".git/objects/${full:0:2}/${full:2}" | awk '{print $6, $7, $8}')"
done

PowerShell (copying out of .git/lost-found, for the reason above):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

git fsck --lost-found | Out-Null
New-Item -ItemType Directory -Force ../recovered | Out-Null
"3f9a2c1", "9c4e7a2", "5d1f8b3", "7e0b4d9" | ForEach-Object {
  $full = git rev-parse $_
  $c = git cat-file -p $_
  $ext = "unknown"
  if ($c -match '<%@ Page') { $ext = "aspx" }
  if ($c -match 'partial class ImageCropping') { $ext = "aspx.cs" }
  if ($c -match '<auto-generated>') { $ext = "aspx.designer.cs" }
  Copy-Item ".git/lost-found/other/$full" "../recovered/$_.$ext"
  $obj = Get-Item ".git/objects/$($full.Substring(0,2))/$($full.Substring(2))"
  "$_  $ext  $(git cat-file -s $_) bytes  $($obj.LastWriteTime)"
}

git fsck --lost-found writes every dangling blob to .git/lost-found/other/, named by its full hash, and git writes those files itself, so the copies are byte for byte what it stored. If git has already packed a blob, it has no file of its own: it won’t be in lost-found and has no staging time to show. You can still save it with git cat-file -p <hash> > file in bash.

If two candidates are versions of the same file, compare them. This works the same in both shells, and doesn’t need either file to be in a repository:

1

git diff --no-index ../recovered/9c4e7a2.aspx.cs ../recovered/5d1f8b3.aspx.cs

5. Put each file back

Copy the version you chose of each file to where it belongs.

bash:

1
2
3

cp ../recovered/<hash>.aspx.cs          path/to/ImageCropping.aspx.cs
cp ../recovered/<hash>.aspx             path/to/ImageCropping.aspx
cp ../recovered/<hash>.aspx.designer.cs path/to/ImageCropping.aspx.designer.cs

PowerShell:

1
2
3

Copy-Item ../recovered/<hash>.aspx.cs          path/to/ImageCropping.aspx.cs
Copy-Item ../recovered/<hash>.aspx             path/to/ImageCropping.aspx
Copy-Item ../recovered/<hash>.aspx.designer.cs path/to/ImageCropping.aspx.designer.cs

Restore only the files that are actually missing. If you didn’t find a designer file, Visual Studio regenerates it from the markup when you open and save the .aspx.

Build the project, open the page, then commit:

1
2
3

git status
git add path/to/ImageCropping.aspx path/to/ImageCropping.aspx.cs path/to/ImageCropping.aspx.designer.cs
git commit -m "Restore ImageCropping page from unreachable blobs"

Commit straight away. A recovered file you haven’t committed can be lost the same way again. Once it’s committed, delete ../recovered and .git/lost-found.

One thing to check after restoring: if core.autocrlf is on, as it often is on Windows, git stored the files with LF line endings, and that is how they come back. Most editors don’t mind. If you want CRLF back, after committing, delete the restored files and check out just those files again, for example git checkout -- path/to/ImageCropping.aspx.cs, which writes them out with CRLF. Name the files: checking out a whole folder also throws away any other uncommitted changes in it.

6. A file deleted in a commit

This is the easy case, and the one people usually mean by “recovering a deleted file”. Find the commit that deleted it, then restore the file from the commit before that one. The ~1 means “the parent of this commit”.

bash:

1
2

sha=$(git log --all --diff-filter=D --format=%h -1 -- "path/to/file")
git restore --source="$sha~1" -- "path/to/file"

PowerShell:

1
2

$sha = git log --all --diff-filter=D --format=%h -1 -- "path/to/file"
git restore --source="$sha~1" -- "path/to/file"

git restore needs git 2.23 or newer. On an older git, use git checkout "$sha~1" -- "path/to/file". I use ~1 rather than ^ on purpose: in cmd.exe ^ is an escape character, and in zsh with extended globbing it is a pattern character.

7. Commits lost to a reset

git reflog lists every position HEAD has been at, including commits no branch points to any more. Find the commit and put a branch on it. Making a branch is safer than git reset --hard, which throws away any uncommitted changes you have now.

1
2

git reflog -20
git branch rescue 'HEAD@{1}'

The command is the same in PowerShell, with the quotes, which bash ignores. Replace HEAD@{1} with whichever entry has your work, or with its hash.

8. A dropped stash

A stash is a commit. After git stash drop or git stash clear it becomes a dangling commit, and git fsck can find it. The entries labelled WIP on <branch> are the stashes.

bash:

1
2
3
4

git fsck --no-reflogs | grep "dangling commit" | awk '{print $3}' | while read c; do
  git log -1 --format="%H %ci %s" "$c"
done
git stash apply <full hash>

PowerShell:

1
2
3
4
5

git fsck --no-reflogs | Select-String "dangling commit" | ForEach-Object {
  $c = $_.ToString().Split(" ")[2]
  git log -1 --format="%H %ci %s" $c
}
git stash apply <full hash>

This lists commits lost to a reset as well. Those will show their own commit messages, not WIP on.

Use the full hash with git stash apply. It reads an argument made only of digits as a stash number, so a short hash like 3298776 turns into stash@{3298776} and fails.

How to avoid needing this

• Commit early, on a branch. A commit called wip costs nothing, and you can squash it before anyone sees it. Every recovery above is easier when there’s a commit.
• Use git reset --keep instead of --hard. It keeps your uncommitted changes, and stops instead of overwriting them.
• Use git stash -u before risky operations. Without -u, untracked files aren’t stashed, and a new file you haven’t run git add on is untracked.
• Read git clean -n before running git clean -f. clean deletes untracked files, and git keeps no copy of those.
• Know the editor you work in. Find out whether it keeps a local history of saved files, and where. Also find out what its git buttons do: a “Discard changes” or “Undo changes” button runs git commands for you, and on a file git doesn’t track yet, it can delete the file outright.

My colleague had given up on that file. It was still in .git/objects, and a search for its class name found it.