I come from a Windows Exe background, and I make regular saves of my source code and of the EXEs the source code generated. This makes it easy for me to go back and find the source of a bug I may have missed in the past. In ASP.NET, apart from the source code, what should I save? The .DLL under bin? Other things?
You should be fine with just the source code. The .dll's get recompiled anyways when you build/rebuild a solution.
Ok... But I've found in Windows EXE programming that even if the source files are the same, rebuilding does not always give the same exe, maybe I'm on a different machine, maybe there have been patches to the compiler in the meantime. Is that not the case with C# and ASP.NET?
It's a bit different. I'll be general but as long as the build system and the target system have similar configurations, the old .dll hell issue is almost non-existent. For example, if you develop a web application using only features found in the .NET Framework 4.5 that's installed on your computer, build the solution and then deploy to a web server, as long as the same version of the .NET Framework is available on the target system, it should run. The biggest challenge now is dealing with package-level configuration management.