Comparing UNIX Password Encoding: MD5, SHA, and Legacy Crypt FormatsPassword storage on UNIX-like systems has evolved alongside hashing algorithms and system requirements. What began as a simple, compact algorithm suitable for constrained systems has grown into a landscape of multiple formats — each with trade-offs in security, compatibility, and performance. This article covers legacy crypt formats, MD5-based schemes, and SHA-based schemes commonly encountered on modern UNIX-like systems, explains how they work, compares their strengths and weaknesses, and gives guidance for choosing and migrating between formats.
Why password encoding matters
Storing raw passwords is never acceptable. Instead, systems store one-way encodings (hashes) so that the original password cannot be trivially recovered even if the hash is leaked. A secure password encoding:
- Is computationally expensive to reverse via brute force or dictionary attacks.
- Uses a per-password salt to prevent precomputed attacks (rainbow tables).
- Is resilient to collisions and other cryptographic weaknesses.
UNIX password storage historically used the crypt(3) interface and a family of algorithms often referred to collectively as “crypt” formats. Over time, new encodings (MD5, SHA variants, bcrypt, scrypt, Argon2, etc.) were introduced. This article focuses on MD5, SHA-based encodings (as used by variants of crypt), and legacy DES-based crypt.
Legacy crypt: the original DES-based scheme
The original UNIX crypt algorithm (often called “DES crypt”) originates from the 1970s and was implemented in the crypt(3) library function. It was designed to produce short, fixed-length password hashes that could be stored easily in /etc/passwd.
How it works (high level)
- Based on a modified Data Encryption Standard (DES).
- Takes a 2-character salt and a password truncated to 8 characters.
- Produces a 13-character encoded result (salt + 11 chars of hash output in a custom base64-like alphabet).
- Salt modifies the DES key schedule to produce different outputs for the same password.
Limitations
- Extremely small salt (2 chars) and limited password length (8 chars) make it weak by modern standards.
- DES itself has a tiny keyspace compared to modern expectations and is computationally fast — convenient for attackers.
- No support for iterated hashing (work factor) to increase computational expense.
Compatibility and legacy
- Still present on very old systems but considered insecure.
- Some systems emulate or allow DES crypt for compatibility, but it is discouraged for new accounts.
MD5-based crypt
MD5-based crypt (often shown in password files with the prefix \(1\)) was proposed as an improvement over DES crypt on systems where DES or its licensing was problematic or where improved hashing was desired.
How it works (high level)
- Uses the MD5 digest algorithm to produce a 128-bit hash of a composite input: password, salt, and repeated patterns per the original algorithm specification.
- Includes a variable-length salt (commonly up to 8 characters).
- Produces a string typically prefixed with \(1\) followed by the salt and a base64-encoded hash.
Benefits
- Supports longer passwords and longer salts than DES crypt.
- MD5 is faster and produces a larger hash than DES crypt.
- Widely supported in glibc and many UNIX implementations for backward compatibility.
Limitations
- MD5 is cryptographically broken for collision resistance; while collisions impact certificates and signatures more than password hashing, MD5’s speed makes brute-force attacks easier compared to slower, memory- or CPU-intensive schemes.
- No configurable work factor (iteration count) in the original MD5-crypt design.
- Considered insufficient for protecting high-value accounts today.
Practical considerations
- MD5-crypt is still used in many legacy environments.
- If migrating from MD5-crypt, ensure users reset passwords to generate a stronger scheme, rather than attempting to transform hashes directly (impossible without the plaintext).
SHA-based crypt variants (SHA-crypt family)
To improve security over MD5 and legacy crypt, several SHA-based crypt formats were introduced. These are typically identified by strings like \(5\) (SHA-256-crypt) and \(6\) (SHA-512-crypt) in password files.
How they work (high level)
- Use SHA-256 or SHA-512 as the underlying digest.
- Include a salt and support an iteration count (work factor) to increase computational cost.
- Produce strings prefixed with \(5\) (SHA-256) or \(6\) (SHA-512), the salt, an optional rounds parameter, and the encoded output.
Key features
- Stronger hash functions (SHA-256 and SHA-512) with larger internal state and outputs.
- Configurable rounds parameter (commonly something like 5000 by default in many implementations; can be increased to tens or hundreds of thousands).
- Longer salts (typically up to 16 or more characters) and longer output encodings.
Security trade-offs
- SHA-256 and SHA-512 are currently considered cryptographically secure as hash functions (collision and preimage resistance) for password hashing use-cases.
- They are still relatively fast and CPU-bound; increasing the rounds raises computational cost linearly but provides less defense against attackers using GPU/ASIC optimized SHA implementations than memory-hard functions like bcrypt/scrypt/Argon2.
- SHA-crypt is widely supported and a pragmatic upgrade over MD5-crypt in many system contexts.
Example format
- \(6\)rounds=50000\(salt\)hash (rounds may be omitted to use system defaults)
Comparison: MD5, SHA-crypt, and legacy DES crypt
| Feature | Legacy DES crypt | MD5-crypt (\(1\)) | SHA-crypt (\(5\)/\(6\)) |
|---|---|---|---|
| Salt length | 2 chars | up to 8 chars (varies) | longer (commonly 16+) |
| Password length handling | truncated to 8 | supports longer | supports longer |
| Underlying primitive | DES-derived | MD5 | SHA-256 / SHA-512 |
| Work factor (configurable) | No | No | Yes (rounds) |
| Speed | Fast (weak) | Fast (broken primitive) | Fast but tunable via rounds |
| Resistance to modern attacks | Poor | Weak | Reasonable, but not memory-hard |
| Typical format prefix | none / traditional | \(1\) | \(5\) / \(6\) |
When to use which format
- For any new deployment: avoid DES crypt and MD5-crypt. Prefer SHA-crypt (SHA-512, \(6\)) only if compatibility with system utilities and ID/password storage formats is required and if you configure a high rounds count.
- For high-security environments: prefer memory-hard algorithms (bcrypt, scrypt, Argon2). These are not always available in the classic /etc/shadow format, but many modern PAM modules, login systems, and authentication backends support them.
- For legacy compatibility: MD5-crypt may be necessary to interoperate with older systems; plan a migration path to SHA-crypt or better.
- For constrained embedded systems: SHA-crypt with tuned rounds may be a pragmatic compromise if bcrypt/Argon2 are unavailable.
Migration and practical steps
- Inventory: identify which accounts use which formats (check /etc/shadow prefixes).
- Policy: choose a target scheme (e.g., SHA-512 with rounds=100000) and set system defaults (e.g., via /etc/login.defs or glibc settings).
- Re-hash during password change: you cannot directly convert old hashes to new ones; force or encourage users to change passwords so the system will store the new format.
- Backwards compatibility: keep support for old hashes temporarily, but require re-authentication to upgrade.
- Rate limiting and MFA: reduce the harm from any leaked hashes by adding multi-factor authentication and throttling login attempts.
- Monitor and iterate: periodically increase rounds as attacker compute gets cheaper.
Example commands and configuration notes
- On Linux with glibc, select SHA-512 as default by setting ENCRYPT_METHOD in /etc/login.defs or using passwd/libxcrypt settings depending on distribution.
- To force a new hash for a user, have the user change their password with passwd or use chpasswd in combination with setting the desired crypt method on the system.
- Check /etc/shadow entries — prefixes like \(1\), \(5\), \(6\) indicate the hash type.
Conclusion
Legacy DES-based crypt is obsolete and unsafe. MD5-crypt improved compatibility and removed some limitations but is no longer recommended due to MD5’s weaknesses and lack of a configurable work factor. SHA-crypt (SHA-⁄512-crypt) offers a practical, widely supported improvement with configurable rounds and larger salts, making it a reasonable default for traditional UNIX password storage — but it remains CPU-bound, so for the highest protection consider memory-hard algorithms (bcrypt/scrypt/Argon2) and additional defenses such as MFA and rate limiting.
Leave a Reply