Maybe I'm doing something wildly wrong but it appears to me that zlib produces invalid gzip headers when the data is above some size ("Hello World\n" compresses fine, with the correct header).
This code:
'use strict'const fs = require('fs')const zlib = require('zlib')const rs = fs.createReadStream(process.argv[2], { highWaterMark: 64 * 1024 })const ws = fs.createWriteStream(process.argv[2] + '.gz', { highWaterMark: 64 * 1024 })const ts = zlib.createGzip()rs.on('error', error => console.error('RS ERROR!', error))ts.on('error', error => console.error('PS ERROR!', error))ws.on('error', error => console.error('WS ERROR!', error))rs.pipe(ts).pipe(ws)Produces a .gz file with a binary header of
1f ef bf bd 08 00 00 00 00 00 00 13when I run the script on a ~88kb test file (a bunch of "Lorem Ipsum"), which is definitely not a valid gzip header. Running gzip -t says not in gzip format, and changing the script to use zlib.createGunzip() (so as to attempt an un-gzip) results in this error:
Error: incorrect header checkat Zlib.zlibOnError [as onerror] (zlib.js:142:17) errno: -3, code: 'Z_DATA_ERROR'According to both the RFC and this simple guide the binary header should look something like
1f 8b 08 00 00 00 00 00 00 13where 1f 8b is the magic bytes as expected. Compressing something small, such as "Hello World\n", produces the header correctly, and gzip -t is happy with it.
(Related gist: https://gist.github.com/Fishrock123/d48aeacec906b79d9cb426ce846cde2f)