This commit is contained in:
Your Name
2026-08-11 17:41:36 +08:00
parent cfe4c82c90
commit 03fe4ddf9d
18771 changed files with 3617239 additions and 0 deletions
+22
View File
@@ -0,0 +1,22 @@
Copyright (c) 2015 Vitaly Puzrin.
Permission is hereby granted, free of charge, to any person
obtaining a copy of this software and associated documentation
files (the "Software"), to deal in the Software without
restriction, including without limitation the rights to use,
copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the
Software is furnished to do so, subject to the following
conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.
+146
View File
@@ -0,0 +1,146 @@
probe-image-size
================
[![CI](https://github.com/nodeca/probe-image-size/actions/workflows/ci.yml/badge.svg)](https://github.com/nodeca/probe-image-size/actions)
[![NPM version](https://img.shields.io/npm/v/probe-image-size.svg?style=flat)](https://www.npmjs.org/package/probe-image-size)
[![Coverage Status](https://coveralls.io/repos/github/nodeca/probe-image-size/badge.svg?branch=master)](https://coveralls.io/github/nodeca/probe-image-size?branch=master)
> Get image size without full download. Supported image types:
> JPG, GIF, PNG, WebP, BMP, TIFF, SVG, PSD, ICO, AVIF, HEIC, HEIF.
Key features:
- small size, no heavy dependencies
- works with remote and local data
- effective with big images (speed/memory), download minimal data from remotes
- extracts orientation value when available
- easy to browserify (splitted to components)
Install
-------
```bash
npm install probe-image-size
```
Example
-------
```js
const probe = require('probe-image-size');
// Get by URL
let result = await probe('http://example.com/image.jpg');
console.log(result); // =>
/*
{
width: xx,
height: yy,
type: 'jpg',
mime: 'image/jpeg',
wUnits: 'px',
hUnits: 'px',
url: 'http://example.com/image.jpg'
}
*/
// By URL with options
let result = await probe('http://example.com/image.jpg', { rejectUnauthorized: false });
console.log(result);
// From the stream
let result = await probe(require('fs').createReadStream('image.jpg'));
console.log(result);
// From a Buffer (sync)
let data = require('fs').readFileSync('image.jpg');
console.log(probe.sync(data));
```
API
---
Note:
- You can access/browserify `stream.js` / `http.js` / `sync.js` directly.
- If you don't like `http.js` dependencies, you can create your own wrapper
for `stream.js`.
### probe(src [, options|keepOpen]) -> Promise
- `src` can be of this types:
- _String_ - URL to fetch
- _Stream_ - readable stream
- `options` - HTTP only. See [`needle` documentation](https://github.com/tomas/needle#request-options), and customized [defaults](https://github.com/nodeca/probe-image-size/blob/master/http.js#L13).
- `keepOpen` (Boolean) - stream only. Keep stream open after parser finishes
(input stream will be closed by default)
`result` (Promise) contains:
```js
{
width: XX,
height: YY,
length: ZZ, // byte length of the file (if available, HTTP only)
type: ..., // image 'type' (usual file name extention)
mime: ..., // mime type
wUnits: 'px', // width units type ('px' by default, can be different for SVG)
hUnits: 'px', // height units type ('px' by default, can be different for SVG)
url: ..., // HTTP only, last url for the image in chain of redirects
// (if no redirects, same as src)
// optional, image orientation (from Exif), number from 1 to 8;
// you may wish to swap width and height if orientation is >= 5
orientation: X,
// optional, full list of sizes for ICO (always) and AVIF (if multiple images)
variants: [ { width, height }, ... ] | undefined
}
```
Width and height in the output object represent image size *before* any transformations
(orientation, cropping) are applied. Orientation is returned separately, which you may
wish to apply afterwards depending on browser support (browsers
[only support JPEG](https://zpl.fi/exif-orientation-in-different-formats/) orientation for now).
See [known issues](known_issues.md) for details.
> [!WARNING]
>
> For untrusted sources, returned `width` and `height` should not be treated
> as trusted values. You may need to additionally check that they are within
> ranges acceptable for your application.
Returned errors can be extended with 2 fields:
- `code` - equals to `ECONTENT` if the library failed to parse the file;
- `status` - equals to a HTTP status code if it receives a non-200 response.
### probe.sync(src) -> result|null
Sync version can eat arrays, typed arrays and buffers. On success it returns
the same result as async version. On fail it returns null.
> [!NOTE]
>
> Formats like JPEG & TIFF can store size anywhere (far from the head).
> That usually does not happens, but if you need guarantees - always provide
> full file content to sync methods. We strongly recommend to use async version
> as memory-friendly.
Similar projects
----------------
- [image-size](https://github.com/netroy/image-size)
Support probe-image-size
------------------------
You can support this project via [Tidelift subscription](https://tidelift.com/subscription/pkg/npm-probe-image-size?utm_source=npm-probe-image-size&utm_medium=referral&utm_campaign=readme).
+100
View File
@@ -0,0 +1,100 @@
'use strict'
var ProbeError = require('./lib/common').ProbeError
var needle = require('needle')
var lodashMerge = require('lodash.merge')
var pkg = require('./package.json')
var probeStream = require('./stream')
var URL = require('url').URL
var defaultAgent = pkg.name + '/' + pkg.version + '(+https://github.com/nodeca/probe-image-size)'
var defaults = {
open_timeout: 10000,
response_timeout: 60000,
read_timeout: 60000,
follow_max: 10,
parse_response: false,
// Use to ignore bad certificates.
// rejectUnauthorized: false,
headers: {
'User-Agent': defaultAgent
}
}
module.exports = function probeHttp (src, options) {
return new Promise(function (resolve, reject) {
var stream
var len
var finalUrl = src
var settled = false
function abortRequest () {
if (stream && stream.request) stream.request.abort()
}
function settle (resolveResult, rejectError) {
if (settled) return
settled = true
if (rejectError) {
reject(rejectError)
abortRequest()
return
}
resolve(resolveResult)
abortRequest()
}
try {
var needleOptions = lodashMerge({}, defaults, options)
stream = needle.get(src, needleOptions)
} catch (err) {
reject(err)
return
}
stream.on('redirect', function (location) {
try {
finalUrl = new URL(location, finalUrl).href
} catch (err) {
settle(null, err)
}
})
stream.on('header', function (statusCode, headers) {
if (statusCode !== 200) {
settle(null, new ProbeError('bad status code: ' + statusCode, null, statusCode))
return
}
len = headers['content-length']
})
stream.on('err', function (err) {
settle(null, err)
})
stream.on('done', function (err) {
if (err) settle(null, err)
})
probeStream(stream, true)
.then(function (result) {
if (len && len.match(/^\d+$/)) result.length = +len
result.url = finalUrl
settle(result)
})
.catch(function (err) {
settle(null, err)
})
})
}
module.exports.parsers = require('./lib/parsers_stream')
+21
View File
@@ -0,0 +1,21 @@
'use strict'
var probeStream = require('./stream')
var probeHttp = require('./http')
module.exports = function get_image_size (src, options) {
if (typeof src.on === 'function' && typeof src.emit === 'function') {
// looks like an EventEmitter, treating it as a stream
return probeStream(src, options)
}
// HTTP (not stream)
return probeHttp(src, options || {})
}
module.exports.parsers = require('./lib/parsers_stream')
module.exports.sync = require('./sync')
module.exports.Error = require('./lib/common').ProbeError
+110
View File
@@ -0,0 +1,110 @@
'use strict'
var Transform = require('stream').Transform
var streamParser = require('stream-parser')
function ParserStream () {
Transform.call(this, { readableObjectMode: true })
}
// Inherit from Transform
ParserStream.prototype = Object.create(Transform.prototype)
ParserStream.prototype.constructor = ParserStream
streamParser(ParserStream.prototype)
exports.ParserStream = ParserStream
exports.sliceEq = function (src, start, dest) {
for (var i = start, j = 0; j < dest.length;) {
if (src[i++] !== dest[j++]) return false
}
return true
}
exports.str2arr = function (str, format) {
var arr = []
var i = 0
if (format && format === 'hex') {
while (i < str.length) {
arr.push(parseInt(str.slice(i, i + 2), 16))
i += 2
}
} else {
for (; i < str.length; i++) {
arr.push(str.charCodeAt(i) & 0xFF)
}
}
return arr
}
exports.readUInt16LE = function (data, offset) {
return data[offset] | (data[offset + 1] << 8)
}
exports.readUInt16BE = function (data, offset) {
return (data[offset] << 8) | data[offset + 1]
}
exports.readInt16LE = function (data, offset) {
return (exports.readUInt16LE(data, offset) << 16) >> 16
}
exports.readInt16BE = function (data, offset) {
return (exports.readUInt16BE(data, offset) << 16) >> 16
}
exports.readUInt32LE = function (data, offset) {
return (data[offset] |
(data[offset + 1] << 8) |
(data[offset + 2] << 16)) +
data[offset + 3] * 0x1000000
}
exports.readUInt32BE = function (data, offset) {
return data[offset] * 0x1000000 +
((data[offset + 1] << 16) |
(data[offset + 2] << 8) |
data[offset + 3])
}
exports.readInt32LE = function (data, offset) {
return exports.readUInt32LE(data, offset) | 0
}
exports.readInt32BE = function (data, offset) {
return exports.readUInt32BE(data, offset) | 0
}
function ProbeError (message, code, statusCode) {
Error.call(this)
// Include stack trace in error object
if (Error.captureStackTrace) {
// Chrome and NodeJS
Error.captureStackTrace(this, this.constructor)
} else {
// FF, IE 10+ and Safari 6+. Fallback for others
this.stack = (new Error()).stack || ''
}
this.name = this.constructor.name
this.message = message
if (code) this.code = code
if (statusCode) this.statusCode = statusCode
}
// Inherit from Error
ProbeError.prototype = Object.create(Error.prototype)
ProbeError.prototype.constructor = ProbeError
exports.ProbeError = ProbeError
+266
View File
@@ -0,0 +1,266 @@
'use strict'
//
// Helpers
//
function error (message, code) {
var err = new Error(message)
err.code = code
return err
}
function utf8_decode (str) {
try {
return decodeURIComponent(escape(str))
} catch (_) {
return str
}
}
//
// Exif parser
//
// Input:
// - jpeg_bin: Uint8Array - jpeg file
// - exif_start: Number - start of TIFF header (after Exif\0\0)
// - exif_end: Number - end of Exif segment
// - on_entry: Number - callback
//
function ExifParser (jpeg_bin, exif_start, exif_end) {
// Uint8Array, exif without signature (which isn't included in offsets)
this.input = jpeg_bin.subarray(exif_start, exif_end)
// offset correction for `on_entry` callback
this.start = exif_start
// Check TIFF header (includes byte alignment and first IFD offset)
var sig = String.fromCharCode.apply(null, this.input.subarray(0, 4))
if (sig !== 'II\x2A\0' && sig !== 'MM\0\x2A') {
throw error('invalid TIFF signature', 'EBADDATA')
}
// true if motorola (big endian) byte alignment, false if intel
this.big_endian = sig[0] === 'M'
}
ExifParser.prototype.each = function (on_entry) {
// allow premature exit
this.aborted = false
var offset = this.read_uint32(4)
this.ifds_to_read = [{
id: 0,
offset: offset
}]
while (this.ifds_to_read.length > 0 && !this.aborted) {
var i = this.ifds_to_read.shift()
if (!i.offset) continue
this.scan_ifd(i.id, i.offset, on_entry)
}
}
ExifParser.prototype.read_uint16 = function (offset) {
var d = this.input
if (offset + 2 > d.length) throw error('unexpected EOF', 'EBADDATA')
return this.big_endian
? d[offset] * 0x100 + d[offset + 1]
: d[offset] + d[offset + 1] * 0x100
}
ExifParser.prototype.read_uint32 = function (offset) {
var d = this.input
if (offset + 4 > d.length) throw error('unexpected EOF', 'EBADDATA')
return this.big_endian
? d[offset] * 0x1000000 + d[offset + 1] * 0x10000 + d[offset + 2] * 0x100 + d[offset + 3]
: d[offset] + d[offset + 1] * 0x100 + d[offset + 2] * 0x10000 + d[offset + 3] * 0x1000000
}
ExifParser.prototype.is_subifd_link = function (ifd, tag) {
return (ifd === 0 && tag === 0x8769) || // SubIFD
(ifd === 0 && tag === 0x8825) || // GPS Info
(ifd === 0x8769 && tag === 0xA005) // Interop IFD
}
// Returns byte length of a single component of a given format
//
ExifParser.prototype.exif_format_length = function (format) {
switch (format) {
case 1: // byte
case 2: // ascii
case 6: // sbyte
case 7: // undefined
return 1
case 3: // short
case 8: // sshort
return 2
case 4: // long
case 9: // slong
case 11: // float
return 4
case 5: // rational
case 10: // srational
case 12: // double
return 8
default:
// unknown type
return 0
}
}
// Reads Exif data
//
ExifParser.prototype.exif_format_read = function (format, offset) {
var v
switch (format) {
case 1: // byte
case 2: // ascii
v = this.input[offset]
return v
case 6: // sbyte
v = this.input[offset]
return v | (v & 0x80) * 0x1fffffe
case 3: // short
v = this.read_uint16(offset)
return v
case 8: // sshort
v = this.read_uint16(offset)
return v | (v & 0x8000) * 0x1fffe
case 4: // long
v = this.read_uint32(offset)
return v
case 9: // slong
v = this.read_uint32(offset)
return v | 0
case 5: // rational
case 10: // srational
case 11: // float
case 12: // double
return null // not implemented
case 7: // undefined
return null // blob
default:
// unknown type
return null
}
}
ExifParser.prototype.scan_ifd = function (ifd_no, offset, on_entry) {
var entry_count = this.read_uint16(offset)
offset += 2
for (var i = 0; i < entry_count; i++) {
var tag = this.read_uint16(offset)
var format = this.read_uint16(offset + 2)
var count = this.read_uint32(offset + 4)
var comp_length = this.exif_format_length(format)
var data_length = count * comp_length
var data_offset = data_length <= 4 ? offset + 8 : this.read_uint32(offset + 8)
var is_subifd_link = false
if (data_offset + data_length > this.input.length) {
throw error('unexpected EOF', 'EBADDATA')
}
var value = []
var comp_offset = data_offset
for (var j = 0; j < count; j++, comp_offset += comp_length) {
var item = this.exif_format_read(format, comp_offset)
if (item === null) {
value = null
break
}
value.push(item)
}
if (Array.isArray(value) && format === 2) {
value = utf8_decode(String.fromCharCode.apply(null, value))
if (value && value[value.length - 1] === '\0') value = value.slice(0, -1)
}
if (this.is_subifd_link(ifd_no, tag)) {
if (Array.isArray(value) && Number.isInteger(value[0]) && value[0] > 0) {
this.ifds_to_read.push({
id: tag,
offset: value[0]
})
is_subifd_link = true
}
}
var entry = {
is_big_endian: this.big_endian,
ifd: ifd_no,
tag: tag,
format: format,
count: count,
entry_offset: offset + this.start,
data_length: data_length,
data_offset: data_offset + this.start,
value: value,
is_subifd_link: is_subifd_link
}
if (on_entry(entry) === false) {
this.aborted = true
return
}
offset += 12
}
if (ifd_no === 0) {
this.ifds_to_read.push({
id: 1,
offset: this.read_uint32(offset)
})
}
}
module.exports.ExifParser = ExifParser
// returns orientation stored in Exif (1-8), 0 if none was found, -1 if error
module.exports.get_orientation = function (data) {
var orientation = 0
try {
new ExifParser(data, 0, data.length).each(function (entry) {
if (entry.ifd === 0 && entry.tag === 0x112 && Array.isArray(entry.value)) {
orientation = entry.value[0]
return false
}
})
return orientation
} catch (err) {
return -1
}
}
+299
View File
@@ -0,0 +1,299 @@
// Utils used to parse miaf-based files (avif/heic/heif)
//
// ISO media file spec:
// https://web.archive.org/web/20180219054429/http://l.web.umkc.edu/lizhu/teaching/2016sp.video-communication/ref/mp4.pdf
//
// ISO image file format spec:
// https://standards.iso.org/ittf/PubliclyAvailableStandards/c066067_ISO_IEC_23008-12_2017.zip
//
'use strict'
var readUInt16BE = require('./common').readUInt16BE
var readUInt32BE = require('./common').readUInt32BE
/*
* interface Box {
* size: uint32; // if size == 0, box lasts until EOF
* boxtype: char[4];
* largesize?: uint64; // only if size == 1
* usertype?: char[16]; // only if boxtype == 'uuid'
* }
*/
function unbox (data, offset) {
if (data.length < 4 + offset) return null
var size = readUInt32BE(data, offset)
// size includes first 4 bytes (length)
if (data.length < size + offset || size < 8) return null
// if size === 1, real size is following uint64 (only for big boxes, not needed)
// if size === 0, real size is until the end of the file (only for big boxes, not needed)
return {
boxtype: String.fromCharCode.apply(null, data.slice(offset + 4, offset + 8)),
data: data.slice(offset + 8, offset + size),
end: offset + size
}
}
module.exports.unbox = unbox
// parses `meta` -> `iprp` -> `ipco` box, returns:
// {
// sizes: [ { width, height } ],
// transforms: [ { type, value } ]
// }
function scan_ipco (data, sandbox) {
var offset = 0
for (;;) {
var box = unbox(data, offset)
if (!box) break
switch (box.boxtype) {
case 'ispe':
sandbox.sizes.push({
width: readUInt32BE(box.data, 4),
height: readUInt32BE(box.data, 8)
})
break
case 'irot':
sandbox.transforms.push({
type: 'irot',
value: box.data[0] & 3
})
break
case 'imir':
sandbox.transforms.push({
type: 'imir',
value: box.data[0] & 1
})
break
}
offset = box.end
}
}
function readUIntBE (data, offset, size) {
var result = 0
for (var i = 0; i < size; i++) {
result = result * 256 + (data[offset + i] || 0)
}
return result
}
// parses `meta` -> `iloc` box
function scan_iloc (data, sandbox) {
var offset_size = (data[4] >> 4) & 0xF
var length_size = data[4] & 0xF
var base_offset_size = (data[5] >> 4) & 0xF
var item_count = readUInt16BE(data, 6)
var offset = 8
for (var i = 0; i < item_count; i++) {
var item_ID = readUInt16BE(data, offset)
offset += 2
var data_reference_index = readUInt16BE(data, offset)
offset += 2
var base_offset = readUIntBE(data, offset, base_offset_size)
offset += base_offset_size
var extent_count = readUInt16BE(data, offset)
offset += 2
if (data_reference_index === 0 && extent_count === 1) {
var first_extent_offset = readUIntBE(data, offset, offset_size)
var first_extent_length = readUIntBE(data, offset + offset_size, length_size)
sandbox.item_loc[item_ID] = { length: first_extent_length, offset: first_extent_offset + base_offset }
}
offset += extent_count * (offset_size + length_size)
}
}
// parses `meta` -> `iinf` box
function scan_iinf (data, sandbox) {
var item_count = readUInt16BE(data, 4)
var offset = 6
for (var i = 0; i < item_count; i++) {
var box = unbox(data, offset)
if (!box) break
if (box.boxtype === 'infe') {
var item_id = readUInt16BE(box.data, 4)
var item_name = ''
for (var pos = 8; pos < box.data.length && box.data[pos]; pos++) {
item_name += String.fromCharCode(box.data[pos])
}
sandbox.item_inf[item_name] = item_id
}
offset = box.end
}
}
// parses `meta` -> `iprp` box
function scan_iprp (data, sandbox) {
var offset = 0
for (;;) {
var box = unbox(data, offset)
if (!box) break
if (box.boxtype === 'ipco') scan_ipco(box.data, sandbox)
offset = box.end
}
}
// parses `meta` box
function scan_meta (data, sandbox) {
var offset = 4 // version + flags
for (;;) {
var box = unbox(data, offset)
if (!box) break
if (box.boxtype === 'iprp') scan_iprp(box.data, sandbox)
if (box.boxtype === 'iloc') scan_iloc(box.data, sandbox)
if (box.boxtype === 'iinf') scan_iinf(box.data, sandbox)
offset = box.end
}
}
// get image with largest single dimension as base
function getMaxSize (sizes) {
var maxWidthSize = sizes.reduce(function (a, b) {
return a.width > b.width || (a.width === b.width && a.height > b.height) ? a : b
})
var maxHeightSize = sizes.reduce(function (a, b) {
return a.height > b.height || (a.height === b.height && a.width > b.width) ? a : b
})
var maxSize
if (maxWidthSize.width > maxHeightSize.height ||
(maxWidthSize.width === maxHeightSize.height && maxWidthSize.height > maxHeightSize.width)) {
maxSize = maxWidthSize
} else {
maxSize = maxHeightSize
}
return maxSize
}
module.exports.readSizeFromMeta = function (data) {
var sandbox = {
sizes: [],
transforms: [],
item_inf: {},
item_loc: {}
}
scan_meta(data, sandbox)
if (!sandbox.sizes.length) return
var maxSize = getMaxSize(sandbox.sizes)
var orientation = 1
// convert imir/irot to exif orientation
sandbox.transforms.forEach(function (transform) {
var rotate_ccw = { 1: 6, 2: 5, 3: 8, 4: 7, 5: 4, 6: 3, 7: 2, 8: 1 }
var mirror_vert = { 1: 4, 2: 3, 3: 2, 4: 1, 5: 6, 6: 5, 7: 8, 8: 7 }
if (transform.type === 'imir') {
if (transform.value === 0) {
// vertical flip
orientation = mirror_vert[orientation]
} else {
// horizontal flip = vertical flip + 180 deg rotation
orientation = mirror_vert[orientation]
orientation = rotate_ccw[orientation]
orientation = rotate_ccw[orientation]
}
}
if (transform.type === 'irot') {
// counter-clockwise rotation 90 deg 0-3 times
for (var i = 0; i < transform.value; i++) {
orientation = rotate_ccw[orientation]
}
}
})
var exif_location = null
if (sandbox.item_inf.Exif) {
exif_location = sandbox.item_loc[sandbox.item_inf.Exif]
}
return {
width: maxSize.width,
height: maxSize.height,
orientation: sandbox.transforms.length ? orientation : null,
variants: sandbox.sizes,
exif_location: exif_location
}
}
module.exports.getMimeType = function (data) {
var brand = String.fromCharCode.apply(null, data.slice(0, 4))
var compat = {}
compat[brand] = true
for (var i = 8; i < data.length; i += 4) {
compat[String.fromCharCode.apply(null, data.slice(i, i + 4))] = true
}
// heic and avif are superset of miaf, so they should all list mif1 as compatible
if (!compat.mif1 && !compat.msf1 && !compat.miaf) return
if (brand === 'avif' || brand === 'avis' || brand === 'avio') {
// `.avifs` and `image/avif-sequence` are removed from spec, all files have single type
return { type: 'avif', mime: 'image/avif' }
}
// https://nokiatech.github.io/heif/technical.html
if (brand === 'heic' || brand === 'heix') {
return { type: 'heic', mime: 'image/heic' }
}
if (brand === 'hevc' || brand === 'hevx') {
return { type: 'heic', mime: 'image/heic-sequence' }
}
if (compat.avif || compat.avis) {
return { type: 'avif', mime: 'image/avif' }
}
if (compat.heic || compat.heix || compat.hevc || compat.hevx || compat.heis) {
if (compat.msf1) {
return { type: 'heif', mime: 'image/heif-sequence' }
}
return { type: 'heif', mime: 'image/heif' }
}
return { type: 'avif', mime: 'image/avif' }
}
+166
View File
@@ -0,0 +1,166 @@
// Utils used to parse miaf-based files (avif/heic/heif)
//
// - image collections are not supported (only last size is reported)
// - images with metadata encoded after image data are not supported
// - images without any `ispe` box are not supported
//
'use strict'
var ParserStream = require('../common').ParserStream
var str2arr = require('../common').str2arr
var sliceEq = require('../common').sliceEq
var readUInt32BE = require('../common').readUInt32BE
var miaf = require('../miaf_utils')
var exif = require('../exif_utils')
var SIG_FTYP = str2arr('ftyp')
function safeSkip (parser, count, callback) {
if (count === 0) { // parser._skipBytes throws error if count === 0
callback()
return
}
parser._skipBytes(count, callback)
}
function readExifOrientation (parser, sandbox, on_orientation) {
if (!sandbox.exif_location || sandbox.exif_location.offset <= sandbox.offset) {
on_orientation(0)
return
}
parser._skipBytes(sandbox.exif_location.offset - sandbox.offset, function () {
sandbox.offset = sandbox.exif_location.offset
parser._bytes(4, function (data) {
sandbox.offset += 4
var sig_offset = readUInt32BE(data, 0)
safeSkip(parser, sig_offset, function () {
sandbox.offset += sig_offset
var byteCount = sandbox.exif_location.length - sig_offset - 4
if (byteCount <= 0) {
on_orientation(0)
return
}
parser._bytes(byteCount, function (exif_data) {
sandbox.offset += byteCount
on_orientation(exif.get_orientation(exif_data))
})
})
})
})
}
// sandbox is a storage for intermediate data retrieved from jpeg while parsing it
function readAvifSize (parser, sandbox) {
parser._bytes(8, function (data) {
sandbox.offset += 8
var size = readUInt32BE(data, 0) - 8
var type = String.fromCharCode.apply(null, data.slice(4, 8))
if (type === 'mdat') {
parser._skipBytes(Infinity)
parser.push(null)
return
}
if (size < 0) {
parser._skipBytes(Infinity)
parser.push(null)
return
}
if (type === 'meta' && size > 0) {
parser._bytes(size, function (data) {
sandbox.offset += size
var imgSize = miaf.readSizeFromMeta(data)
if (!imgSize) {
parser._skipBytes(Infinity)
parser.push(null)
return
}
var result = {
width: imgSize.width,
height: imgSize.height,
type: sandbox.fileType.type,
mime: sandbox.fileType.mime,
wUnits: 'px',
hUnits: 'px'
}
if (imgSize.variants.length > 1) {
result.variants = imgSize.variants
}
if (imgSize.orientation) {
result.orientation = imgSize.orientation
}
sandbox.exif_location = imgSize.exif_location
readExifOrientation(parser, sandbox, function (orientation) {
if (orientation > 0) result.orientation = orientation
parser._skipBytes(Infinity)
parser.push(result)
parser.push(null)
})
})
} else {
safeSkip(parser, size, function () {
sandbox.offset += size
readAvifSize(parser, sandbox)
})
}
})
}
module.exports = function () {
var parser = new ParserStream()
var sandbox = { offset: 0, fileType: null }
parser._bytes(8, function (data) {
sandbox.offset += 8
if (!sliceEq(data, 4, SIG_FTYP)) {
parser._skipBytes(Infinity)
parser.push(null)
return
}
var size = readUInt32BE(data, 0) - 8
if (size <= 0) {
parser._skipBytes(Infinity)
parser.push(null)
return
}
parser._bytes(size, function (data) {
sandbox.offset += size
sandbox.fileType = miaf.getMimeType(data)
if (!sandbox.fileType) {
parser._skipBytes(Infinity)
parser.push(null)
return
}
readAvifSize(parser, sandbox)
})
})
return parser
}
+53
View File
@@ -0,0 +1,53 @@
'use strict'
var ParserStream = require('../common').ParserStream
var str2arr = require('../common').str2arr
var sliceEq = require('../common').sliceEq
var SIG_BM = str2arr('BM')
module.exports = function () {
var parser = new ParserStream()
parser._bytes(26, function (data) {
parser._skipBytes(Infinity)
if (!sliceEq(data, 0, SIG_BM)) {
parser.push(null)
return
}
var w, h
var headerSize = data.readUInt32LE(14)
if (headerSize === 12) {
// BMP v2 header
w = data.readInt16LE(18)
h = data.readInt16LE(20)
} else if (headerSize > 12) {
// BMP v3+ header
w = data.readInt32LE(18)
h = data.readInt32LE(22)
} else {
parser.push(null)
return
}
parser.push({
width: w,
// Height can be negative to indicate a top-down bitmap
height: Math.abs(h),
type: 'bmp',
mime: 'image/bmp',
wUnits: 'px',
hUnits: 'px'
})
parser.push(null)
})
return parser
}
+37
View File
@@ -0,0 +1,37 @@
'use strict'
var ParserStream = require('../common').ParserStream
var str2arr = require('../common').str2arr
var sliceEq = require('../common').sliceEq
var SIG_GIF87a = str2arr('GIF87a')
var SIG_GIF89a = str2arr('GIF89a')
module.exports = function () {
var parser = new ParserStream()
parser._bytes(10, function (data) {
parser._skipBytes(Infinity)
if (!sliceEq(data, 0, SIG_GIF87a) && !sliceEq(data, 0, SIG_GIF89a)) {
parser.push(null)
return
}
parser.push({
width: data.readUInt16LE(6),
height: data.readUInt16LE(8),
type: 'gif',
mime: 'image/gif',
wUnits: 'px',
hUnits: 'px'
})
parser.push(null)
})
return parser
}
+56
View File
@@ -0,0 +1,56 @@
'use strict'
var ParserStream = require('../common').ParserStream
var HEADER = 0
var TYPE_ICO = 1
var INDEX_SIZE = 16
// Format specification:
// https://en.wikipedia.org/wiki/ICO_(file_format)#Icon_resource_structure
module.exports = function () {
var parser = new ParserStream()
parser._bytes(6, function (data) {
var header = data.readUInt16LE(0)
var type = data.readUInt16LE(2)
var numImages = data.readUInt16LE(4)
if (header !== HEADER || type !== TYPE_ICO || !numImages) {
parser._skipBytes(Infinity)
parser.push(null)
return
}
parser._bytes(numImages * INDEX_SIZE, function (indexData) {
parser._skipBytes(Infinity)
var variants = []
var maxSize = { width: 0, height: 0 }
for (var i = 0; i < numImages; i++) {
var width = indexData.readUInt8(INDEX_SIZE * i + 0) || 256
var height = indexData.readUInt8(INDEX_SIZE * i + 1) || 256
var size = { width: width, height: height }
variants.push(size)
if (width > maxSize.width || height > maxSize.height) {
maxSize = size
}
}
parser.push({
width: maxSize.width,
height: maxSize.height,
variants: variants,
type: 'ico',
mime: 'image/x-icon',
wUnits: 'px',
hUnits: 'px'
})
parser.push(null)
})
})
return parser
}
+148
View File
@@ -0,0 +1,148 @@
'use strict'
var ParserStream = require('../common').ParserStream
var str2arr = require('../common').str2arr
var sliceEq = require('../common').sliceEq
var exif = require('../exif_utils')
var SIG_EXIF = str2arr('Exif\0\0')
// part of parseJpegMarker called after skipping initial FF
function parseJpegMarker_afterFF (parser, callback) {
parser._bytes(1, function (data) {
var code = data[0]
if (code === 0xFF) {
// padding byte, skip it
parseJpegMarker_afterFF(parser, callback)
return
}
// standalone markers, according to JPEG 1992,
// http://www.w3.org/Graphics/JPEG/itu-t81.pdf, see Table B.1
if ((code >= 0xD0 && code <= 0xD9) || code === 0x01) {
callback(code, 0)
return
}
// the rest of the unreserved markers
if (code >= 0xC0 && code <= 0xFE) {
parser._bytes(2, function (length) {
callback(code, length.readUInt16BE(0) - 2)
})
return
}
// unknown markers
callback()
})
}
function parseJpegMarker (parser, sandbox, callback) {
var start = sandbox.start
sandbox.start = false
parser._bytes(1, function (data) {
if (data[0] !== 0xFF) {
// not a JPEG marker
if (start) {
// expect JPEG file to start with `FFD8 FFE0`, `FFD8 FFE2` or `FFD8 FFE1`,
// don't allow garbage before second marker
callback()
} else {
// skip until we see 0xFF, see https://github.com/nodeca/probe-image-size/issues/68
parseJpegMarker(parser, sandbox, callback)
}
return
}
parseJpegMarker_afterFF(parser, callback)
})
}
// sandbox is a storage for intermediate data retrieved from jpeg while parsing it
function getJpegSize (parser, sandbox) {
parseJpegMarker(parser, sandbox, function (code, length) {
if (!code || length < 0) {
// invalid jpeg
parser._skipBytes(Infinity)
parser.push(null)
return
}
if (code === 0xD9 /* EOI */ || code === 0xDA /* SOS */) {
// end of the datastream
parser._skipBytes(Infinity)
parser.push(null)
return
}
// try to get orientation from Exif segment
if (code === 0xE1 && length >= 10) {
parser._bytes(length, function (data) {
if (sliceEq(data, 0, SIG_EXIF)) {
sandbox.orientation = exif.get_orientation(data.slice(6, 6 + length))
}
getJpegSize(parser, sandbox)
})
return
}
if (length <= 0) {
// e.g. empty comment
getJpegSize(parser, sandbox)
return
}
if (length >= 5 &&
(code >= 0xC0 && code <= 0xCF) &&
code !== 0xC4 && code !== 0xC8 && code !== 0xCC) {
parser._bytes(length, function (data) {
parser._skipBytes(Infinity)
var result = {
width: data.readUInt16BE(3),
height: data.readUInt16BE(1),
type: 'jpg',
mime: 'image/jpeg',
wUnits: 'px',
hUnits: 'px'
}
if (sandbox.orientation > 0) result.orientation = sandbox.orientation
parser.push(result)
parser.push(null)
})
return
}
parser._skipBytes(length, function () {
getJpegSize(parser, sandbox)
})
})
}
module.exports = function () {
var parser = new ParserStream()
parser._bytes(2, function (data) {
if (data[0] !== 0xFF || data[1] !== 0xD8) {
// first marker of the file MUST be 0xFFD8
parser._skipBytes(Infinity)
parser.push(null)
return
}
getJpegSize(parser, { start: true })
})
return parser
}
+44
View File
@@ -0,0 +1,44 @@
'use strict'
var ParserStream = require('../common').ParserStream
var str2arr = require('../common').str2arr
var sliceEq = require('../common').sliceEq
var SIG_PNG = str2arr('\x89PNG\r\n\x1a\n')
var SIG_IHDR = str2arr('IHDR')
module.exports = function () {
var parser = new ParserStream()
parser._bytes(24, function (data) {
parser._skipBytes(Infinity)
// check PNG signature
if (!sliceEq(data, 0, SIG_PNG)) {
parser.push(null)
return
}
// check that first chunk is IHDR
if (!sliceEq(data, 12, SIG_IHDR)) {
parser.push(null)
return
}
parser.push({
width: data.readUInt32BE(16),
height: data.readUInt32BE(20),
type: 'png',
mime: 'image/png',
wUnits: 'px',
hUnits: 'px'
})
parser.push(null)
})
return parser
}
+40
View File
@@ -0,0 +1,40 @@
'use strict'
var ParserStream = require('../common').ParserStream
var str2arr = require('../common').str2arr
var sliceEq = require('../common').sliceEq
var SIG_8BPS = str2arr('8BPS\x00\x01')
module.exports = function () {
var parser = new ParserStream()
parser._bytes(6, function (data) {
// signature + version
if (!sliceEq(data, 0, SIG_8BPS)) {
parser._skipBytes(Infinity)
parser.push(null)
return
}
parser._bytes(16, function (data) {
parser._skipBytes(Infinity)
parser.push({
width: data.readUInt32BE(12),
height: data.readUInt32BE(8),
type: 'psd',
mime: 'image/vnd.adobe.photoshop',
wUnits: 'px',
hUnits: 'px'
})
parser.push(null)
})
})
return parser
}
+217
View File
@@ -0,0 +1,217 @@
'use strict'
var Transform = require('stream').Transform
var STATE_IDENTIFY = 0 // look for '<'
var STATE_PARSE = 1 // extract width and height from svg tag
var STATE_IGNORE = 2 // we got all the data we want, skip the rest
// max size for pre-svg-tag comments plus svg tag itself
var MAX_DATA_LENGTH = 65536
// skip `<?` (comments), `<!` (directives, cdata, doctype),
// looking for `<svg>` or `<NAMESPACE:svg>`
var SVG_HEADER_RE = /<[-_.:a-zA-Z0-9][^>]*>/
// test if the top level element is svg + optional namespace,
// used to skip svg embedded in html
var SVG_TAG_RE = /^<([-_.:a-zA-Z0-9]+:)?svg\s/
var SVG_WIDTH_RE = /[^-]\bwidth="([^%]+?)"|[^-]\bwidth='([^%]+?)'/
var SVG_HEIGHT_RE = /\bheight="([^%]+?)"|\bheight='([^%]+?)'/
var SVG_VIEWBOX_RE = /\bview[bB]ox="(.+?)"|\bview[bB]ox='(.+?)'/
var SVG_UNITS_RE = /in$|mm$|cm$|pt$|pc$|px$|em$|ex$/
function isWhiteSpace (chr) {
return chr === 0x20 || chr === 0x09 || chr === 0x0D || chr === 0x0A
}
// Filter NaN, Infinity, < 0
function isFinitePositive (val) {
return typeof val === 'number' && isFinite(val) && val > 0
}
function svgAttrs (str) {
var width = str.match(SVG_WIDTH_RE)
var height = str.match(SVG_HEIGHT_RE)
var viewbox = str.match(SVG_VIEWBOX_RE)
return {
width: width && (width[1] || width[2]),
height: height && (height[1] || height[2]),
viewbox: viewbox && (viewbox[1] || viewbox[2])
}
}
function units (str) {
if (!SVG_UNITS_RE.test(str)) return 'px'
return str.match(SVG_UNITS_RE)[0]
}
function parseSvg (str) {
// get top level element
var svgTag = (str.match(SVG_HEADER_RE) || [''])[0]
// test if top level element is <svg>
if (!SVG_TAG_RE.test(svgTag)) return
var attrs = svgAttrs(svgTag)
var width = parseFloat(attrs.width)
var height = parseFloat(attrs.height)
// Extract from direct values
if (attrs.width && attrs.height) {
if (!isFinitePositive(width) || !isFinitePositive(height)) return
return {
width: width,
height: height,
type: 'svg',
mime: 'image/svg+xml',
wUnits: units(attrs.width),
hUnits: units(attrs.height)
}
}
// Extract from viewbox
var parts = (attrs.viewbox || '').split(' ')
var viewbox = {
width: parts[2],
height: parts[3]
}
var vbWidth = parseFloat(viewbox.width)
var vbHeight = parseFloat(viewbox.height)
if (!isFinitePositive(vbWidth) || !isFinitePositive(vbHeight)) return
if (units(viewbox.width) !== units(viewbox.height)) return
var ratio = vbWidth / vbHeight
if (attrs.width) {
if (!isFinitePositive(width)) return
return {
width: width,
height: width / ratio,
type: 'svg',
mime: 'image/svg+xml',
wUnits: units(attrs.width),
hUnits: units(attrs.width)
}
}
if (attrs.height) {
if (!isFinitePositive(height)) return
return {
width: height * ratio,
height: height,
type: 'svg',
mime: 'image/svg+xml',
wUnits: units(attrs.height),
hUnits: units(attrs.height)
}
}
return {
width: vbWidth,
height: vbHeight,
type: 'svg',
mime: 'image/svg+xml',
wUnits: units(viewbox.width),
hUnits: units(viewbox.height)
}
}
module.exports = function () {
var state = STATE_IDENTIFY
var data_len = 0
var str = ''
var buf = null // used to manage first chunk in IDENTIFY
var parser = new Transform({
readableObjectMode: true,
transform: function transform (chunk, encoding, next) {
switch (state) {
// identify step is needed to fail fast if the file isn't SVG
case STATE_IDENTIFY:
if (buf) {
// make sure that first chunk is at least 4 bytes (to do BOM skip later),
// last chunk was small
chunk = Buffer.concat([buf, chunk])
buf = null
}
if (data_len === 0 && chunk.length < 4) {
// make sure that first chunk is at least 4 bytes (to do BOM skip later),
// current chunk is small
buf = chunk
break
}
var i = 0
var max = chunk.length
// byte order mark, https://github.com/nodeca/probe-image-size/issues/57
if (data_len === 0 && chunk[0] === 0xEF && chunk[1] === 0xBB && chunk[2] === 0xBF) i = 3
while (i < max && isWhiteSpace(chunk[i])) i++
if (i >= max) {
data_len += chunk.length
if (data_len > MAX_DATA_LENGTH) {
state = STATE_IGNORE
parser.push(null)
}
} else if (chunk[i] === 0x3c /* < */) {
state = STATE_PARSE
return transform(chunk, encoding, next)
} else {
state = STATE_IGNORE
parser.push(null)
}
break
case STATE_PARSE:
str += chunk.toString()
var result = parseSvg(str)
if (result) {
state = STATE_IGNORE
parser.push(result)
parser.push(null)
break
}
data_len += chunk.length
if (data_len > MAX_DATA_LENGTH) {
state = STATE_IGNORE
parser.push(null)
}
break
}
next()
},
flush: function () {
state = STATE_IGNORE
parser.push(null)
}
})
return parser
}
+112
View File
@@ -0,0 +1,112 @@
'use strict'
var ParserStream = require('../common').ParserStream
var str2arr = require('../common').str2arr
var sliceEq = require('../common').sliceEq
var SIG_1 = str2arr('II\x2A\0')
var SIG_2 = str2arr('MM\0\x2A')
function readUInt16 (buffer, offset, is_big_endian) {
return is_big_endian ? buffer.readUInt16BE(offset) : buffer.readUInt16LE(offset)
}
function readUInt32 (buffer, offset, is_big_endian) {
return is_big_endian ? buffer.readUInt32BE(offset) : buffer.readUInt32LE(offset)
}
function readIFDValue (data, data_offset, is_big_endian) {
var type = readUInt16(data, data_offset + 2, is_big_endian)
var values = readUInt32(data, data_offset + 4, is_big_endian)
if (values !== 1 || (type !== 3 && type !== 4)) {
return null
}
if (type === 3) {
return readUInt16(data, data_offset + 8, is_big_endian)
}
return readUInt32(data, data_offset + 8, is_big_endian)
}
module.exports = function () {
var parser = new ParserStream()
// read header
parser._bytes(8, function (data) {
// check TIFF signature
if (!sliceEq(data, 0, SIG_1) && !sliceEq(data, 0, SIG_2)) {
parser._skipBytes(Infinity)
parser.push(null)
return
}
var is_big_endian = (data[0] === 77 /* 'MM' */)
var count = readUInt32(data, 4, is_big_endian) - 8
if (count < 0) {
parser._skipBytes(Infinity)
parser.push(null)
return
}
function safeSkip (parser, count, callback) {
if (count === 0) { // parser._skipBytes throws error if count === 0
callback()
return
}
parser._skipBytes(count, callback)
}
// skip until IFD
safeSkip(parser, count, function () {
// read number of IFD entries
parser._bytes(2, function (data) {
var ifd_size = readUInt16(data, 0, is_big_endian) * 12
if (ifd_size <= 0) {
parser._skipBytes(Infinity)
parser.push(null)
return
}
// read all IFD entries
parser._bytes(ifd_size, function (data) {
parser._skipBytes(Infinity)
var i, width, height, tag
for (i = 0; i < ifd_size; i += 12) {
tag = readUInt16(data, i, is_big_endian)
if (tag === 256) {
width = readIFDValue(data, i, is_big_endian)
} else if (tag === 257) {
height = readIFDValue(data, i, is_big_endian)
}
}
if (width && height) {
parser.push({
width: width,
height: height,
type: 'tiff',
mime: 'image/tiff',
wUnits: 'px',
hUnits: 'px'
})
}
parser.push(null)
})
})
})
})
return parser
}
+177
View File
@@ -0,0 +1,177 @@
'use strict'
var ParserStream = require('../common').ParserStream
var str2arr = require('../common').str2arr
var sliceEq = require('../common').sliceEq
var exif = require('../exif_utils')
var SIG_RIFF = str2arr('RIFF')
var SIG_WEBP = str2arr('WEBP')
function safeSkip (parser, count, callback) {
if (count === 0) { // parser._skipBytes throws error if count === 0
callback()
return
}
parser._skipBytes(count, callback)
}
function parseVP8 (parser, length, sandbox) {
parser._bytes(10, function (data) {
// check code block signature
if (data[3] === 0x9D && data[4] === 0x01 && data[5] === 0x2A) {
sandbox.result = sandbox.result || {
width: data.readUInt16LE(6) & 0x3FFF,
height: data.readUInt16LE(8) & 0x3FFF,
type: 'webp',
mime: 'image/webp',
wUnits: 'px',
hUnits: 'px'
}
}
safeSkip(parser, length - 10, function () {
sandbox.offset += length
getWebpSize(parser, sandbox)
})
})
}
function parseVP8L (parser, length, sandbox) {
parser._bytes(5, function (data) {
// check code block signature
if (data[0] === 0x2F) {
var bits = data.readUInt32LE(1)
sandbox.result = sandbox.result || {
width: (bits & 0x3FFF) + 1,
height: ((bits >> 14) & 0x3FFF) + 1,
type: 'webp',
mime: 'image/webp',
wUnits: 'px',
hUnits: 'px'
}
}
safeSkip(parser, length - 5, function () {
sandbox.offset += length
getWebpSize(parser, sandbox)
})
})
}
function parseVP8X (parser, length, sandbox) {
parser._bytes(10, function (data) {
sandbox.result = sandbox.result || {
// TODO: replace with `data.readUIntLE(8, 3) + 1`
// when 0.10 support is dropped
width: ((data[6] << 16) | (data[5] << 8) | data[4]) + 1,
height: ((data[9] << 16) | (data[8] << 8) | data[7]) + 1,
type: 'webp',
mime: 'image/webp',
wUnits: 'px',
hUnits: 'px'
}
safeSkip(parser, length - 10, function () {
sandbox.offset += length
getWebpSize(parser, sandbox)
})
})
}
function parseExif (parser, length, sandbox) {
parser._bytes(length, function (data) {
// exif is the last chunk we care about, stop after it
sandbox.offset = Infinity
sandbox.exif_orientation = exif.get_orientation(data)
getWebpSize(parser, sandbox)
})
}
function getWebpSize (parser, sandbox) {
if (sandbox.fileLength - 8 <= sandbox.offset) {
parser._skipBytes(Infinity)
if (sandbox.result) {
var result = sandbox.result
if (sandbox.exif_orientation > 0) {
result.orientation = sandbox.exif_orientation
}
parser.push(result)
}
parser.push(null)
return
}
parser._bytes(4 - sandbox.bufferedChunkHeader.length, function (data) {
sandbox.offset += 4 - sandbox.bufferedChunkHeader.length
var header = sandbox.bufferedChunkHeader + String.fromCharCode.apply(null, data)
// after each chunk of odd size there should be 0 byte of padding, skip those
header = header.replace(/^\0+/, '')
if (header.length < 4) {
sandbox.bufferedChunkHeader = header
getWebpSize(parser, sandbox)
return
}
sandbox.bufferedChunkHeader = ''
parser._bytes(4, function (data) {
sandbox.offset += 4
var length = data.readUInt32LE(0)
if (header === 'VP8 ' && length >= 10) {
parseVP8(parser, length, sandbox)
} else if (header === 'VP8L' && length >= 5) {
parseVP8L(parser, length, sandbox)
} else if (header === 'VP8X' && length >= 10) {
parseVP8X(parser, length, sandbox)
} else if (header === 'EXIF' && length >= 4) {
parseExif(parser, length, sandbox)
} else {
safeSkip(parser, length, function () {
sandbox.offset += length
getWebpSize(parser, sandbox)
})
}
})
})
}
module.exports = function () {
var parser = new ParserStream()
parser._bytes(12, function (data) {
// check /^RIFF....WEBPVP8([ LX])$/ signature
if (sliceEq(data, 0, SIG_RIFF) && sliceEq(data, 8, SIG_WEBP)) {
getWebpSize(parser, {
fileLength: data.readUInt32LE(4) + 8,
offset: 12,
exif_orientation: 0,
bufferedChunkHeader: '' // for dealing with padding
})
} else {
parser._skipBytes(Infinity)
parser.push(null)
}
})
return parser
}
+87
View File
@@ -0,0 +1,87 @@
// Utils used to parse miaf-based files (avif/heic/heif)
//
// - image collections are not supported (only last size is reported)
// - images with metadata encoded after image data are not supported
// - images without any `ispe` box are not supported
//
'use strict'
var str2arr = require('../common').str2arr
var sliceEq = require('../common').sliceEq
var readUInt32BE = require('../common').readUInt32BE
var miaf = require('../miaf_utils')
var exif = require('../exif_utils')
var SIG_FTYP = str2arr('ftyp')
module.exports = function (data) {
// ISO media file (avif format) starts with ftyp box:
// 0000 0020 6674 7970 6176 6966
// (length) f t y p a v i f
//
if (!sliceEq(data, 4, SIG_FTYP)) return
var firstBox = miaf.unbox(data, 0)
if (!firstBox) return
var fileType = miaf.getMimeType(firstBox.data)
if (!fileType) return
var meta
var offset = firstBox.end
for (;;) {
var box = miaf.unbox(data, offset)
if (!box) break
offset = box.end
// mdat block SHOULD be last (but not strictly required),
// so it's unlikely that metadata is after it
if (box.boxtype === 'mdat') return
if (box.boxtype === 'meta') {
meta = box.data
break
}
}
if (!meta) return
var imgSize = miaf.readSizeFromMeta(meta)
if (!imgSize) return
var result = {
width: imgSize.width,
height: imgSize.height,
type: fileType.type,
mime: fileType.mime,
wUnits: 'px',
hUnits: 'px'
}
if (imgSize.variants.length > 1) {
result.variants = imgSize.variants
}
if (imgSize.orientation) {
result.orientation = imgSize.orientation
}
if (imgSize.exif_location &&
imgSize.exif_location.offset + imgSize.exif_location.length <= data.length) {
var sig_offset = readUInt32BE(data, imgSize.exif_location.offset)
var exif_data = data.slice(
imgSize.exif_location.offset + sig_offset + 4,
imgSize.exif_location.offset + imgSize.exif_location.length)
var orientation = exif.get_orientation(exif_data)
if (orientation > 0) result.orientation = orientation
}
return result
}
+44
View File
@@ -0,0 +1,44 @@
'use strict'
var str2arr = require('../common').str2arr
var sliceEq = require('../common').sliceEq
var readInt16LE = require('../common').readInt16LE
var readInt32LE = require('../common').readInt32LE
var readUInt32LE = require('../common').readUInt32LE
var SIG_BM = str2arr('BM')
module.exports = function (data) {
if (data.length < 26) return
if (!sliceEq(data, 0, SIG_BM)) return
var h
var w
var headerSize = readUInt32LE(data, 14)
if (headerSize === 12) {
// BMP v2 header
w = readInt16LE(data, 18)
h = readInt16LE(data, 20)
} else if (headerSize > 12) {
// BMP v3+ header
w = readInt32LE(data, 18)
h = readInt32LE(data, 22)
} else {
// BPM v1 and other garbage (10 bytes usually)
return
}
return {
width: w,
// Height can be negative to indicate a top-down bitmap
height: Math.abs(h),
type: 'bmp',
mime: 'image/bmp',
wUnits: 'px',
hUnits: 'px'
}
}
+26
View File
@@ -0,0 +1,26 @@
'use strict'
var str2arr = require('../common').str2arr
var sliceEq = require('../common').sliceEq
var readUInt16LE = require('../common').readUInt16LE
var SIG_GIF87a = str2arr('GIF87a')
var SIG_GIF89a = str2arr('GIF89a')
module.exports = function (data) {
if (data.length < 10) return
if (!sliceEq(data, 0, SIG_GIF87a) && !sliceEq(data, 0, SIG_GIF89a)) return
return {
width: readUInt16LE(data, 6),
height: readUInt16LE(data, 8),
type: 'gif',
mime: 'image/gif',
wUnits: 'px',
hUnits: 'px'
}
}
+46
View File
@@ -0,0 +1,46 @@
'use strict'
var readUInt16LE = require('../common').readUInt16LE
var HEADER = 0
var TYPE_ICO = 1
var INDEX_SIZE = 16
// Format specification:
// https://en.wikipedia.org/wiki/ICO_(file_format)#Icon_resource_structure
module.exports = function (data) {
var header = readUInt16LE(data, 0)
var type = readUInt16LE(data, 2)
var numImages = readUInt16LE(data, 4)
if (header !== HEADER || type !== TYPE_ICO || !numImages) {
return
}
if (data.length < 6 + numImages * INDEX_SIZE) return
var variants = []
var maxSize = { width: 0, height: 0 }
for (var i = 0; i < numImages; i++) {
var width = data[6 + INDEX_SIZE * i] || 256
var height = data[6 + INDEX_SIZE * i + 1] || 256
var size = { width: width, height: height }
variants.push(size)
if (width > maxSize.width || height > maxSize.height) {
maxSize = size
}
}
return {
width: maxSize.width,
height: maxSize.height,
variants: variants,
type: 'ico',
mime: 'image/x-icon',
wUnits: 'px',
hUnits: 'px'
}
}
+85
View File
@@ -0,0 +1,85 @@
'use strict'
var readUInt16BE = require('../common').readUInt16BE
var str2arr = require('../common').str2arr
var sliceEq = require('../common').sliceEq
var exif = require('../exif_utils')
var SIG_EXIF = str2arr('Exif\0\0')
module.exports = function (data) {
if (data.length < 2) return
// first marker of the file MUST be 0xFFD8,
// following by either 0xFFE0, 0xFFE2 or 0xFFE3
if (data[0] !== 0xFF || data[1] !== 0xD8 || data[2] !== 0xFF) return
var offset = 2
for (;;) {
// skip until we see 0xFF, see https://github.com/nodeca/probe-image-size/issues/68
for (;;) {
if (data.length - offset < 2) return
if (data[offset++] === 0xFF) break
}
var code = data[offset++]
var length
// skip padding bytes
while (code === 0xFF) code = data[offset++]
// standalone markers, according to JPEG 1992,
// http://www.w3.org/Graphics/JPEG/itu-t81.pdf, see Table B.1
if ((code >= 0xD0 && code <= 0xD9) || code === 0x01) {
length = 0
} else if (code >= 0xC0 && code <= 0xFE) {
// the rest of the unreserved markers
if (data.length - offset < 2) return
length = readUInt16BE(data, offset) - 2
offset += 2
} else {
// unknown markers
return
}
if (code === 0xD9 /* EOI */ || code === 0xDA /* SOS */) {
// end of the datastream
return
}
var orientation
// try to get orientation from Exif segment
if (code === 0xE1 && length >= 10 && sliceEq(data, offset, SIG_EXIF)) {
orientation = exif.get_orientation(data.slice(offset + 6, offset + length))
}
if (length >= 5 &&
(code >= 0xC0 && code <= 0xCF) &&
code !== 0xC4 && code !== 0xC8 && code !== 0xCC) {
if (data.length - offset < length) return
var result = {
width: readUInt16BE(data, offset + 3),
height: readUInt16BE(data, offset + 1),
type: 'jpg',
mime: 'image/jpeg',
wUnits: 'px',
hUnits: 'px'
}
if (orientation > 0) {
result.orientation = orientation
}
return result
}
offset += length
}
}
+30
View File
@@ -0,0 +1,30 @@
'use strict'
var str2arr = require('../common').str2arr
var sliceEq = require('../common').sliceEq
var readUInt32BE = require('../common').readUInt32BE
var SIG_PNG = str2arr('\x89PNG\r\n\x1a\n')
var SIG_IHDR = str2arr('IHDR')
module.exports = function (data) {
if (data.length < 24) return
// check PNG signature
if (!sliceEq(data, 0, SIG_PNG)) return
// check that first chunk is IHDR
if (!sliceEq(data, 12, SIG_IHDR)) return
return {
width: readUInt32BE(data, 16),
height: readUInt32BE(data, 20),
type: 'png',
mime: 'image/png',
wUnits: 'px',
hUnits: 'px'
}
}
+26
View File
@@ -0,0 +1,26 @@
'use strict'
var str2arr = require('../common').str2arr
var sliceEq = require('../common').sliceEq
var readUInt32BE = require('../common').readUInt32BE
var SIG_8BPS = str2arr('8BPS\x00\x01')
module.exports = function (data) {
if (data.length < 6 + 16) return
// signature + version
if (!sliceEq(data, 0, SIG_8BPS)) return
return {
width: readUInt32BE(data, 6 + 12),
height: readUInt32BE(data, 6 + 8),
type: 'psd',
mime: 'image/vnd.adobe.photoshop',
wUnits: 'px',
hUnits: 'px'
}
}
+145
View File
@@ -0,0 +1,145 @@
'use strict'
function isWhiteSpace (chr) {
return chr === 0x20 || chr === 0x09 || chr === 0x0D || chr === 0x0A
}
// Filter NaN, Infinity, < 0
function isFinitePositive (val) {
return typeof val === 'number' && isFinite(val) && val > 0
}
function canBeSvg (buf) {
var i = 0
var max = buf.length
// byte order mark, https://github.com/nodeca/probe-image-size/issues/57
if (buf[0] === 0xEF && buf[1] === 0xBB && buf[2] === 0xBF) i = 3
while (i < max && isWhiteSpace(buf[i])) i++
if (i === max) return false
return buf[i] === 0x3c /* < */
}
// skip `<?` (comments), `<!` (directives, cdata, doctype),
// looking for `<svg>` or `<NAMESPACE:svg>`
var SVG_HEADER_RE = /<[-_.:a-zA-Z0-9][^>]*>/
// test if the top level element is svg + optional namespace,
// used to skip svg embedded in html
var SVG_TAG_RE = /^<([-_.:a-zA-Z0-9]+:)?svg\s/
var SVG_WIDTH_RE = /[^-]\bwidth="([^%]+?)"|[^-]\bwidth='([^%]+?)'/
var SVG_HEIGHT_RE = /\bheight="([^%]+?)"|\bheight='([^%]+?)'/
var SVG_VIEWBOX_RE = /\bview[bB]ox="(.+?)"|\bview[bB]ox='(.+?)'/
var SVG_UNITS_RE = /in$|mm$|cm$|pt$|pc$|px$|em$|ex$/
function svgAttrs (str) {
var width = str.match(SVG_WIDTH_RE)
var height = str.match(SVG_HEIGHT_RE)
var viewbox = str.match(SVG_VIEWBOX_RE)
return {
width: width && (width[1] || width[2]),
height: height && (height[1] || height[2]),
viewbox: viewbox && (viewbox[1] || viewbox[2])
}
}
function units (str) {
if (!SVG_UNITS_RE.test(str)) return 'px'
return str.match(SVG_UNITS_RE)[0]
}
module.exports = function (data) {
if (!canBeSvg(data)) return
var str = ''
for (var i = 0; i < data.length; i++) {
// 1. We can't rely on buffer features
// 2. Don't care about UTF16 because ascii is enougth for our goals
str += String.fromCharCode(data[i])
}
// get top level element
var svgTag = (str.match(SVG_HEADER_RE) || [''])[0]
// test if top level element is <svg>
if (!SVG_TAG_RE.test(svgTag)) return
var attrs = svgAttrs(svgTag)
var width = parseFloat(attrs.width)
var height = parseFloat(attrs.height)
// Extract from direct values
if (attrs.width && attrs.height) {
if (!isFinitePositive(width) || !isFinitePositive(height)) return
return {
width: width,
height: height,
type: 'svg',
mime: 'image/svg+xml',
wUnits: units(attrs.width),
hUnits: units(attrs.height)
}
}
// Extract from viewbox
var parts = (attrs.viewbox || '').split(' ')
var viewbox = {
width: parts[2],
height: parts[3]
}
var vbWidth = parseFloat(viewbox.width)
var vbHeight = parseFloat(viewbox.height)
if (!isFinitePositive(vbWidth) || !isFinitePositive(vbHeight)) return
if (units(viewbox.width) !== units(viewbox.height)) return
var ratio = vbWidth / vbHeight
if (attrs.width) {
if (!isFinitePositive(width)) return
return {
width: width,
height: width / ratio,
type: 'svg',
mime: 'image/svg+xml',
wUnits: units(attrs.width),
hUnits: units(attrs.width)
}
}
if (attrs.height) {
if (!isFinitePositive(height)) return
return {
width: height * ratio,
height: height,
type: 'svg',
mime: 'image/svg+xml',
wUnits: units(attrs.height),
hUnits: units(attrs.height)
}
}
return {
width: vbWidth,
height: vbHeight,
type: 'svg',
mime: 'image/svg+xml',
wUnits: units(viewbox.width),
hUnits: units(viewbox.height)
}
}
+85
View File
@@ -0,0 +1,85 @@
'use strict'
var str2arr = require('../common').str2arr
var sliceEq = require('../common').sliceEq
var readUInt16LE = require('../common').readUInt16LE
var readUInt16BE = require('../common').readUInt16BE
var readUInt32LE = require('../common').readUInt32LE
var readUInt32BE = require('../common').readUInt32BE
var SIG_1 = str2arr('II\x2A\0')
var SIG_2 = str2arr('MM\0\x2A')
function readUInt16 (buffer, offset, is_big_endian) {
return is_big_endian ? readUInt16BE(buffer, offset) : readUInt16LE(buffer, offset)
}
function readUInt32 (buffer, offset, is_big_endian) {
return is_big_endian ? readUInt32BE(buffer, offset) : readUInt32LE(buffer, offset)
}
function readIFDValue (data, data_offset, is_big_endian) {
var type = readUInt16(data, data_offset + 2, is_big_endian)
var values = readUInt32(data, data_offset + 4, is_big_endian)
if (values !== 1 || (type !== 3 && type !== 4)) return null
if (type === 3) {
return readUInt16(data, data_offset + 8, is_big_endian)
}
return readUInt32(data, data_offset + 8, is_big_endian)
}
module.exports = function (data) {
if (data.length < 8) return
// check TIFF signature
if (!sliceEq(data, 0, SIG_1) && !sliceEq(data, 0, SIG_2)) return
var is_big_endian = (data[0] === 77 /* 'MM' */)
var count = readUInt32(data, 4, is_big_endian) - 8
if (count < 0) return
// skip until IFD
var offset = count + 8
if (data.length - offset < 2) return
// read number of IFD entries
var ifd_size = readUInt16(data, offset + 0, is_big_endian) * 12
if (ifd_size <= 0) return
offset += 2
// read all IFD entries
if (data.length - offset < ifd_size) return
var i, width, height, tag
for (i = 0; i < ifd_size; i += 12) {
tag = readUInt16(data, offset + i, is_big_endian)
if (tag === 256) {
width = readIFDValue(data, offset + i, is_big_endian)
} else if (tag === 257) {
height = readIFDValue(data, offset + i, is_big_endian)
}
}
if (width && height) {
return {
width: width,
height: height,
type: 'tiff',
mime: 'image/tiff',
wUnits: 'px',
hUnits: 'px'
}
}
}
+108
View File
@@ -0,0 +1,108 @@
'use strict'
var str2arr = require('../common').str2arr
var sliceEq = require('../common').sliceEq
var readUInt16LE = require('../common').readUInt16LE
var readUInt32LE = require('../common').readUInt32LE
var exif = require('../exif_utils')
var SIG_RIFF = str2arr('RIFF')
var SIG_WEBP = str2arr('WEBP')
function parseVP8 (data, offset) {
if (data[offset + 3] !== 0x9D || data[offset + 4] !== 0x01 || data[offset + 5] !== 0x2A) {
// bad code block signature
return
}
return {
width: readUInt16LE(data, offset + 6) & 0x3FFF,
height: readUInt16LE(data, offset + 8) & 0x3FFF,
type: 'webp',
mime: 'image/webp',
wUnits: 'px',
hUnits: 'px'
}
}
function parseVP8L (data, offset) {
if (data[offset] !== 0x2F) return
var bits = readUInt32LE(data, offset + 1)
return {
width: (bits & 0x3FFF) + 1,
height: ((bits >> 14) & 0x3FFF) + 1,
type: 'webp',
mime: 'image/webp',
wUnits: 'px',
hUnits: 'px'
}
}
function parseVP8X (data, offset) {
return {
// TODO: replace with `data.readUIntLE(8, 3) + 1`
// when 0.10 support is dropped
width: ((data[offset + 6] << 16) | (data[offset + 5] << 8) | data[offset + 4]) + 1,
height: ((data[offset + 9] << 16) | (data[offset + 8] << 8) | data[offset + 7]) + 1,
type: 'webp',
mime: 'image/webp',
wUnits: 'px',
hUnits: 'px'
}
}
module.exports = function (data) {
if (data.length < 16) return
// check /^RIFF....WEBPVP8([ LX])$/ signature
if (!sliceEq(data, 0, SIG_RIFF) || !sliceEq(data, 8, SIG_WEBP)) return
var offset = 12
var result = null
var exif_orientation = 0
var fileLength = readUInt32LE(data, 4) + 8
if (fileLength > data.length) return
while (offset + 8 < fileLength) {
if (data[offset] === 0) {
// after each chunk of odd size there should be 0 byte of padding, skip those
offset++
continue
}
var header = String.fromCharCode.apply(null, data.slice(offset, offset + 4))
var length = readUInt32LE(data, offset + 4)
if (header === 'VP8 ' && length >= 10) {
result = result || parseVP8(data, offset + 8)
} else if (header === 'VP8L' && length >= 5) {
result = result || parseVP8L(data, offset + 8)
} else if (header === 'VP8X' && length >= 10) {
result = result || parseVP8X(data, offset + 8)
} else if (header === 'EXIF') {
exif_orientation = exif.get_orientation(data.slice(offset + 8, offset + 8 + length))
// exif is the last chunk we care about, stop after it
offset = Infinity
}
offset += 8 + length
}
if (!result) return
if (exif_orientation > 0) {
result.orientation = exif_orientation
}
return result
}
+14
View File
@@ -0,0 +1,14 @@
'use strict'
module.exports = {
avif: require('./parse_stream/avif'),
bmp: require('./parse_stream/bmp'),
gif: require('./parse_stream/gif'),
ico: require('./parse_stream/ico'),
jpeg: require('./parse_stream/jpeg'),
png: require('./parse_stream/png'),
psd: require('./parse_stream/psd'),
svg: require('./parse_stream/svg'),
tiff: require('./parse_stream/tiff'),
webp: require('./parse_stream/webp')
}
+15
View File
@@ -0,0 +1,15 @@
'use strict'
module.exports = {
avif: require('./parse_sync/avif'),
bmp: require('./parse_sync/bmp'),
gif: require('./parse_sync/gif'),
ico: require('./parse_sync/ico'),
jpeg: require('./parse_sync/jpeg'),
png: require('./parse_sync/png'),
psd: require('./parse_sync/psd'),
svg: require('./parse_sync/svg'),
tiff: require('./parse_sync/tiff'),
webp: require('./parse_sync/webp')
}
+16
View File
@@ -0,0 +1,16 @@
#!/bin/sh
basedir=$(dirname "$(echo "$0" | sed -e 's,\\,/,g')")
case `uname` in
*CYGWIN*|*MINGW*|*MSYS*)
if command -v cygpath > /dev/null 2>&1; then
basedir=`cygpath -w "$basedir"`
fi
;;
esac
if [ -x "$basedir/node" ]; then
exec "$basedir/node" "$basedir/../needle/bin/needle" "$@"
else
exec node "$basedir/../needle/bin/needle" "$@"
fi
@@ -0,0 +1,17 @@
@ECHO off
GOTO start
:find_dp0
SET dp0=%~dp0
EXIT /b
:start
SETLOCAL
CALL :find_dp0
IF EXIST "%dp0%\node.exe" (
SET "_prog=%dp0%\node.exe"
) ELSE (
SET "_prog=node"
SET PATHEXT=%PATHEXT:;.JS;=;%
)
endLocal & goto #_undefined_# 2>NUL || title %COMSPEC% & "%_prog%" "%dp0%\..\needle\bin\needle" %*
@@ -0,0 +1,28 @@
#!/usr/bin/env pwsh
$basedir=Split-Path $MyInvocation.MyCommand.Definition -Parent
$exe=""
if ($PSVersionTable.PSVersion -lt "6.0" -or $IsWindows) {
# Fix case when both the Windows and Linux builds of Node
# are installed in the same directory
$exe=".exe"
}
$ret=0
if (Test-Path "$basedir/node$exe") {
# Support pipeline input
if ($MyInvocation.ExpectingInput) {
$input | & "$basedir/node$exe" "$basedir/../needle/bin/needle" $args
} else {
& "$basedir/node$exe" "$basedir/../needle/bin/needle" $args
}
$ret=$LASTEXITCODE
} else {
# Support pipeline input
if ($MyInvocation.ExpectingInput) {
$input | & "node$exe" "$basedir/../needle/bin/needle" $args
} else {
& "node$exe" "$basedir/../needle/bin/needle" $args
}
$ret=$LASTEXITCODE
}
exit $ret
@@ -0,0 +1,395 @@
3.1.0 / 2017-09-26
==================
* Add `DEBUG_HIDE_DATE` env var (#486)
* Remove ReDoS regexp in %o formatter (#504)
* Remove "component" from package.json
* Remove `component.json`
* Ignore package-lock.json
* Examples: fix colors printout
* Fix: browser detection
* Fix: spelling mistake (#496, @EdwardBetts)
3.0.1 / 2017-08-24
==================
* Fix: Disable colors in Edge and Internet Explorer (#489)
3.0.0 / 2017-08-08
==================
* Breaking: Remove DEBUG_FD (#406)
* Breaking: Use `Date#toISOString()` instead to `Date#toUTCString()` when output is not a TTY (#418)
* Breaking: Make millisecond timer namespace specific and allow 'always enabled' output (#408)
* Addition: document `enabled` flag (#465)
* Addition: add 256 colors mode (#481)
* Addition: `enabled()` updates existing debug instances, add `destroy()` function (#440)
* Update: component: update "ms" to v2.0.0
* Update: separate the Node and Browser tests in Travis-CI
* Update: refactor Readme, fixed documentation, added "Namespace Colors" section, redid screenshots
* Update: separate Node.js and web browser examples for organization
* Update: update "browserify" to v14.4.0
* Fix: fix Readme typo (#473)
2.6.9 / 2017-09-22
==================
* remove ReDoS regexp in %o formatter (#504)
2.6.8 / 2017-05-18
==================
* Fix: Check for undefined on browser globals (#462, @marbemac)
2.6.7 / 2017-05-16
==================
* Fix: Update ms to 2.0.0 to fix regular expression denial of service vulnerability (#458, @hubdotcom)
* Fix: Inline extend function in node implementation (#452, @dougwilson)
* Docs: Fix typo (#455, @msasad)
2.6.5 / 2017-04-27
==================
* Fix: null reference check on window.documentElement.style.WebkitAppearance (#447, @thebigredgeek)
* Misc: clean up browser reference checks (#447, @thebigredgeek)
* Misc: add npm-debug.log to .gitignore (@thebigredgeek)
2.6.4 / 2017-04-20
==================
* Fix: bug that would occur if process.env.DEBUG is a non-string value. (#444, @LucianBuzzo)
* Chore: ignore bower.json in npm installations. (#437, @joaovieira)
* Misc: update "ms" to v0.7.3 (@tootallnate)
2.6.3 / 2017-03-13
==================
* Fix: Electron reference to `process.env.DEBUG` (#431, @paulcbetts)
* Docs: Changelog fix (@thebigredgeek)
2.6.2 / 2017-03-10
==================
* Fix: DEBUG_MAX_ARRAY_LENGTH (#420, @slavaGanzin)
* Docs: Add backers and sponsors from Open Collective (#422, @piamancini)
* Docs: Add Slackin invite badge (@tootallnate)
2.6.1 / 2017-02-10
==================
* Fix: Module's `export default` syntax fix for IE8 `Expected identifier` error
* Fix: Whitelist DEBUG_FD for values 1 and 2 only (#415, @pi0)
* Fix: IE8 "Expected identifier" error (#414, @vgoma)
* Fix: Namespaces would not disable once enabled (#409, @musikov)
2.6.0 / 2016-12-28
==================
* Fix: added better null pointer checks for browser useColors (@thebigredgeek)
* Improvement: removed explicit `window.debug` export (#404, @tootallnate)
* Improvement: deprecated `DEBUG_FD` environment variable (#405, @tootallnate)
2.5.2 / 2016-12-25
==================
* Fix: reference error on window within webworkers (#393, @KlausTrainer)
* Docs: fixed README typo (#391, @lurch)
* Docs: added notice about v3 api discussion (@thebigredgeek)
2.5.1 / 2016-12-20
==================
* Fix: babel-core compatibility
2.5.0 / 2016-12-20
==================
* Fix: wrong reference in bower file (@thebigredgeek)
* Fix: webworker compatibility (@thebigredgeek)
* Fix: output formatting issue (#388, @kribblo)
* Fix: babel-loader compatibility (#383, @escwald)
* Misc: removed built asset from repo and publications (@thebigredgeek)
* Misc: moved source files to /src (#378, @yamikuronue)
* Test: added karma integration and replaced babel with browserify for browser tests (#378, @yamikuronue)
* Test: coveralls integration (#378, @yamikuronue)
* Docs: simplified language in the opening paragraph (#373, @yamikuronue)
2.4.5 / 2016-12-17
==================
* Fix: `navigator` undefined in Rhino (#376, @jochenberger)
* Fix: custom log function (#379, @hsiliev)
* Improvement: bit of cleanup + linting fixes (@thebigredgeek)
* Improvement: rm non-maintainted `dist/` dir (#375, @freewil)
* Docs: simplified language in the opening paragraph. (#373, @yamikuronue)
2.4.4 / 2016-12-14
==================
* Fix: work around debug being loaded in preload scripts for electron (#368, @paulcbetts)
2.4.3 / 2016-12-14
==================
* Fix: navigation.userAgent error for react native (#364, @escwald)
2.4.2 / 2016-12-14
==================
* Fix: browser colors (#367, @tootallnate)
* Misc: travis ci integration (@thebigredgeek)
* Misc: added linting and testing boilerplate with sanity check (@thebigredgeek)
2.4.1 / 2016-12-13
==================
* Fix: typo that broke the package (#356)
2.4.0 / 2016-12-13
==================
* Fix: bower.json references unbuilt src entry point (#342, @justmatt)
* Fix: revert "handle regex special characters" (@tootallnate)
* Feature: configurable util.inspect()`options for NodeJS (#327, @tootallnate)
* Feature: %O`(big O) pretty-prints objects (#322, @tootallnate)
* Improvement: allow colors in workers (#335, @botverse)
* Improvement: use same color for same namespace. (#338, @lchenay)
2.3.3 / 2016-11-09
==================
* Fix: Catch `JSON.stringify()` errors (#195, Jovan Alleyne)
* Fix: Returning `localStorage` saved values (#331, Levi Thomason)
* Improvement: Don't create an empty object when no `process` (Nathan Rajlich)
2.3.2 / 2016-11-09
==================
* Fix: be super-safe in index.js as well (@TooTallNate)
* Fix: should check whether process exists (Tom Newby)
2.3.1 / 2016-11-09
==================
* Fix: Added electron compatibility (#324, @paulcbetts)
* Improvement: Added performance optimizations (@tootallnate)
* Readme: Corrected PowerShell environment variable example (#252, @gimre)
* Misc: Removed yarn lock file from source control (#321, @fengmk2)
2.3.0 / 2016-11-07
==================
* Fix: Consistent placement of ms diff at end of output (#215, @gorangajic)
* Fix: Escaping of regex special characters in namespace strings (#250, @zacronos)
* Fix: Fixed bug causing crash on react-native (#282, @vkarpov15)
* Feature: Enabled ES6+ compatible import via default export (#212 @bucaran)
* Feature: Added %O formatter to reflect Chrome's console.log capability (#279, @oncletom)
* Package: Update "ms" to 0.7.2 (#315, @DevSide)
* Package: removed superfluous version property from bower.json (#207 @kkirsche)
* Readme: fix USE_COLORS to DEBUG_COLORS
* Readme: Doc fixes for format string sugar (#269, @mlucool)
* Readme: Updated docs for DEBUG_FD and DEBUG_COLORS environment variables (#232, @mattlyons0)
* Readme: doc fixes for PowerShell (#271 #243, @exoticknight @unreadable)
* Readme: better docs for browser support (#224, @matthewmueller)
* Tooling: Added yarn integration for development (#317, @thebigredgeek)
* Misc: Renamed History.md to CHANGELOG.md (@thebigredgeek)
* Misc: Added license file (#226 #274, @CantemoInternal @sdaitzman)
* Misc: Updated contributors (@thebigredgeek)
2.2.0 / 2015-05-09
==================
* package: update "ms" to v0.7.1 (#202, @dougwilson)
* README: add logging to file example (#193, @DanielOchoa)
* README: fixed a typo (#191, @amir-s)
* browser: expose `storage` (#190, @stephenmathieson)
* Makefile: add a `distclean` target (#189, @stephenmathieson)
2.1.3 / 2015-03-13
==================
* Updated stdout/stderr example (#186)
* Updated example/stdout.js to match debug current behaviour
* Renamed example/stderr.js to stdout.js
* Update Readme.md (#184)
* replace high intensity foreground color for bold (#182, #183)
2.1.2 / 2015-03-01
==================
* dist: recompile
* update "ms" to v0.7.0
* package: update "browserify" to v9.0.3
* component: fix "ms.js" repo location
* changed bower package name
* updated documentation about using debug in a browser
* fix: security error on safari (#167, #168, @yields)
2.1.1 / 2014-12-29
==================
* browser: use `typeof` to check for `console` existence
* browser: check for `console.log` truthiness (fix IE 8/9)
* browser: add support for Chrome apps
* Readme: added Windows usage remarks
* Add `bower.json` to properly support bower install
2.1.0 / 2014-10-15
==================
* node: implement `DEBUG_FD` env variable support
* package: update "browserify" to v6.1.0
* package: add "license" field to package.json (#135, @panuhorsmalahti)
2.0.0 / 2014-09-01
==================
* package: update "browserify" to v5.11.0
* node: use stderr rather than stdout for logging (#29, @stephenmathieson)
1.0.4 / 2014-07-15
==================
* dist: recompile
* example: remove `console.info()` log usage
* example: add "Content-Type" UTF-8 header to browser example
* browser: place %c marker after the space character
* browser: reset the "content" color via `color: inherit`
* browser: add colors support for Firefox >= v31
* debug: prefer an instance `log()` function over the global one (#119)
* Readme: update documentation about styled console logs for FF v31 (#116, @wryk)
1.0.3 / 2014-07-09
==================
* Add support for multiple wildcards in namespaces (#122, @seegno)
* browser: fix lint
1.0.2 / 2014-06-10
==================
* browser: update color palette (#113, @gscottolson)
* common: make console logging function configurable (#108, @timoxley)
* node: fix %o colors on old node <= 0.8.x
* Makefile: find node path using shell/which (#109, @timoxley)
1.0.1 / 2014-06-06
==================
* browser: use `removeItem()` to clear localStorage
* browser, node: don't set DEBUG if namespaces is undefined (#107, @leedm777)
* package: add "contributors" section
* node: fix comment typo
* README: list authors
1.0.0 / 2014-06-04
==================
* make ms diff be global, not be scope
* debug: ignore empty strings in enable()
* node: make DEBUG_COLORS able to disable coloring
* *: export the `colors` array
* npmignore: don't publish the `dist` dir
* Makefile: refactor to use browserify
* package: add "browserify" as a dev dependency
* Readme: add Web Inspector Colors section
* node: reset terminal color for the debug content
* node: map "%o" to `util.inspect()`
* browser: map "%j" to `JSON.stringify()`
* debug: add custom "formatters"
* debug: use "ms" module for humanizing the diff
* Readme: add "bash" syntax highlighting
* browser: add Firebug color support
* browser: add colors for WebKit browsers
* node: apply log to `console`
* rewrite: abstract common logic for Node & browsers
* add .jshintrc file
0.8.1 / 2014-04-14
==================
* package: re-add the "component" section
0.8.0 / 2014-03-30
==================
* add `enable()` method for nodejs. Closes #27
* change from stderr to stdout
* remove unnecessary index.js file
0.7.4 / 2013-11-13
==================
* remove "browserify" key from package.json (fixes something in browserify)
0.7.3 / 2013-10-30
==================
* fix: catch localStorage security error when cookies are blocked (Chrome)
* add debug(err) support. Closes #46
* add .browser prop to package.json. Closes #42
0.7.2 / 2013-02-06
==================
* fix package.json
* fix: Mobile Safari (private mode) is broken with debug
* fix: Use unicode to send escape character to shell instead of octal to work with strict mode javascript
0.7.1 / 2013-02-05
==================
* add repository URL to package.json
* add DEBUG_COLORED to force colored output
* add browserify support
* fix component. Closes #24
0.7.0 / 2012-05-04
==================
* Added .component to package.json
* Added debug.component.js build
0.6.0 / 2012-03-16
==================
* Added support for "-" prefix in DEBUG [Vinay Pulim]
* Added `.enabled` flag to the node version [TooTallNate]
0.5.0 / 2012-02-02
==================
* Added: humanize diffs. Closes #8
* Added `debug.disable()` to the CS variant
* Removed padding. Closes #10
* Fixed: persist client-side variant again. Closes #9
0.4.0 / 2012-02-01
==================
* Added browser variant support for older browsers [TooTallNate]
* Added `debug.enable('project:*')` to browser variant [TooTallNate]
* Added padding to diff (moved it to the right)
0.3.0 / 2012-01-26
==================
* Added millisecond diff when isatty, otherwise UTC string
0.2.0 / 2012-01-22
==================
* Added wildcard support
0.1.0 / 2011-12-02
==================
* Added: remove colors unless stderr isatty [TooTallNate]
0.0.1 / 2010-01-03
==================
* Initial release
+19
View File
@@ -0,0 +1,19 @@
(The MIT License)
Copyright (c) 2014 TJ Holowaychuk <tj@vision-media.ca>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software
and associated documentation files (the 'Software'), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial
portions of the Software.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT
LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+437
View File
@@ -0,0 +1,437 @@
# debug
[![Build Status](https://travis-ci.org/visionmedia/debug.svg?branch=master)](https://travis-ci.org/visionmedia/debug) [![Coverage Status](https://coveralls.io/repos/github/visionmedia/debug/badge.svg?branch=master)](https://coveralls.io/github/visionmedia/debug?branch=master) [![Slack](https://visionmedia-community-slackin.now.sh/badge.svg)](https://visionmedia-community-slackin.now.sh/) [![OpenCollective](https://opencollective.com/debug/backers/badge.svg)](#backers)
[![OpenCollective](https://opencollective.com/debug/sponsors/badge.svg)](#sponsors)
<img width="647" src="https://user-images.githubusercontent.com/71256/29091486-fa38524c-7c37-11e7-895f-e7ec8e1039b6.png">
A tiny JavaScript debugging utility modelled after Node.js core's debugging
technique. Works in Node.js and web browsers.
## Installation
```bash
$ npm install debug
```
## Usage
`debug` exposes a function; simply pass this function the name of your module, and it will return a decorated version of `console.error` for you to pass debug statements to. This will allow you to toggle the debug output for different parts of your module as well as the module as a whole.
Example [_app.js_](./examples/node/app.js):
```js
var debug = require('debug')('http')
, http = require('http')
, name = 'My App';
// fake app
debug('booting %o', name);
http.createServer(function(req, res){
debug(req.method + ' ' + req.url);
res.end('hello\n');
}).listen(3000, function(){
debug('listening');
});
// fake worker of some kind
require('./worker');
```
Example [_worker.js_](./examples/node/worker.js):
```js
var a = require('debug')('worker:a')
, b = require('debug')('worker:b');
function work() {
a('doing lots of uninteresting work');
setTimeout(work, Math.random() * 1000);
}
work();
function workb() {
b('doing some work');
setTimeout(workb, Math.random() * 2000);
}
workb();
```
The `DEBUG` environment variable is then used to enable these based on space or
comma-delimited names.
Here are some examples:
<img width="647" alt="screen shot 2017-08-08 at 12 53 04 pm" src="https://user-images.githubusercontent.com/71256/29091703-a6302cdc-7c38-11e7-8304-7c0b3bc600cd.png">
<img width="647" alt="screen shot 2017-08-08 at 12 53 38 pm" src="https://user-images.githubusercontent.com/71256/29091700-a62a6888-7c38-11e7-800b-db911291ca2b.png">
<img width="647" alt="screen shot 2017-08-08 at 12 53 25 pm" src="https://user-images.githubusercontent.com/71256/29091701-a62ea114-7c38-11e7-826a-2692bedca740.png">
#### Windows command prompt notes
##### CMD
On Windows the environment variable is set using the `set` command.
```cmd
set DEBUG=*,-not_this
```
Example:
```cmd
set DEBUG=* & node app.js
```
##### PowerShell (VS Code default)
PowerShell uses different syntax to set environment variables.
```cmd
$env:DEBUG = "*,-not_this"
```
Example:
```cmd
$env:DEBUG='app';node app.js
```
Then, run the program to be debugged as usual.
npm script example:
```js
"windowsDebug": "@powershell -Command $env:DEBUG='*';node app.js",
```
## Namespace Colors
Every debug instance has a color generated for it based on its namespace name.
This helps when visually parsing the debug output to identify which debug instance
a debug line belongs to.
#### Node.js
In Node.js, colors are enabled when stderr is a TTY. You also _should_ install
the [`supports-color`](https://npmjs.org/supports-color) module alongside debug,
otherwise debug will only use a small handful of basic colors.
<img width="521" src="https://user-images.githubusercontent.com/71256/29092181-47f6a9e6-7c3a-11e7-9a14-1928d8a711cd.png">
#### Web Browser
Colors are also enabled on "Web Inspectors" that understand the `%c` formatting
option. These are WebKit web inspectors, Firefox ([since version
31](https://hacks.mozilla.org/2014/05/editable-box-model-multiple-selection-sublime-text-keys-much-more-firefox-developer-tools-episode-31/))
and the Firebug plugin for Firefox (any version).
<img width="524" src="https://user-images.githubusercontent.com/71256/29092033-b65f9f2e-7c39-11e7-8e32-f6f0d8e865c1.png">
## Millisecond diff
When actively developing an application it can be useful to see when the time spent between one `debug()` call and the next. Suppose for example you invoke `debug()` before requesting a resource, and after as well, the "+NNNms" will show you how much time was spent between calls.
<img width="647" src="https://user-images.githubusercontent.com/71256/29091486-fa38524c-7c37-11e7-895f-e7ec8e1039b6.png">
When stdout is not a TTY, `Date#toISOString()` is used, making it more useful for logging the debug information as shown below:
<img width="647" src="https://user-images.githubusercontent.com/71256/29091956-6bd78372-7c39-11e7-8c55-c948396d6edd.png">
## Conventions
If you're using this in one or more of your libraries, you _should_ use the name of your library so that developers may toggle debugging as desired without guessing names. If you have more than one debuggers you _should_ prefix them with your library name and use ":" to separate features. For example "bodyParser" from Connect would then be "connect:bodyParser". If you append a "*" to the end of your name, it will always be enabled regardless of the setting of the DEBUG environment variable. You can then use it for normal output as well as debug output.
## Wildcards
The `*` character may be used as a wildcard. Suppose for example your library has
debuggers named "connect:bodyParser", "connect:compress", "connect:session",
instead of listing all three with
`DEBUG=connect:bodyParser,connect:compress,connect:session`, you may simply do
`DEBUG=connect:*`, or to run everything using this module simply use `DEBUG=*`.
You can also exclude specific debuggers by prefixing them with a "-" character.
For example, `DEBUG=*,-connect:*` would include all debuggers except those
starting with "connect:".
## Environment Variables
When running through Node.js, you can set a few environment variables that will
change the behavior of the debug logging:
| Name | Purpose |
|-----------|-------------------------------------------------|
| `DEBUG` | Enables/disables specific debugging namespaces. |
| `DEBUG_HIDE_DATE` | Hide date from debug output (non-TTY). |
| `DEBUG_COLORS`| Whether or not to use colors in the debug output. |
| `DEBUG_DEPTH` | Object inspection depth. |
| `DEBUG_SHOW_HIDDEN` | Shows hidden properties on inspected objects. |
__Note:__ The environment variables beginning with `DEBUG_` end up being
converted into an Options object that gets used with `%o`/`%O` formatters.
See the Node.js documentation for
[`util.inspect()`](https://nodejs.org/api/util.html#util_util_inspect_object_options)
for the complete list.
## Formatters
Debug uses [printf-style](https://wikipedia.org/wiki/Printf_format_string) formatting.
Below are the officially supported formatters:
| Formatter | Representation |
|-----------|----------------|
| `%O` | Pretty-print an Object on multiple lines. |
| `%o` | Pretty-print an Object all on a single line. |
| `%s` | String. |
| `%d` | Number (both integer and float). |
| `%j` | JSON. Replaced with the string '[Circular]' if the argument contains circular references. |
| `%%` | Single percent sign ('%'). This does not consume an argument. |
### Custom formatters
You can add custom formatters by extending the `debug.formatters` object.
For example, if you wanted to add support for rendering a Buffer as hex with
`%h`, you could do something like:
```js
const createDebug = require('debug')
createDebug.formatters.h = (v) => {
return v.toString('hex')
}
// …elsewhere
const debug = createDebug('foo')
debug('this is hex: %h', new Buffer('hello world'))
// foo this is hex: 68656c6c6f20776f726c6421 +0ms
```
## Browser Support
You can build a browser-ready script using [browserify](https://github.com/substack/node-browserify),
or just use the [browserify-as-a-service](https://wzrd.in/) [build](https://wzrd.in/standalone/debug@latest),
if you don't want to build it yourself.
Debug's enable state is currently persisted by `localStorage`.
Consider the situation shown below where you have `worker:a` and `worker:b`,
and wish to debug both. You can enable this using `localStorage.debug`:
```js
localStorage.debug = 'worker:*'
```
And then refresh the page.
```js
a = debug('worker:a');
b = debug('worker:b');
setInterval(function(){
a('doing some work');
}, 1000);
setInterval(function(){
b('doing some work');
}, 1200);
```
## Output streams
By default `debug` will log to stderr, however this can be configured per-namespace by overriding the `log` method:
Example [_stdout.js_](./examples/node/stdout.js):
```js
var debug = require('debug');
var error = debug('app:error');
// by default stderr is used
error('goes to stderr!');
var log = debug('app:log');
// set this namespace to log via console.log
log.log = console.log.bind(console); // don't forget to bind to console!
log('goes to stdout');
error('still goes to stderr!');
// set all output to go via console.info
// overrides all per-namespace log settings
debug.log = console.info.bind(console);
error('now goes to stdout via console.info');
log('still goes to stdout, but via console.info now');
```
## Extend
You can simply extend debugger
```js
const log = require('debug')('auth');
//creates new debug instance with extended namespace
const logSign = log.extend('sign');
const logLogin = log.extend('login');
log('hello'); // auth hello
logSign('hello'); //auth:sign hello
logLogin('hello'); //auth:login hello
```
## Set dynamically
You can also enable debug dynamically by calling the `enable()` method :
```js
let debug = require('debug');
console.log(1, debug.enabled('test'));
debug.enable('test');
console.log(2, debug.enabled('test'));
debug.disable();
console.log(3, debug.enabled('test'));
```
print :
```
1 false
2 true
3 false
```
Usage :
`enable(namespaces)`
`namespaces` can include modes separated by a colon and wildcards.
Note that calling `enable()` completely overrides previously set DEBUG variable :
```
$ DEBUG=foo node -e 'var dbg = require("debug"); dbg.enable("bar"); console.log(dbg.enabled("foo"))'
=> false
```
## Checking whether a debug target is enabled
After you've created a debug instance, you can determine whether or not it is
enabled by checking the `enabled` property:
```javascript
const debug = require('debug')('http');
if (debug.enabled) {
// do stuff...
}
```
You can also manually toggle this property to force the debug instance to be
enabled or disabled.
## Authors
- TJ Holowaychuk
- Nathan Rajlich
- Andrew Rhyne
## Backers
Support us with a monthly donation and help us continue our activities. [[Become a backer](https://opencollective.com/debug#backer)]
<a href="https://opencollective.com/debug/backer/0/website" target="_blank"><img src="https://opencollective.com/debug/backer/0/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/1/website" target="_blank"><img src="https://opencollective.com/debug/backer/1/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/2/website" target="_blank"><img src="https://opencollective.com/debug/backer/2/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/3/website" target="_blank"><img src="https://opencollective.com/debug/backer/3/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/4/website" target="_blank"><img src="https://opencollective.com/debug/backer/4/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/5/website" target="_blank"><img src="https://opencollective.com/debug/backer/5/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/6/website" target="_blank"><img src="https://opencollective.com/debug/backer/6/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/7/website" target="_blank"><img src="https://opencollective.com/debug/backer/7/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/8/website" target="_blank"><img src="https://opencollective.com/debug/backer/8/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/9/website" target="_blank"><img src="https://opencollective.com/debug/backer/9/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/10/website" target="_blank"><img src="https://opencollective.com/debug/backer/10/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/11/website" target="_blank"><img src="https://opencollective.com/debug/backer/11/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/12/website" target="_blank"><img src="https://opencollective.com/debug/backer/12/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/13/website" target="_blank"><img src="https://opencollective.com/debug/backer/13/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/14/website" target="_blank"><img src="https://opencollective.com/debug/backer/14/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/15/website" target="_blank"><img src="https://opencollective.com/debug/backer/15/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/16/website" target="_blank"><img src="https://opencollective.com/debug/backer/16/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/17/website" target="_blank"><img src="https://opencollective.com/debug/backer/17/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/18/website" target="_blank"><img src="https://opencollective.com/debug/backer/18/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/19/website" target="_blank"><img src="https://opencollective.com/debug/backer/19/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/20/website" target="_blank"><img src="https://opencollective.com/debug/backer/20/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/21/website" target="_blank"><img src="https://opencollective.com/debug/backer/21/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/22/website" target="_blank"><img src="https://opencollective.com/debug/backer/22/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/23/website" target="_blank"><img src="https://opencollective.com/debug/backer/23/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/24/website" target="_blank"><img src="https://opencollective.com/debug/backer/24/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/25/website" target="_blank"><img src="https://opencollective.com/debug/backer/25/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/26/website" target="_blank"><img src="https://opencollective.com/debug/backer/26/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/27/website" target="_blank"><img src="https://opencollective.com/debug/backer/27/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/28/website" target="_blank"><img src="https://opencollective.com/debug/backer/28/avatar.svg"></a>
<a href="https://opencollective.com/debug/backer/29/website" target="_blank"><img src="https://opencollective.com/debug/backer/29/avatar.svg"></a>
## Sponsors
Become a sponsor and get your logo on our README on Github with a link to your site. [[Become a sponsor](https://opencollective.com/debug#sponsor)]
<a href="https://opencollective.com/debug/sponsor/0/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/0/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/1/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/1/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/2/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/2/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/3/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/3/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/4/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/4/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/5/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/5/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/6/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/6/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/7/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/7/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/8/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/8/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/9/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/9/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/10/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/10/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/11/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/11/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/12/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/12/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/13/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/13/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/14/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/14/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/15/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/15/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/16/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/16/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/17/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/17/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/18/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/18/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/19/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/19/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/20/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/20/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/21/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/21/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/22/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/22/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/23/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/23/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/24/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/24/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/25/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/25/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/26/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/26/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/27/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/27/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/28/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/28/avatar.svg"></a>
<a href="https://opencollective.com/debug/sponsor/29/website" target="_blank"><img src="https://opencollective.com/debug/sponsor/29/avatar.svg"></a>
## License
(The MIT License)
Copyright (c) 2014-2017 TJ Holowaychuk &lt;tj@vision-media.ca&gt;
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1 @@
module.exports = require('./src/node');
@@ -0,0 +1,51 @@
{
"name": "debug",
"version": "3.2.7",
"repository": {
"type": "git",
"url": "git://github.com/visionmedia/debug.git"
},
"description": "small debugging utility",
"keywords": [
"debug",
"log",
"debugger"
],
"files": [
"src",
"node.js",
"dist/debug.js",
"LICENSE",
"README.md"
],
"author": "TJ Holowaychuk <tj@vision-media.ca>",
"contributors": [
"Nathan Rajlich <nathan@tootallnate.net> (http://n8.io)",
"Andrew Rhyne <rhyneandrew@gmail.com>"
],
"license": "MIT",
"dependencies": {
"ms": "^2.1.1"
},
"devDependencies": {
"@babel/cli": "^7.0.0",
"@babel/core": "^7.0.0",
"@babel/preset-env": "^7.0.0",
"browserify": "14.4.0",
"chai": "^3.5.0",
"concurrently": "^3.1.0",
"coveralls": "^3.0.2",
"istanbul": "^0.4.5",
"karma": "^3.0.0",
"karma-chai": "^0.1.0",
"karma-mocha": "^1.3.0",
"karma-phantomjs-launcher": "^1.0.2",
"mocha": "^5.2.0",
"mocha-lcov-reporter": "^1.2.0",
"rimraf": "^2.5.4",
"xo": "^0.23.0"
},
"main": "./src/index.js",
"browser": "./src/browser.js",
"unpkg": "./dist/debug.js"
}
@@ -0,0 +1,180 @@
"use strict";
function _typeof(obj) { if (typeof Symbol === "function" && typeof Symbol.iterator === "symbol") { _typeof = function _typeof(obj) { return typeof obj; }; } else { _typeof = function _typeof(obj) { return obj && typeof Symbol === "function" && obj.constructor === Symbol && obj !== Symbol.prototype ? "symbol" : typeof obj; }; } return _typeof(obj); }
/* eslint-env browser */
/**
* This is the web browser implementation of `debug()`.
*/
exports.log = log;
exports.formatArgs = formatArgs;
exports.save = save;
exports.load = load;
exports.useColors = useColors;
exports.storage = localstorage();
/**
* Colors.
*/
exports.colors = ['#0000CC', '#0000FF', '#0033CC', '#0033FF', '#0066CC', '#0066FF', '#0099CC', '#0099FF', '#00CC00', '#00CC33', '#00CC66', '#00CC99', '#00CCCC', '#00CCFF', '#3300CC', '#3300FF', '#3333CC', '#3333FF', '#3366CC', '#3366FF', '#3399CC', '#3399FF', '#33CC00', '#33CC33', '#33CC66', '#33CC99', '#33CCCC', '#33CCFF', '#6600CC', '#6600FF', '#6633CC', '#6633FF', '#66CC00', '#66CC33', '#9900CC', '#9900FF', '#9933CC', '#9933FF', '#99CC00', '#99CC33', '#CC0000', '#CC0033', '#CC0066', '#CC0099', '#CC00CC', '#CC00FF', '#CC3300', '#CC3333', '#CC3366', '#CC3399', '#CC33CC', '#CC33FF', '#CC6600', '#CC6633', '#CC9900', '#CC9933', '#CCCC00', '#CCCC33', '#FF0000', '#FF0033', '#FF0066', '#FF0099', '#FF00CC', '#FF00FF', '#FF3300', '#FF3333', '#FF3366', '#FF3399', '#FF33CC', '#FF33FF', '#FF6600', '#FF6633', '#FF9900', '#FF9933', '#FFCC00', '#FFCC33'];
/**
* Currently only WebKit-based Web Inspectors, Firefox >= v31,
* and the Firebug extension (any Firefox version) are known
* to support "%c" CSS customizations.
*
* TODO: add a `localStorage` variable to explicitly enable/disable colors
*/
// eslint-disable-next-line complexity
function useColors() {
// NB: In an Electron preload script, document will be defined but not fully
// initialized. Since we know we're in Chrome, we'll just detect this case
// explicitly
if (typeof window !== 'undefined' && window.process && (window.process.type === 'renderer' || window.process.__nwjs)) {
return true;
} // Internet Explorer and Edge do not support colors.
if (typeof navigator !== 'undefined' && navigator.userAgent && navigator.userAgent.toLowerCase().match(/(edge|trident)\/(\d+)/)) {
return false;
} // Is webkit? http://stackoverflow.com/a/16459606/376773
// document is undefined in react-native: https://github.com/facebook/react-native/pull/1632
return typeof document !== 'undefined' && document.documentElement && document.documentElement.style && document.documentElement.style.WebkitAppearance || // Is firebug? http://stackoverflow.com/a/398120/376773
typeof window !== 'undefined' && window.console && (window.console.firebug || window.console.exception && window.console.table) || // Is firefox >= v31?
// https://developer.mozilla.org/en-US/docs/Tools/Web_Console#Styling_messages
typeof navigator !== 'undefined' && navigator.userAgent && navigator.userAgent.toLowerCase().match(/firefox\/(\d+)/) && parseInt(RegExp.$1, 10) >= 31 || // Double check webkit in userAgent just in case we are in a worker
typeof navigator !== 'undefined' && navigator.userAgent && navigator.userAgent.toLowerCase().match(/applewebkit\/(\d+)/);
}
/**
* Colorize log arguments if enabled.
*
* @api public
*/
function formatArgs(args) {
args[0] = (this.useColors ? '%c' : '') + this.namespace + (this.useColors ? ' %c' : ' ') + args[0] + (this.useColors ? '%c ' : ' ') + '+' + module.exports.humanize(this.diff);
if (!this.useColors) {
return;
}
var c = 'color: ' + this.color;
args.splice(1, 0, c, 'color: inherit'); // The final "%c" is somewhat tricky, because there could be other
// arguments passed either before or after the %c, so we need to
// figure out the correct index to insert the CSS into
var index = 0;
var lastC = 0;
args[0].replace(/%[a-zA-Z%]/g, function (match) {
if (match === '%%') {
return;
}
index++;
if (match === '%c') {
// We only are interested in the *last* %c
// (the user may have provided their own)
lastC = index;
}
});
args.splice(lastC, 0, c);
}
/**
* Invokes `console.log()` when available.
* No-op when `console.log` is not a "function".
*
* @api public
*/
function log() {
var _console;
// This hackery is required for IE8/9, where
// the `console.log` function doesn't have 'apply'
return (typeof console === "undefined" ? "undefined" : _typeof(console)) === 'object' && console.log && (_console = console).log.apply(_console, arguments);
}
/**
* Save `namespaces`.
*
* @param {String} namespaces
* @api private
*/
function save(namespaces) {
try {
if (namespaces) {
exports.storage.setItem('debug', namespaces);
} else {
exports.storage.removeItem('debug');
}
} catch (error) {// Swallow
// XXX (@Qix-) should we be logging these?
}
}
/**
* Load `namespaces`.
*
* @return {String} returns the previously persisted debug modes
* @api private
*/
function load() {
var r;
try {
r = exports.storage.getItem('debug');
} catch (error) {} // Swallow
// XXX (@Qix-) should we be logging these?
// If debug isn't set in LS, and we're in Electron, try to load $DEBUG
if (!r && typeof process !== 'undefined' && 'env' in process) {
r = process.env.DEBUG;
}
return r;
}
/**
* Localstorage attempts to return the localstorage.
*
* This is necessary because safari throws
* when a user disables cookies/localstorage
* and you attempt to access it.
*
* @return {LocalStorage}
* @api private
*/
function localstorage() {
try {
// TVMLKit (Apple TV JS Runtime) does not have a window object, just localStorage in the global context
// The Browser also has localStorage in the global context.
return localStorage;
} catch (error) {// Swallow
// XXX (@Qix-) should we be logging these?
}
}
module.exports = require('./common')(exports);
var formatters = module.exports.formatters;
/**
* Map %j to `JSON.stringify()`, since no Web Inspectors do that by default.
*/
formatters.j = function (v) {
try {
return JSON.stringify(v);
} catch (error) {
return '[UnexpectedJSONParseError]: ' + error.message;
}
};
@@ -0,0 +1,249 @@
"use strict";
/**
* This is the common logic for both the Node.js and web browser
* implementations of `debug()`.
*/
function setup(env) {
createDebug.debug = createDebug;
createDebug.default = createDebug;
createDebug.coerce = coerce;
createDebug.disable = disable;
createDebug.enable = enable;
createDebug.enabled = enabled;
createDebug.humanize = require('ms');
Object.keys(env).forEach(function (key) {
createDebug[key] = env[key];
});
/**
* Active `debug` instances.
*/
createDebug.instances = [];
/**
* The currently active debug mode names, and names to skip.
*/
createDebug.names = [];
createDebug.skips = [];
/**
* Map of special "%n" handling functions, for the debug "format" argument.
*
* Valid key names are a single, lower or upper-case letter, i.e. "n" and "N".
*/
createDebug.formatters = {};
/**
* Selects a color for a debug namespace
* @param {String} namespace The namespace string for the for the debug instance to be colored
* @return {Number|String} An ANSI color code for the given namespace
* @api private
*/
function selectColor(namespace) {
var hash = 0;
for (var i = 0; i < namespace.length; i++) {
hash = (hash << 5) - hash + namespace.charCodeAt(i);
hash |= 0; // Convert to 32bit integer
}
return createDebug.colors[Math.abs(hash) % createDebug.colors.length];
}
createDebug.selectColor = selectColor;
/**
* Create a debugger with the given `namespace`.
*
* @param {String} namespace
* @return {Function}
* @api public
*/
function createDebug(namespace) {
var prevTime;
function debug() {
// Disabled?
if (!debug.enabled) {
return;
}
for (var _len = arguments.length, args = new Array(_len), _key = 0; _key < _len; _key++) {
args[_key] = arguments[_key];
}
var self = debug; // Set `diff` timestamp
var curr = Number(new Date());
var ms = curr - (prevTime || curr);
self.diff = ms;
self.prev = prevTime;
self.curr = curr;
prevTime = curr;
args[0] = createDebug.coerce(args[0]);
if (typeof args[0] !== 'string') {
// Anything else let's inspect with %O
args.unshift('%O');
} // Apply any `formatters` transformations
var index = 0;
args[0] = args[0].replace(/%([a-zA-Z%])/g, function (match, format) {
// If we encounter an escaped % then don't increase the array index
if (match === '%%') {
return match;
}
index++;
var formatter = createDebug.formatters[format];
if (typeof formatter === 'function') {
var val = args[index];
match = formatter.call(self, val); // Now we need to remove `args[index]` since it's inlined in the `format`
args.splice(index, 1);
index--;
}
return match;
}); // Apply env-specific formatting (colors, etc.)
createDebug.formatArgs.call(self, args);
var logFn = self.log || createDebug.log;
logFn.apply(self, args);
}
debug.namespace = namespace;
debug.enabled = createDebug.enabled(namespace);
debug.useColors = createDebug.useColors();
debug.color = selectColor(namespace);
debug.destroy = destroy;
debug.extend = extend; // Debug.formatArgs = formatArgs;
// debug.rawLog = rawLog;
// env-specific initialization logic for debug instances
if (typeof createDebug.init === 'function') {
createDebug.init(debug);
}
createDebug.instances.push(debug);
return debug;
}
function destroy() {
var index = createDebug.instances.indexOf(this);
if (index !== -1) {
createDebug.instances.splice(index, 1);
return true;
}
return false;
}
function extend(namespace, delimiter) {
return createDebug(this.namespace + (typeof delimiter === 'undefined' ? ':' : delimiter) + namespace);
}
/**
* Enables a debug mode by namespaces. This can include modes
* separated by a colon and wildcards.
*
* @param {String} namespaces
* @api public
*/
function enable(namespaces) {
createDebug.save(namespaces);
createDebug.names = [];
createDebug.skips = [];
var i;
var split = (typeof namespaces === 'string' ? namespaces : '').split(/[\s,]+/);
var len = split.length;
for (i = 0; i < len; i++) {
if (!split[i]) {
// ignore empty strings
continue;
}
namespaces = split[i].replace(/\*/g, '.*?');
if (namespaces[0] === '-') {
createDebug.skips.push(new RegExp('^' + namespaces.substr(1) + '$'));
} else {
createDebug.names.push(new RegExp('^' + namespaces + '$'));
}
}
for (i = 0; i < createDebug.instances.length; i++) {
var instance = createDebug.instances[i];
instance.enabled = createDebug.enabled(instance.namespace);
}
}
/**
* Disable debug output.
*
* @api public
*/
function disable() {
createDebug.enable('');
}
/**
* Returns true if the given mode name is enabled, false otherwise.
*
* @param {String} name
* @return {Boolean}
* @api public
*/
function enabled(name) {
if (name[name.length - 1] === '*') {
return true;
}
var i;
var len;
for (i = 0, len = createDebug.skips.length; i < len; i++) {
if (createDebug.skips[i].test(name)) {
return false;
}
}
for (i = 0, len = createDebug.names.length; i < len; i++) {
if (createDebug.names[i].test(name)) {
return true;
}
}
return false;
}
/**
* Coerce `val`.
*
* @param {Mixed} val
* @return {Mixed}
* @api private
*/
function coerce(val) {
if (val instanceof Error) {
return val.stack || val.message;
}
return val;
}
createDebug.enable(createDebug.load());
return createDebug;
}
module.exports = setup;
@@ -0,0 +1,12 @@
"use strict";
/**
* Detect Electron renderer / nwjs process, which is node, but we should
* treat as a browser.
*/
if (typeof process === 'undefined' || process.type === 'renderer' || process.browser === true || process.__nwjs) {
module.exports = require('./browser.js');
} else {
module.exports = require('./node.js');
}
@@ -0,0 +1,177 @@
"use strict";
/**
* Module dependencies.
*/
var tty = require('tty');
var util = require('util');
/**
* This is the Node.js implementation of `debug()`.
*/
exports.init = init;
exports.log = log;
exports.formatArgs = formatArgs;
exports.save = save;
exports.load = load;
exports.useColors = useColors;
/**
* Colors.
*/
exports.colors = [6, 2, 3, 4, 5, 1];
try {
// Optional dependency (as in, doesn't need to be installed, NOT like optionalDependencies in package.json)
// eslint-disable-next-line import/no-extraneous-dependencies
var supportsColor = require('supports-color');
if (supportsColor && (supportsColor.stderr || supportsColor).level >= 2) {
exports.colors = [20, 21, 26, 27, 32, 33, 38, 39, 40, 41, 42, 43, 44, 45, 56, 57, 62, 63, 68, 69, 74, 75, 76, 77, 78, 79, 80, 81, 92, 93, 98, 99, 112, 113, 128, 129, 134, 135, 148, 149, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 178, 179, 184, 185, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 214, 215, 220, 221];
}
} catch (error) {} // Swallow - we only care if `supports-color` is available; it doesn't have to be.
/**
* Build up the default `inspectOpts` object from the environment variables.
*
* $ DEBUG_COLORS=no DEBUG_DEPTH=10 DEBUG_SHOW_HIDDEN=enabled node script.js
*/
exports.inspectOpts = Object.keys(process.env).filter(function (key) {
return /^debug_/i.test(key);
}).reduce(function (obj, key) {
// Camel-case
var prop = key.substring(6).toLowerCase().replace(/_([a-z])/g, function (_, k) {
return k.toUpperCase();
}); // Coerce string value into JS value
var val = process.env[key];
if (/^(yes|on|true|enabled)$/i.test(val)) {
val = true;
} else if (/^(no|off|false|disabled)$/i.test(val)) {
val = false;
} else if (val === 'null') {
val = null;
} else {
val = Number(val);
}
obj[prop] = val;
return obj;
}, {});
/**
* Is stdout a TTY? Colored output is enabled when `true`.
*/
function useColors() {
return 'colors' in exports.inspectOpts ? Boolean(exports.inspectOpts.colors) : tty.isatty(process.stderr.fd);
}
/**
* Adds ANSI color escape codes if enabled.
*
* @api public
*/
function formatArgs(args) {
var name = this.namespace,
useColors = this.useColors;
if (useColors) {
var c = this.color;
var colorCode = "\x1B[3" + (c < 8 ? c : '8;5;' + c);
var prefix = " ".concat(colorCode, ";1m").concat(name, " \x1B[0m");
args[0] = prefix + args[0].split('\n').join('\n' + prefix);
args.push(colorCode + 'm+' + module.exports.humanize(this.diff) + "\x1B[0m");
} else {
args[0] = getDate() + name + ' ' + args[0];
}
}
function getDate() {
if (exports.inspectOpts.hideDate) {
return '';
}
return new Date().toISOString() + ' ';
}
/**
* Invokes `util.format()` with the specified arguments and writes to stderr.
*/
function log() {
return process.stderr.write(util.format.apply(util, arguments) + '\n');
}
/**
* Save `namespaces`.
*
* @param {String} namespaces
* @api private
*/
function save(namespaces) {
if (namespaces) {
process.env.DEBUG = namespaces;
} else {
// If you set a process.env field to null or undefined, it gets cast to the
// string 'null' or 'undefined'. Just delete instead.
delete process.env.DEBUG;
}
}
/**
* Load `namespaces`.
*
* @return {String} returns the previously persisted debug modes
* @api private
*/
function load() {
return process.env.DEBUG;
}
/**
* Init logic for `debug` instances.
*
* Create a new `inspectOpts` object in case `useColors` is set
* differently for a particular `debug` instance.
*/
function init(debug) {
debug.inspectOpts = {};
var keys = Object.keys(exports.inspectOpts);
for (var i = 0; i < keys.length; i++) {
debug.inspectOpts[keys[i]] = exports.inspectOpts[keys[i]];
}
}
module.exports = require('./common')(exports);
var formatters = module.exports.formatters;
/**
* Map %o to `util.inspect()`, all on a single line.
*/
formatters.o = function (v) {
this.inspectOpts.colors = this.useColors;
return util.inspect(v, this.inspectOpts)
.split('\n')
.map(function (str) { return str.trim(); })
.join(' ');
};
/**
* Map %O to `util.inspect()`, allowing multiple lines if needed.
*/
formatters.O = function (v) {
this.inspectOpts.colors = this.useColors;
return util.inspect(v, this.inspectOpts);
};
@@ -0,0 +1,27 @@
name: Node CI
on: [push]
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [4.x, 6.x, 8.x, 10.x, 12.x]
steps:
- uses: actions/checkout@v1
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v1
with:
node-version: ${{ matrix.node-version }}
- name: npm install, build, and test
run: |
npm install
mkdir -p test/keys
openssl genrsa -out test/keys/ssl.key 2048
openssl req -new -key test/keys/ssl.key -subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=www.example.com" -x509 -days 999 -out test/keys/ssl.cert
npm run build --if-present
npm test
env:
CI: true
@@ -0,0 +1,612 @@
Needle
======
[![NPM](https://nodei.co/npm/needle.png)](https://nodei.co/npm/needle/)
The leanest and most handsome HTTP client in the Nodelands.
```js
var needle = require('needle');
needle.get('http://www.google.com', function(error, response) {
if (!error && response.statusCode == 200)
console.log(response.body);
});
```
Callbacks not floating your boat? Needle got your back.
``` js
var data = {
file: '/home/johnlennon/walrus.png',
content_type: 'image/png'
};
// the callback is optional, and needle returns a `readableStream` object
// that triggers a 'done' event when the request/response process is complete.
needle
.post('https://my.server.com/foo', data, { multipart: true })
.on('readable', function() { /* eat your chunks */ })
.on('done', function(err) {
console.log('Ready-o!');
})
```
From version 2.0.x up, Promises are also supported. Just call `needle()` directly and you'll get a native Promise object.
```js
needle('put', 'https://hacking.the.gibson/login', { password: 'god' }, { json: true })
.then(function(response) {
return doSomethingWith(response)
})
.catch(function(err) {
console.log('Call the locksmith!')
})
```
With only two real dependencies, Needle supports:
- HTTP/HTTPS requests, with the usual verbs you would expect
- All of Node's native TLS options, such as 'rejectUnauthorized' (see below)
- Basic & Digest authentication with auto-detection
- Multipart form-data (e.g. file uploads)
- HTTP Proxy forwarding, optionally with authentication
- Streaming gzip, deflate, and brotli decompression
- Automatic XML & JSON parsing
- 301/302/303 redirect following, with fine-grained tuning, and
- Streaming non-UTF-8 charset decoding, via `iconv-lite`
And yes, Mr. Wayne, it does come in black.
This makes Needle an ideal alternative for performing quick HTTP requests in Node, either for API interaction, downloading or uploading streams of data, and so on. If you need OAuth, AWS support or anything fancier, you should check out mikeal's request module.
Install
-------
```
$ npm install needle
```
Usage
-----
```js
// using promises
needle('get', 'https://server.com/posts/123')
.then(function(resp) {
// ...
})
.catch(function(err) {
// ...
});
// with callback
needle.get('ifconfig.me/all.json', function(error, response, body) {
if (error) throw error;
// body is an alias for `response.body`,
// that in this case holds a JSON-decoded object.
console.log(body.ip_addr);
});
// no callback, using streams
needle.get('https://google.com/images/logo.png')
.pipe(fs.createWriteStream('logo.png'))
.on('done', function(err) {
console.log('Pipe finished!');
});
```
As you can see, you can use Needle with Promises or without them. When using Promises or when a callback is passed, the response's body will be buffered and written to `response.body`, and the callback will be fired when all of the data has been collected and processed (e.g. decompressed, decoded and/or parsed).
When no callback is passed, however, the buffering logic will be skipped but the response stream will still go through Needle's processing pipeline, so you get all the benefits of post-processing while keeping the streamishness we all love from Node.
Response pipeline
-----------------
Depending on the response's Content-Type, Needle will either attempt to parse JSON or XML streams, or, if a text response was received, will ensure that the final encoding you get is UTF-8.
You can also request a gzip/deflated/brotli response, which, if sent by the server, will be processed before parsing or decoding is performed. (Note: brotli is only supported on Node 10.16.0 or above, and will not be requested or processed on earlier versions.)
```js
needle.get('http://stackoverflow.com/feeds', { compressed: true }, function(err, resp) {
console.log(resp.body); // this little guy won't be a Gzipped binary blob
// but a nice object containing all the latest entries
});
```
Or in anti-callback mode, using a few other options:
```js
var options = {
compressed : true, // sets 'Accept-Encoding' to 'gzip, deflate, br'
follow_max : 5, // follow up to five redirects
rejectUnauthorized : true // verify SSL certificate
}
var stream = needle.get('https://backend.server.com/everything.html', options);
// read the chunks from the 'readable' event, so the stream gets consumed.
stream.on('readable', function() {
while (data = this.read()) {
console.log(data.toString());
}
})
stream.on('done', function(err) {
// if our request had an error, our 'done' event will tell us.
if (!err) console.log('Great success!');
})
```
API
---
### needle(method, url[, data][, options][, callback]) `(> 2.0.x)`
Calling `needle()` directly returns a Promise. Besides `method` and `url`, all parameters are optional, although when sending a `post`, `put` or `patch` request you will get an error if `data` is not present.
```js
needle('get', 'http://some.url.com')
.then(function(resp) { console.log(resp.body) })
.catch(function(err) { console.error(err) })
```
Except from the above, all of Needle's request methods return a Readable stream, and both `options` and `callback` are optional. If passed, the callback will return three arguments: `error`, `response` and `body`, which is basically an alias for `response.body`.
### needle.head(url[, options][, callback])
```js
needle.head('https://my.backend.server.com', {
open_timeout: 5000 // if we're not able to open a connection in 5 seconds, boom.
}, function(err, resp) {
if (err)
console.log('Shoot! Something is wrong: ' + err.message)
else
console.log('Yup, still alive.')
})
```
### needle.get(url[, options][, callback])
```js
needle.get('google.com/search?q=syd+barrett', function(err, resp) {
// if no http:// is found, Needle will automagically prepend it.
});
```
### needle.post(url, data[, options][, callback])
```js
var options = {
headers: { 'X-Custom-Header': 'Bumbaway atuna' }
}
needle.post('https://my.app.com/endpoint', 'foo=bar', options, function(err, resp) {
// you can pass params as a string or as an object.
});
```
### needle.put(url, data[, options][, callback])
```js
var nested = {
params: {
are: {
also: 'supported'
}
}
}
needle.put('https://api.app.com/v2', nested, function(err, resp) {
console.log('Got ' + resp.bytes + ' bytes.') // another nice treat from this handsome fella.
});
```
### needle.patch(url, data[, options][, callback])
Same behaviour as PUT.
### needle.delete(url, data[, options][, callback])
```js
var options = {
username: 'fidelio',
password: 'x'
}
needle.delete('https://api.app.com/messages/123', null, options, function(err, resp) {
// in this case, data may be null, but you need to explicity pass it.
});
```
### needle.request(method, url, data[, options][, callback])
Generic request. This not only allows for flexibility, but also lets you perform a GET request with data, in which case will be appended to the request as a query string, unless you pass a `json: true` option (read below).
```js
var params = {
q : 'a very smart query',
page : 2
}
needle.request('get', 'forum.com/search', params, function(err, resp) {
if (!err && resp.statusCode == 200)
console.log(resp.body); // here you go, mister.
});
```
Now, if you set pass `json: true` among the options, Needle won't set your params as a querystring but instead send a JSON representation of your data through the request's body, as well as set the `Content-Type` and `Accept` headers to `application/json`.
```js
needle.request('get', 'forum.com/search', params, { json: true }, function(err, resp) {
if (resp.statusCode == 200) console.log('It worked!');
});
```
Events
------
The [Readable stream](https://nodejs.org/api/stream.html#stream_class_stream_readable) object returned by the above request methods emits the following events, in addition to the regular ones (e.g. `end`, `close`, `data`, `pipe`, `readable`).
### Event: `'response'`
- `response <http.IncomingMessage>`
Emitted when the underlying [http.ClientRequest](https://nodejs.org/api/http.html#http_class_http_clientrequest) emits a response event. This is after the connection is established and the header received, but before any of it is processed (e.g. authorization required or redirect to be followed). No data has been consumed at this point.
### Event: `'redirect'`
- `location <String>`
Indicates that the a redirect is being followed. This means that the response code was a redirect (`301`, `302`, `303`, `307`) and the given [redirect options](#redirect-options) allowed following the URL received in the `Location` header.
### Event: `'header'`
- `statusCode <Integer>`
- `headers <Object>`
Triggered after the header has been processed, and just before the data is to be consumed. This implies that no redirect was followed and/or authentication header was received. In other words, we got a "valid" response.
### Event: `'done'` (previously 'end')
- `exception <Error>` (optional)
Emitted when the request/response process has finished, either because all data was consumed or an error ocurred somewhere in between. Unlike a regular stream's `end` event, Needle's `done` will be fired either on success or on failure, which is why the first argument may be an Error object. In other words:
```js
var resp = needle.get('something.worthy/of/being/streamed/by/needle');
resp.pipe(someWritableStream);
resp.on('done', function(err) {
if (err) console.log('An error ocurred: ' + err.message);
else console.log('Great success!');
})
```
### Event: `'err'`
- `exception <Error>`
Emitted when an error ocurrs. This should only happen once in the lifecycle of a Needle request.
### Event: `'timeout'`
- `type <String>`
Emitted when an timeout error occurs. Type can be either 'open', 'response', or 'read'. This will called right before aborting the request, which will also trigger an `err` event, a described above, with an `ECONNRESET` (Socket hang up) exception.
Request options
---------------
For information about options that've changed, there's always [the changelog](https://github.com/tomas/needle/releases).
- `agent` : Uses an [http.Agent](https://nodejs.org/api/http.html#http_class_http_agent) of your choice, instead of the global, default one. Useful for tweaking the behaviour at the connection level, such as when doing tunneling (see below for an example).
- `json` : When `true`, sets content type to `application/json` and sends request body as JSON string, instead of a query string.
- `open_timeout`: (or `timeout`) Returns error if connection takes longer than X milisecs to establish. Defaults to `10000` (10 secs). `0` means no timeout.
- `response_timeout`: Returns error if no response headers are received in X milisecs, counting from when the connection is opened. Defaults to `0` (no response timeout).
- `read_timeout`: Returns error if data transfer takes longer than X milisecs, once response headers are received. Defaults to `0` (no timeout).
- `follow_max` : (or `follow`) Number of redirects to follow. Defaults to `0`. See below for more redirect options.
- `multipart` : Enables multipart/form-data encoding. Defaults to `false`. Use it when uploading files.
- `proxy` : Forwards request through HTTP(s) proxy. Eg. `proxy: 'http://user:pass@proxy.server.com:3128'`. For more advanced proxying/tunneling use a custom `agent`, as described below.
- `headers` : Object containing custom HTTP headers for request. Overrides defaults described below.
- `auth` : Determines what to do with provided username/password. Options are `auto`, `digest` or `basic` (default). `auto` will detect the type of authentication depending on the response headers.
- `stream_length`: When sending streams, this lets you manually set the Content-Length header --if the stream's bytecount is known beforehand--, preventing ECONNRESET (socket hang up) errors on some servers that misbehave when receiving payloads of unknown size. Set it to `0` and Needle will get and set the stream's length for you, or leave unset for the default behaviour, which is no Content-Length header for stream payloads.
- `localAddress`: <string>, IP address. Passed to http/https request. Local interface from which the request should be emitted.
- `uri_modifier`: Anonymous function taking request (or redirect location if following redirects) URI as an argument and modifying it given logic. It has to return a valid URI string for successful request.
Response options
----------------
- `decode_response` : (or `decode`) Whether to decode the text responses to UTF-8, if Content-Type header shows a different charset. Defaults to `true`.
- `parse_response` : (or `parse`) Whether to parse XML or JSON response bodies automagically. Defaults to `true`. You can also set this to 'xml' or 'json' in which case Needle will *only* parse the response if the content type matches.
- `output` : Dump response output to file. This occurs after parsing and charset decoding is done.
- `parse_cookies` : Whether to parse responses `Set-Cookie` header. Defaults to `true`. If parsed, response cookies will be available at `resp.cookies`.
HTTP Header options
-------------------
These are basically shortcuts to the `headers` option described above.
- `cookies` : Builds and sets a Cookie header from a `{ key: 'value' }` object.
- `compressed`: If `true`, sets 'Accept-Encoding' header to 'gzip,deflate', and inflates content if zipped. Defaults to `false`.
- `username` : For HTTP basic auth.
- `password` : For HTTP basic auth. Requires username to be passed, but is optional.
- `accept` : Sets 'Accept' HTTP header. Defaults to `*/*`.
- `connection`: Sets 'Connection' HTTP header. Not set by default, unless running Node < 0.11.4 in which case it defaults to `close`. More info about this below.
- `user_agent`: Sets the 'User-Agent' HTTP header. Defaults to `Needle/{version} (Node.js {node_version})`.
- `content_type`: Sets the 'Content-Type' header. Unset by default, unless you're sending data in which case it's set accordingly to whatever is being sent (`application/x-www-form-urlencoded`, `application/json` or `multipart/form-data`). That is, of course, unless the option is passed, either here or through `options.headers`. You're the boss.
Node.js TLS Options
-------------------
These options are passed directly to `https.request` if present. Taken from the [original documentation](http://nodejs.org/docs/latest/api/https.html):
- `pfx` : Certificate, Private key and CA certificates to use for SSL.
- `key` : Private key to use for SSL.
- `passphrase` : A string of passphrase for the private key or pfx.
- `cert` : Public x509 certificate to use.
- `ca` : An authority certificate or array of authority certificates to check the remote host against.
- `ciphers` : A string describing the ciphers to use or exclude.
- `rejectUnauthorized` : If true, the server certificate is verified against the list of supplied CAs. An 'error' event is emitted if verification fails. Verification happens at the connection level, before the HTTP request is sent.
- `secureProtocol` : The SSL method to use, e.g. SSLv3_method to force SSL version 3.
- `family` : IP address family to use when resolving host and hostname. Valid values are 4 or 6. When unspecified, both IP v4 and v6 will be used.
Redirect options
----------------
These options only apply if the `follow_max` (or `follow`) option is higher than 0.
- `follow_set_cookies` : Sends the cookies received in the `set-cookie` header as part of the following request. `false` by default.
- `follow_set_referer` : Sets the 'Referer' header to the requested URI when following a redirect. `false` by default.
- `follow_keep_method` : If enabled, resends the request using the original verb instead of being rewritten to `get` with no data. `false` by default.
- `follow_if_same_host` : When true, Needle will only follow redirects that point to the same host as the original request. `false` by default.
- `follow_if_same_protocol` : When true, Needle will only follow redirects that point to the same protocol as the original request. `false` by default.
- `follow_if_same_location` : Unless true, Needle will not follow redirects that point to same location (as set in the response header) as the original request URL. `false` by default.
Overriding Defaults
-------------------
Yes sir, we have it. Needle includes a `defaults()` method, that lets you override some of the defaults for all future requests. Like this:
```js
needle.defaults({
open_timeout: 60000,
user_agent: 'MyApp/1.2.3',
parse_response: false });
```
This will override Needle's default user agent and 10-second timeout, and disable response parsing, so you don't need to pass those options in every other request.
More advanced Proxy support
---------------------------
Since you can pass a custom HTTPAgent to Needle you can do all sorts of neat stuff. For example, if you want to use the [`tunnel`](https://github.com/koichik/node-tunnel) module for HTTPS proxying, you can do this:
```js
var tunnel = require('tunnel');
var myAgent = tunnel.httpOverHttp({
proxy: { host: 'localhost' }
});
needle.get('foobar.com', { agent: myAgent });
```
Otherwise, you can use the [`hpagent`](https://github.com/delvedor/hpagent) package, which keeps the internal sockets alive to be reused.
```js
const { HttpsProxyAgent } = require('hpagent');
needle('get', 'https://localhost:9200', {
agent: new HttpsProxyAgent({
keepAlive: true,
keepAliveMsecs: 1000,
maxSockets: 256,
maxFreeSockets: 256,
scheduling: 'lifo',
proxy: 'https://localhost:8080'
})
});
```
Regarding the 'Connection' header
---------------------------------
Unless you're running an old version of Node (< 0.11.4), by default Needle won't set the Connection header on requests, yielding Node's default behaviour of keeping the connection alive with the target server. This speeds up immensely the process of sending several requests to the same host.
On older versions, however, this has the unwanted behaviour of preventing the runtime from exiting, either because of a bug or 'feature' that was changed on 0.11.4. To overcome this Needle does set the 'Connection' header to 'close' on those versions, however this also means that making new requests to the same host doesn't benefit from Keep-Alive.
So if you're stuck on 0.10 or even lower and want full speed, you can simply set the Connection header to 'Keep-Alive' by using `{ connection: 'Keep-Alive' }`. Please note, though, that an event loop handler will prevent the runtime from exiting so you'll need to manually call `process.exit()` or the universe will collapse.
Examples Galore
---------------
### HTTPS GET with Basic Auth
```js
needle.get('https://api.server.com', { username: 'you', password: 'secret' },
function(err, resp) {
// used HTTP auth
});
```
Or use [RFC-1738](http://tools.ietf.org/html/rfc1738#section-3.1) basic auth URL syntax:
```js
needle.get('https://username:password@api.server.com', function(err, resp) {
// used HTTP auth from URL
});
```
### Digest Auth
```js
needle.get('other.server.com', { username: 'you', password: 'secret', auth: 'digest' },
function(err, resp, body) {
// needle prepends 'http://' to your URL, if missing
});
```
### Custom Accept header, deflate
```js
var options = {
compressed : true,
follow : 10,
accept : 'application/vnd.github.full+json'
}
needle.get('api.github.com/users/tomas', options, function(err, resp, body) {
// body will contain a JSON.parse(d) object
// if parsing fails, you'll simply get the original body
});
```
### GET XML object
```js
needle.get('https://news.ycombinator.com/rss', function(err, resp, body) {
// you'll get a nice object containing the nodes in the RSS
});
```
### GET binary, output to file
```js
needle.get('http://upload.server.com/tux.png', { output: '/tmp/tux.png' }, function(err, resp, body) {
// you can dump any response to a file, not only binaries.
});
```
### GET through proxy
```js
needle.get('http://search.npmjs.org', { proxy: 'http://localhost:1234' }, function(err, resp, body) {
// request passed through proxy
});
```
### GET a very large document in a stream (from 0.7+)
```js
var stream = needle.get('http://www.as35662.net/100.log');
stream.on('readable', function() {
var chunk;
while (chunk = this.read()) {
console.log('got data: ', chunk);
}
});
```
### GET JSON object in a stream (from 0.7+)
```js
var stream = needle.get('http://jsonplaceholder.typicode.com/db', { parse: true });
stream.on('readable', function() {
var node;
// our stream will only emit a single JSON root node.
while (node = this.read()) {
console.log('got data: ', node);
}
});
```
### GET JSONStream flexible parser with search query (from 0.7+)
```js
// The 'data' element of this stream will be the string representation
// of the titles of all posts.
needle.get('http://jsonplaceholder.typicode.com/db', { parse: true })
.pipe(new JSONStream.parse('posts.*.title'));
.on('data', function (obj) {
console.log('got post title: %s', obj);
});
```
### File upload using multipart, passing file path
```js
var data = {
foo: 'bar',
image: { file: '/home/tomas/linux.png', content_type: 'image/png' }
}
needle.post('http://my.other.app.com', data, { multipart: true }, function(err, resp, body) {
// needle will read the file and include it in the form-data as binary
});
```
### Stream upload, PUT or POST
``` js
needle.put('https://api.app.com/v2', fs.createReadStream('myfile.txt'), function(err, resp, body) {
// stream content is uploaded verbatim
});
```
### Multipart POST, passing data buffer
```js
var buffer = fs.readFileSync('/path/to/package.zip');
var data = {
zip_file: {
buffer : buffer,
filename : 'mypackage.zip',
content_type : 'application/octet-stream'
}
}
needle.post('http://somewhere.com/over/the/rainbow', data, { multipart: true }, function(err, resp, body) {
// if you see, when using buffers we need to pass the filename for the multipart body.
// you can also pass a filename when using the file path method, in case you want to override
// the default filename to be received on the other end.
});
```
### Multipart with custom Content-Type
```js
var data = {
token: 'verysecret',
payload: {
value: JSON.stringify({ title: 'test', version: 1 }),
content_type: 'application/json'
}
}
needle.post('http://test.com/', data, { timeout: 5000, multipart: true }, function(err, resp, body) {
// in this case, if the request takes more than 5 seconds
// the callback will return a [Socket closed] error
});
```
For even more examples, check out the examples directory in the repo.
### Testing
To run tests, you need to generate a self-signed SSL certificate in the `test` directory. After cloning the repository, run the following commands:
$ mkdir -p test/keys
$ openssl genrsa -out test/keys/ssl.key 2048
$ openssl req -new -key test/keys/ssl.key -x509 -days 999 -out test/keys/ssl.cert
Then you should be able to run `npm test` once you have the dependencies in place.
> Note: Tests currently only work on linux-based environments that have `/proc/self/fd`. They *do not* work on MacOS environments.
> You can use Docker to run tests by creating a container and mounting the needle project directory on `/app`
> `docker create --name Needle -v /app -w /app -v /app/node_modules -i node:argon`
Credits
-------
Written by Tomás Pollak, with the help of contributors.
Copyright
---------
(c) Fork Ltd. Licensed under the MIT license.
@@ -0,0 +1,22 @@
var fs = require('fs'),
stream = require('stream'),
needle = require('./../');
var url = 'http://ibl.gamechaser.net/f/tagqfxtteucbuldhezkz/bt_level1.gz';
var resp = needle.get(url, { compressed: true, follow_max: 10 });
console.log('Downloading...');
resp.on('readable', function() {
while (data = this.read()) {
var lines = data.toString().split('\n');
console.log('Got ' + lines.length + ' items.');
// console.log(lines);
}
})
resp.on('done', function(data) {
console.log('Done');
})
@@ -0,0 +1,16 @@
var needle = require('./..');
var opts = {
username: 'user3',
password: 'user3',
auth: 'digest'
}
needle.get('http://test.webdav.org/auth-digest/', opts, function(err, resp, body) {
console.log(resp.headers);
if (resp.statusCode == 401)
console.log('\nIt failed.')
else
console.log('\nIt worked!')
});
@@ -0,0 +1,18 @@
var fs = require('fs'),
needle = require('./..'),
path = require('path');
var url = process.argv[2] || 'https://upload.wikimedia.org/wikipedia/commons/a/af/Tux.png';
var file = path.basename(url);
console.log('Downloading ' + file);
needle.get(url, { output: file, follow: 3 }, function(err, resp, data){
console.log('File saved: ' + process.cwd() + '/' + file);
var size = fs.statSync(file).size;
if (size == resp.bytes)
console.log(resp.bytes + ' bytes written to file.');
else
throw new Error('File size mismatch: ' + size + ' != ' + resp.bytes);
});
@@ -0,0 +1,25 @@
var needle = require('./../');
var url = 'http://posttestserver.com/post.php?dir=needle';
var black_pixel = Buffer.from("R0lGODlhAQABAIAAAAUEBAAAACwAAAAAAQABAAACAkQBADs=", 'base64');
var data = {
foo: 'bar',
nested: {
test: 123
},
image: { buffer: black_pixel, content_type: 'image/gif' }
}
var resp = needle.post(url, data, { multipart: true });
resp.on('readable', function() {
while (data = this.read()) {
console.log(data.toString());
}
})
resp.on('done', function(data) {
console.log('Done.');
})
@@ -0,0 +1,23 @@
//////////////////////////////////////////
// This example demonstrates what happends
// when you use the built-in JSON parser.
//////////////////////////////////////////
var fs = require('fs'),
stream = require('stream'),
needle = require('./../');
var url = 'http://ip.jsontest.com/',
resp = needle.get(url, { parse: true });
resp.on('readable', function(obj) {
var chunk;
while (chunk = this.read()) {
console.log('root = ', chunk);
}
});
resp.on('done', function() {
console.log('Done.');
});
@@ -0,0 +1,21 @@
//////////////////////////////////////////
// This example illustrates a more complex
// example of parsing a JSON stream.
//////////////////////////////////////////
var needle = require('./../'),
JSONStream = require('JSONStream');
var url = 'http://jsonplaceholder.typicode.com/db';
// Initialize our GET request with our default (JSON)
// parsers disabled.
var json = new needle.get(url, {parse: false})
// And now interpret the stream as JSON, returning only the
// title of all the posts.
.pipe(new JSONStream.parse('posts.*.title'));
json.on('data', function (obj) {
console.log('got title: \'' + obj + '\'');
})
@@ -0,0 +1,23 @@
var needle = require('./..');
var resp = needle.get('google.com', { follow_max: 10, timeout: 5000 });
resp.on('readable', function() {
var chunk;
while (chunk = this.read()) {
console.log('Got ' + chunk.length + ' bytes');
}
})
resp.on('headers', function(headers) {
console.log('Got headers', headers);
})
resp.on('redirect', function(url) {
console.log('Redirected to url ' + url);
})
resp.on('done', function(err) {
console.log('Finished. No more data to receive.');
if (err) console.log('With error', err)
})
@@ -0,0 +1,14 @@
var fs = require('fs'),
needle = require('./..'),
path = require('path');
var url = process.argv[2] || 'http://www.google.com/images/errors/robot.png';
var file = path.basename(url);
console.log('Downloading ' + file + '...');
needle
.get(url)
.pipe(fs.createWriteStream(file))
.on('done', function() {
console.log('Done!')
})
@@ -0,0 +1,51 @@
var needle = require('../'),
path = require('path');
var image = 'https://upload.wikimedia.org/wikipedia/commons/a/af/Tux.png';
function upload(obj, cb) {
console.log('Uploading image...');
var url = 'http://deviantsart.com';
var opts = {
timeout: 10000,
follow: 3,
multipart: true
};
var params = {
file: obj
}
needle.post(url, params, opts, function(err, resp) {
if (err || !resp.body.match('url'))
return cb(err || new Error('No image URL found.'))
cb(null, JSON.parse(resp.body).url)
})
}
function download(url, cb) {
console.log('Getting ' + url);
needle.get(url, function(err, resp) {
if (err) throw err;
cb(null, resp.body);
})
}
////////////////////////////////////////
// ok, now go.
download(image, function(err, buffer) {
if (err) throw err;
var obj = { buffer: buffer, content_type: 'image/png' };
upload(obj, function(err, url) {
if (err) throw err;
console.log('Image uploaded to ' + url);
})
})
@@ -0,0 +1,112 @@
var createHash = require('crypto').createHash;
function get_header(header, credentials, opts) {
var type = header.split(' ')[0],
user = credentials[0],
pass = credentials[1];
if (type == 'Digest') {
return digest.generate(header, user, pass, opts.method, opts.path);
} else if (type == 'Basic') {
return basic(user, pass);
}
}
////////////////////
// basic
function md5(string) {
return createHash('md5').update(string).digest('hex');
}
function basic(user, pass) {
var str = typeof pass == 'undefined' ? user : [user, pass].join(':');
return 'Basic ' + Buffer.from(str).toString('base64');
}
////////////////////
// digest
// logic inspired from https://github.com/simme/node-http-digest-client
var digest = {};
digest.parse_header = function(header) {
var challenge = {},
matches = header.match(/([a-z0-9_-]+)="?([a-z0-9_=\/\.@\s-\+)()]+)"?/gi);
for (var i = 0, l = matches.length; i < l; i++) {
var parts = matches[i].split('='),
key = parts.shift(),
val = parts.join('=').replace(/^"/, '').replace(/"$/, '');
challenge[key] = val;
}
return challenge;
}
digest.update_nc = function(nc) {
var max = 99999999;
nc++;
if (nc > max)
nc = 1;
var padding = new Array(8).join('0') + '';
nc = nc + '';
return padding.substr(0, 8 - nc.length) + nc;
}
digest.generate = function(header, user, pass, method, path) {
var nc = 1,
cnonce = null,
challenge = digest.parse_header(header);
var ha1 = md5(user + ':' + challenge.realm + ':' + pass),
ha2 = md5(method.toUpperCase() + ':' + path),
resp = [ha1, challenge.nonce];
if (typeof challenge.qop === 'string') {
cnonce = md5(Math.random().toString(36)).substr(0, 8);
nc = digest.update_nc(nc);
resp = resp.concat(nc, cnonce);
resp = resp.concat(challenge.qop, ha2);
} else {
resp = resp.concat(ha2);
}
var params = {
uri : path,
realm : challenge.realm,
nonce : challenge.nonce,
username : user,
response : md5(resp.join(':'))
}
if (challenge.qop) {
params.qop = challenge.qop;
}
if (challenge.opaque) {
params.opaque = challenge.opaque;
}
if (cnonce) {
params.nc = nc;
params.cnonce = cnonce;
}
header = []
for (var k in params)
header.push(k + '="' + params[k] + '"')
return 'Digest ' + header.join(', ');
}
module.exports = {
header : get_header,
basic : basic,
digest : digest.generate
}
@@ -0,0 +1,79 @@
// Simple cookie handling implementation based on the standard RFC 6265.
//
// This module just has two functionalities:
// - Parse a set-cookie-header as a key value object
// - Write a cookie-string from a key value object
//
// All cookie attributes are ignored.
var unescape = require('querystring').unescape;
var COOKIE_PAIR = /^([^=\s]+)\s*=\s*("?)\s*(.*)\s*\2\s*$/;
var EXCLUDED_CHARS = /[\x00-\x1F\x7F\x3B\x3B\s\"\,\\"%]/g;
var TRAILING_SEMICOLON = /\x3B+$/;
var SEP_SEMICOLON = /\s*\x3B\s*/;
// i know these should be 'const', but I'd like to keep
// supporting earlier node.js versions as long as I can. :)
var KEY_INDEX = 1; // index of key from COOKIE_PAIR match
var VALUE_INDEX = 3; // index of value from COOKIE_PAIR match
// Returns a copy str trimmed and without trainling semicolon.
function cleanCookieString(str) {
return str.trim().replace(/\x3B+$/, '');
}
function getFirstPair(str) {
var index = str.indexOf('\x3B');
return index === -1 ? str : str.substr(0, index);
}
// Returns a encoded copy of str based on RFC6265 S4.1.1.
function encodeCookieComponent(str) {
return str.toString().replace(EXCLUDED_CHARS, encodeURIComponent);
}
// Parses a set-cookie-string based on the standard defined in RFC6265 S4.1.1.
function parseSetCookieString(str) {
str = cleanCookieString(str);
str = getFirstPair(str);
var res = COOKIE_PAIR.exec(str);
if (!res || !res[VALUE_INDEX]) return null;
return {
name : unescape(res[KEY_INDEX]),
value : unescape(res[VALUE_INDEX])
};
}
// Parses a set-cookie-header and returns a key/value object.
// Each key represents the name of a cookie.
function parseSetCookieHeader(header) {
if (!header) return {};
header = Array.isArray(header) ? header : [header];
return header.reduce(function(res, str) {
var cookie = parseSetCookieString(str);
if (cookie) res[cookie.name] = cookie.value;
return res;
}, {});
}
// Writes a set-cookie-string based on the standard definded in RFC6265 S4.1.1.
function writeCookieString(obj) {
return Object.keys(obj).reduce(function(str, name) {
var encodedName = encodeCookieComponent(name);
var encodedValue = encodeCookieComponent(obj[name]);
str += (str ? '; ' : '') + encodedName + '=' + encodedValue;
return str;
}, '');
}
// returns a key/val object from an array of cookie strings
exports.read = parseSetCookieHeader;
// writes a cookie string header
exports.write = writeCookieString;
@@ -0,0 +1,53 @@
var iconv,
inherits = require('util').inherits,
stream = require('stream');
var regex = /(?:charset|encoding)\s*=\s*['"]? *([\w\-]+)/i;
inherits(StreamDecoder, stream.Transform);
function StreamDecoder(charset) {
if (!(this instanceof StreamDecoder))
return new StreamDecoder(charset);
stream.Transform.call(this, charset);
this.charset = charset;
this.parsed_chunk = false;
}
StreamDecoder.prototype._transform = function(chunk, encoding, done) {
var res, found;
// try get charset from chunk, just once
if (this.charset == 'utf8' && !this.parsed_chunk) {
this.parsed_chunk = true;
var matches = regex.exec(chunk.toString());
if (matches) {
found = matches[1].toLowerCase();
this.charset = found == 'utf-8' ? 'utf8' : found;
}
}
try {
res = iconv.decode(chunk, this.charset);
} catch(e) { // something went wrong, just return original chunk
res = chunk;
}
this.push(res);
done();
}
module.exports = function(charset) {
try {
if (!iconv) iconv = require('iconv-lite');
} catch(e) {
/* iconv not found */
}
if (iconv)
return new StreamDecoder(charset);
else
return new stream.PassThrough;
}
@@ -0,0 +1,98 @@
var readFile = require('fs').readFile,
basename = require('path').basename;
exports.build = function(data, boundary, callback) {
if (typeof data != 'object' || typeof data.pipe == 'function')
return callback(new Error('Multipart builder expects data as key/val object.'));
var body = '',
object = flatten(data),
count = Object.keys(object).length;
if (count === 0)
return callback(new Error('Empty multipart body. Invalid data.'))
function done(err, section) {
if (err) return callback(err);
if (section) body += section;
--count || callback(null, body + '--' + boundary + '--');
};
for (var key in object) {
var value = object[key];
if (value === null || typeof value == 'undefined') {
done();
} else if (Buffer.isBuffer(value)) {
var part = { buffer: value, content_type: 'application/octet-stream' };
generate_part(key, part, boundary, done);
} else {
var part = (value.buffer || value.file || value.content_type) ? value : { value: value };
generate_part(key, part, boundary, done);
}
}
}
function generate_part(name, part, boundary, callback) {
var return_part = '--' + boundary + '\r\n';
return_part += 'Content-Disposition: form-data; name="' + name + '"';
function append(data, filename) {
if (data) {
var binary = part.content_type.indexOf('text') == -1;
return_part += '; filename="' + encodeURIComponent(filename) + '"\r\n';
if (binary) return_part += 'Content-Transfer-Encoding: binary\r\n';
return_part += 'Content-Type: ' + part.content_type + '\r\n\r\n';
return_part += binary ? data.toString('binary') : data.toString('utf8');
}
callback(null, return_part + '\r\n');
};
if ((part.file || part.buffer) && part.content_type) {
var filename = part.filename ? part.filename : part.file ? basename(part.file) : name;
if (part.buffer) return append(part.buffer, filename);
readFile(part.file, function(err, data) {
if (err) return callback(err);
append(data, filename);
});
} else {
if (typeof part.value == 'object')
return callback(new Error('Object received for ' + name + ', expected string.'))
if (part.content_type) {
return_part += '\r\n';
return_part += 'Content-Type: ' + part.content_type;
}
return_part += '\r\n\r\n';
return_part += Buffer.from(String(part.value), 'utf8').toString('binary');
append();
}
}
// flattens nested objects for multipart body
function flatten(object, into, prefix) {
into = into || {};
for(var key in object) {
var prefix_key = prefix ? prefix + '[' + key + ']' : key;
var prop = object[key];
if (prop && typeof prop === 'object' && !(prop.buffer || prop.file || prop.content_type))
flatten(prop, into, prefix_key)
else
into[prefix_key] = prop;
}
return into;
}
@@ -0,0 +1,879 @@
//////////////////////////////////////////
// Needle -- HTTP Client for Node.js
// Written by Tomás Pollak <tomas@forkhq.com>
// (c) 2012-2020 - Fork Ltd.
// MIT Licensed
//////////////////////////////////////////
var fs = require('fs'),
http = require('http'),
https = require('https'),
url = require('url'),
stream = require('stream'),
debug = require('debug')('needle'),
stringify = require('./querystring').build,
multipart = require('./multipart'),
auth = require('./auth'),
cookies = require('./cookies'),
parsers = require('./parsers'),
decoder = require('./decoder');
//////////////////////////////////////////
// variabilia
var version = require('../package.json').version;
var user_agent = 'Needle/' + version;
user_agent += ' (Node.js ' + process.version + '; ' + process.platform + ' ' + process.arch + ')';
var tls_options = 'agent pfx key passphrase cert ca ciphers rejectUnauthorized secureProtocol checkServerIdentity family';
// older versions of node (< 0.11.4) prevent the runtime from exiting
// because of connections in keep-alive state. so if this is the case
// we'll default new requests to set a Connection: close header.
var close_by_default = !http.Agent || http.Agent.defaultMaxSockets != Infinity;
// see if we have Object.assign. otherwise fall back to util._extend
var extend = Object.assign ? Object.assign : require('util')._extend;
// these are the status codes that Needle interprets as redirects.
var redirect_codes = [301, 302, 303, 307, 308];
//////////////////////////////////////////
// decompressors for gzip/deflate/br bodies
function bind_opts(fn, options) {
return fn.bind(null, options);
}
var decompressors = {};
try {
var zlib = require('zlib');
// Enable Z_SYNC_FLUSH to avoid Z_BUF_ERROR errors (Node PR #2595)
var zlib_options = {
flush: zlib.Z_SYNC_FLUSH,
finishFlush: zlib.Z_SYNC_FLUSH
};
var br_options = {
flush: zlib.BROTLI_OPERATION_FLUSH,
finishFlush: zlib.BROTLI_OPERATION_FLUSH
};
decompressors['x-deflate'] = bind_opts(zlib.Inflate, zlib_options);
decompressors['deflate'] = bind_opts(zlib.Inflate, zlib_options);
decompressors['x-gzip'] = bind_opts(zlib.Gunzip, zlib_options);
decompressors['gzip'] = bind_opts(zlib.Gunzip, zlib_options);
if (typeof zlib.BrotliDecompress === 'function') {
decompressors['br'] = bind_opts(zlib.BrotliDecompress, br_options);
}
} catch(e) { /* zlib not available */ }
//////////////////////////////////////////
// options and aliases
var defaults = {
// data
boundary : '--------------------NODENEEDLEHTTPCLIENT',
encoding : 'utf8',
parse_response : 'all', // same as true. valid options: 'json', 'xml' or false/null
proxy : null,
// headers
headers : {},
accept : '*/*',
user_agent : user_agent,
// numbers
open_timeout : 10000,
response_timeout : 0,
read_timeout : 0,
follow_max : 0,
stream_length : -1,
// booleans
compressed : false,
decode_response : true,
parse_cookies : true,
follow_set_cookies : false,
follow_set_referer : false,
follow_keep_method : false,
follow_if_same_host : false,
follow_if_same_protocol : false,
follow_if_same_location : false
}
var aliased = {
options: {
decode : 'decode_response',
parse : 'parse_response',
timeout : 'open_timeout',
follow : 'follow_max'
},
inverted: {}
}
// only once, invert aliased keys so we can get passed options.
Object.keys(aliased.options).map(function(k) {
var value = aliased.options[k];
aliased.inverted[value] = k;
});
//////////////////////////////////////////
// helpers
function keys_by_type(type) {
return Object.keys(defaults).map(function(el) {
if (defaults[el] !== null && defaults[el].constructor == type)
return el;
}).filter(function(el) { return el })
}
function parse_content_type(header) {
if (!header || header === '') return {};
var found, charset = 'utf8', arr = header.split(';');
if (arr.length > 1 && (found = arr[1].match(/charset=(.+)/)))
charset = found[1];
return { type: arr[0], charset: charset };
}
function is_stream(obj) {
return typeof obj.pipe === 'function';
}
function get_stream_length(stream, given_length, cb) {
if (given_length > 0)
return cb(given_length);
if (stream.end !== void 0 && stream.end !== Infinity && stream.start !== void 0)
return cb((stream.end + 1) - (stream.start || 0));
fs.stat(stream.path, function(err, stat) {
cb(stat ? stat.size - (stream.start || 0) : null);
});
}
function resolve_url(href, base) {
if (url.URL)
return new url.URL(href, base);
// older Node version (< v6.13)
return url.resolve(base, href);
}
function pump_streams(streams, cb) {
if (stream.pipeline)
return stream.pipeline.apply(null, streams.concat(cb));
var tmp = streams.shift();
while (streams.length) {
tmp = tmp.pipe(streams.shift());
tmp.once('error', function(e) {
cb && cb(e);
cb = null;
})
}
}
//////////////////////////////////////////
// the main act
function Needle(method, uri, data, options, callback) {
// if (!(this instanceof Needle)) {
// return new Needle(method, uri, data, options, callback);
// }
if (typeof uri !== 'string')
throw new TypeError('URL must be a string, not ' + uri);
this.method = method.toLowerCase();
this.uri = uri;
this.data = data;
if (typeof options == 'function') {
this.callback = options;
this.options = {};
} else {
this.callback = callback;
this.options = options;
}
}
Needle.prototype.setup = function(uri, options) {
function get_option(key, fallback) {
// if original is in options, return that value
if (typeof options[key] != 'undefined') return options[key];
// otherwise, return value from alias or fallback/undefined
return typeof options[aliased.inverted[key]] != 'undefined'
? options[aliased.inverted[key]] : fallback;
}
function check_value(expected, key) {
var value = get_option(key),
type = typeof value;
if (type != 'undefined' && type != expected)
throw new TypeError(type + ' received for ' + key + ', but expected a ' + expected);
return (type == expected) ? value : defaults[key];
}
//////////////////////////////////////////////////
// the basics
var config = {
http_opts : {
localAddress: get_option('localAddress', undefined),
lookup: get_option('lookup', undefined)
}, // passed later to http.request() directly
headers : {},
output : options.output,
proxy : get_option('proxy', defaults.proxy),
parser : get_option('parse_response', defaults.parse_response),
encoding : options.encoding || (options.multipart ? 'binary' : defaults.encoding)
}
keys_by_type(Boolean).forEach(function(key) {
config[key] = check_value('boolean', key);
})
keys_by_type(Number).forEach(function(key) {
config[key] = check_value('number', key);
})
// populate http_opts with given TLS options
tls_options.split(' ').forEach(function(key) {
if (typeof options[key] != 'undefined') {
config.http_opts[key] = options[key];
if (typeof options.agent == 'undefined')
config.http_opts.agent = false; // otherwise tls options are skipped
}
});
//////////////////////////////////////////////////
// headers, cookies
for (var key in defaults.headers)
config.headers[key] = defaults.headers[key];
config.headers['accept'] = options.accept || defaults.accept;
config.headers['user-agent'] = options.user_agent || defaults.user_agent;
if (options.content_type)
config.headers['content-type'] = options.content_type;
// set connection header if opts.connection was passed, or if node < 0.11.4 (close)
if (options.connection || close_by_default)
config.headers['connection'] = options.connection || 'close';
if ((options.compressed || defaults.compressed) && typeof zlib != 'undefined')
config.headers['accept-encoding'] = decompressors['br'] ? 'gzip, deflate, br' : 'gzip, deflate';
if (options.cookies)
config.headers['cookie'] = cookies.write(options.cookies);
//////////////////////////////////////////////////
// basic/digest auth
if (uri.match(/[^\/]@/)) { // url contains user:pass@host, so parse it.
var parts = (url.parse(uri).auth || '').split(':');
options.username = parts[0];
options.password = parts[1];
}
if (options.username) {
if (options.auth && (options.auth == 'auto' || options.auth == 'digest')) {
config.credentials = [options.username, options.password];
} else {
config.headers['authorization'] = auth.basic(options.username, options.password);
}
}
// if proxy is present, set auth header from either url or proxy_user option.
if (config.proxy) {
if (config.proxy.indexOf('http') === -1)
config.proxy = 'http://' + config.proxy;
if (config.proxy.indexOf('@') !== -1) {
var proxy = (url.parse(config.proxy).auth || '').split(':');
options.proxy_user = proxy[0];
options.proxy_pass = proxy[1];
}
if (options.proxy_user)
config.headers['proxy-authorization'] = auth.basic(options.proxy_user, options.proxy_pass);
}
// now that all our headers are set, overwrite them if instructed.
for (var h in options.headers)
config.headers[h.toLowerCase()] = options.headers[h];
config.uri_modifier = get_option('uri_modifier', null);
return config;
}
Needle.prototype.start = function() {
var out = new stream.PassThrough({ objectMode: false }),
uri = this.uri,
data = this.data,
method = this.method,
callback = (typeof this.options == 'function') ? this.options : this.callback,
options = this.options || {};
// if no 'http' is found on URL, prepend it.
if (uri.indexOf('http') === -1)
uri = uri.replace(/^(\/\/)?/, 'http://');
var self = this, body, waiting = false, config = this.setup(uri, options);
// unless options.json was set to false, assume boss also wants JSON if content-type matches.
var json = options.json || (options.json !== false && config.headers['content-type'] == 'application/json');
if (data) {
if (options.multipart) { // boss says we do multipart. so we do it.
var boundary = options.boundary || defaults.boundary;
waiting = true;
multipart.build(data, boundary, function(err, parts) {
if (err) throw(err);
config.headers['content-type'] = 'multipart/form-data; boundary=' + boundary;
next(parts);
});
} else if (is_stream(data)) {
if (method == 'get')
throw new Error('Refusing to pipe() a stream via GET. Did you mean .post?');
if (config.stream_length > 0 || (config.stream_length === 0 && data.path)) {
// ok, let's get the stream's length and set it as the content-length header.
// this prevents some servers from cutting us off before all the data is sent.
waiting = true;
get_stream_length(data, config.stream_length, function(length) {
data.length = length;
next(data);
})
} else {
// if the boss doesn't want us to get the stream's length, or if it doesn't
// have a file descriptor for that purpose, then just head on.
body = data;
}
} else if (Buffer.isBuffer(data)) {
body = data; // use the raw buffer as request body.
} else if (method == 'get' && !json) {
// append the data to the URI as a querystring.
uri = uri.replace(/\?.*|$/, '?' + stringify(data));
} else { // string or object data, no multipart.
// if string, leave it as it is, otherwise, stringify.
body = (typeof(data) === 'string') ? data
: json ? JSON.stringify(data) : stringify(data);
// ensure we have a buffer so bytecount is correct.
body = Buffer.from(body, config.encoding);
}
}
function next(body) {
if (body) {
if (body.length) config.headers['content-length'] = body.length;
// if no content-type was passed, determine if json or not.
if (!config.headers['content-type']) {
config.headers['content-type'] = json
? 'application/json; charset=utf-8'
: 'application/x-www-form-urlencoded'; // no charset says W3 spec.
}
}
// unless a specific accept header was set, assume json: true wants JSON back.
if (options.json && (!options.accept && !(options.headers || {}).accept))
config.headers['accept'] = 'application/json';
self.send_request(1, method, uri, config, body, out, callback);
}
if (!waiting) next(body);
return out;
}
Needle.prototype.get_request_opts = function(method, uri, config) {
var opts = config.http_opts,
proxy = config.proxy,
remote = proxy ? url.parse(proxy) : url.parse(uri);
opts.protocol = remote.protocol;
opts.host = remote.hostname;
opts.port = remote.port || (remote.protocol == 'https:' ? 443 : 80);
opts.path = proxy ? uri : remote.pathname + (remote.search || '');
opts.method = method;
opts.headers = config.headers;
if (!opts.headers['host']) {
// if using proxy, make sure the host header shows the final destination
var target = proxy ? url.parse(uri) : remote;
opts.headers['host'] = target.hostname;
// and if a non standard port was passed, append it to the port header
if (target.port && [80, 443].indexOf(target.port) === -1) {
opts.headers['host'] += ':' + target.port;
}
}
return opts;
}
Needle.prototype.should_follow = function(location, config, original) {
if (!location) return false;
// returns true if location contains matching property (host or protocol)
function matches(property) {
var property = original[property];
return location.indexOf(property) !== -1;
}
// first, check whether the requested location is actually different from the original
if (!config.follow_if_same_location && location === original)
return false;
if (config.follow_if_same_host && !matches('host'))
return false; // host does not match, so not following
if (config.follow_if_same_protocol && !matches('protocol'))
return false; // procotol does not match, so not following
return true;
}
Needle.prototype.send_request = function(count, method, uri, config, post_data, out, callback) {
if (typeof config.uri_modifier === 'function') {
var modified_uri = config.uri_modifier(uri);
debug('Modifying request URI', uri + ' => ' + modified_uri);
uri = modified_uri;
}
var request,
timer,
returned = 0,
self = this,
request_opts = this.get_request_opts(method, uri, config),
protocol = request_opts.protocol == 'https:' ? https : http;
function done(err, resp) {
if (returned++ > 0)
return debug('Already finished, stopping here.');
if (timer) clearTimeout(timer);
request.removeListener('error', had_error);
out.done = true;
if (callback)
return callback(err, resp, resp ? resp.body : undefined);
// NOTE: this event used to be called 'end', but the behaviour was confusing
// when errors ocurred, because the stream would still emit an 'end' event.
out.emit('done', err);
// trigger the 'done' event on streams we're being piped to, if any
var pipes = out._readableState.pipes || [];
if (!pipes.forEach) pipes = [pipes];
pipes.forEach(function(st) { st.emit('done', err); })
}
function had_error(err) {
debug('Request error', err);
out.emit('err', err);
done(err || new Error('Unknown error when making request.'));
}
function set_timeout(type, milisecs) {
if (timer) clearTimeout(timer);
if (milisecs <= 0) return;
timer = setTimeout(function() {
out.emit('timeout', type);
request.abort();
// also invoke done() to terminate job on read_timeout
if (type == 'read') done(new Error(type + ' timeout'));
}, milisecs);
}
// handle errors on the underlying socket, that may be closed while writing
// for an example case, see test/long_string_spec.js. we make sure this
// scenario ocurred by verifying the socket's writable & destroyed states.
function on_socket_end() {
if (returned && !this.writable && this.destroyed === false) {
this.destroy();
had_error(new Error('Remote end closed socket abruptly.'))
}
}
debug('Making request #' + count, request_opts);
request = protocol.request(request_opts, function(resp) {
var headers = resp.headers;
debug('Got response', resp.statusCode, headers);
out.emit('response', resp);
set_timeout('read', config.read_timeout);
// if we got cookies, parse them unless we were instructed not to. make sure to include any
// cookies that might have been set on previous redirects.
if (config.parse_cookies && (headers['set-cookie'] || config.previous_resp_cookies)) {
resp.cookies = extend(config.previous_resp_cookies || {}, cookies.read(headers['set-cookie']));
debug('Got cookies', resp.cookies);
}
// if redirect code is found, determine if we should follow it according to the given options.
if (redirect_codes.indexOf(resp.statusCode) !== -1 && self.should_follow(headers.location, config, uri)) {
// clear timer before following redirects to prevent unexpected setTimeout consequence
clearTimeout(timer);
if (count <= config.follow_max) {
out.emit('redirect', headers.location);
// unless 'follow_keep_method' is true, rewrite the request to GET before continuing.
if (!config.follow_keep_method) {
method = 'GET';
post_data = null;
delete config.headers['content-length']; // in case the original was a multipart POST request.
}
// if follow_set_cookies is true, insert cookies in the next request's headers.
// we set both the original request cookies plus any response cookies we might have received.
if (config.follow_set_cookies) {
var request_cookies = cookies.read(config.headers['cookie']);
config.previous_resp_cookies = resp.cookies;
if (Object.keys(request_cookies).length || Object.keys(resp.cookies || {}).length) {
config.headers['cookie'] = cookies.write(extend(request_cookies, resp.cookies));
}
} else if (config.headers['cookie']) {
debug('Clearing original request cookie', config.headers['cookie']);
delete config.headers['cookie'];
}
if (config.follow_set_referer)
config.headers['referer'] = encodeURI(uri); // the original, not the destination URL.
config.headers['host'] = null; // clear previous Host header to avoid conflicts.
var redirect_url = resolve_url(headers.location, uri);
debug('Redirecting to ' + redirect_url.toString());
return self.send_request(++count, method, redirect_url.toString(), config, post_data, out, callback);
} else if (config.follow_max > 0) {
return done(new Error('Max redirects reached. Possible loop in: ' + headers.location));
}
}
// if auth is requested and credentials were not passed, resend request, provided we have user/pass.
if (resp.statusCode == 401 && headers['www-authenticate'] && config.credentials) {
if (!config.headers['authorization']) { // only if authentication hasn't been sent
var auth_header = auth.header(headers['www-authenticate'], config.credentials, request_opts);
if (auth_header) {
config.headers['authorization'] = auth_header;
return self.send_request(count, method, uri, config, post_data, out, callback);
}
}
}
// ok, so we got a valid (non-redirect & authorized) response. let's notify the stream guys.
out.emit('header', resp.statusCode, headers);
out.emit('headers', headers);
var pipeline = [],
mime = parse_content_type(headers['content-type']),
text_response = mime.type && (mime.type.indexOf('text/') != -1 || !!mime.type.match(/(\/|\+)(xml|json)$/));
// To start, if our body is compressed and we're able to inflate it, do it.
if (headers['content-encoding'] && decompressors[headers['content-encoding']]) {
var decompressor = decompressors[headers['content-encoding']]();
// make sure we catch errors triggered by the decompressor.
decompressor.on('error', had_error);
pipeline.push(decompressor);
}
// If parse is enabled and we have a parser for it, then go for it.
if (config.parser && parsers[mime.type]) {
// If a specific parser was requested, make sure we don't parse other types.
var parser_name = config.parser.toString().toLowerCase();
if (['xml', 'json'].indexOf(parser_name) == -1 || parsers[mime.type].name == parser_name) {
// OK, so either we're parsing all content types or the one requested matches.
out.parser = parsers[mime.type].name;
pipeline.push(parsers[mime.type].fn());
// Set objectMode on out stream to improve performance.
out._writableState.objectMode = true;
out._readableState.objectMode = true;
}
// If we're not parsing, and unless decoding was disabled, we'll try
// decoding non UTF-8 bodies to UTF-8, using the iconv-lite library.
} else if (text_response && config.decode_response && mime.charset) {
pipeline.push(decoder(mime.charset));
}
// And `out` is the stream we finally push the decoded/parsed output to.
pipeline.push(out);
// Now, release the kraken!
pump_streams([resp].concat(pipeline), function(err) {
if (err) debug(err)
// on node v8.x, if an error ocurrs on the receiving end,
// then we want to abort the request to avoid having dangling sockets
if (err && err.message == 'write after end') request.destroy();
});
// If the user has requested and output file, pipe the output stream to it.
// In stream mode, we will still get the response stream to play with.
if (config.output && resp.statusCode == 200) {
// for some reason, simply piping resp to the writable stream doesn't
// work all the time (stream gets cut in the middle with no warning).
// so we'll manually need to do the readable/write(chunk) trick.
var file = fs.createWriteStream(config.output);
file.on('error', had_error);
out.on('end', function() {
if (file.writable) file.end();
});
file.on('close', function() {
delete out.file;
})
out.on('readable', function() {
var chunk;
while ((chunk = this.read()) !== null) {
if (file.writable) file.write(chunk);
// if callback was requested, also push it to resp.body
if (resp.body) resp.body.push(chunk);
}
})
out.file = file;
}
// Only aggregate the full body if a callback was requested.
if (callback) {
resp.raw = [];
resp.body = [];
resp.bytes = 0;
// Gather and count the amount of (raw) bytes using a PassThrough stream.
var clean_pipe = new stream.PassThrough();
clean_pipe.on('readable', function() {
var chunk;
while ((chunk = this.read()) != null) {
resp.bytes += chunk.length;
resp.raw.push(chunk);
}
})
pump_streams([resp, clean_pipe], function(err) {
if (err) debug(err);
});
// Listen on the 'readable' event to aggregate the chunks, but only if
// file output wasn't requested. Otherwise we'd have two stream readers.
if (!config.output || resp.statusCode != 200) {
out.on('readable', function() {
var chunk;
while ((chunk = this.read()) !== null) {
// We're either pushing buffers or objects, never strings.
if (typeof chunk == 'string') chunk = Buffer.from(chunk);
// Push all chunks to resp.body. We'll bind them in resp.end().
resp.body.push(chunk);
}
})
}
}
// And set the .body property once all data is in.
out.on('end', function() {
if (resp.body) { // callback mode
// we want to be able to access to the raw data later, so keep a reference.
resp.raw = Buffer.concat(resp.raw);
// if parse was successful, we should have an array with one object
if (resp.body[0] !== undefined && !Buffer.isBuffer(resp.body[0])) {
// that's our body right there.
resp.body = resp.body[0];
// set the parser property on our response. we may want to check.
if (out.parser) resp.parser = out.parser;
} else { // we got one or several buffers. string or binary.
resp.body = Buffer.concat(resp.body);
// if we're here and parsed is true, it means we tried to but it didn't work.
// so given that we got a text response, let's stringify it.
if (text_response || out.parser) {
resp.body = resp.body.toString();
}
}
}
// if an output file is being written to, make sure the callback
// is triggered after all data has been written to it.
if (out.file) {
out.file.on('close', function() {
done(null, resp);
})
} else { // elvis has left the building.
done(null, resp);
}
});
// out.on('error', function(err) {
// had_error(err);
// if (err.code == 'ERR_STREAM_DESTROYED' || err.code == 'ERR_STREAM_PREMATURE_CLOSE') {
// request.abort();
// }
// })
}); // end request call
// unless open_timeout was disabled, set a timeout to abort the request.
set_timeout('open', config.open_timeout);
// handle errors on the request object. things might get bumpy.
request.on('error', had_error);
// make sure timer is cleared if request is aborted (issue #257)
request.once('abort', function() {
if (timer) clearTimeout(timer);
})
// handle socket 'end' event to ensure we don't get delayed EPIPE errors.
request.once('socket', function(socket) {
if (socket.connecting) {
socket.once('connect', function() {
set_timeout('response', config.response_timeout);
})
} else {
set_timeout('response', config.response_timeout);
}
// socket.once('close', function(e) {
// console.log('socket closed!', e);
// })
if (!socket.on_socket_end) {
socket.on_socket_end = on_socket_end;
socket.once('end', function() { process.nextTick(on_socket_end.bind(socket)) });
}
})
if (post_data) {
if (is_stream(post_data)) {
pump_streams([post_data, request], function(err) {
if (err) debug(err);
});
} else {
request.write(post_data, config.encoding);
request.end();
}
} else {
request.end();
}
out.abort = function() { request.abort() }; // easier access
out.request = request;
return out;
}
//////////////////////////////////////////
// exports
if (typeof Promise !== 'undefined') {
module.exports = function() {
var verb, args = [].slice.call(arguments);
if (args[0].match(/\.|\//)) // first argument looks like a URL
verb = (args.length > 2) ? 'post' : 'get';
else
verb = args.shift();
if (verb.match(/get|head/i) && args.length == 2)
args.splice(1, 0, null); // assume no data if head/get with two args (url, options)
return new Promise(function(resolve, reject) {
module.exports.request(verb, args[0], args[1], args[2], function(err, resp) {
return err ? reject(err) : resolve(resp);
});
})
}
}
module.exports.version = version;
module.exports.defaults = function(obj) {
for (var key in obj) {
var target_key = aliased.options[key] || key;
if (defaults.hasOwnProperty(target_key) && typeof obj[key] != 'undefined') {
if (target_key != 'parse_response' && target_key != 'proxy') {
// ensure type matches the original, except for proxy/parse_response that can be null/bool or string
var valid_type = defaults[target_key].constructor.name;
if (obj[key].constructor.name != valid_type)
throw new TypeError('Invalid type for ' + key + ', should be ' + valid_type);
}
defaults[target_key] = obj[key];
} else {
throw new Error('Invalid property for defaults:' + target_key);
}
}
return defaults;
}
'head get'.split(' ').forEach(function(method) {
module.exports[method] = function(uri, options, callback) {
return new Needle(method, uri, null, options, callback).start();
}
})
'post put patch delete'.split(' ').forEach(function(method) {
module.exports[method] = function(uri, data, options, callback) {
return new Needle(method, uri, data, options, callback).start();
}
})
module.exports.request = function(method, uri, data, opts, callback) {
return new Needle(method, uri, data, opts, callback).start();
};
@@ -0,0 +1,121 @@
//////////////////////////////////////////
// Defines mappings between content-type
// and the appropriate parsers.
//////////////////////////////////////////
var Transform = require('stream').Transform;
var sax = require('sax');
function parseXML(str, cb) {
var obj, current, parser = sax.parser(true, { trim: true, lowercase: true })
parser.onerror = parser.onend = done;
function done(err) {
parser.onerror = parser.onend = function() { }
cb(err, obj)
}
function newElement(name, attributes) {
return {
name: name || '',
value: '',
attributes: attributes || {},
children: []
}
}
parser.oncdata = parser.ontext = function(t) {
if (current) current.value += t
}
parser.onopentag = function(node) {
var element = newElement(node.name, node.attributes)
if (current) {
element.parent = current
current.children.push(element)
} else { // root object
obj = element
}
current = element
};
parser.onclosetag = function() {
if (typeof current.parent !== 'undefined') {
var just_closed = current
current = current.parent
delete just_closed.parent
}
}
parser.write(str).close()
}
function parserFactory(name, fn) {
function parser() {
var chunks = [],
stream = new Transform({ objectMode: true });
// Buffer all our data
stream._transform = function(chunk, encoding, done) {
chunks.push(chunk);
done();
}
// And call the parser when all is there.
stream._flush = function(done) {
var self = this,
data = Buffer.concat(chunks);
try {
fn(data, function(err, result) {
if (err) throw err;
self.push(result);
});
} catch (err) {
self.push(data); // just pass the original data
} finally {
done();
}
}
return stream;
}
return { fn: parser, name: name };
}
var parsers = {}
function buildParser(name, types, fn) {
var parser = parserFactory(name, fn);
types.forEach(function(type) {
parsers[type] = parser;
})
}
buildParser('json', [
'application/json',
'text/javascript',
'application/vnd.api+json'
], function(buffer, cb) {
var err, data;
try { data = JSON.parse(buffer); } catch (e) { err = e; }
cb(err, data);
});
buildParser('xml', [
'text/xml',
'application/xml',
'application/rdf+xml',
'application/rss+xml',
'application/atom+xml'
], function(buffer, cb) {
parseXML(buffer.toString(), function(err, obj) {
cb(err, obj)
})
});
module.exports = parsers;
module.exports.use = buildParser;
@@ -0,0 +1,49 @@
// based on the qs module, but handles null objects as expected
// fixes by Tomas Pollak.
var toString = Object.prototype.toString;
function stringify(obj, prefix) {
if (prefix && (obj === null || typeof obj == 'undefined')) {
return prefix + '=';
} else if (toString.call(obj) == '[object Array]') {
return stringifyArray(obj, prefix);
} else if (toString.call(obj) == '[object Object]') {
return stringifyObject(obj, prefix);
} else if (toString.call(obj) == '[object Date]') {
return obj.toISOString();
} else if (prefix) { // string inside array or hash
return prefix + '=' + encodeURIComponent(String(obj));
} else if (String(obj).indexOf('=') !== -1) { // string with equal sign
return String(obj);
} else {
throw new TypeError('Cannot build a querystring out of: ' + obj);
}
};
function stringifyArray(arr, prefix) {
var ret = [];
for (var i = 0, len = arr.length; i < len; i++) {
if (prefix)
ret.push(stringify(arr[i], prefix + '[]'));
else
ret.push(stringify(arr[i]));
}
return ret.join('&');
}
function stringifyObject(obj, prefix) {
var ret = [];
Object.keys(obj).forEach(function(key) {
ret.push(stringify(obj[key], prefix
? prefix + '[' + encodeURIComponent(key) + ']'
: encodeURIComponent(key)));
})
return ret.join('&');
}
exports.build = stringify;
@@ -0,0 +1,19 @@
Copyright (c) Fork, Ltd.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
@@ -0,0 +1,71 @@
{
"name": "needle",
"version": "2.9.1",
"description": "The leanest and most handsome HTTP client in the Nodelands.",
"keywords": [
"http",
"https",
"simple",
"request",
"client",
"multipart",
"upload",
"proxy",
"deflate",
"timeout",
"charset",
"iconv",
"cookie",
"redirect"
],
"tags": [
"http",
"https",
"simple",
"request",
"client",
"multipart",
"upload",
"proxy",
"deflate",
"timeout",
"charset",
"iconv",
"cookie",
"redirect"
],
"author": "Tomás Pollak <tomas@forkhq.com>",
"repository": {
"type": "git",
"url": "https://github.com/tomas/needle.git"
},
"dependencies": {
"debug": "^3.2.6",
"iconv-lite": "^0.4.4",
"sax": "^1.2.4"
},
"devDependencies": {
"JSONStream": "^1.3.5",
"jschardet": "^1.6.0",
"mocha": "^5.2.0",
"pump": "^3.0.0",
"q": "^1.5.1",
"should": "^13.2.3",
"sinon": "^2.3.0",
"xml2js": "^0.4.19"
},
"scripts": {
"test": "mocha test"
},
"directories": {
"lib": "./lib"
},
"main": "./lib/needle",
"bin": {
"needle": "./bin/needle"
},
"license": "MIT",
"engines": {
"node": ">= 4.4.x"
}
}
@@ -0,0 +1,192 @@
var needle = require('../'),
auth = require('../lib/auth'),
sinon = require('sinon'),
should = require('should'),
http = require('http'),
helpers = require('./helpers');
var createHash = require('crypto').createHash;
function md5(string) {
return createHash('md5').update(string).digest('hex');
}
function parse_header(header) {
var challenge = {},
matches = header.match(/([a-z0-9_-]+)="?([a-z0-9=\/\.@\s-\+]+)"?/gi);
for (var i = 0, l = matches.length; i < l; i++) {
var parts = matches[i].split('='),
key = parts.shift(),
val = parts.join('=').replace(/^"/, '').replace(/"$/, '');
challenge[key] = val;
}
return challenge;
}
describe('auth_digest', function() {
describe('With qop (RFC 2617)', function() {
it('should generate a proper header', function() {
// from https://tools.ietf.org/html/rfc2617
var performDigest = function() {
var header = 'Digest realm="testrealm@host.com", qop="auth,auth-int", nonce="dcd98b7102dd2f0e8b11d0f600bfb0c093", opaque="5ccc069c403ebaf9f0171e9517f40e41"';
var user = 'Mufasa';
var pass = 'Circle Of Life';
var method = 'get';
var path = '/dir/index.html';
var updatedHeader = auth.digest(header, user, pass, method, path);
var parsedUpdatedHeader = parse_header(updatedHeader);
var ha1 = md5(user + ':' + parsedUpdatedHeader.realm + ':' + pass);
var ha2 = md5(method.toUpperCase() + ':' + path);
var expectedResponse = md5([
ha1,
parsedUpdatedHeader.nonce,
parsedUpdatedHeader.nc,
parsedUpdatedHeader.cnonce,
parsedUpdatedHeader.qop,
ha2
].join(':'));
return {
header: updatedHeader,
parsed: parsedUpdatedHeader,
expectedResponse: expectedResponse,
}
}
const result = performDigest();
(result.header).should
.match(/qop="auth"/)
.match(/uri="\/dir\/index.html"/)
.match(/opaque="5ccc069c403ebaf9f0171e9517f40e41"/)
.match(/realm="testrealm@host\.com"/)
.match(/response=/)
.match(/nc=/)
.match(/nonce=/)
.match(/cnonce=/);
(result.parsed.response).should.be.eql(result.expectedResponse);
});
});
describe('With plus character in nonce header', function() {
it('should generate a proper header', function() {
// from https://tools.ietf.org/html/rfc2617
var performDigest = function() {
var header = 'Digest realm="testrealm@host.com", qop="auth,auth-int", nonce="dcd98b7102dd2f0e8b11d0f6+00bfb0c093", opaque="5ccc069c403ebaf9f0171e9517f40e41"';
var user = 'Mufasa';
var pass = 'Circle Of Life';
var method = 'get';
var path = '/dir/index.html';
var updatedHeader = auth.digest(header, user, pass, method, path);
var parsedUpdatedHeader = parse_header(updatedHeader);
var ha1 = md5(user + ':' + parsedUpdatedHeader.realm + ':' + pass);
var ha2 = md5(method.toUpperCase() + ':' + path);
var expectedResponse = md5([
ha1,
parsedUpdatedHeader.nonce,
parsedUpdatedHeader.nc,
parsedUpdatedHeader.cnonce,
parsedUpdatedHeader.qop,
ha2
].join(':'));
return {
header: updatedHeader,
parsed: parsedUpdatedHeader,
expectedResponse: expectedResponse,
}
}
const result = performDigest();
(result.header).should
.match(/nonce="dcd98b7102dd2f0e8b11d0f6\+00bfb0c093"/)
});
});
describe('With brackets in realm header', function() {
it('should generate a proper header', function() {
// from https://tools.ietf.org/html/rfc2617
var performDigest = function() {
var header = 'Digest qop="auth", realm="IP Camera(76475)", nonce="4e4449794d575269597a706b5a575935595441324d673d3d", stale="FALSE", Basic realm="IP Camera(76475)"';
var user = 'Mufasa';
var pass = 'Circle Of Life';
var method = 'get';
var path = '/dir/index.html';
var updatedHeader = auth.digest(header, user, pass, method, path);
var parsedUpdatedHeader = parse_header(updatedHeader);
var ha1 = md5(user + ':' + parsedUpdatedHeader.realm + ':' + pass);
var ha2 = md5(method.toUpperCase() + ':' + path);
var expectedResponse = md5([
ha1,
parsedUpdatedHeader.nonce,
parsedUpdatedHeader.nc,
parsedUpdatedHeader.cnonce,
parsedUpdatedHeader.qop,
ha2
].join(':'));
return {
header: updatedHeader,
parsed: parsedUpdatedHeader,
expectedResponse: expectedResponse,
}
}
const result = performDigest();
(result.header).should
.match(/realm="IP Camera\(76475\)"/)
});
});
describe('Without qop (RFC 2617)', function() {
it('should generate a proper header', function() {
// from https://tools.ietf.org/html/rfc2069
var performDigest = function() {
var header = 'Digest realm="testrealm@host.com", nonce="dcd98b7102dd2f0e8b11d0f600bfb0c093", opaque="5ccc069c403ebaf9f0171e9517f40e41"';
var user = 'Mufasa';
var pass = 'Circle Of Life';
var method = 'get';
var path = '/dir/index.html';
var updatedHeader = auth.digest(header, user, pass, method, path);
var parsedUpdatedHeader = parse_header(updatedHeader);
var ha1 = md5(user + ':' + parsedUpdatedHeader.realm + ':' + pass);
var ha2 = md5(method.toUpperCase() + ':' + path);
var expectedResponse = md5([ha1, parsedUpdatedHeader.nonce, ha2].join(':'));
return {
header: updatedHeader,
parsed: parsedUpdatedHeader,
expectedResponse: expectedResponse,
}
}
const result = performDigest();
(result.header).should
.not.match(/qop=/)
.match(/uri="\/dir\/index.html"/)
.match(/opaque="5ccc069c403ebaf9f0171e9517f40e41"/)
.match(/realm="testrealm@host\.com"/)
.match(/response=/)
.not.match(/nc=/)
.match(/nonce=/)
.not.match(/cnonce=/);
(result.parsed.response).should.be.eql(result.expectedResponse);
});
});
})
@@ -0,0 +1,196 @@
var helpers = require('./helpers'),
should = require('should'),
needle = require('./../'),
server;
var port = 7707;
describe('Basic Auth', function() {
before(function(done) {
server = helpers.server({ port: port }, done);
})
after(function(done) {
server.close(done);
})
///////////////// helpers
var get_auth = function(header) {
var token = header.split(/\s+/).pop();
return token && Buffer.from(token, 'base64').toString().split(':');
}
describe('when neither username or password are passed', function() {
it('doesnt send any Authorization headers', function(done) {
needle.get('localhost:' + port, { parse: true }, function(err, resp) {
var sent_headers = resp.body.headers;
Object.keys(sent_headers).should.not.containEql('authorization');
done();
})
})
})
describe('when username is an empty string, and password is a valid string', function() {
var opts = { username: '', password: 'foobar', parse: true };
it('doesnt send any Authorization headers', function(done) {
needle.get('localhost:' + port, { parse: true }, function(err, resp) {
var sent_headers = resp.body.headers;
Object.keys(sent_headers).should.not.containEql('authorization');
done();
})
})
});
describe('when username is a valid string, but no username is passed', function() {
var opts = { username: 'foobar', parse: true };
it('sends Authorization header', function(done) {
needle.get('localhost:' + port, opts, function(err, resp) {
var sent_headers = resp.body.headers;
Object.keys(sent_headers).should.containEql('authorization');
done();
})
})
it('Basic Auth only includes username, without colon', function(done) {
needle.get('localhost:' + port, opts, function(err, resp) {
var sent_headers = resp.body.headers;
var auth = get_auth(sent_headers['authorization']);
auth[0].should.equal('foobar');
auth.should.have.lengthOf(1);
done();
})
})
})
describe('when username is a valid string, and password is null', function() {
var opts = { username: 'foobar', password: null, parse: true };
it('sends Authorization header', function(done) {
needle.get('localhost:' + port, opts, function(err, resp) {
var sent_headers = resp.body.headers;
Object.keys(sent_headers).should.containEql('authorization');
done();
})
})
it('Basic Auth only includes both username and password', function(done) {
needle.get('localhost:' + port, opts, function(err, resp) {
var sent_headers = resp.body.headers;
var auth = get_auth(sent_headers['authorization']);
auth[0].should.equal('foobar');
auth[1].should.equal('');
done();
})
})
})
describe('when username is a valid string, and password is an empty string', function() {
var opts = { username: 'foobar', password: '', parse: true };
it('sends Authorization header', function(done) {
needle.get('localhost:' + port, opts, function(err, resp) {
var sent_headers = resp.body.headers;
Object.keys(sent_headers).should.containEql('authorization');
done();
})
})
it('Basic Auth only includes both username and password', function(done) {
needle.get('localhost:' + port, opts, function(err, resp) {
var sent_headers = resp.body.headers;
var auth = get_auth(sent_headers['authorization']);
auth[0].should.equal('foobar');
auth[1].should.equal('');
auth.should.have.lengthOf(2);
done();
})
})
})
describe('when username AND password are non empty strings', function() {
var opts = { username: 'foobar', password: 'jakub', parse: true };
it('sends Authorization header', function(done) {
needle.get('localhost:' + port, opts, function(err, resp) {
var sent_headers = resp.body.headers;
Object.keys(sent_headers).should.containEql('authorization');
done();
})
})
it('Basic Auth only includes both user and password', function(done) {
needle.get('localhost:' + port, opts, function(err, resp) {
var sent_headers = resp.body.headers;
var auth = get_auth(sent_headers['authorization']);
auth[0].should.equal('foobar');
auth[1].should.equal('jakub');
auth.should.have.lengthOf(2);
done();
})
})
})
describe('URL with @ but not username/pass', function() {
it('doesnt send Authorization header', function(done) {
var url = 'localhost:' + port + '/abc/@def/xyz.zip';
needle.get(url, {}, function(err, resp) {
var sent_headers = resp.body.headers;
Object.keys(sent_headers).should.not.containEql('authorization');
done();
})
})
it('sends user:pass headers if passed via options', function(done) {
var url = 'localhost:' + port + '/abc/@def/xyz.zip';
needle.get(url, { username: 'foo' }, function(err, resp) {
var sent_headers = resp.body.headers;
Object.keys(sent_headers).should.containEql('authorization');
sent_headers['authorization'].should.eql('Basic Zm9v')
done();
})
})
})
describe('when username/password are included in URL', function() {
var opts = { parse: true };
it('sends Authorization header', function(done) {
needle.get('foobar:jakub@localhost:' + port, opts, function(err, resp) {
var sent_headers = resp.body.headers;
Object.keys(sent_headers).should.containEql('authorization');
done();
})
})
it('Basic Auth only includes both user and password', function(done) {
needle.get('foobar:jakub@localhost:' + port, opts, function(err, resp) {
var sent_headers = resp.body.headers;
var auth = get_auth(sent_headers['authorization']);
auth[0].should.equal('foobar');
auth[1].should.equal('jakub');
auth.should.have.lengthOf(2);
done();
})
})
})
})
@@ -0,0 +1,112 @@
var should = require('should'),
needle = require('./../'),
http = require('http'),
zlib = require('zlib'),
stream = require('stream'),
port = 11123,
server;
describe('compression', function(){
require.bind(null, 'zlib').should.not.throw()
var jsonData = '{"foo":"bar"}';
describe('when server supports compression', function(){
before(function(){
server = http.createServer(function(req, res) {
var raw = new stream.PassThrough();
var acceptEncoding = req.headers['accept-encoding'];
if (!acceptEncoding) {
acceptEncoding = '';
}
if (acceptEncoding.match(/\bdeflate\b/)) {
res.setHeader('Content-Encoding', 'deflate');
raw.pipe(zlib.createDeflate()).pipe(res);
} else if (acceptEncoding.match(/\bgzip\b/)) {
res.setHeader('Content-Encoding', 'gzip');
raw.pipe(zlib.createGzip()).pipe(res);
} else if (acceptEncoding.match(/\bbr\b/)) {
res.setHeader('Content-Encoding', 'br');
raw.pipe(zlib.createBrotliCompress()).pipe(res);
} else {
raw.pipe(res);
}
res.setHeader('Content-Type', 'application/json')
if (req.headers['with-bad']) {
res.end('foo'); // end, no deflate data
} else {
raw.end(jsonData)
}
})
server.listen(port);
});
after(function(done){
server.close(done);
})
describe('and client requests no compression', function() {
it('should have the body decompressed', function(done){
needle.get('localhost:' + port, function(err, response, body){
should.ifError(err);
body.should.have.property('foo', 'bar');
response.bytes.should.equal(jsonData.length);
done();
})
})
})
describe('and client requests gzip compression', function() {
it('should have the body decompressed', function(done){
needle.get('localhost:' + port, {headers: {'Accept-Encoding': 'gzip'}}, function(err, response, body){
should.ifError(err);
body.should.have.property('foo', 'bar');
response.bytes.should.not.equal(jsonData.length);
done();
})
})
})
describe('and client requests deflate compression', function() {
it('should have the body decompressed', function(done){
needle.get('localhost:' + port, {headers: {'Accept-Encoding': 'deflate'}}, function(err, response, body){
should.ifError(err);
body.should.have.property('foo', 'bar');
response.bytes.should.not.equal(jsonData.length);
done();
})
})
it('should rethrow errors from decompressors', function(done){
needle.get('localhost:' + port, {headers: {'Accept-Encoding': 'deflate', 'With-Bad': 'true'}}, function(err, response, body) {
should.exist(err);
err.message.should.equal("incorrect header check");
err.code.should.equal("Z_DATA_ERROR")
done();
})
})
})
describe('and client requests brotli compression', function() {
it('should have the body decompressed', function(done){
// Skip this test if Brotli is not supported
if (typeof zlib.BrotliDecompress !== 'function') {
return done();
}
needle.get('localhost:' + port, {headers: {'Accept-Encoding': 'br'}}, function(err, response, body){
should.ifError(err);
body.should.have.property('foo', 'bar');
response.bytes.should.not.equal(jsonData.length);
done();
})
})
})
})
})
@@ -0,0 +1,396 @@
var needle = require('../'),
cookies = require('../lib/cookies'),
sinon = require('sinon'),
http = require('http'),
should = require('should');
var WEIRD_COOKIE_NAME = 'wc',
BASE64_COOKIE_NAME = 'bc',
FORBIDDEN_COOKIE_NAME = 'fc',
NUMBER_COOKIE_NAME = 'nc';
var WEIRD_COOKIE_VALUE = '!\'*+#()&-./0123456789:<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~',
BASE64_COOKIE_VALUE = 'Y29va2llCg==',
FORBIDDEN_COOKIE_VALUE = ' ;"\\,',
NUMBER_COOKIE_VALUE = 12354342;
var NO_COOKIES_TEST_PORT = 11112,
ALL_COOKIES_TEST_PORT = 11113;
describe('cookies', function() {
var setCookieHeader, headers, server, opts;
function decode(str) {
return decodeURIComponent(str);
}
function encode(str) {
str = str.toString().replace(/[\x00-\x1F\x7F]/g, encodeURIComponent);
return str.replace(/[\s\"\,;\\%]/g, encodeURIComponent);
}
before(function() {
setCookieHeader = [
WEIRD_COOKIE_NAME + '=' + encode(WEIRD_COOKIE_VALUE) + ';',
BASE64_COOKIE_NAME + '=' + encode(BASE64_COOKIE_VALUE) + ';',
FORBIDDEN_COOKIE_NAME + '=' + encode(FORBIDDEN_COOKIE_VALUE) + ';',
NUMBER_COOKIE_NAME + '=' + encode(NUMBER_COOKIE_VALUE) + ';'
];
});
before(function(done) {
serverAllCookies = http.createServer(function(req, res) {
res.setHeader('Content-Type', 'text/html');
res.setHeader('Set-Cookie', setCookieHeader);
res.end('200');
}).listen(ALL_COOKIES_TEST_PORT, done);
});
after(function(done) {
serverAllCookies.close(done);
});
describe('with default options', function() {
it('no cookie header is set on request', function(done) {
needle.get(
'localhost:' + ALL_COOKIES_TEST_PORT, function(err, response) {
should.not.exist(response.req._headers.cookie);
done();
});
});
});
describe('if response does not contain cookies', function() {
before(function(done) {
serverNoCookies = http.createServer(function(req, res) {
res.setHeader('Content-Type', 'text/html');
res.end('200');
}).listen(NO_COOKIES_TEST_PORT, done);
});
it('response.cookies is undefined', function(done) {
needle.get(
'localhost:' + NO_COOKIES_TEST_PORT, function(error, response) {
should.not.exist(response.cookies);
done();
});
});
after(function(done) {
serverNoCookies.close(done);
});
});
describe('if response contains cookies', function() {
it('puts them on resp.cookies', function(done) {
needle.get(
'localhost:' + ALL_COOKIES_TEST_PORT, function(error, response) {
response.should.have.property('cookies');
done();
});
});
it('parses them as a object', function(done) {
needle.get(
'localhost:' + ALL_COOKIES_TEST_PORT, function(error, response) {
response.cookies.should.be.an.instanceOf(Object)
.and.have.property(WEIRD_COOKIE_NAME);
response.cookies.should.have.property(BASE64_COOKIE_NAME);
response.cookies.should.have.property(FORBIDDEN_COOKIE_NAME);
response.cookies.should.have.property(NUMBER_COOKIE_NAME);
done();
});
});
it('must decode it', function(done) {
needle.get(
'localhost:' + ALL_COOKIES_TEST_PORT, function(error, response) {
response.cookies.wc.should.be.eql(WEIRD_COOKIE_VALUE);
response.cookies.bc.should.be.eql(BASE64_COOKIE_VALUE);
response.cookies.fc.should.be.eql(FORBIDDEN_COOKIE_VALUE);
response.cookies.nc.should.be.eql(NUMBER_COOKIE_VALUE.toString());
done();
});
});
describe('when a cookie value is invalid', function() {
before(function() {
setCookieHeader = [
'geo_city=%D1%E0%ED%EA%F2-%CF%E5%F2%E5%F0%E1%F3%F0%E3'
];
})
it('doesnt blow up', function(done) {
needle.get('localhost:' + ALL_COOKIES_TEST_PORT, function(error, response) {
should.not.exist(error)
var whatever = 'efbfbdefbfbdefbfbdefbfbdefbfbd2defbfbdefbfbdefbfbdefbfbdefbfbdefbfbdefbfbdefbfbdefbfbd';
Buffer.from(response.cookies.geo_city).toString('hex').should.eql(whatever)
done();
});
})
})
describe('and response is a redirect', function() {
var redirectServer, testPort = 22222;
var requestCookies = [];
var responseCookies = [
[ // first req
WEIRD_COOKIE_NAME + '=' + encode(WEIRD_COOKIE_VALUE) + ';',
BASE64_COOKIE_NAME + '=' + encode(BASE64_COOKIE_VALUE) + ';',
'FOO=123;'
], [ // second req
FORBIDDEN_COOKIE_NAME + '=' + encode(FORBIDDEN_COOKIE_VALUE) + ';',
NUMBER_COOKIE_NAME + '=' + encode(NUMBER_COOKIE_VALUE) + ';'
], [ // third red
'FOO=BAR;'
]
]
before(function(done) {
redirectServer = http.createServer(function(req, res) {
var number = parseInt(req.url.replace('/', ''));
var nextUrl = 'http://' + 'localhost:' + testPort + '/' + (number + 1);
if (number == 0) requestCookies = []; // reset
requestCookies.push(req.headers['cookie']);
if (responseCookies[number]) { // we should send cookies for this request
res.statusCode = 302;
res.setHeader('Set-Cookie', responseCookies[number]);
res.setHeader('Location', nextUrl);
} else if (number == 3) {
res.statusCode = 302; // redirect but without cookies
res.setHeader('Location', nextUrl);
}
res.end('OK');
}).listen(22222, done);
});
after(function(done) {
redirectServer.close(done);
})
describe('and follow_set_cookies is false', function() {
describe('with original request cookie', function() {
var opts = {
follow_set_cookies: false,
follow_max: 4,
cookies: { 'xxx': 123 }
};
it('request cookie is not passed to redirects', function(done) {
needle.get('localhost:' + testPort + '/0', opts, function(err, resp) {
requestCookies.should.eql(["xxx=123", undefined, undefined, undefined, undefined])
done();
});
});
it('response cookies are not passed either', function(done) {
needle.get('localhost:' + testPort + '/0', opts, function(err, resp) {
should.not.exist(resp.cookies);
done();
});
});
})
describe('without original request cookie', function() {
var opts = {
follow_set_cookies: false,
follow_max: 4,
};
it('no request cookies are sent', function(done) {
needle.get('localhost:' + testPort + '/0', opts, function(err, resp) {
requestCookies.should.eql([undefined, undefined, undefined, undefined, undefined])
done();
});
});
it('response cookies are not passed either', function(done) {
needle.get('localhost:' + testPort + '/0', opts, function(err, resp) {
should.not.exist(resp.cookies);
done();
});
});
})
});
describe('and follow_set_cookies is true', function() {
describe('with original request cookie', function() {
var opts = {
follow_set_cookies: true,
follow_max: 4,
cookies: { 'xxx': 123 }
};
it('request cookie is passed passed to redirects, and response cookies are added too', function(done) {
needle.get('localhost:' + testPort + '/0', opts, function(err, resp) {
requestCookies.should.eql([
"xxx=123",
"xxx=123; wc=!'*+#()&-./0123456789:<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~; bc=Y29va2llCg==; FOO=123",
"xxx=123; wc=!\'*+#()&-./0123456789:<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~; bc=Y29va2llCg==; FOO=123; fc=%20%3B%22%5C%2C; nc=12354342",
"xxx=123; wc=!\'*+#()&-./0123456789:<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~; bc=Y29va2llCg==; FOO=BAR; fc=%20%3B%22%5C%2C; nc=12354342",
"xxx=123; wc=!\'*+#()&-./0123456789:<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~; bc=Y29va2llCg==; FOO=BAR; fc=%20%3B%22%5C%2C; nc=12354342"
])
done();
});
});
it('response cookies are passed as well', function(done) {
needle.get('localhost:' + testPort + '/0', opts, function(err, resp) {
resp.cookies.should.have.property(WEIRD_COOKIE_NAME);
resp.cookies.should.have.property(BASE64_COOKIE_NAME);
resp.cookies.should.have.property(FORBIDDEN_COOKIE_NAME);
resp.cookies.should.have.property(NUMBER_COOKIE_NAME);
resp.cookies.should.have.property('FOO');
resp.cookies.FOO.should.eql('BAR'); // should overwrite previous one
done();
});
});
})
describe('without original request cookie', function() {
var opts = {
follow_set_cookies: true,
follow_max: 4,
};
it('response cookies are passed to redirects', function(done) {
needle.get('localhost:' + testPort + '/0', opts, function(err, resp) {
requestCookies.should.eql([
undefined,
"wc=!'*+#()&-./0123456789:<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~; bc=Y29va2llCg==; FOO=123",
"wc=!\'*+#()&-./0123456789:<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~; bc=Y29va2llCg==; FOO=123; fc=%20%3B%22%5C%2C; nc=12354342",
"wc=!\'*+#()&-./0123456789:<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~; bc=Y29va2llCg==; FOO=BAR; fc=%20%3B%22%5C%2C; nc=12354342",
"wc=!\'*+#()&-./0123456789:<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~; bc=Y29va2llCg==; FOO=BAR; fc=%20%3B%22%5C%2C; nc=12354342"
])
done();
});
});
it('response cookies are passed as well', function(done) {
needle.get('localhost:' + testPort + '/0', opts, function(err, resp) {
// resp.cookies.should.have.property(WEIRD_COOKIE_NAME);
// resp.cookies.should.have.property(BASE64_COOKIE_NAME);
// resp.cookies.should.have.property(FORBIDDEN_COOKIE_NAME);
// resp.cookies.should.have.property(NUMBER_COOKIE_NAME);
// resp.cookies.should.have.property('FOO');
// resp.cookies.FOO.should.eql('BAR'); // should overwrite previous one
done();
});
});
})
});
});
describe('with parse_cookies = false', function() {
it('does not parse them', function(done) {
needle.get(
'localhost:' + ALL_COOKIES_TEST_PORT, { parse_cookies: false }, function(error, response) {
should.not.exist(response.cookies);
done();
});
});
});
});
describe('if request contains cookie header', function() {
var opts = {
cookies: {}
};
before(function() {
opts.cookies[WEIRD_COOKIE_NAME] = WEIRD_COOKIE_VALUE;
opts.cookies[BASE64_COOKIE_NAME] = BASE64_COOKIE_VALUE;
opts.cookies[FORBIDDEN_COOKIE_NAME] = FORBIDDEN_COOKIE_VALUE;
opts.cookies[NUMBER_COOKIE_NAME] = NUMBER_COOKIE_VALUE;
});
it('must be a valid cookie string', function(done) {
var COOKIE_PAIR = /^([^=\s]+)\s*=\s*("?)\s*(.*)\s*\2\s*$/;
var full_header = [
WEIRD_COOKIE_NAME + '=' + WEIRD_COOKIE_VALUE,
BASE64_COOKIE_NAME + '=' + BASE64_COOKIE_VALUE,
FORBIDDEN_COOKIE_NAME + '=' + encode(FORBIDDEN_COOKIE_VALUE),
NUMBER_COOKIE_NAME + '=' + NUMBER_COOKIE_VALUE
].join('; ')
needle.get('localhost:' + ALL_COOKIES_TEST_PORT, opts, function(error, response) {
var cookieString = response.req._headers.cookie;
cookieString.should.be.type('string');
cookieString.split(/\s*;\s*/).forEach(function(pair) {
COOKIE_PAIR.test(pair).should.be.exactly(true);
});
cookieString.should.be.exactly(full_header);
done();
});
});
it('dont have to encode allowed characters', function(done) {
var COOKIE_PAIR = /^([^=\s]+)\s*=\s*("?)\s*(.*)\s*\2\s*$/,
KEY_INDEX = 1,
VALUE_INEX = 3;
needle.get('localhost:' + ALL_COOKIES_TEST_PORT, opts, function(error, response) {
var cookieObj = {},
cookieString = response.req._headers.cookie;
cookieString.split(/\s*;\s*/).forEach(function(str) {
var pair = COOKIE_PAIR.exec(str);
cookieObj[pair[KEY_INDEX]] = pair[VALUE_INEX];
});
cookieObj[WEIRD_COOKIE_NAME].should.be.exactly(WEIRD_COOKIE_VALUE);
cookieObj[BASE64_COOKIE_NAME].should.be.exactly(BASE64_COOKIE_VALUE);
done();
});
});
it('must encode forbidden characters', function(done) {
var COOKIE_PAIR = /^([^=\s]+)\s*=\s*("?)\s*(.*)\s*\2\s*$/,
KEY_INDEX = 1,
VALUE_INEX = 3;
needle.get('localhost:' + ALL_COOKIES_TEST_PORT, opts, function(error, response) {
var cookieObj = {},
cookieString = response.req._headers.cookie;
cookieString.split(/\s*;\s*/).forEach(function(str) {
var pair = COOKIE_PAIR.exec(str);
cookieObj[pair[KEY_INDEX]] = pair[VALUE_INEX];
});
cookieObj[FORBIDDEN_COOKIE_NAME].should.not.be.eql(
FORBIDDEN_COOKIE_VALUE);
cookieObj[FORBIDDEN_COOKIE_NAME].should.be.exactly(
encode(FORBIDDEN_COOKIE_VALUE));
cookieObj[FORBIDDEN_COOKIE_NAME].should.be.exactly(
encodeURIComponent(FORBIDDEN_COOKIE_VALUE));
done();
});
});
});
});
@@ -0,0 +1,121 @@
var should = require('should'),
needle = require('./../'),
Q = require('q'),
chardet = require('jschardet'),
helpers = require('./helpers');
describe('character encoding', function() {
var url;
this.timeout(5000);
describe('Given content-type: "text/html; charset=EUC-JP"', function() {
before(function() {
url = 'http://www.nina.jp/server/slackware/webapp/tomcat_charset.html';
})
describe('with decode = false', function() {
it('does not decode', function(done) {
needle.get(url, { decode: false }, function(err, resp) {
resp.body.should.be.a.String;
chardet.detect(resp.body).encoding.should.eql('windows-1252');
resp.body.indexOf('EUCを使う').should.eql(-1);
done();
})
})
})
describe('with decode = true', function() {
it('decodes', function(done) {
needle.get(url, { decode: true }, function(err, resp) {
resp.body.should.be.a.String;
chardet.detect(resp.body).encoding.should.eql('ascii');
resp.body.indexOf('EUCを使う').should.not.eql(-1);
done();
})
})
})
})
describe('Given content-type: "text/html but file is charset: gb2312', function() {
it('encodes to UTF-8', function(done) {
// Our Needle wrapper that requests a chinese website.
var task = Q.nbind(needle.get, needle, 'http://www.chinesetop100.com/');
// Different instantiations of this task
var tasks = [Q.fcall(task, {decode: true}),
Q.fcall(task, {decode: false})];
var results = tasks.map(function(task) {
return task.then(function(obj) {
return obj[0].body;
});
});
// Execute all requests concurrently
Q.all(results).done(function(bodies) {
var charsets = [
chardet.detect(bodies[0]).encoding,
chardet.detect(bodies[1]).encoding,
]
// We wanted to decode our first stream as specified by options
charsets[0].should.equal('ascii');
bodies[0].indexOf('全球中文网站前二十强').should.not.equal(-1);
// But not our second stream
charsets[1].should.equal('windows-1252');
bodies[1].indexOf('全球中文网站前二十强').should.equal(-1);
done();
});
})
})
describe('Given content-type: "text/html"', function () {
var server,
port = 54321,
text = 'Magyarországi Fióktelepe'
before(function(done) {
server = helpers.server({
port: port,
response: text,
headers: { 'Content-Type': 'text/html' }
}, done);
})
after(function(done) {
server.close(done)
})
describe('with decode = false', function () {
it('decodes by default to utf-8', function (done) {
needle.get('http://localhost:' + port, { decode: false }, function (err, resp) {
resp.body.should.be.a.String;
chardet.detect(resp.body).encoding.should.eql('ISO-8859-2');
resp.body.should.eql('Magyarországi Fióktelepe')
done();
})
})
})
})
})
@@ -0,0 +1,272 @@
var needle = require('../'),
sinon = require('sinon'),
should = require('should'),
http = require('http'),
Emitter = require('events').EventEmitter,
helpers = require('./helpers');
var get_catch = function(url, opts) {
var err;
try {
needle.get(url, opts);
} catch(e) {
err = e;
}
return err;
}
describe('errors', function() {
after(function(done) {
setTimeout(done, 100)
})
describe('when host does not exist', function() {
var url = 'http://unexistinghost/foo';
describe('with callback', function() {
it('does not throw', function() {
var ex = get_catch(url);
should.not.exist(ex);
})
it('callbacks an error', function(done) {
needle.get(url, function(err) {
err.should.be.a.Error;
done();
})
})
it('error should be ENOTFOUND or EADDRINFO or EAI_AGAIN', function(done) {
needle.get(url, function(err) {
err.code.should.match(/ENOTFOUND|EADDRINFO|EAI_AGAIN/)
done();
})
})
it('does not callback a response', function(done) {
needle.get(url, function(err, resp) {
should.not.exist(resp);
done();
})
})
it('does not emit an error event', function(done) {
var emitted = false;
var req = needle.get(url, function(err, resp) { })
req.on('error', function() {
emitted = true;
})
setTimeout(function() {
emitted.should.eql(false);
done();
}, 100);
})
})
describe('without callback', function() {
it('does not throw', function() {
var ex = get_catch(url);
should.not.exist(ex);
})
it('emits end event once, with error', function(done) {
var callcount = 0,
stream = needle.get(url);
stream.on('done', function(err) {
err.code.should.match(/ENOTFOUND|EADDRINFO|EAI_AGAIN/)
callcount++;
})
setTimeout(function() {
callcount.should.equal(1);
done();
}, 200)
})
it('does not emit a readable event', function(done) {
var called = false,
stream = needle.get(url);
stream.on('readable', function() {
called = true;
})
stream.on('done', function(err) {
called.should.be.false;
done();
})
})
it('does not emit an error event', function(done) {
var emitted = false,
stream = needle.get(url);
stream.on('error', function() {
emitted = true;
})
stream.on('done', function(err) {
emitted.should.eql(false);
done();
})
})
})
})
describe('when request times out waiting for response', function() {
var server,
url = 'http://localhost:3333/foo';
var send_request = function(cb) {
return needle.get(url, { response_timeout: 200 }, cb);
}
before(function() {
server = helpers.server({ port: 3333, wait: 1000 });
})
after(function() {
server.close();
})
describe('with callback', function() {
it('aborts the request', function(done) {
var time = new Date();
send_request(function(err) {
var timediff = (new Date() - time);
timediff.should.be.within(200, 300);
done();
})
})
it('callbacks an error', function(done) {
send_request(function(err) {
err.should.be.a.Error;
done();
})
})
it('error should be ECONNRESET', function(done) {
send_request(function(err) {
err.code.should.equal('ECONNRESET')
done();
})
})
it('does not callback a response', function(done) {
send_request(function(err, resp) {
should.not.exist(resp);
done();
})
})
it('does not emit an error event', function(done) {
var emitted = false;
var req = send_request(function(err, resp) {
should.not.exist(resp);
})
req.on('error', function() {
emitted = true;
})
setTimeout(function() {
emitted.should.eql(false);
done();
}, 350);
})
})
describe('without callback', function() {
it('emits done event once, with error', function(done) {
var error,
called = 0,
stream = send_request();
stream.on('done', function(err) {
err.code.should.equal('ECONNRESET');
called++;
})
setTimeout(function() {
called.should.equal(1);
done();
}, 250)
})
it('aborts the request', function(done) {
var time = new Date();
var stream = send_request();
stream.on('done', function(err) {
var timediff = (new Date() - time);
timediff.should.be.within(200, 300);
done();
})
})
it('error should be ECONNRESET', function(done) {
var error,
stream = send_request();
stream.on('done', function(err) {
err.code.should.equal('ECONNRESET')
done();
})
})
it('does not emit a readable event', function(done) {
var called = false,
stream = send_request();
stream.on('readable', function() {
called = true;
})
stream.on('done', function(err) {
called.should.be.false;
done();
})
})
it('does not emit an error event', function(done) {
var emitted = false;
var stream = send_request();
stream.on('error', function() {
emitted = true;
})
stream.on('done', function(err) {
err.should.be.a.Error;
err.code.should.equal('ECONNRESET')
emitted.should.eql(false);
done();
})
})
})
})
})
@@ -0,0 +1,203 @@
var http = require('http'),
helpers = require('./helpers'),
should = require('should');
var port = 54321;
describe('request headers', function() {
var needle,
server,
existing_sockets,
original_defaultMaxSockets;
before(function(done) {
setTimeout(function() {
existing_sockets = get_active_sockets().length;
server = helpers.server({ port: port }, done);
}, 100);
})
after(function(done) {
server.close(done);
})
function send_request(opts, cb) {
needle.get('http://localhost:' + port, opts, cb);
}
function get_active_sockets() {
var handles = process._getActiveHandles();
return handles.filter(function(el) {
if (el.constructor.name.toString() == 'Socket') {
return el.destroyed !== true;
}
})
}
describe('old node versions (<0.11.4) with persistent keep-alive connections', function() {
before(function() {
delete require.cache[require.resolve('..')] // in case it was already loaded
original_defaultMaxSockets = http.Agent.defaultMaxSockets;
http.Agent.defaultMaxSockets = 5;
needle = require('..');
})
after(function() {
http.Agent.defaultMaxSockets = original_defaultMaxSockets;
delete require.cache[require.resolve('..')]
})
describe('default options', function() {
it('sends a Connection: close header', function(done) {
send_request({}, function(err, resp) {
resp.body.headers['connection'].should.eql('close');
done();
})
})
it('no open sockets remain after request', function(done) {
send_request({}, function(err, resp) {
setTimeout(function() {
get_active_sockets().length.should.eql(existing_sockets);
done();
}, 10)
});
})
})
describe('passing connection: close', function() {
it('sends a Connection: close header', function(done) {
send_request({ connection: 'close' }, function(err, resp) {
resp.body.headers['connection'].should.eql('close');
done();
})
})
it('no open sockets remain after request', function(done) {
send_request({ connection: 'close' }, function(err, resp) {
setTimeout(function() {
get_active_sockets().length.should.eql(existing_sockets);
done();
}, 10)
});
})
})
describe('passing connection: keep-alive', function() {
it('sends a Connection: keep-alive header (using options.headers.connection)', function(done) {
send_request({ headers: { connection: 'keep-alive' }}, function(err, resp) {
resp.body.headers['connection'].should.eql('keep-alive');
done();
})
})
it('sends a Connection: keep-alive header (using options.connection)', function(done) {
send_request({ connection: 'keep-alive' }, function(err, resp) {
resp.body.headers['connection'].should.eql('keep-alive');
done();
})
})
it('one open socket remain after request', function(done) {
send_request({ connection: 'keep-alive' }, function(err, resp) {
get_active_sockets().length.should.eql(existing_sockets + 1);
done();
});
})
})
})
describe('new node versions with smarter connection disposing', function() {
before(function() {
delete require.cache[require.resolve('..')]
original_defaultMaxSockets = http.Agent.defaultMaxSockets;
http.Agent.defaultMaxSockets = Infinity;
needle = require('..');
})
after(function() {
http.Agent.defaultMaxSockets = original_defaultMaxSockets;
delete require.cache[require.resolve('..')]
})
describe('default options', function() {
// TODO:
// this is weird. by default, new node versions set a 'close' header
// while older versions set a keep-alive header
it.skip('sets a Connection header', function(done) {
send_request({}, function(err, resp) {
// should.not.exist(resp.body.headers['connection']);
// done();
})
})
it.skip('one open sockets remain after request', function(done) {
send_request({}, function(err, resp) {
// get_active_sockets().length.should.eql(1);
// done();
});
})
})
describe('passing connection: close', function() {
it('sends a Connection: close header', function(done) {
send_request({ connection: 'close' }, function(err, resp) {
resp.body.headers['connection'].should.eql('close');
done();
})
})
it('no open sockets remain after request', function(done) {
send_request({ connection: 'close' }, function(err, resp) {
setTimeout(function() {
get_active_sockets().length.should.eql(existing_sockets);
done();
}, 10);
});
})
})
describe('passing connection: keep-alive', function() {
it('sends a Connection: keep-alive header (using options.headers.connection)', function(done) {
send_request({ headers: { connection: 'keep-alive' }}, function(err, resp) {
resp.body.headers['connection'].should.eql('keep-alive');
done();
})
})
it('sends a Connection: keep-alive header (using options.connection)', function(done) {
send_request({ connection: 'keep-alive' }, function(err, resp) {
resp.body.headers['connection'].should.eql('keep-alive');
done();
})
})
it('one open socket remain after request', function(done) {
send_request({ connection: 'keep-alive' }, function(err, resp) {
get_active_sockets().length.should.eql(existing_sockets + 1);
done();
});
})
})
})
})
@@ -0,0 +1,74 @@
var fs = require('fs');
var protocols = {
http : require('http'),
https : require('https')
}
var keys = {
cert : fs.readFileSync(__dirname + '/keys/ssl.cert'),
key : fs.readFileSync(__dirname + '/keys/ssl.key')
}
var helpers = {};
helpers.server = function(opts, cb) {
var defaults = {
code : 200,
headers : {'Content-Type': 'application/json'}
}
var mirror_response = function(req) {
return JSON.stringify({
headers: req.headers,
body: req.body
})
}
var get = function(what) {
if (!opts[what])
return defaults[what];
if (typeof opts[what] == 'function')
return opts[what](); // set them at runtime
else
return opts[what];
}
var finish = function(req, res) {
if (opts.handler) return opts.handler(req, res);
res.writeHead(get('code'), get('headers'));
res.end(opts.response || mirror_response(req));
}
var handler = function(req, res) {
req.setEncoding('utf8'); // get as string
req.body = '';
req.on('data', function(str) { req.body += str })
req.socket.on('error', function(e) {
// res.writeHead(500, {'Content-Type': 'text/plain'});
// res.end('Error: ' + e.message);
})
setTimeout(function(){
finish(req, res);
}, opts.wait || 0);
};
var protocol = opts.protocol || 'http';
var server;
if (protocol == 'https')
server = protocols[protocol].createServer(keys, handler);
else
server = protocols[protocol].createServer(handler);
server.listen(opts.port, cb);
return server;
}
module.exports = helpers;
@@ -0,0 +1,43 @@
var needle = require('../'),
should = require('should');
describe('when posting a very long string', function() {
this.timeout(20000);
function get_string(length) {
var str = '';
for (var i = 0; i < length; i++) {
str += 'x';
}
return str;
}
var major_version = process.version.split('.')[0];
it("shouldn't throw an EPIPE error out of nowhere", function(done) {
// for some reason this test fails in Github Actions with Node v8.x
// although in my Linux box passes without issues
if (process.env.CI && (major_version == 'v8' || major_version == 'v6')) {
return done();
}
var error;
function finished() {
setTimeout(function() {
should.not.exist(error);
done();
}, 300);
}
try {
needle.post('https://google.com', { data: get_string(Math.pow(2, 20)) }, finished)
} catch(e) {
error = e;
}
})
})
@@ -0,0 +1,81 @@
var should = require('should'),
needle = require('./../'),
helpers = require('./helpers');
describe('receiving json and xml content as string', function() {
this.timeout(5000);
["text/plain", "application/json", "application/ld+json", "application/xml", "image/svg+xml"].forEach(function(mimetype, offset){
describe('Given content-type: "'+mimetype+'"', function () {
var server, port = 54330+offset;
before(function(done) {
server = helpers.server({
port: port,
response: 'content',
headers: { 'Content-Type': mimetype }
}, done);
})
after(function(done) {
server.close(done)
})
describe('with parse = false', function () {
it('delivers by default as string', function (done) {
needle.get('http://localhost:' + port, { parse: false }, function (err, resp) {
resp.body.should.be.a.String;
(typeof resp.body).should.eql('string')
done();
})
})
})
})
});
["application/octet-stream", "image/png"].forEach(function(mimetype, offset){
describe('Given content-type: "'+mimetype+'"', function () {
var server, port = 54340+offset;
before(function(done) {
server = helpers.server({
port: port,
response: 'content',
headers: { 'Content-Type': mimetype }
}, done);
})
after(function(done) {
server.close(done)
})
describe('with parse = false', function () {
it('delivers by default as Buffer', function (done) {
needle.get('http://localhost:' + port, { parse: false }, function (err, resp) {
resp.body.should.be.a.Buffer;
(resp.body instanceof Buffer).should.eql(true)
done();
})
})
})
})
})
})
@@ -0,0 +1,254 @@
var should = require('should'),
needle = require('./../'),
http = require('http'),
sinon = require('sinon'),
stream = require('stream'),
fs = require('fs'),
port = 11111,
server;
describe('with output option', function() {
var server, handler, file = '/tmp/foobar.out';
function send_request_cb(where, cb) {
var url = 'http://localhost:' + port + '/whatever.file';
return needle.get(url, { output: where }, cb);
}
function send_request_stream(where, cb) {
var url = 'http://localhost:' + port + '/whatever.file';
var stream = needle.get(url, { output: where });
stream.on('end', cb);
}
// this will only work in UNICES
function get_open_file_descriptors() {
var list = fs.readdirSync('/proc/self/fd');
return list.length;
}
var send_request = send_request_cb;
before(function(){
server = http.createServer(function(req, res) {
handler(req, res);
}).listen(port);
});
after(function() {
server.close();
})
beforeEach(function() {
try { fs.unlinkSync(file) } catch(e) { };
})
describe('and a 404 response', function() {
before(function() {
handler = function(req, res) {
res.writeHead(404, {'Content-Type': 'text/plain' });
res.end();
}
})
it('doesnt attempt to write a file', function(done) {
var spy = sinon.spy(fs, 'createWriteStream');
send_request(file, function(err, resp) {
resp.statusCode.should.eql(404);
spy.called.should.eql(false);
spy.restore();
done();
})
})
it('doesnt actually write a file', function(done) {
send_request(file, function(err, resp) {
resp.statusCode.should.eql(404);
fs.existsSync(file).should.eql(false);
done();
})
})
})
describe('and a 200 response', function() {
describe('for an empty response', function() {
before(function() {
handler = function(req, res) {
res.writeHead(200, { 'Content-Type': 'text/plain' });
res.end();
}
})
it('uses a writableStream', function(done) {
var spy = sinon.spy(fs, 'createWriteStream');
send_request(file, function(err, resp) {
resp.statusCode.should.eql(200);
spy.called.should.eql(true);
spy.restore();
done();
})
})
it('writes a file', function(done) {
fs.existsSync(file).should.eql(false);
send_request(file, function(err, resp) {
fs.existsSync(file).should.eql(true);
done();
})
})
it('file is zero bytes in length', function(done) {
send_request(file, function(err, resp) {
fs.statSync(file).size.should.equal(0);
done();
})
})
if (process.platform != 'win32') {
it('closes the file descriptor', function(done) {
var open_descriptors = get_open_file_descriptors();
send_request(file + Math.random(), function(err, resp) {
var current_descriptors = get_open_file_descriptors();
open_descriptors.should.eql(current_descriptors);
done()
})
})
}
})
describe('for a JSON response', function() {
before(function() {
handler = function(req, res) {
res.writeHead(200, { 'Content-Type': 'application/javascript' });
res.end(JSON.stringify({foo: 'bar'}));
}
})
it('uses a writableStream', function(done) {
var spy = sinon.spy(fs, 'createWriteStream');
send_request(file, function(err, resp) {
resp.statusCode.should.eql(200);
spy.called.should.eql(true);
spy.restore();
done();
})
})
it('writes a file', function(done) {
fs.existsSync(file).should.eql(false);
send_request(file, function(err, resp) {
fs.existsSync(file).should.eql(true);
done();
})
})
it('file size equals response length', function(done) {
send_request(file, function(err, resp) {
fs.statSync(file).size.should.equal(resp.bytes);
done();
})
})
it('response pipeline is honoured (JSON is decoded by default)', function(done) {
send_request_stream(file, function(err, resp) {
// we need to wait a bit since writing to config.output
// happens independently of needle's callback logic.
setTimeout(function() {
fs.readFileSync(file).toString().should.eql('{\"foo\":\"bar\"}');
done();
}, 20);
})
})
it('closes the file descriptor', function(done) {
var open_descriptors = get_open_file_descriptors();
send_request(file + Math.random(), function(err, resp) {
var current_descriptors = get_open_file_descriptors();
open_descriptors.should.eql(current_descriptors);
done()
})
})
})
describe('for a binary file', function() {
var pixel = Buffer.from("base64,R0lGODlhAQABAIAAAAUEBAAAACwAAAAAAQABAAACAkQBADs", "base64");
before(function() {
handler = function(req, res) {
res.writeHead(200, { 'Content-Type': 'application/octet-stream', 'Transfer-Encoding': 'chunked' });
res.write(pixel.slice(0, 10));
res.write(pixel.slice(10, 20));
res.write(pixel.slice(20, 30));
res.write(pixel.slice(30));
res.end();
}
})
it('uses a writableStream', function(done) {
var spy = sinon.spy(fs, 'createWriteStream');
send_request(file, function(err, resp) {
resp.statusCode.should.eql(200);
spy.called.should.eql(true);
spy.restore();
done();
})
})
it('writes a file', function(done) {
fs.existsSync(file).should.eql(false);
send_request(file, function(err, resp) {
fs.existsSync(file).should.eql(true);
done();
})
})
it('file size equals response length', function(done) {
send_request(file, function(err, resp) {
fs.statSync(file).size.should.equal(resp.bytes);
done();
})
})
it('file is equal to original buffer', function(done) {
send_request(file, function(err, resp) {
// we need to wait a bit since writing to config.output
// happens independently of needle's callback logic.
setTimeout(function() {
fs.readFileSync(file).should.eql(pixel);
done();
}, 20);
})
})
it('returns the data in resp.body too', function(done) {
send_request(file, function(err, resp) {
resp.body.should.eql(pixel);
done();
})
})
if (process.platform != 'win32') {
it('closes the file descriptor', function(done) {
var open_descriptors = get_open_file_descriptors();
send_request(file + Math.random(), function(err, resp) {
var current_descriptors = get_open_file_descriptors();
open_descriptors.should.eql(current_descriptors);
done()
})
})
}
})
})
})
@@ -0,0 +1,547 @@
var should = require('should'),
needle = require('./../'),
http = require('http'),
port = 11111,
server;
describe('parsing', function(){
describe('when response is an JSON string', function(){
var json_string = '{"foo":"bar"}';
before(function(done){
server = http.createServer(function(req, res) {
res.setHeader('Content-Type', 'application/json');
res.end(json_string);
}).listen(port, done);
});
after(function(done){
server.close(done);
})
describe('and parse option is not passed', function() {
describe('with default parse_response', function() {
before(function() {
needle.defaults().parse_response.should.eql('all')
})
it('should return object', function(done){
needle.get('localhost:' + port, function(err, response, body){
should.ifError(err);
body.should.have.property('foo', 'bar');
done();
})
})
})
describe('and default parse_response is set to false', function() {
it('does NOT return object when disabled using .defaults', function(done){
needle.defaults({ parse_response: false })
needle.get('localhost:' + port, function(err, response, body) {
should.not.exist(err);
body.should.be.an.instanceof(String)
body.toString().should.eql('{"foo":"bar"}');
needle.defaults({ parse_response: 'all' });
done();
})
})
})
})
describe('and parse option is true', function() {
describe('and JSON is valid', function() {
it('should return object', function(done) {
needle.get('localhost:' + port, { parse: true }, function(err, response, body){
should.not.exist(err);
body.should.have.property('foo', 'bar')
done();
})
})
it('should have a .parser = json property', function(done) {
needle.get('localhost:' + port, { parse: true }, function(err, resp) {
should.not.exist(err);
resp.parser.should.eql('json');
done();
})
})
});
describe('and response is empty', function() {
var old_json_string;
before(function() {
old_json_string = json_string;
json_string = "";
});
after(function() {
json_string = old_json_string;
});
it('should return an empty string', function(done) {
needle.get('localhost:' + port, { parse: true }, function(err, resp) {
should.not.exist(err);
resp.body.should.equal('');
done();
})
})
})
describe('and JSON is invalid', function() {
var old_json_string;
before(function() {
old_json_string = json_string;
json_string = "this is not going to work";
});
after(function() {
json_string = old_json_string;
});
it('does not throw', function(done) {
(function(){
needle.get('localhost:' + port, { parse: true }, done);
}).should.not.throw();
});
it('does NOT return object', function(done) {
needle.get('localhost:' + port, { parse: true }, function(err, response, body) {
should.not.exist(err);
body.should.be.a.String;
body.toString().should.eql('this is not going to work');
done();
})
})
});
})
describe('and parse option is false', function() {
it('does NOT return object', function(done){
needle.get('localhost:' + port, { parse: false }, function(err, response, body) {
should.not.exist(err);
body.should.be.an.instanceof(String)
body.toString().should.eql('{"foo":"bar"}');
done();
})
})
it('should NOT have a .parser = json property', function(done) {
needle.get('localhost:' + port, { parse: false }, function(err, resp) {
should.not.exist(err);
should.not.exist(resp.parser);
done();
})
})
})
describe('and parse option is "xml"', function() {
it('does NOT return object', function(done){
needle.get('localhost:' + port, { parse: 'xml' }, function(err, response, body) {
should.not.exist(err);
body.should.be.an.instanceof(String)
body.toString().should.eql('{"foo":"bar"}');
done();
})
})
it('should NOT have a .parser = json property', function(done) {
needle.get('localhost:' + port, { parse: 'xml' }, function(err, resp) {
should.not.exist(err);
should.not.exist(resp.parser);
done();
})
})
})
});
describe('when response is JSON \'false\'', function(){
var json_string = 'false';
before(function(done){
server = http.createServer(function(req, res) {
res.setHeader('Content-Type', 'application/json');
res.end(json_string);
}).listen(port, done);
});
after(function(done){
server.close(done);
})
describe('and parse option is not passed', function() {
it('should return object', function(done){
needle.get('localhost:' + port, function(err, response, body){
should.ifError(err);
body.should.equal(false);
done();
})
})
})
describe('and parse option is true', function() {
describe('and JSON is valid', function() {
it('should return object', function(done){
needle.get('localhost:' + port, { parse: true }, function(err, response, body){
should.not.exist(err);
body.should.equal(false)
done();
})
})
});
describe('and response is empty', function() {
var old_json_string;
before(function() {
old_json_string = json_string;
json_string = "";
});
after(function() {
json_string = old_json_string;
});
it('should return an empty string', function(done) {
needle.get('localhost:' + port, { parse: true }, function(err, resp) {
should.not.exist(err);
resp.body.should.equal('');
done();
})
})
})
describe('and JSON is invalid', function() {
var old_json_string;
before(function() {
old_json_string = json_string;
json_string = "this is not going to work";
});
after(function() {
json_string = old_json_string;
});
it('does not throw', function(done) {
(function(){
needle.get('localhost:' + port, { parse: true }, done);
}).should.not.throw();
});
it('does NOT return object', function(done) {
needle.get('localhost:' + port, { parse: true }, function(err, response, body) {
should.not.exist(err);
body.should.be.a.String;
body.toString().should.eql('this is not going to work');
done();
})
})
});
})
describe('and parse option is false', function() {
it('does NOT return object', function(done){
needle.get('localhost:' + port, { parse: false }, function(err, response, body) {
should.not.exist(err);
body.should.be.an.instanceof(String)
body.toString().should.eql('false');
done();
})
})
})
describe('and parse option is "xml"', function() {
it('does NOT return object', function(done){
needle.get('localhost:' + port, { parse: 'xml' }, function(err, response, body) {
should.not.exist(err);
body.should.be.an.instanceof(String)
body.toString().should.eql('false');
done();
})
})
})
});
describe('when response is an invalid XML string', function(){
before(function(done){
server = http.createServer(function(req, res) {
res.writeHeader(200, {'Content-Type': 'application/xml'})
res.end("<post><body there11post>")
}).listen(port, done);
});
after(function(done){
server.close(done);
})
describe('and parse_response is true', function(){
it('should return original string', function(done) {
needle.get('localhost:' + port, { parse_response: true }, function(err, response, body) {
should.not.exist(err);
body.should.eql('<post><body there11post>')
should.not.exist(body.name);
done();
})
})
it('should not have a .parser = xml property', function(done) {
needle.get('localhost:' + port, { parse_response: true }, function(err, resp) {
should.not.exist(err);
should.not.exist(resp.parser);
done();
})
})
})
describe('and parse response is false', function(){
it('should return valid object', function(done) {
needle.get('localhost:' + port, { parse_response: false }, function(err, response, body){
should.not.exist(err);
body.toString().should.eql('<post><body there11post>')
done();
})
})
it('should not have a .parser property', function(done) {
needle.get('localhost:' + port, { parse_response: false }, function(err, resp) {
should.not.exist(err);
should.not.exist(resp.parser)
done();
})
})
})
})
describe('when response is a valid XML string', function(){
before(function(done) {
server = http.createServer(function(req, res) {
res.writeHeader(200, {'Content-Type': 'application/xml'})
res.end("<post><p>hello</p><p><![CDATA[world]]></p></post>")
}).listen(port, done);
});
after(function(done) {
server.close(done);
})
describe('and parse_response is true', function(){
it('should return valid object', function(done) {
needle.get('localhost:' + port, { parse_response: true }, function(err, response, body) {
should.not.exist(err);
body.name.should.eql('post')
body.children[0].name.should.eql('p')
body.children[0].value.should.eql('hello')
body.children[1].name.should.eql('p')
body.children[1].value.should.eql('world')
done();
})
})
it('should have a .parser = xml property', function(done) {
needle.get('localhost:' + port, { parse_response: true }, function(err, resp) {
should.not.exist(err);
resp.parser.should.eql('xml');
done();
})
})
})
describe('and parse response is false', function(){
it('should return valid object', function(done) {
needle.get('localhost:' + port, { parse_response: false }, function(err, response, body){
should.not.exist(err);
body.toString().should.eql('<post><p>hello</p><p><![CDATA[world]]></p></post>')
done();
})
})
it('should not have a .parser property', function(done) {
needle.get('localhost:' + port, { parse_response: false }, function(err, resp) {
should.not.exist(err);
should.not.exist(resp.parser)
done();
})
})
})
})
describe('valid XML, using xml2js', function() {
var parsers, origParser;
before(function(done) {
var xml2js = require('xml2js')
parsers = require('../lib/parsers');
origParser = parsers['application/xml'];
var customParser = require('xml2js').parseString;
parsers.use('xml2js', ['application/xml'], function(buff, cb) {
var opts = { explicitRoot: true, explicitArray: false };
customParser(buff, opts, cb);
})
server = http.createServer(function(req, res) {
res.writeHeader(200, {'Content-Type': 'application/xml'})
res.end("<post><p>hello</p><p>world</p></post>")
}).listen(port, done);
});
after(function(done) {
parsers['application/xml'] = origParser;
server.close(done);
})
describe('and parse_response is true', function(){
it('should return valid object', function(done) {
needle.get('localhost:' + port, { parse_response: true }, function(err, response, body) {
should.not.exist(err);
body.should.eql({ post: { p: ['hello', 'world' ]}})
done();
})
})
it('should have a .parser = xml property', function(done) {
needle.get('localhost:' + port, { parse_response: true }, function(err, resp) {
should.not.exist(err);
resp.parser.should.eql('xml2js');
done();
})
})
})
describe('and parse response is false', function(){
it('should return valid object', function(done) {
needle.get('localhost:' + port, { parse_response: false }, function(err, response, body){
should.not.exist(err);
body.toString().should.eql('<post><p>hello</p><p>world</p></post>')
done();
})
})
it('should not have a .parser property', function(done) {
needle.get('localhost:' + port, { parse_response: false }, function(err, resp) {
should.not.exist(err);
should.not.exist(resp.parser)
done();
})
})
})
})
describe('when response is a JSON API flavored JSON string', function () {
var json_string = '{"data":[{"type":"articles","id":"1","attributes":{"title":"Needle","body":"The leanest and most handsome HTTP client in the Nodelands."}}],"included":[{"type":"people","id":"42","attributes":{"name":"Tomás"}}]}';
before(function(done){
server = http.createServer(function(req, res) {
res.setHeader('Content-Type', 'application/vnd.api+json');
res.end(json_string);
}).listen(port, done);
});
after(function(done){
server.close(done);
});
describe('and parse option is not passed', function() {
describe('with default parse_response', function() {
before(function() {
needle.defaults().parse_response.should.eql('all')
})
it('should return object', function(done){
needle.get('localhost:' + port, function(err, response, body){
should.ifError(err);
body.should.deepEqual({
"data": [{
"type": "articles",
"id": "1",
"attributes": {
"title": "Needle",
"body": "The leanest and most handsome HTTP client in the Nodelands."
}
}],
"included": [
{
"type": "people",
"id": "42",
"attributes": {
"name": "Tomás"
}
}
]
});
done();
});
});
});
})
});
})
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,202 @@
var helpers = require('./helpers'),
should = require('should'),
sinon = require('sinon'),
http = require('http'),
needle = require('./../');
var port = 7707;
var url = 'localhost:' + port;
var nonexisting_host = 'awepfokawepofawe.com';
describe('proxy option', function() {
var spy, opts;
function send_request(opts, done) {
if (spy) spy.restore();
spy = sinon.spy(http, 'request');
needle.get(url, opts, done);
}
//////////////////////
// proxy opts helpers
function not_proxied(done) {
return function(err, resp) {
var path = spy.args[0][0].path;
path.should.eql('/'); // not the full original URI
spy.restore();
done();
}
}
function proxied(host, port, done) {
return function(err, resp) {
var path = spy.args[0][0].path;
path.should.eql('http://' + url); // the full original URI
var http_host = spy.args[0][0].host;
if (http_host) http_host.should.eql(host);
var http_port = spy.args[0][0].port;
if (http_port) http_port.should.eql(port);
spy.restore();
done();
}
}
//////////////////////
// auth helpers
function get_auth(header) {
var token = header.split(/\s+/).pop();
return token && Buffer.from(token, 'base64').toString().split(':');
}
function no_proxy_auth(done) {
return function(err, resp) {
var headers = spy.args[0][0].headers;
Object.keys(headers).should.not.containEql('proxy-authorization');
done();
}
}
function header_set(name, user, pass, done) {
return function(err, resp) {
var headers = spy.args[0][0].headers;
var auth = get_auth(headers[name]);
auth[0].should.eql(user);
auth[1].should.eql(pass);
done();
}
}
function proxy_auth_set(user, pass, done) {
return header_set('proxy-authorization', user, pass, done);
}
function basic_auth_set(user, pass, done) {
return header_set('authorization', user, pass, done);
}
after(function() {
spy.restore();
})
describe('when null proxy is passed', function() {
it('does not proxy', function(done) {
send_request({ proxy: null }, not_proxied(done))
})
describe('but defaults has been set', function() {
before(function() {
needle.defaults({ proxy: 'foobar' });
})
after(function() {
needle.defaults({ proxy: null });
})
it('tries to proxy anyway', function(done) {
send_request({}, proxied('foobar', 80, done))
})
})
})
describe('when weird string is passed', function() {
it('tries to proxy anyway', function(done) {
send_request({ proxy: 'alfalfa' }, proxied('alfalfa', 80, done))
})
})
describe('when valid url is passed', function() {
it('proxies request', function(done) {
send_request({ proxy: nonexisting_host + ':123/done' }, proxied(nonexisting_host, '123', done))
})
it('does not set a Proxy-Authorization header', function(done) {
send_request({ proxy: nonexisting_host + ':123/done' }, no_proxy_auth(done));
})
describe('and proxy url contains user:pass', function() {
before(function() {
opts = {
proxy: 'http://mj:x@' + nonexisting_host + ':123/done'
}
})
it('proxies request', function(done) {
send_request(opts, proxied(nonexisting_host, '123', done))
})
it('sets Proxy-Authorization header', function(done) {
send_request(opts, proxy_auth_set('mj', 'x', done));
})
})
describe('and a proxy_user is passed', function() {
before(function() {
opts = {
proxy: nonexisting_host + ':123',
proxy_user: 'someone',
proxy_pass: 'else'
}
})
it('proxies request', function(done) {
send_request(opts, proxied(nonexisting_host, '123', done))
})
it('sets Proxy-Authorization header', function(done) {
send_request(opts, proxy_auth_set('someone', 'else', done));
})
describe('and url also contains user:pass', function() {
it('url user:pass wins', function(done) {
var opts = {
proxy: 'http://xxx:yyy@' + nonexisting_host + ':123',
proxy_user: 'someone',
proxy_pass: 'else'
}
send_request(opts, proxy_auth_set('xxx', 'yyy', done));
})
})
describe('and options.username is also present', function() {
before(function() {
opts = { proxy_user: 'foobar', username: 'someone' };
})
it('a separate Authorization header is set', function(done) {
var opts = {
proxy: nonexisting_host + ':123',
proxy_user: 'someone',
proxy_pass: 'else',
username: 'test',
password: 'X'
}
send_request(opts, basic_auth_set('test', 'X', done));
})
})
})
})
})
@@ -0,0 +1,128 @@
var should = require('should'),
stringify = require('../lib/querystring').build;
describe('stringify', function() {
describe('with null', function() {
it('throws', function() {
(function() {
var res = stringify(null);
}).should.throw();
})
})
describe('with a number', function() {
it('throws', function() {
(function() {
var res = stringify(100);
}).should.throw();
})
})
describe('with a string', function() {
describe('that is empty', function() {
it('throws', function() {
(function() {
var res = stringify('');
}).should.throw();
})
})
describe('that doesnt contain an equal sign', function() {
it('throws', function() {
(function() {
var res = stringify('boomshagalaga');
}).should.throw();
})
})
describe('that contains an equal sign', function() {
it('works', function() {
var res = stringify('hello=123');
res.should.eql('hello=123');
})
})
})
describe('with an array', function() {
describe('with key val objects', function() {
it('works', function() {
var res = stringify([ {foo: 'bar'} ]);
res.should.eql('foo=bar');
})
})
describe('where all elements are strings with an equal sign', function() {
it('works', function() {
var res = stringify([ 'bar=123', 'quux=' ]);
res.should.eql('bar=123&quux=');
})
})
describe('with random words', function() {
it('throws', function() {
(function() {
var res = stringify(['hello', 'there']);
}).should.throw();
})
})
describe('with integers', function() {
it('throws', function() {
(function() {
var res = stringify([123, 432]);
}).should.throw();
})
})
})
describe('with an object', function() {
it('works', function() {
var res = stringify({ test: 100 });
res.should.eql('test=100');
})
describe('with object where val is an array', function() {
it('works', function() {
var res = stringify({ foo: ['bar', 'baz'] });
res.should.eql('foo[]=bar&foo[]=baz');
})
})
describe('with object where val is an array of key val objects', function() {
it('works', function() {
var res = stringify({ foo: [{'1': 'bar'}, {'2': 'baz'}] });
res.should.eql('foo[][1]=bar&foo[][2]=baz');
})
})
})
})
@@ -0,0 +1,394 @@
var helpers = require('./helpers'),
should = require('should'),
sinon = require('sinon'),
needle = require('./../');
var ports = {
http : 8888,
https : 9999
}
var protocols = {
http : require('http'),
https : require('https')
}
var code = 301;
var location; // var to set the response location
function response_code() {
return code;
}
function response_headers() {
return { 'Content-Type': 'text/plain', 'Location': location }
}
describe('redirects', function() {
var spies = {},
servers = {};
var current_protocol;
var hostname = require('os').hostname();
// open two servers, one that responds to a redirect
before(function(done) {
var conf = {
port : ports.http,
code : response_code,
headers : response_headers
}
servers.http = helpers.server(conf, function() {
conf.port = ports.https;
conf.protocol = 'https';
servers.https = helpers.server(conf, done);
});
})
after(function(done) {
servers.http.close(function() {
servers.https.close(done);
});
})
var prots = {'http': 'https'};
Object.keys(prots).forEach(function(protocol) {
current_protocol = protocol;
var other_protocol = protocol == 'http' ? 'https' : 'http';
var opts, // each test will modify this
host = '127.0.0.1',
url = protocol + '://' + host + ':' + ports[protocol] + '/hello';
function send_request(opts, cb) {
opts.rejectUnauthorized = false;
// console.log(' -- sending request ' + url + ' -- redirect to ' + location);
needle.post(url, { foo: 'bar' }, opts, cb);
}
function not_followed(done) {
send_request(opts, function(err, resp) {
resp.statusCode.should.eql(301);
if (current_protocol == 'http') {
spies.http.callCount.should.eql(1); // only original request
spies.https.callCount.should.eql(0);
} else {
spies.http.callCount.should.eql(0);
spies.https.callCount.should.eql(1); // only original request
}
done();
})
}
function followed_same_protocol(done) {
send_request(opts, function(err, resp) {
// the original request plus the redirect one
spies[current_protocol].callCount.should.eql(2);
done();
})
}
function followed_other_protocol(done) {
send_request(opts, function(err, resp) {
// on new-ish node versions, https.request calls http.request internally,
// so we need to amount for that additional call.
// update: this doesn't happen on node > 10.x
var node_major_ver = process.version.split('.')[0].replace('v', '');
var http_calls = protocols.http.Agent.defaultMaxSockets == Infinity && parseInt(node_major_ver) < 10 ? 2 : 1;
spies.http.callCount.should.eql(http_calls); // the one(s) from http.request
spies.https.callCount.should.eql(1); // the one from https.request (redirect)
done();
})
}
// set a spy on [protocol].request
// so we can see how many times a request was made
before(function() {
spies.http = sinon.spy(protocols.http, 'request');
spies.https = sinon.spy(protocols.https, 'request');
})
// and make sure it is restored after each test
afterEach(function() {
spies.http.reset();
spies.https.reset();
})
after(function() {
spies.http.restore();
spies.https.restore();
})
describe('when overriding defaults', function() {
before(function() {
needle.defaults({ follow_max: 10 });
opts = {};
})
after(function() {
// reset values to previous
needle.defaults({ follow_max: 0 });
})
describe('and redirected to the same path on same host and protocol', function() {
before(function() {
location = url;
})
it('does not follow redirect', not_followed);
})
describe('and redirected to the same path on same host and different protocol', function() {
before(function() {
location = url.replace(protocol, other_protocol).replace(ports[protocol], ports[other_protocol]);
})
it('follows redirect', followed_other_protocol);
})
describe('and redirected to a different path on same host, same protocol', function() {
before(function() {
location = url.replace('/hello', '/goodbye');
})
it('follows redirect', followed_same_protocol);
})
describe('and redirected to a different path on same host, different protocol', function() {
before(function() {
location = url.replace('/hello', '/goodbye').replace(protocol, other_protocol).replace(ports[protocol], ports[other_protocol]);
})
it('follows redirect', followed_other_protocol);
})
describe('and redirected to same path on another host, same protocol', function() {
before(function() {
location = url.replace(host, hostname);
})
it('follows redirect', followed_same_protocol);
})
describe('and redirected to same path on another host, different protocol', function() {
before(function() {
location = url.replace(host, hostname).replace(protocol, other_protocol).replace(ports[protocol], ports[other_protocol]);
})
it('follows redirect', followed_other_protocol);
})
})
// false and null have the same result
var values = [false, null];
values.forEach(function(value) {
describe('when follow is ' + value, function() {
before(function() {
opts = { follow: value };
})
describe('and redirected to the same path on same host and protocol', function() {
before(function() {
location = url;
})
it('throws an error', function() {
(function() {
send_request(opts, function() { });
}).should.throw;
})
})
})
})
describe('when follow is true', function() {
before(function() {
opts = { follow: true };
})
describe('and redirected to the same path on same host and protocol', function() {
before(function() { location = url })
it('throws an error', function() {
(function() {
send_request(opts, function() { });
}).should.throw;
})
})
})
describe('when follow is > 0', function() {
before(function() {
needle.defaults({ follow: 10 });
})
after(function() {
needle.defaults({ follow: 0 });
})
describe('when keep_method is false', function() {
before(function() {
opts = { follow_keep_method: false };
})
// defaults to follow host and protocol
describe('and redirected to the same path on same host and different protocol', function() {
before(function() {
location = url.replace(protocol, other_protocol);
})
it('follows redirect', followed_other_protocol);
it('sends a GET request with no data', function(done) {
send_request(opts, function(err, resp) {
spies.http.args[0][0].method.should.eql('GET');
// spy.args[0][3].should.eql(null);
done();
})
})
})
})
describe('and set_referer is true', function() {
before(function() {
opts = { follow_set_referer: true };
})
// defaults to follow host and protocol
describe('and redirected to the same path on same host and different protocol', function() {
before(function() {
location = url.replace(protocol, other_protocol);
})
it('follows redirect', followed_other_protocol);
it('sets Referer header when following redirect', function(done) {
send_request(opts, function(err, resp) {
spies.http.args[0][0].headers['referer'].should.eql("http://" + host + ":8888/hello");
// spies.http.args[0][3].should.eql({ foo: 'bar'});
done();
})
})
})
})
describe('and keep_method is true', function() {
before(function() {
opts = { follow_keep_method: true };
})
// defaults to follow host and protocol
describe('and redirected to the same path on same host and different protocol', function() {
before(function() {
location = url.replace(protocol, other_protocol);
})
it('follows redirect', followed_other_protocol);
it('sends a POST request with the original data', function(done) {
send_request(opts, function(err, resp) {
spies.http.args[0][0].method.should.eql('post');
// spies.http.args[0][3].should.eql({ foo: 'bar'});
done();
})
})
})
})
describe('and if_same_host is false', function() {
before(function() {
opts = { follow_if_same_host: false };
})
// by default it will follow other protocols
describe('and redirected to same path on another domain, same protocol', function() {
before(function() {
location = url.replace(host, hostname);
})
it('follows redirect', followed_same_protocol);
})
})
describe('and if_same_host is true', function() {
before(function() {
opts = { follow_if_same_host: true };
})
// by default it will follow other protocols
describe('and redirected to same path on another domain, same protocol', function() {
before(function() {
location = url.replace(host, hostname);
})
it('does not follow redirect', not_followed);
})
})
describe('and if_same_protocol is false', function() {
before(function() {
opts = { follow_if_same_protocol: false };
})
// by default it will follow other hosts
describe('and redirected to same path on another domain, different protocol', function() {
before(function() {
location = url.replace(host, hostname).replace(protocol, other_protocol).replace(ports[protocol], ports[other_protocol]);
})
it('follows redirect', followed_other_protocol);
})
})
describe('and if_same_protocol is true', function() {
before(function() {
opts = { follow_if_same_protocol: true };
})
// by default it will follow other hosts
describe('and redirected to same path on another domain, different protocol', function() {
before(function() {
location = url.replace(host, hostname).replace(protocol, other_protocol).replace(ports[protocol], ports[other_protocol]);
})
it('does not follow redirect', not_followed);
})
})
})
})
});
@@ -0,0 +1,45 @@
var should = require('should')
var needle = require('./../')
describe('follow redirects when read_timeout is set', function () {
it('clear timeout before following redirect', function (done) {
var opts = {
open_timeout: 1000,
read_timeout: 3000,
follow: 5,
user_agent: 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36'
}
var timedOut = 0
var redirects = 0
var timer = setTimeout(function () {
var hasRedirects = redirects > 0
hasRedirects.should.equal(true)
done()
}, opts.read_timeout || 3000)
var resp = needle.get('http://google.com/', opts, function (err, resp, body) {
var noErr = err === null
var hasBody = body.length > 0
noErr.should.equal(true);
hasBody.should.equal(true);
});
resp.on('redirect', function (location) {
redirects++
// console.info(' Redirected to ', location)
})
resp.on('timeout', function (type) {
timedOut++
timedOut.should.equal(0)
// console.error(' ', type, 'timeout')
clearTimeout(timer)
done()
})
}).timeout(30000)
})
@@ -0,0 +1,237 @@
var fs = require('fs'),
needle = require('..'),
stream = require('stream'),
http = require('http'),
should = require('should'),
sinon = require('sinon');
var port = 2233;
var node_major_ver = parseInt(process.version.split('.')[0].replace('v', ''));
var node_minor_ver = parseInt(process.version.split('.')[1]);
describe('request stream length', function() {
var server, writable;
function createServer() {
return http.createServer(function(req, res) {
req.on('data', function(chunk) {
// console.log(chunk.length);
})
req.on('end', function() {
res.writeHeader(200, { 'Content-Type': 'application/json'})
res.end(JSON.stringify({ headers: req.headers }))
})
})
}
before(function(done) {
server = createServer();
server.listen(port, done)
})
beforeEach(function() {
writable = new stream.Readable();
writable._read = function() {
this.push('hello world');
this.push(null);
}
})
after(function(done) {
server.close(done)
})
function send_request(opts, cb) {
needle.post('http://localhost:' + port, writable, opts, cb)
}
describe('no stream_length set', function() {
it('doesnt set Content-Length header', function(done) {
send_request({}, function(err, resp) {
should.not.exist(resp.body.headers['content-length']);
done()
})
})
if (node_major_ver >= 10) {
it('returns 400 if Transfer-Encoding is set to a blank string', function(done) {
send_request({ headers: { 'Transfer-Encoding': '' }}, function(err, resp) {
should.not.exist(err);
resp.statusCode.should.eql(400);
done()
})
})
} else {
it('doesnt work if Transfer-Encoding is set to a blank string', function(done) {
send_request({ headers: { 'Transfer-Encoding': '' }}, function(err, resp) {
err.code.should.eql('ECONNRESET');
done()
})
})
}
it('works if Transfer-Encoding is not set', function(done) {
send_request({}, function(err, resp) {
should.not.exist(err);
resp.statusCode.should.eql(200);
done()
})
})
})
describe('stream_length set to invalid value', function() {
if (node_major_ver >= 10) {
it('returns 400 if Transfer-Encoding is set to a blank string', function(done) {
send_request({ stream_length: 5, headers: { 'Transfer-Encoding': '' }}, function(err, resp) {
should.not.exist(err);
resp.statusCode.should.eql(400);
done()
})
})
it('returns 400 if Transfer-Encoding is not set', function(done) {
send_request({ stream_length: 5 }, function(err, resp) {
should.not.exist(err);
resp.statusCode.should.eql(400);
done()
})
})
} else {
it('doesnt work if Transfer-Encoding is set to a blank string', function(done) {
send_request({ stream_length: 5, headers: { 'Transfer-Encoding': '' }}, function(err, resp) {
err.code.should.eql('ECONNRESET');
done()
})
})
it('doesnt work if Transfer-Encoding is not set', function(done) {
send_request({ stream_length: 5 }, function(err, resp) {
err.code.should.eql('ECONNRESET');
done()
})
})
}
})
describe('stream_length is set to valid value', function() {
it('sets Content-Length header to that value', function(done) {
send_request({ stream_length: 11 }, function(err, resp) {
resp.body.headers['content-length'].should.eql('11');
done()
})
})
it('works if Transfer-Encoding is set to a blank string', function(done) {
send_request({ stream_length: 11, headers: { 'Transfer-Encoding': '' }}, function(err, resp) {
should.not.exist(err);
var code = node_major_ver == 10 && node_minor_ver > 15 ? 400 : 200;
resp.statusCode.should.eql(code);
done()
})
})
it('works if Transfer-Encoding is not set', function(done) {
send_request({ stream_length: 11 }, function(err, resp) {
should.not.exist(err);
resp.statusCode.should.eql(200);
done()
})
})
})
describe('stream_length set to 0', function() {
describe('stream with path', function() {
var stub;
beforeEach(function() {
writable.path = '/foo/bar';
stub = sinon.stub(fs, 'stat').callsFake(function(path, cb) {
cb(null, { size: 11 })
})
})
afterEach(function() {
stub.restore();
})
it('sets Content-Length header to streams length', function(done) {
send_request({ stream_length: 0 }, function(err, resp) {
resp.body.headers['content-length'].should.eql('11');
done()
})
})
it('works if Transfer-Encoding is set to a blank string', function(done) {
send_request({ stream_length: 0, headers: { 'Transfer-Encoding': '' }}, function(err, resp) {
should.not.exist(err);
var code = node_major_ver == 10 && node_minor_ver > 15 ? 400 : 200;
resp.statusCode.should.eql(code);
done()
})
})
it('works if Transfer-Encoding is not set', function(done) {
send_request({ stream_length: 0 }, function(err, resp) {
should.not.exist(err);
resp.statusCode.should.eql(200);
done()
})
})
})
describe('stream without path', function() {
it('does not set Content-Length header', function(done) {
send_request({ stream_length: 0 }, function(err, resp) {
should.not.exist(resp.body.headers['content-length']);
done()
})
})
if (node_major_ver >= 10) {
it('returns 400 if Transfer-Encoding is set to a blank string', function(done) {
send_request({ stream_length: 0, headers: { 'Transfer-Encoding': '' }}, function(err, resp) {
should.not.exist(err);
resp.statusCode.should.eql(400);
done()
})
})
} else {
it('throws ECONNRESET if Transfer-Encoding is set to a blank string', function(done) {
send_request({ stream_length: 0, headers: { 'Transfer-Encoding': '' }}, function(err, resp) {
err.code.should.eql('ECONNRESET');
done()
})
})
}
it('works if Transfer-Encoding is not set', function(done) {
send_request({ stream_length: 0 }, function(err, resp) {
should.not.exist(err);
resp.statusCode.should.eql(200);
done()
})
})
})
})
})
@@ -0,0 +1,138 @@
var should = require('should'),
needle = require('./../'),
http = require('http'),
stream = require('stream'),
fs = require('fs'),
port = 11111,
server;
describe('response streams', function() {
describe('when the server sends back json', function(){
before(function(done) {
server = http.createServer(function(req, res) {
res.setHeader('Content-Type', 'application/json')
res.end('{"foo":"bar"}')
}).listen(port, done);
});
after(function(done) {
server.close(done);
})
describe('and the client uses streams', function(){
it('creates a proper streams2 stream', function(done) {
var stream = needle.get('localhost:' + port)
// newer node versions set this to null instead of false
var bool = !!stream._readableState.flowing;
should.equal(false, bool);
var readableCalled = false;
stream.on('readable', function() {
readableCalled = true;
})
stream.on('finish', function() {
readableCalled.should.be.true;
done();
});
stream.resume();
})
it('emits a single data item which is our JSON object', function(done) {
var stream = needle.get('localhost:' + port)
var chunks = [];
stream.on('readable', function () {
while (chunk = this.read()) {
chunk.should.be.an.Object;
chunks.push(chunk);
}
})
stream.on('done', function () {
chunks.should.have.length(1)
chunks[0].should.have.property('foo', 'bar');
done();
});
})
it('emits a raw buffer if we do not want to parse JSON', function(done) {
var stream = needle.get('localhost:' + port, { parse: false })
var chunks = [];
stream.on('readable', function () {
while (chunk = this.read()) {
Buffer.isBuffer(chunk).should.be.true;
chunks.push(chunk);
}
})
stream.on('done', function() {
var body = Buffer.concat(chunks).toString();
body.should.equal('{"foo":"bar"}')
done();
});
})
})
})
describe('when the server sends back what was posted to it', function () {
var file = 'asdf.txt';
before(function(done){
server = http.createServer(function(req, res) {
res.setHeader('Content-Type', 'application/octet')
req.pipe(res);
}).listen(port);
fs.writeFile(file, 'contents of stream', done);
});
after(function(done){
server.close();
fs.unlink(file, done);
})
it('can PUT a stream', function (done) {
var stream = needle.put('localhost:' + port, fs.createReadStream(file), { stream: true });
var chunks = [];
stream.on('readable', function () {
while (chunk = this.read()) {
Buffer.isBuffer(chunk).should.be.true;
chunks.push(chunk);
}
})
stream.on('end', function () {
var body = Buffer.concat(chunks).toString();
body.should.equal('contents of stream')
done();
});
});
it('can PATCH a stream', function (done) {
var stream = needle.patch('localhost:' + port, fs.createReadStream(file), { stream: true });
var chunks = [];
stream.on('readable', function () {
while (chunk = this.read()) {
Buffer.isBuffer(chunk).should.be.true;
chunks.push(chunk);
}
})
stream.on('end', function () {
var body = Buffer.concat(chunks).toString();
body.should.equal('contents of stream')
done();
});
});
})
})
@@ -0,0 +1,79 @@
var should = require('should'),
needle = require('./../'),
fs = require('fs'),
https = require('https'),
stream = require('stream');
describe('socket cleanup', function(){
var outFile = 'test/tmp';
var httpAgent, readStream, writeStream
var file = 'ubuntu-21.04-desktop-amd64.iso',
url = 'https://releases.ubuntu.com/21.04/' + file;
function getActiveSockets() {
return Object.keys(httpAgent.sockets).length
}
before(function() {
httpAgent = new https.Agent({
keepAlive : true,
maxSockets : 1
});
})
after(function() {
httpAgent.destroy()
fs.unlinkSync(outFile);
})
it('should cleanup sockets on ERR_STREAM_PREMATURE_CLOSE (using .pipe)', function(done) {
getActiveSockets().should.eql(0);
var resp = needle.get(url, { agent: httpAgent });
var writable = fs.createWriteStream(outFile);
resp.pipe(writable);
writable.on('close', function(e) {
if (!resp.done) resp.abort();
})
setTimeout(function() {
getActiveSockets().should.eql(1);
writable.destroy();
}, 50);
setTimeout(function() {
getActiveSockets().should.eql(0);
done();
}, 500); // takes a bit
})
it('should cleanup sockets on ERR_STREAM_PREMATURE_CLOSE (using stream.pipeline)', function(done) {
if (!stream.pipeline)
return done()
getActiveSockets().should.eql(0);
var resp = needle.get(url, { agent: httpAgent });
var writable = fs.createWriteStream(outFile);
stream.pipeline(resp, writable, function(err) {
err.code.should.eql('ERR_STREAM_PREMATURE_CLOSE')
// if (err) resp.request.destroy();
});
setTimeout(function() {
getActiveSockets().should.eql(1);
writable.destroy();
}, 50);
setTimeout(function() {
getActiveSockets().should.eql(0);
done();
}, 1000); // takes a bit
})
})
@@ -0,0 +1,67 @@
var needle = require('../'),
should = require('should'),
http = require('http');
var server, port = 11112;
describe('socket reuse', function() {
var httpAgent = new http.Agent({
keepAlive : true,
maxSockets : 1
});
before(function(done) {
server = http.createServer(function(req, res) {
res.setHeader('Content-Type', 'application/json');
setTimeout(function() {
res.end('{"foo":"bar"}');
}, 50);
}).listen(port, done);
});
after(function(done) {
httpAgent.destroy();
server.close(done);
});
describe('when sockets are reused', function() {
it('does not duplicate listeners on .end', function(done) {
var last_error;
var count = 10;
function completed(err) {
--count || done(last_error);
}
function send() {
needle.get('localhost:' + port, { agent: httpAgent }, function(err, resp) {
if (err)
throw new Error("Unexpected error: " + err);
// lets go through all sockets and inspect all socket objects
for (hostTarget in httpAgent.sockets) {
httpAgent.sockets[hostTarget].forEach(function(socket) {
// normally, there are 2 internal listeners and 1 needle sets up,
// but to be sure the test does not fail even if newer node versions
// introduce additional listeners, we use a higher limit.
try {
socket.listeners('end').length.should.be.below(5, "too many listeners on the socket object's end event");
} catch (e) {
last_error = e;
}
});
}
completed();
});
}
for (var i = 0; i < count; i++) {
send();
}
});
});
});
@@ -0,0 +1,144 @@
var needle = require('../'),
fs = require('fs'),
should = require('should'),
helpers = require('./helpers');
describe('stream events', function() {
var server,
port = 3456,
responseData,
serverOpts = {},
requestHandler = function(req, res) { res.end('OK') }
before(function() {
var opts = {
port: port,
handler: function(req, res) { requestHandler(req, res) }
}
server = helpers.server(opts);
})
after(function() {
server.close();
})
beforeEach(function() {
responseData = '';
})
describe('when consuming data directly', function() {
function send_request(opts, cb) {
return needle
.get('http://localhost:' + port, opts)
.on('data', function(data) { responseData += data })
}
describe('and request stream fails', function() {
it('emits done event with error', function(done) {
requestHandler = function(req, res) { req.socket.destroy() }
send_request({}).on('done', function(err) {
err.code.should.eql('ECONNRESET');
responseData.should.eql('');
done()
})
})
})
describe('and request succeeds but decoding fails', function() {
it('emits done event without error', function(done) {
requestHandler = function(req, res) {
res.setHeader('Content-Type', 'application/json')
res.end('invalid:json')
}
send_request({ json: true }).on('done', function(err) {
should.not.exist(err);
responseData.should.eql('invalid:json');
done()
})
})
})
describe('and request succeeds and pipeline works ok', function() {
it('emits done event without error', function(done) {
requestHandler = function(req, res) { res.end('{"ok":1}') }
send_request({ json: true }).on('done', function(err) {
should.not.exist(err);
responseData.should.eql('{"ok":1}');
done()
})
})
})
})
describe('when piping to a fs writableStream', function() {
var outFile = 'test/tmp.dat';
function send_request(opts, cb) {
return needle
.get('http://localhost:' + port, opts)
.pipe(fs.createWriteStream(outFile))
.on('data', function(data) { responseData += data })
}
after(function(done) {
fs.unlink(outFile, done)
})
describe('and request stream fails', function() {
it('final stream emits done event with error', function(done) {
requestHandler = function(req, res) { req.socket.destroy() }
send_request({}).on('done', function(err) {
err.code.should.eql('ECONNRESET');
done()
})
})
})
describe('and request succeeds but decoding fails', function() {
it('final stream emits done event without error', function(done) {
requestHandler = function(req, res) {
res.setHeader('Content-Type', 'application/json')
res.end('invalid:json')
}
send_request({ json: true }).on('done', function(err) {
should.not.exist(err);
done()
})
})
})
describe('and request succeeds and pipeline works ok', function() {
it('final stream emits done event without error', function(done) {
requestHandler = function(req, res) { res.end('{"ok":1}') }
send_request({ json: true }).on('done', function(err) {
should.not.exist(err);
done()
})
})
})
})
})
@@ -0,0 +1,46 @@
var needle = require('../'),
sinon = require('sinon'),
should = require('should'),
http = require('http'),
helpers = require('./helpers');
var port = 3456;
describe('uri_modifier config parameter function', function() {
var server, uri;
function send_request(mw, cb) {
needle.get(uri, { uri_modifier: mw }, cb);
}
before(function(done){
server = helpers.server({ port: port }, done);
})
after(function(done) {
server.close(done);
})
describe('modifies uri', function() {
var path = '/foo/replace';
before(function() {
uri = 'localhost:' + port + path
});
it('should modify path', function(done) {
send_request(function(uri) {
return uri.replace('/replace', '');
}, function(err, res) {
should.not.exist(err);
should(res.req.path).be.exactly('/foo');
done();
});
});
})
})
@@ -0,0 +1,155 @@
var needle = require('../'),
sinon = require('sinon'),
should = require('should'),
http = require('http'),
helpers = require('./helpers');
var port = 3456;
describe('urls', function() {
var server, url;
function send_request(cb) {
return needle.get(url, cb);
}
before(function(done){
server = helpers.server({ port: port }, done);
})
after(function(done) {
server.close(done);
})
describe('null URL', function(){
it('throws', function(){
(function() {
send_request()
}).should.throw();
})
})
describe('invalid protocol', function(){
before(function() {
url = 'foo://google.com/what'
})
it('does not throw', function(done) {
(function() {
send_request(function(err) {
done();
})
}).should.not.throw()
})
it('returns an error', function(done) {
send_request(function(err) {
err.should.be.an.Error;
err.code.should.match(/ENOTFOUND|EADDRINFO|EAI_AGAIN/)
done();
})
})
})
describe('invalid host', function(){
before(function() {
url = 'http://s1\\\u0002.com/'
})
it('fails', function(done) {
(function() {
send_request(function(){ })
}.should.throw(TypeError))
done()
})
})
/*
describe('invalid path', function(){
before(function() {
url = 'http://www.google.com\\\/x\\\ %^&*() /x2.com/'
})
it('fails', function(done) {
send_request(function(err) {
err.should.be.an.Error;
done();
})
})
})
*/
describe('valid protocol and path', function() {
before(function() {
url = 'http://localhost:' + port + '/foo';
})
it('works', function(done) {
send_request(function(err){
should.not.exist(err);
done();
})
})
})
describe('no protocol but with slashes and valid path', function() {
before(function() {
url = '//localhost:' + port + '/foo';
})
it('works', function(done) {
send_request(function(err){
should.not.exist(err);
done();
})
})
})
describe('no protocol nor slashes and valid path', function() {
before(function() {
url = 'localhost:' + port + '/foo';
})
it('works', function(done) {
send_request(function(err){
should.not.exist(err);
done();
})
})
})
describe('double encoding', function() {
var path = '/foo?email=' + encodeURIComponent('what-ever@Example.Com');
before(function() {
url = 'localhost:' + port + path
});
it('should not occur', function(done) {
send_request(function(err, res) {
should.not.exist(err);
should(res.req.path).be.exactly(path);
done();
});
});
})
})
@@ -0,0 +1,17 @@
var formidable = require('formidable'),
http = require('http'),
util = require('util');
var port = process.argv[2] || 8888;
http.createServer(function(req, res) {
var form = new formidable.IncomingForm();
form.parse(req, function(err, fields, files) {
res.writeHead(200, {'content-type': 'text/plain'});
res.write('received upload:\n\n');
console.log(util.inspect({fields: fields, files: files}))
res.end(util.inspect({fields: fields, files: files}));
});
}).listen(port);
console.log('HTTP server listening on port ' + port);
@@ -0,0 +1,62 @@
var http = require('http'),
https = require('https'),
url = require('url');
var port = 1234,
log = true,
request_auth = false;
http.createServer(function(request, response) {
console.log(request.headers);
console.log("Got request: " + request.url);
console.log("Forwarding request to " + request.headers['host']);
if (request_auth) {
if (!request.headers['proxy-authorization']) {
response.writeHead(407, {'Proxy-Authenticate': 'Basic realm="proxy.com"'})
return response.end('Hello.');
}
}
var remote = url.parse(request.url);
var protocol = remote.protocol == 'https:' ? https : http;
var opts = {
host: request.headers['host'],
port: remote.port || (remote.protocol == 'https:' ? 443 : 80),
method: request.method,
path: remote.pathname,
headers: request.headers
}
var proxy_request = protocol.request(opts, function(proxy_response){
proxy_response.on('data', function(chunk) {
if (log) console.log(chunk.toString());
response.write(chunk, 'binary');
});
proxy_response.on('end', function() {
response.end();
});
response.writeHead(proxy_response.statusCode, proxy_response.headers);
});
request.on('data', function(chunk) {
if (log) console.log(chunk.toString());
proxy_request.write(chunk, 'binary');
});
request.on('end', function() {
proxy_request.end();
});
}).listen(port);
process.on('uncaughtException', function(err){
console.log('Uncaught exception!');
console.log(err);
});
console.log("Proxy server listening on port " + port);
@@ -0,0 +1,104 @@
// TODO: write specs. :)
var fs = require('fs'),
client = require('./../../');
process.env.DEBUG = true;
var response_callback = function(err, resp, body){
console.log(err);
if(resp) console.log("Got status code " + resp.statusCode)
console.log(body);
}
function simple_head(){
client.head('http://www.amazon.com', response_callback);
}
function simple_get(){
client.get('http://www.nodejs.org', response_callback);
}
function proxy_get(){
client.get('https://www.google.com/search?q=nodejs', {proxy: 'http://localhost:1234'}, response_callback);
}
function auth_get(){
client.get('https://www.twitter.com', {username: 'asd', password: '123'}, response_callback);
}
function simple_post(url){
var data = {
foo: 'bar',
baz: {
nested: 'attribute'
}
}
client.post(url, data, response_callback);
}
function multipart_post(url){
var filename = 'test_file.txt';
var data = 'Plain text data.\nLorem ipsum dolor sit amet.\nBla bla bla.\n';
fs.writeFileSync(filename, data);
var black_pixel = Buffer.from("data:image/gif;base64,R0lGODlhAQABAIAAAAUEBAAAACwAAAAAAQABAAACAkQBADs=".replace(/^data:image\/\w+;base64,/, ""), "base64");
var data = {
foo: 'bar',
bar: 'baz',
nested: {
my_document: { file: filename, content_type: 'text/plain' },
even: {
more: 'nesting'
}
},
pixel: { filename: 'black_pixel.gif', buffer: black_pixel, content_type: 'image/gif' },
field2: {value: JSON.stringify({"json":[ {"one":1}, {"two":2} ]}), content_type: 'application/json' }
}
client.post(url, data, {multipart: true}, function(err, resp, body){
console.log(err);
console.log("Got status code " + resp.statusCode)
console.log(body);
fs.unlink(filename);
});
}
switch(process.argv[2]){
case 'head':
simple_head();
break;
case 'get':
simple_get();
break;
case 'auth':
auth_get();
break;
case 'proxy':
proxy_get();
break;
case 'post':
simple_post(process.argv[3] || 'http://posttestserver.com/post.php');
break;
case 'multipart':
multipart_post(process.argv[3] || 'http://posttestserver.com/post.php?dir=example');
break;
case 'all':
simple_head();
simple_get();
auth_get();
proxy_get();
simple_post(process.argv[3] || 'http://posttestserver.com/post.php');
multipart_post(process.argv[3] || 'http://posttestserver.com/post.php?dir=example');
break;
default:
console.log("Usage: ./test.js [head|get|auth|proxy|multipart]")
}
+54
View File
@@ -0,0 +1,54 @@
{
"name": "probe-image-size",
"version": "7.3.0",
"description": "Get image size without full download (JPG, GIF, PNG, WebP, BMP, TIFF, PSD)",
"keywords": [
"image",
"size",
"jpg",
"jpeg",
"ico",
"gif",
"png",
"webp",
"tiff",
"bmp",
"svg",
"psd"
],
"repository": "nodeca/probe-image-size",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/puzrin"
},
{
"type": "github",
"url": "https://github.com/sponsors/nodeca"
}
],
"license": "MIT",
"files": [
"index.js",
"http.js",
"stream.js",
"sync.js",
"lib/"
],
"scripts": {
"lint": "eslint .",
"test": "npm run lint && nyc node --test",
"coverage": "npm run test && nyc report --reporter html",
"report-coveralls": "nyc --reporter=lcov node --test"
},
"dependencies": {
"lodash.merge": "^4.6.2",
"needle": "^2.5.2",
"stream-parser": "~0.3.1"
},
"devDependencies": {
"eslint": "^9.39.4",
"neostandard": "^0.13.0",
"nyc": "^18.0.0"
}
}
+80
View File
@@ -0,0 +1,80 @@
'use strict'
var ProbeError = require('./lib/common').ProbeError
var parsers = require('./lib/parsers_stream')
var PassThrough = require('stream').PassThrough
var pipeline = require('stream').pipeline
module.exports = function probeStream (src, keepOpen) {
var proxy = new PassThrough()
// increase max number of listeners to stop memory leak warning
proxy.setMaxListeners(Object.keys(parsers).length + 10)
var result = new Promise(function (resolve, reject) {
src.on('error', reject)
proxy.on('error', reject)
var alive_parsers = []
var last_error
function parserEnd (err) {
var idx = alive_parsers.indexOf[this]
/* istanbul ignore if */
if (idx < 0) return
/* istanbul ignore if */
if (err) last_error = err
proxy.unpipe(this)
this.removeAllListeners()
alive_parsers.splice(idx, 1)
if (alive_parsers.length) return
// if all parsers finished without success -> fail.
reject(last_error || new ProbeError('unrecognized file format', 'ECONTENT'))
}
Object.keys(parsers).forEach(function (type) {
var pStream = parsers[type]()
alive_parsers.push(pStream)
pStream.once('data', function (data) {
if (data.width > 0 && data.height > 0) {
resolve(data)
return
}
// Manually finish parser to get standars "unrecognized" error.
parserEnd.call(pStream)
})
pStream.once('end', parserEnd)
// User does not need to know that something wrong in parser
// Process error the same was unrecognized format (end without data)
pStream.on('error', parserEnd)
proxy.pipe(pStream)
})
})
function cleanup () {
// request stream doesn't have unpipe, https://github.com/request/request/issues/874
if (keepOpen && typeof src.unpipe === 'function') src.unpipe(proxy)
proxy.destroy()
}
result.then(cleanup).catch(cleanup)
if (keepOpen) src.pipe(proxy)
else pipeline(src, proxy, function () {})
return result
}
module.exports.parsers = parsers
+27
View File
@@ -0,0 +1,27 @@
'use strict'
var parsers = require('./lib/parsers_sync')
function probeBuffer (buffer) {
var parser_names = Object.keys(parsers)
for (var i = 0; i < parser_names.length; i++) {
var result = parsers[parser_names[i]](buffer)
if (result) {
if (result.width > 0 && result.height > 0) return result
return null
}
}
return null
}
module.exports = function get_image_size (src) {
return probeBuffer(src)
}
module.exports.parsers = parsers