ARCHIVE/xahaud
3
0
Fork 0
mirror of https://github.com/Xahau/xahaud.git synced 2025-12-06 17:27:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

Squashed 'src/lz4/' changes from e25b51d..baf78e7

Browse Source 
baf78e7 Merge pull request #111 from Cyan4973/dev
6f50184 Updated NEWS
acae59a Fixed : default sparse mode disabled on stdout, to support ` >>`  redirection scenario reported by Takayuki Matsuoka (#110)
91c1b9a Performance fix : big compression speed boost for clang (+30%)
0fb0392 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
bb22103 Merge pull request #107 from t-mat/issue/103pr2
7d72c0c Added LZF
e769a0e Combine unique .lz4 file info
ffff9ee Support iterative edit and testing
a8fdeb4 Add log output of succeeded decompression to test script
5151c30 Add log of same files and sha1 hash of unique files to versionstest
26065c3 Fixed  : LZ4IO exits too early when frame crc not present, reported by Yongwoon Cho (#106)
87e560e minor tests improvements
a9ff13a minor test refactor
58b5aad Fixed sparse issue with non seekable streams (#105)
60d657a removed "flush" argument to please Travis's python version
cdb136d cosmetic changes, 2nd try
c779c80 minor cosmetic changes for Takayuki's testVersions
26021db Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
e3b5bf3 Merge pull request #104 from t-mat/issue/103pr2
fe11e0b Issue #103 : Add compatibility test between releases
f02c467 Added a few more interoperability tests (32bits vs 64 bits)
05c3f66 Updated a few comments
9607848 Fixed minor typo
45e1995 minor introduction update
7d182b8 Merge pull request #102 from Cyan4973/dev
fdd0029 minor parsing update
672bfde Updated comments
b4ef93a Fixed typo
efbebd2 Added : LZ4_compress_destSize()
1c3e633 Added compilation flag -Wcast-qual
05b0aa6 Updated readme
e05088d Updated lz4hc API
b4348a4 Fixed minor Visual warning
1171303 Updated streaming examples
1b17bf2 New lz4 API, using LZ4_compress_fast()
b495c91 Merge pull request #99 from eberge/dev
791512c Fixed bug 9318
2a974d7 refactored lz4hc
1e751a7 Install the lz4frame.h header in the cmake build
ad2dd6d moved lz4frame context types to incomplete typedef
c9cbb8f Increased aligment requirement for lz4frame context pointer
ef55dfb Modified lz4frame context typedef, to enforce stricter alignment condition
87a1c70 Fixed minor static analyzer warning
d6dc0a4 streaming API : Improved ring buffer management
bda38fd Merge pull request #98 from eberge/dev
a9a24e8 cmake support for AIX, HPUX, Solaris and Windows
9c6fb8b Added LZ4_compress_fast_extState()
b05d3d7 Frame content size disabled by default when using LZ4F_compressFrame(), to be in better coherence with the advanced API LZ4F_compress_update()
b805d58 Removed obsolete functions from lz4 cli
f11afaf Removed LZ4_compress() (obsolete) from lz4
72e6794 Updated LZ4F_freeDecompressionContext(), to provide stage hint as result
a01e10d Changed LZ4F compressionLevel from unsigned to signed, in anticipation for LZ4_compress_fast() integration.
cbcdd88 Fixed frame concatenation
e18aa90 Fixed frametest
c035b7a Restored make-lz4
197982e Fixed unfinished frame (issue #75)
409f816 Updated LZ4F_getFrameInfo() behavior, related to uncomplete frame header decoding attempts
47c3040 added --no-sparse test
9fd4f1f Sparse file support is now enabled by default
7644bee test error message in multiple files mode
bce2eeb Reclassified some notification messages as errors
9e92bee stronger arm tests
2ed9dcc fix minor "divide by zero" risk
633c1ca fixed minor leak
0ed2e71 Static analyzer generates error codes on bug suspicion
2cf8a19 minor header refactoring
634e4ee Merge pull request #96 from t-mat/improve-pr-95
e328d41 minor optimization for small files
13c6e16 Removed status notification in multiple-files mode
d153aaa Add LZ4F_OBSOLETE_ENUM() to describe obsolete enums
a430b85 Multiple files decompression refactoring
4e574e7 Updated lz4frame error names
d37926b Merge pull request #95 from t-mat/issue/90
240b554 Merge pull request #94 from t-mat/dev
3d46d4b Fix LZ4_DEPRECATED() in lz4hc.h
5f732e1 Merge pull request #91 from t-mat/dev
175890f Issue#90 : Change old enum names to new one
585bab8 Issue#90 : Change old enum to macro to maximize compatibility
081bcca Issue#90 : Add LZ4F_ prefix
e1283c7 Fix LZ4_DEPRECATED() for older/non-gcc/clang/MSVC compilers
d7298d2 Replace GCC_VERSION with LZ4_GCC_VERSION
9851583 Merge pull request #93 from drcrallen/descriptiveFrameErrors
b664a72 Revert "Revert "Add more descriptive frame errors""
3f4f623 Valgrind tests generate errors
83e350d Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
066e9d3 Merge pull request #92 from Cyan4973/revert-89-descriptiveFrameErrors
5a66527 Revert "Add more descriptive frame errors"
0dc8308 Merge pull request #89 from drcrallen/descriptiveFrameErrors
05a46fc Changes LZ4F_compressBound() definition using NULL prefsPtr to cover worst case instead of default.
348f509 lz4io refactoring
eabc6d8 New valgrind test with multiple files
113b150 Fix leak issue with compression of multiple files
c64200d Improved performance when compressing a lot of small files
cc24124 minor compatibility fixes
ccba7a0 Merge pull request #86 from KyleJHarper/origin/r129/multiple_inputs_patch
d535214 Add more descriptive frame errors
bc28fc1 Merge pull request #87 from t-mat/fix-example2
fd77bad Replace obsolete functions
b036eaa Add snprintf macro for MSVC
7f2f1fc Added support for continuation of file compression and decompression if input files are missing.  Should more closely match gzip/bzip2/xz and so forth.  Also removed a debug print accidentally left in.
0169502 Added new LZ4IO_decompressMultipleFilenames to allow decompression of multiple files with the -m switch added in r128 (ref: google code issue 151).  Limitation: will only process files matching LZ4_EXTENSION macro, which for now seems reasonable.
da11725 new memory leak test for fullbench using multi-files
2c79887 Shortened tests durations
42e5bc4 Updated badges
2852b9e Fixed issue #84
8f49666 Fixed : minor coverity warning
8a61000 Fixed a few coverity warnings
138673d fixed minor g++ warning
cc8d617 Merge pull request #82 from t-mat/add-lz4-prefix
81fdd9d Fixed a few Valgrind warnings
ad86910 Add LZ4 prefix to deprecation macros
66b8a4a Fixed : minor Visual warnings
62ed153 Fixed : a few minor coverity warnings
9443f3d Extended obsolete warning messages to lz4hc
973e385 Implemented obsolete warning message
be9d248 Update lz4hc API : LZ4_compressHC_safe()
a07db74 Clarified lz4frame.h inline doc
8b8e5ef fixed minor sanitize warning
c22a0e1 Updated : fuzzer tests can be programmed for a timelength
a2864fd Fixed a few minor sanitize warnings
33134fb Added : sanitize test
f344fbd Fixed a few warnings from -fsanitize=undefined
2f8a4c3 New LZ4_compress_safe() API
1853622 fixed over-cautious visual warning
b41137f minor Makefile test refactoring
327cb04 minor memory leak fix and test
43e0535 fix g++ typecast
61d7416 updated doc
17f8614 added : memtest on fullbench
d38b0b6 Merge pull request #73 from funcodeio/dev
979a991 memcpy speed as reference
157a739 Merge pull request #74 from Cyan4973/fastMode
dd69902 Removed unused lines.
43eaf8f Merge pull request #72 from fzort/master
f72761f new tests for large files with content size support (#70)
90c0104 Added : progress indicator, in fast and decompression modes
1d3ab5d Cygwin has fileno, so there's no need to use _fileno.
78d2dfd fullbench : tests of _limitedOutput variants intentionnally provides less memory space than safe (LZ4_compressBound())
ef7cd83 Fixed issue 160, reported by Eric Berge
28e237e simplified LZ4_compress_limitedOutput()
89eee0d Removed make dist
0615eb4 Stricter tests : treat warnings as errors
76a03c1 simplified LZ4_compress_withState()
6625068 simplified LZ4_compress()
886b199 Modified files rights
7b5e945 Removed Visual 2013 solution, as AppVeyor automated mode only works with a single solution
117ab8c Added : Visual 2013 solution
08b24af Updated Visual 2012 solution : + 3 projects (fullbench, frametest, datagen)
a761546 Fix : minor warning under Visual
bf146ec Removed .suo & .user files from Visual solutions
7db6678 Restored proper credit
3bba55c Fixed : Windows compilation Added : Appveyor badge
160661c Merge pull request #69 from Cyan4973/dev
8437a0e Fixed : Visual compilation
7c26b03 Updated make dist
f174964 Added : Frame documentation in MarkDown format
880381c Removed HTML Frame Format documentation
5b9fb69 minor tweak
4783cb8 Updated readme
4c227a4 Added LZ4_compress_fast()
003af71 Merge pull request #67 from Cyan4973/dev
2a82619 fixed fullbench memory allocation error
6c69dc1 faster compression in 64 bits mode
44793b8 Updated documentation
b93f629 changed file name
eeb8bea Updated comments on LZ4F_getFrameInfo()
002ec60 restored lz4hc compression ratio
987e78c Merge pull request #66 from Cyan4973/dev
8cb06d5 lz4frame validates contentSize during decompression
d5da787 Changed struct member to contentSize
2d4fed5 Merge pull request #65 from Cyan4973/dev
ce71b07 converted to markdown friendly syntax
1ba37f3 Reference format doc
5780864 Fixed : Makefile
b009767 windows friendly make clean
27f7d06 minor beautifier (make clean)
b4755c7 Added : arm cross-compilation test
a357f43 Fixed cast-align warnings on 32-bits
4a9335b Added : doc authorship
e652285 Merge pull request #63 from t-mat/comment-on-example-directory
2af52a9 Add "Examples" subtree
679afea Add README.md as table of contents
19665c9 Add document for "Line by Line Text Compression" example
438fee9 Add document for "Double Buffer" example
a38166b Add document : "Streaming API Basics"
80e71c6 Updated man page : "--[no-]X"
5950f72 Updated tests
6b923d5 Updated long commands, with reverse "--no-" variants
d0f8d40 updated dist list
00c3208 Merge pull request #61 from Cyan4973/dev
7f436a1 lz4 cli supports frame content size
a28b147 removed useless man pages
7cf4e5c Updates tests & Man pages
f02adc7 new long commands
86715b2 Some more tests related to frame content size
7ee7256 frame content size support
7d87d43 Updated lz4io sparse file support (alignment properties)
b54d256 minor lz4frame optimisation (no more malloc() on using LZ4F_compressFrame() in fast mode)
da9402c minor lz4frame refactoring
859fe3b Updated LZ4 frame format documentation
8edb7f1 Added : Readme into lib directory, to explain what does each file
e7fb4d1 lz4 utility supports "pass-through" mode
2a02455 minor refactoring
3a68324 skippable frames support
93849d1 minor CMakeLists update
471eabe Merge pull request #60 from Cyberunner23/master
ef029a1 Removed checking of CMAKE_SYSTEM_PROCESSOR when adding -fPIC, breaks when that var is '64bit'.
c9a2b14 removed -s command from lz4c specific list of legacy commands
207aafd Added :  unlz4 symbolic link to "lz4 -d"
2b55752 changed "make install" default install directory to /usr/local
bbcfe21 Added : clang test
8a87769 Fix : static analyzer test on Travis
45b0642 scan-build tests
a18fb43 Merge pull request #58 from Cyan4973/sparseFile
248b761 windows sparse file support
d11ac40 Improved sparse files support
45a357f Improved sparse file support
e38c268 Fixed minor g++ warnings
12ab415 Preliminary support for sparse files
e3f33d2 Fixed minor warnings
74a6b14 Merge pull request #57 from alexDarcy/master
f2cc4be Updated Cmake configuration for non-gnu compiler
ceec6fa g++ compatibility
6b0c39b Updated datagen (can create sparse files)
e277511 Merge pull request #54 from t-mat/gc-issue/155
e1d9b59 Fixed : static library (x64 binary)
32a85fc NetBSD compatibility (#48)
488029e Updated : compress multiple files
046bd3a Merge pull request #52 from KyleJHarper/r128/multiple_inputs
dcdd628 Fix sentinel bit pattern
de5c930 Fix sentinel size miscalculation
eed7952 Add GNU coreutil's is_nul() method to isSparse()
b372f45 Add Neil's method to isSparse()
01a24af Improve isSparse()
4a5d92b Adjust coding style
97679fa Google Code Issue 155: lz4 cli should support sparse file https://code.google.com/p/lz4/issues/detail?id=155
fa27d23 Added support for multiple input files to act more like other compressors. For example: gzip file1 file2 file3. You can now do: lz4 [args] -m file1 file2 file3. Fixes 151.
67f3b41 Merge pull request #49 from t-mat/msvc-fseeki64
e68d1c9 restored lz4 hc compression ratio
41b6ed3 Replace fseek with _fseeki64 to avoid MSVC's 2GiB barrier
8f4e201 Fix : lz4frame.h within uninstaller
9fd92de Added : Visual project directory

git-subtree-dir: src/lz4
git-subtree-split: baf78e7e4dcbdf824a76f990ffeb573d113bbbdb
This commit is contained in:
JoelKatz
2015-05-29 15:03:27 -07:00
parent dad460dcfc
commit df26c08a34
62 changed files with 6254 additions and 2490 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
.gitattributes vendored
View File

@@ -8,3 +8,8 @@
# Denote files that should not be modified.
*.odt binary
*.png binary
# Visual Studio
*.sln binary
*.suo binary
*.vcxproj* binary

11
.travis.yml
View File

@@ -3,14 +3,23 @@ compiler: gcc
script: make test-travis
before_install:
- sudo apt-get update -qq
- sudo apt-get install -qq gcc-arm-linux-gnueabi
- sudo apt-get install -qq clang
- sudo apt-get install -qq g++-multilib
- sudo apt-get install -qq gcc-multilib
- sudo apt-get install -qq valgrind
- sudo apt-get install -qq python3
env:
- LZ4_TRAVIS_CI_ENV=travis-install
- LZ4_TRAVIS_CI_ENV=streaming-examples
- LZ4_TRAVIS_CI_ENV=cmake
- LZ4_TRAVIS_CI_ENV=dist
- LZ4_TRAVIS_CI_ENV=clangtest
- LZ4_TRAVIS_CI_ENV=sanitize
- LZ4_TRAVIS_CI_ENV=staticAnalyze
- LZ4_TRAVIS_CI_ENV=gpptest
- LZ4_TRAVIS_CI_ENV=armtest
- LZ4_TRAVIS_CI_ENV=versionstest
- LZ4_TRAVIS_CI_ENV=test-lz4
- LZ4_TRAVIS_CI_ENV=test-lz4c
- LZ4_TRAVIS_CI_ENV=test-lz4c32

1
LZ4_Frame_Format.html
View File

File diff suppressed because one or more lines are too long

76
Makefile
View File

@@ -26,39 +26,21 @@
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# You can contact the author at :
# - LZ4 source repository : http://code.google.com/p/lz4/
# - LZ4 source repository : https://github.com/Cyan4973/lz4
# - LZ4 forum froup : https://groups.google.com/forum/#!forum/lz4c
# ################################################################
# Version number
export VERSION=126
export VERSION=130
export RELEASE=r$(VERSION)
DESTDIR?=
PREFIX ?= /usr
PREFIX ?= /usr/local
LIBDIR ?= $(PREFIX)/lib
INCLUDEDIR=$(PREFIX)/include
PRGDIR = programs
LZ4DIR = lib
DISTRIBNAME=lz4-$(RELEASE).tar.gz
TEXT = $(LZ4DIR)/lz4.c $(LZ4DIR)/lz4.h $(LZ4DIR)/lz4hc.c $(LZ4DIR)/lz4hc.h \
$(LZ4DIR)/lz4frame.c $(LZ4DIR)/lz4frame.h $(LZ4DIR)/lz4frame_static.h \
$(LZ4DIR)/xxhash.c $(LZ4DIR)/xxhash.h \
$(LZ4DIR)/liblz4.pc.in $(LZ4DIR)/Makefile $(LZ4DIR)/LICENSE \
Makefile lz4_block_format.txt LZ4_Frame_Format.html NEWS README.md \
cmake_unofficial/CMakeLists.txt \
$(PRGDIR)/fullbench.c $(PRGDIR)/lz4cli.c \
$(PRGDIR)/datagen.c $(PRGDIR)/fuzzer.c \
$(PRGDIR)/lz4io.c $(PRGDIR)/lz4io.h \
$(PRGDIR)/bench.c $(PRGDIR)/bench.h \
$(PRGDIR)/lz4.1 $(PRGDIR)/lz4c.1 $(PRGDIR)/lz4cat.1 \
$(PRGDIR)/Makefile $(PRGDIR)/COPYING
NONTEXT = images/image00.png images/image01.png images/image02.png \
images/image03.png images/image04.png images/image05.png \
images/image06.png
SOURCES = $(TEXT) $(NONTEXT)
# Select test target for Travis CI's Build Matrix
@@ -68,6 +50,13 @@ else
TRAVIS_TARGET=$(LZ4_TRAVIS_CI_ENV)
endif
# Define nul output
ifneq (,$(filter Windows%,$(OS)))
VOID = nul
else
VOID = /dev/null
endif
default: lz4programs
@@ -79,10 +68,10 @@ lz4programs:
@cd $(PRGDIR); $(MAKE) -e
clean:
@rm -f $(DISTRIBNAME) *.sha1
@cd $(PRGDIR); $(MAKE) clean
@cd $(LZ4DIR); $(MAKE) clean
@cd examples; $(MAKE) clean
@cd $(PRGDIR); $(MAKE) clean > $(VOID)
@cd $(LZ4DIR); $(MAKE) clean > $(VOID)
@cd examples; $(MAKE) clean > $(VOID)
@cd test; $(MAKE) clean > $(VOID)
@echo Cleaning completed
@@ -101,24 +90,6 @@ uninstall:
travis-install:
sudo $(MAKE) install
dist: clean
@install -dD -m 700 lz4-$(RELEASE)/lib/
@install -dD -m 700 lz4-$(RELEASE)/programs/
@install -dD -m 700 lz4-$(RELEASE)/cmake_unofficial/
@install -dD -m 700 lz4-$(RELEASE)/images/
@for f in $(TEXT); do \
tr -d '\r' < $$f > .tmp; \
install -m 600 .tmp lz4-$(RELEASE)/$$f; \
done
@rm .tmp
@for f in $(NONTEXT); do \
install -m 600 $$f lz4-$(RELEASE)/$$f; \
done
@tar -czf $(DISTRIBNAME) lz4-$(RELEASE)/
@rm -rf lz4-$(RELEASE)
@sha1sum $(DISTRIBNAME) > $(DISTRIBNAME).sha1
@echo Distribution $(DISTRIBNAME) built
test:
@cd $(PRGDIR); $(MAKE) -e test
@@ -127,6 +98,25 @@ test-travis: $(TRAVIS_TARGET)
cmake:
@cd cmake_unofficial; cmake CMakeLists.txt; $(MAKE)
gpptest: clean
$(MAKE) all CC=g++ CFLAGS="-O3 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Werror"
clangtest: clean
$(MAKE) all CC=clang CPPFLAGS="-Werror -Wconversion -Wno-sign-conversion"
sanitize: clean
$(MAKE) test CC=clang CPPFLAGS="-g -fsanitize=undefined" FUZZER_TIME="-T1mn" NB_LOOPS=-i1
staticAnalyze: clean
scan-build --status-bugs -v $(MAKE) all CFLAGS=-g
armtest: clean
cd lib; $(MAKE) -e all CC=arm-linux-gnueabi-gcc CPPFLAGS="-Werror"
cd programs; $(MAKE) -e bins CC=arm-linux-gnueabi-gcc CPPFLAGS="-Werror"
versionstest: clean
@cd test; $(MAKE)
streaming-examples:
cd examples; $(MAKE) -e test

42
NEWS
View File

@@ -1,5 +1,45 @@
r130:
Fixed : incompatibility sparse mode vs console, reported by Yongwoon Cho (#105)
Fixed : LZ4IO exits too early when frame crc not present, reported by Yongwoon Cho (#106)
Fixed : incompatibility sparse mode vs append mode, reported by Takayuki Matsuoka (#110)
Performance fix : big compression speed boost for clang (+30%)
New : cross-version test, by Takayuki Matsuoka
r129:
Added : LZ4_compress_fast(), LZ4_compress_fast_continue()
Added : LZ4_compress_destSize()
Changed: New lz4 and lz4hc compression API. Previous function prototypes still supported.
Changed: Sparse file support enabled by default
New : LZ4 CLI improved performance compressing/decompressing multiple files (#86, kind contribution from Kyle J. Harper & Takayuki Matsuoka)
Fixed : GCC 4.9+ optimization bug - Reported by Markus Trippelsdorf, Greg Slazinski & Evan Nemerson
Changed: Enums converted to LZ4F_ namespace convention - by Takayuki Matsuoka
Added : AppVeyor CI environment, for Visual tests - Suggested by Takayuki Matsuoka
Modified:Obsolete functions generate warnings - Suggested by Evan Nemerson, contributed by Takayuki Matsuoka
Fixed : Bug #75 (unfinished stream), reported by Yongwoon Cho
Updated: Documentation converted to MarkDown format
r128:
New : lz4cli sparse file support (Requested by Neil Wilson, and contributed by Takayuki Matsuoka)
New : command -m, to compress multiple files in a single command (suggested by Kyle J. Harper)
Fixed : Restored lz4hc compression ratio (slightly lower since r124)
New : lz4 cli supports long commands (suggested by Takayuki Matsuoka)
New : lz4frame & lz4cli frame content size support
New : lz4frame supports skippable frames, as requested by Sergey Cherepanov
Changed: Default "make install" directory is /usr/local, as notified by Ron Johnson
New : lz4 cli supports "pass-through" mode, requested by Neil Wilson
New : datagen can generate sparse files
New : scan-build tests, thanks to kind help by Takayuki Matsuoka
New : g++ compatibility tests
New : arm cross-compilation test, thanks to kind help by Takayuki Matsuoka
Fixed : Fuzzer + frametest compatibility with NetBSD (issue #48, reported by Thomas Klausner)
Added : Visual project directory
Updated: Man page & Specification
r127:
N/A : added a file on SVN
r126:
New : lz4frame API is now integrated into liblz4
New : lz4frame API is now integrated into liblz4
Fixed : GCC 4.9 bug on highest performance settings, reported by Greg Slazinski
Fixed : bug within LZ4 HC streaming mode, reported by James Boyle
Fixed : older compiler don't like nameless unions, reported by Cheyi Lin

107
README.md
View File

@@ -1,56 +1,81 @@
LZ4 - Extremely fast compression
================================
LZ4 is lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems.
A high compression derivative, called LZ4_HC, is also provided. It trades CPU time for compression ratio.
LZ4 is lossless compression algorithm,
providing compression speed at 400 MB/s per core,
scalable with multi-cores CPU.
It also features an extremely fast decoder,
with speed in multiple GB/s per core,
typically reaching RAM speed limits on multi-core systems.
Speed can be tuned dynamically, selecting an "acceleration" factor
which trades compression ratio for more speed up.
On the other end, a high compression derivative, LZ4_HC, is also provided,
trading CPU time for improved compression ratio.
All versions feature the same excellent decompression speed.
|Branch |Status |
|------------|---------|
|master | [![Build Status](https://travis-ci.org/Cyan4973/lz4.svg?branch=master)](https://travis-ci.org/Cyan4973/lz4) |
|dev | [![Build Status](https://travis-ci.org/Cyan4973/lz4.svg?branch=dev)](https://travis-ci.org/Cyan4973/lz4) |
|master | [![Build Status](https://travis-ci.org/Cyan4973/lz4.svg?branch=master)](https://travis-ci.org/Cyan4973/lz4) [![Build status](https://ci.appveyor.com/api/projects/status/v6kxv9si529477cq/branch/master?svg=true)](https://ci.appveyor.com/project/YannCollet/lz4) |
|dev | [![Build Status](https://travis-ci.org/Cyan4973/lz4.svg?branch=dev)](https://travis-ci.org/Cyan4973/lz4) [![Build status](https://ci.appveyor.com/api/projects/status/v6kxv9si529477cq/branch/dev?svg=true)](https://ci.appveyor.com/project/YannCollet/lz4) |
This is an official mirror of LZ4 project, [hosted on Google Code](http://code.google.com/p/lz4/).
The intention is to offer github's capabilities to lz4 users, such as cloning, branch, pull requests or source download.
The "master" branch will reflect, the status of lz4 at its official homepage.
The "dev" branch is the one where all contributions will be merged. If you plan to propose a patch, please commit into the "dev" branch. Direct commit to "master" are not permitted.
Feature branches will also exist, typically to introduce new requirements, and be temporarily available for testing before merge into "dev" branch.
> **Branch Policy:**
> - The "master" branch is considered stable, at all times.
> - The "dev" branch is the one where all contributions must be merged
before being promoted to master.
> + If you plan to propose a patch, please commit into the "dev" branch,
or its own feature branch.
Direct commit to "master" are not permitted.
Benchmarks
-------------------------
The benchmark uses the [Open-Source Benchmark program by m^2 (v0.14.2)](http://encode.ru/threads/1371-Filesystem-benchmark?p=33548&viewfull=1#post33548) compiled with GCC v4.6.1 on Linux Ubuntu 64-bits v11.10,
The reference system uses a Core i5-3340M @2.7GHz.
Benchmark evaluates the compression of reference [Silesia Corpus](http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia) in single-thread mode.
The benchmark uses the [Open-Source Benchmark program by m^2 (v0.14.3)]
compiled with GCC v4.8.2 on Linux Mint 64-bits v17.
The reference system uses a Core i5-4300U @1.9GHz.
Benchmark evaluates the compression of reference [Silesia Corpus]
in single-thread mode.
<table>
<tr>
<th>Compressor</th><th>Ratio</th><th>Compression</th><th>Decompression</th>
</tr>
<tr>
<th>LZ4 (r101)</th><th>2.084</th><th>422 MB/s</th><th>1820 MB/s</th>
</tr>
<tr>
<th>LZO 2.06</th><th>2.106</th><th>414 MB/s</th><th>600 MB/s</th>
</tr>
<tr>
<th>QuickLZ 1.5.1b6</th><th>2.237</th><th>373 MB/s</th><th>420 MB/s</th>
</tr>
<tr>
<th>Snappy 1.1.0</th><th>2.091</th><th>323 MB/s</th><th>1070 MB/s</th>
</tr>
<tr>
<th>LZF</th><th>2.077</th><th>270 MB/s</th><th>570 MB/s</th>
</tr>
<tr>
<th>zlib 1.2.8 -1</th><th>2.730</th><th>65 MB/s</th><th>280 MB/s</th>
</tr>
<tr>
<th>LZ4 HC (r101)</th><th>2.720</th><th>25 MB/s</th><th>2080 MB/s</th>
</tr>
<tr>
<th>zlib 1.2.8 -6</th><th>3.099</th><th>21 MB/s</th><th>300 MB/s</th>
</tr>
</table>
| Compressor | Ratio | Compression | Decompression |
| ---------- | ----- | ----------- | ------------- |
| memcpy | 1.000 | 4200 MB/s | 4200 MB/s |
|**LZ4 fast 17 (r129)**| 1.607 |**690 MB/s** | **2220 MB/s** |
|**LZ4 default (r129)**|**2.101**|**385 MB/s** | **1850 MB/s** |
| LZO 2.06 | 2.108 | 350 MB/s | 510 MB/s |
| QuickLZ 1.5.1.b6 | 2.238 | 320 MB/s | 380 MB/s |
| Snappy 1.1.0 | 2.091 | 250 MB/s | 960 MB/s |
| LZF v3.6 | 2.073 | 175 MB/s | 500 MB/s |
| zlib 1.2.8 -1 | 2.730 | 59 MB/s | 250 MB/s |
|**LZ4 HC (r129)** |**2.720**| 22 MB/s | **1830 MB/s** |
| zlib 1.2.8 -6 | 3.099 | 18 MB/s | 270 MB/s |
Documentation
-------------------------
The raw LZ4 block compression format is detailed within [lz4_Block_format].
To compress an arbitrarily long file or data stream, multiple blocks are required.
Organizing these blocks and providing a common header format to handle their content
is the purpose of the Frame format, defined into [lz4_Frame_format].
Interoperable versions of LZ4 must respect this frame format.
Other source versions
-------------------------
Beyond the C reference source,
many contributors have created versions of lz4 in multiple languages.
A list of these sources is maintained on the [LZ4 Homepage].
[Open-Source Benchmark program by m^2 (v0.14.3)]: http://encode.ru/threads/1371-Filesystem-benchmark?p=34029&viewfull=1#post34029
[Silesia Corpus]: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia
[lz4_Block_format]: lz4_Block_format.md
[lz4_Frame_format]: lz4_Frame_format.md
[LZ4 Homepage]: http://www.lz4.org

28
cmake_unofficial/CMakeLists.txt
View File

@@ -2,7 +2,7 @@ PROJECT(LZ4 C)
set(CPACK_PACKAGE_DESCRIPTION_SUMMARY "LZ4 compression library")
set(CPACK_PACKAGE_VERSION_MAJOR 1)
set(CPACK_PACKAGE_VERSION_MINOR 5)
set(CPACK_PACKAGE_VERSION_PATCH r126)
set(CPACK_PACKAGE_VERSION_PATCH r128)
set(VERSION_STRING " \"${CPACK_PACKAGE_VERSION_MAJOR}.${CPACK_PACKAGE_VERSION_MINOR}.${CPACK_PACKAGE_VERSION_PATCH}\" ")
include(CPack)
@@ -15,17 +15,17 @@ IF( ${SIZEOF_VOID_P} STREQUAL "8" )
ENDIF()
option(BUILD_TOOLS "Build the command line tools" ON)
option(BUILD_LIBS "Build the libraries in addition to the tools" OFF)
option(BUILD_LIBS "Build the libraries in addition to the tools" ON)
if(CMAKE_COMPILER_IS_GNUCC OR CMAKE_COMPILER_IS_GNUCXX)
if(UNIX AND BUILD_LIBS)
if(CMAKE_SYSTEM_PROCESSOR MATCHES "x86_64")
add_definitions(-fPIC)
endif(CMAKE_SYSTEM_PROCESSOR MATCHES "x86_64")
add_definitions(-fPIC)
endif()
endif()
set(LZ4_DIR ../lib/)
set(PRG_DIR ../programs/)
set(LZ4_SRCS_LIB ${LZ4_DIR}lz4.c ${LZ4_DIR}lz4hc.c ${LZ4_DIR}lz4.h ${LZ4_DIR}lz4hc.h)
set(LZ4_SRCS_LIB ${LZ4_DIR}lz4.c ${LZ4_DIR}lz4hc.c ${LZ4_DIR}lz4.h ${LZ4_DIR}lz4hc.h ${LZ4_DIR}lz4frame.c ${LZ4_DIR}lz4frame.h ${LZ4_DIR}xxhash.c)
set(LZ4_SRCS ${LZ4_DIR}lz4frame.c ${LZ4_DIR}xxhash.c ${PRG_DIR}bench.c ${PRG_DIR}lz4cli.c ${PRG_DIR}lz4io.c)
if(BUILD_TOOLS AND NOT BUILD_LIBS)
@@ -34,10 +34,7 @@ endif()
if(BUILD_TOOLS)
add_executable(lz4 ${LZ4_SRCS})
set_target_properties(lz4 PROPERTIES COMPILE_DEFINITIONS DISABLE_LZ4C_LEGACY_OPTIONS)
install(TARGETS lz4 RUNTIME DESTINATION "bin/")
add_executable(lz4c ${LZ4_SRCS})
install(TARGETS lz4c RUNTIME DESTINATION "bin/")
endif()
if(BUILD_LIBS)
@@ -56,27 +53,38 @@ if(BUILD_LIBS)
install(FILES
${LZ4_DIR}/lz4.h
${LZ4_DIR}/lz4hc.h
${LZ4_DIR}/lz4frame.h
DESTINATION include
)
if(BUILD_TOOLS)
target_link_libraries(lz4 liblz4)
target_link_libraries(lz4c liblz4)
endif()
endif()
#warnings
if(MSVC)
ADD_DEFINITIONS("-W4")
endif()
if(CMAKE_COMPILER_IS_GNUCC OR CMAKE_COMPILER_IS_GNUCXX)
ADD_DEFINITIONS("-Wall")
endif()
if(CMAKE_COMPILER_IS_GNUCXX)
ADD_DEFINITIONS("-Wextra")
ADD_DEFINITIONS("-Wundef")
ADD_DEFINITIONS("-Wshadow")
ADD_DEFINITIONS("-Wcast-align")
ADD_DEFINITIONS("-Wstrict-prototypes")
endif(CMAKE_COMPILER_IS_GNUCXX)
if((CMAKE_COMPILER_IS_GNUCC OR CMAKE_COMPILER_IS_GNUCXX) AND
(NOT CMAKE_SYSTEM_NAME MATCHES "SunOS"))
ADD_DEFINITIONS("-std=c99")
endif()
ADD_DEFINITIONS("-DLZ4_VERSION=\"${CPACK_PACKAGE_VERSION_PATCH}\"")
INCLUDE_DIRECTORIES (${LZ4_DIR})

25
examples/Makefile
View File

@@ -1,6 +1,7 @@
# ##########################################################################
# LZ4 examples - Makefile
# Copyright (C) Yann Collet 2011-2014
#
# GPL v2 License
#
# This program is free software; you can redistribute it and/or modify
@@ -21,29 +22,17 @@
# - LZ4 source repository : http://code.google.com/p/lz4/
# - LZ4 forum froup : https://groups.google.com/forum/#!forum/lz4c
# ##########################################################################
# lz4 : Command Line Utility, supporting gzip-like arguments
# lz4c : CLU, supporting also legacy lz4demo arguments
# lz4c32: Same as lz4c, but forced to compile in 32-bits mode
# fuzzer : Test tool, to check lz4 integrity on target platform
# fuzzer32: Same as fuzzer, but forced to compile in 32-bits mode
# fullbench : Precisely measure speed for each LZ4 function variant
# fullbench32: Same as fullbench, but forced to compile in 32-bits mode
# This makefile compile and test
# example programs, using (mostly) LZ4 streaming library,
# kindly provided by Takayuki Matsuoka
# ##########################################################################
CC := $(CC)
CFLAGS ?= -O3
CFLAGS += -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wstrict-prototypes -Wno-missing-braces # Wno-missing-braces required due to GCC <4.8.3 bug
FLAGS = -I../lib $(CPPFLAGS) $(CFLAGS) $(LDFLAGS)
CFLAGS += -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wstrict-prototypes
FLAGS := -I../lib $(CPPFLAGS) $(CFLAGS) $(LDFLAGS)
TESTFILE= Makefile
LZ4DIR = ../lib
# Minimize test target for Travis CI's Build Matrix
ifeq ($(LZ4_TRAVIS_CI_ENV),-m32)
CFLAGS += -m32
else ifeq ($(LZ4_TRAVIS_CI_ENV),-m64)
endif
LZ4DIR := ../lib
# Define *.exe as extension for Windows systems

8
examples/README.md Normal file
View File

@@ -0,0 +1,8 @@
# LZ4 examples
## Documents
- [Streaming API Basics](streaming_api_basics.md)
- Examples
- [Double Buffer](blockStreaming_doubleBuffer.md)
- [Line by Line Text Compression](blockStreaming_lineByLine.md)

17
examples/blockStreaming_doubleBuffer.c
View File

@@ -2,7 +2,10 @@
// Copyright : Takayuki Matsuoka
#define _CRT_SECURE_NO_WARNINGS // for MSVC
#ifdef _MSC_VER /* Visual Studio */
# define _CRT_SECURE_NO_WARNINGS
# define snprintf sprintf_s
#endif
#include "lz4.h"
#include <stdio.h>
@@ -35,11 +38,13 @@ size_t read_bin(FILE* fp, void* array, size_t arrayBytes) {
void test_compress(FILE* outFp, FILE* inpFp)
{
LZ4_stream_t lz4Stream_body = { 0 };
LZ4_stream_t lz4Stream_body;
LZ4_stream_t* lz4Stream = &lz4Stream_body;
char inpBuf[2][BLOCK_BYTES];
int inpBufIndex = 0;
LZ4_resetStream(lz4Stream);
for(;;) {
char* const inpPtr = inpBuf[inpBufIndex];
@@ -50,8 +55,8 @@ void test_compress(FILE* outFp, FILE* inpFp)
{
char cmpBuf[LZ4_COMPRESSBOUND(BLOCK_BYTES)];
const int cmpBytes = LZ4_compress_continue(
lz4Stream, inpPtr, cmpBuf, inpBytes);
const int cmpBytes = LZ4_compress_fast_continue(
lz4Stream, inpPtr, cmpBuf, inpBytes, sizeof(cmpBuf), 1);
if(cmpBytes <= 0) {
break;
}
@@ -68,12 +73,14 @@ void test_compress(FILE* outFp, FILE* inpFp)
void test_decompress(FILE* outFp, FILE* inpFp)
{
LZ4_streamDecode_t lz4StreamDecode_body = { 0 };
LZ4_streamDecode_t lz4StreamDecode_body;
LZ4_streamDecode_t* lz4StreamDecode = &lz4StreamDecode_body;
char decBuf[2][BLOCK_BYTES];
int decBufIndex = 0;
LZ4_setStreamDecode(lz4StreamDecode, NULL, 0);
for(;;) {
char cmpBuf[LZ4_COMPRESSBOUND(BLOCK_BYTES)];
int cmpBytes = 0;

100
examples/blockStreaming_doubleBuffer.md Normal file
View File

@@ -0,0 +1,100 @@
# LZ4 Streaming API Example : Double Buffer
by *Takayuki Matsuoka*
`blockStreaming_doubleBuffer.c` is LZ4 Straming API example which implements double buffer (de)compression.
Please note :
- Firstly, read "LZ4 Streaming API Basics".
- This is relatively advanced application example.
- Output file is not compatible with lz4frame and platform dependent.
## What's the point of this example ?
- Handle huge file in small amount of memory
- Always better compression ratio than Block API
- Uniform block size
## How the compression works
First of all, allocate "Double Buffer" for input and LZ4 compressed data buffer for output.
Double buffer has two pages, "first" page (Page#1) and "second" page (Page#2).
```
Double Buffer
Page#1 Page#2
+---------+---------+
| Block#1 | |
+----+----+---------+
|
v
{Out#1}
Prefix Dependency
+---------+
| |
v |
+---------+----+----+
| Block#1 | Block#2 |
+---------+----+----+
|
v
{Out#2}
External Dictionary Mode
+---------+
| |
| v
+----+----+---------+
| Block#3 | Block#2 |
+----+----+---------+
|
v
{Out#3}
Prefix Dependency
+---------+
| |
v |
+---------+----+----+
| Block#3 | Block#4 |
+---------+----+----+
|
v
{Out#4}
```
Next, read first block to double buffer's first page. And compress it by `LZ4_compress_continue()`.
For the first time, LZ4 doesn't know any previous dependencies,
so it just compress the line without dependencies and generates compressed block {Out#1} to LZ4 compressed data buffer.
After that, write {Out#1} to the file.
Next, read second block to double buffer's second page. And compress it.
In this time, LZ4 can use dependency to Block#1 to improve compression ratio.
This dependency is called "Prefix mode".
Next, read third block to double buffer's *first* page. And compress it.
Also this time, LZ4 can use dependency to Block#2.
This dependency is called "External Dictonaly mode".
Continue these procedure to the end of the file.
## How the decompression works
Decompression will do reverse order.
- Read first compressed block.
- Decompress it to the first page and write that page to the file.
- Read second compressed block.
- Decompress it to the second page and write that page to the file.
- Read third compressed block.
- Decompress it to the *first* page and write that page to the file.
Continue these procedure to the end of the compressed file.

22
examples/blockStreaming_lineByLine.c
View File

@@ -2,7 +2,10 @@
// Copyright : Takayuki Matsuoka
#define _CRT_SECURE_NO_WARNINGS // for MSVC
#ifdef _MSC_VER /* Visual Studio */
# define _CRT_SECURE_NO_WARNINGS
# define snprintf sprintf_s
#endif
#include "lz4.h"
#include <stdio.h>
@@ -38,8 +41,9 @@ static void test_compress(
size_t ringBufferBytes)
{
LZ4_stream_t* const lz4Stream = LZ4_createStream();
char* const cmpBuf = malloc(LZ4_COMPRESSBOUND(messageMaxBytes));
char* const inpBuf = malloc(ringBufferBytes);
const size_t cmpBufBytes = LZ4_COMPRESSBOUND(messageMaxBytes);
char* const cmpBuf = (char*) malloc(cmpBufBytes);
char* const inpBuf = (char*) malloc(ringBufferBytes);
int inpOffset = 0;
for ( ; ; )
@@ -60,8 +64,8 @@ static void test_compress(
#endif
{
const int cmpBytes = LZ4_compress_continue(
lz4Stream, inpPtr, cmpBuf, inpBytes);
const int cmpBytes = LZ4_compress_fast_continue(
lz4Stream, inpPtr, cmpBuf, inpBytes, cmpBufBytes, 1);
if (cmpBytes <= 0) break;
write_uint16(outFp, (uint16_t) cmpBytes);
write_bin(outFp, cmpBuf, cmpBytes);
@@ -86,8 +90,8 @@ static void test_decompress(
size_t ringBufferBytes)
{
LZ4_streamDecode_t* const lz4StreamDecode = LZ4_createStreamDecode();
char* const cmpBuf = malloc(LZ4_COMPRESSBOUND(messageMaxBytes));
char* const decBuf = malloc(ringBufferBytes);
char* const cmpBuf = (char*) malloc(LZ4_COMPRESSBOUND(messageMaxBytes));
char* const decBuf = (char*) malloc(ringBufferBytes);
int decOffset = 0;
for ( ; ; )
@@ -121,8 +125,8 @@ static int compare(FILE* f0, FILE* f1)
{
int result = 0;
const size_t tempBufferBytes = 65536;
char* const b0 = malloc(tempBufferBytes);
char* const b1 = malloc(tempBufferBytes);
char* const b0 = (char*) malloc(tempBufferBytes);
char* const b1 = (char*) malloc(tempBufferBytes);
while(0 == result)
{

122
examples/blockStreaming_lineByLine.md Normal file
View File

@@ -0,0 +1,122 @@
# LZ4 Streaming API Example : Line by Line Text Compression
by *Takayuki Matsuoka*
`blockStreaming_lineByLine.c` is LZ4 Straming API example which implements line by line incremental (de)compression.
Please note the following restrictions :
- Firstly, read "LZ4 Streaming API Basics".
- This is relatively advanced application example.
- Output file is not compatible with lz4frame and platform dependent.
## What's the point of this example ?
- Line by line incremental (de)compression.
- Handle huge file in small amount of memory
- Generally better compression ratio than Block API
- Non-uniform block size
## How the compression works
First of all, allocate "Ring Buffer" for input and LZ4 compressed data buffer for output.
```
(1)
Ring Buffer
+--------+
| Line#1 |
+---+----+
|
v
{Out#1}
(2)
Prefix Mode Dependency
+----+
| |
v |
+--------+-+------+
| Line#1 | Line#2 |
+--------+---+----+
|
v
{Out#2}
(3)
Prefix Prefix
+----+ +----+
| | | |
v | v |
+--------+-+------+-+------+
| Line#1 | Line#2 | Line#3 |
+--------+--------+---+----+
|
v
{Out#3}
(4)
External Dictionary Mode
+----+ +----+
| | | |
v | v |
------+--------+-+------+-+--------+
| .... | Line#X | Line#X+1 |
------+--------+--------+-----+----+
^ |
| v
| {Out#X+1}
|
Reset
(5)
Prefix
+-----+
| |
v |
------+--------+--------+----------+--+-------+
| .... | Line#X | Line#X+1 | Line#X+2 |
------+--------+--------+----------+-----+----+
^ |
| v
| {Out#X+2}
|
Reset
```
Next (see (1)), read first line to ringbuffer and compress it by `LZ4_compress_continue()`.
For the first time, LZ4 doesn't know any previous dependencies,
so it just compress the line without dependencies and generates compressed line {Out#1} to LZ4 compressed data buffer.
After that, write {Out#1} to the file and forward ringbuffer offset.
Do the same things to second line (see (2)).
But in this time, LZ4 can use dependency to Line#1 to improve compression ratio.
This dependency is called "Prefix mode".
Eventually, we'll reach end of ringbuffer at Line#X (see (4)).
This time, we should reset ringbuffer offset.
After resetting, at Line#X+1 pointer is not adjacent, but LZ4 still maintain its memory.
This is called "External Dictionary Mode".
In Line#X+2 (see (5)), finally LZ4 forget almost all memories but still remains Line#X+1.
This is the same situation as Line#2.
Continue these procedure to the end of text file.
## How the decompression works
Decompression will do reverse order.
- Read compressed line from the file to buffer.
- Decompress it to the ringbuffer.
- Output decompressed plain text line to the file.
- Forward ringbuffer offset. If offset exceedes end of the ringbuffer, reset it.
Continue these procedure to the end of the compressed file.

87
examples/streaming_api_basics.md Normal file
View File

@@ -0,0 +1,87 @@
# LZ4 Streaming API Basics
by *Takayuki Matsuoka*
## LZ4 API sets
LZ4 has the following API sets :
- "Auto Framing" API (lz4frame.h) :
This is most recommended API for usual application.
It guarantees interoperability with other LZ4 framing format compliant tools/libraries
such as LZ4 command line utility, node-lz4, etc.
- "Block" API : This is recommended for simple purpose.
It compress single raw memory block to LZ4 memory block and vice versa.
- "Streaming" API : This is designed for complex thing.
For example, compress huge stream data in restricted memory environment.
Basically, you should use "Auto Framing" API.
But if you want to write advanced application, it's time to use Block or Streaming APIs.
## What is difference between Block and Streaming API ?
Block API (de)compresses single contiguous memory block.
In other words, LZ4 library find redundancy from single contiguous memory block.
Streaming API does same thing but (de)compress multiple adjacent contiguous memory block.
So LZ4 library could find more redundancy than Block API.
The following figure shows difference between API and block sizes.
In these figures, original data is splitted to 4KiBytes contiguous chunks.
```
Original Data
+---------------+---------------+----+----+----+
| 4KiB Chunk A | 4KiB Chunk B | C | D |... |
+---------------+---------------+----+----+----+
Example (1) : Block API, 4KiB Block
+---------------+---------------+----+----+----+
| 4KiB Chunk A | 4KiB Chunk B | C | D |... |
+---------------+---------------+----+----+----+
| Block #1 | Block #2 | #3 | #4 |... |
+---------------+---------------+----+----+----+
(No Dependency)
Example (2) : Block API, 8KiB Block
+---------------+---------------+----+----+----+
| 4KiB Chunk A | 4KiB Chunk B | C | D |... |
+---------------+---------------+----+----+----+
| Block #1 |Block #2 |... |
+--------------------+----------+-------+-+----+
^ | ^ |
| | | |
+--------------+ +----+
Internal Dependency Internal Dependency
Example (3) : Streaming API, 4KiB Block
+---------------+---------------+-----+----+----+
| 4KiB Chunk A | 4KiB Chunk B | C | D |... |
+---------------+---------------+-----+----+----+
| Block #1 | Block #2 | #3 | #4 |... |
+---------------+----+----------+-+---+-+--+----+
^ | ^ | ^ |
| | | | | |
+--------------+ +--------+ +---+
Dependency Dependency Dependency
```
- In example (1), there is no dependency.
All blocks are compressed independently.
- In example (2), naturally 8KiBytes block has internal dependency.
But still block #1 and #2 are compressed independently.
- In example (3), block #2 has dependency to #1,
also #3 has dependency to #2 and #1, #4 has #3, #2 and #1, and so on.
Here, we can observe difference between example (2) and (3).
In (2), there's no dependency between chunk B and C, but (3) has dependency between B and C.
This dependency improves compression ratio.
## Restriction of Streaming API
For the efficiency, Streaming API doesn't keep mirror copy of dependent (de)compressed memory.
This means users should keep these dependent (de)compressed memory explicitly.
Usually, "Dependent memory" is previous adjacent contiguous memory up to 64KiBytes.
LZ4 will not access further memories.

BIN
images/image00.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 7.8 KiB

BIN
images/image01.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 3.0 KiB

BIN
images/image02.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 6.2 KiB

BIN
images/image03.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 3.4 KiB

BIN
images/image04.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 4.0 KiB

BIN
images/image05.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 4.1 KiB

BIN
images/image06.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 6.1 KiB

12
lib/Makefile
View File

@@ -26,22 +26,21 @@
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# You can contact the author at :
# - LZ4 source repository : http://code.google.com/p/lz4/
# - LZ4 source mirror : https://github.com/Cyan4973/lz4
# - LZ4 source repository : https://github.com/Cyan4973/lz4
# - LZ4 forum froup : https://groups.google.com/forum/#!forum/lz4c
# ################################################################
# Version numbers
VERSION ?= 126
VERSION ?= 129
LIBVER_MAJOR=`sed -n '/define LZ4_VERSION_MAJOR/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < lz4.h`
LIBVER_MINOR=`sed -n '/define LZ4_VERSION_MINOR/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < lz4.h`
LIBVER_PATCH=`sed -n '/define LZ4_VERSION_RELEASE/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < lz4.h`
LIBVER=$(LIBVER_MAJOR).$(LIBVER_MINOR).$(LIBVER_PATCH)
DESTDIR?=
PREFIX ?= /usr
PREFIX ?= /usr/local
CFLAGS ?= -O3
CFLAGS += -I. -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wstrict-prototypes -pedantic
CFLAGS += -I. -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wcast-qual -Wstrict-prototypes -pedantic
LIBDIR?= $(PREFIX)/lib
INCLUDEDIR=$(PREFIX)/include
@@ -67,7 +66,7 @@ all: liblz4
liblz4: lz4.c lz4hc.c lz4frame.c xxhash.c
@echo compiling static library
@$(CC) $(CPPFLAGS) $(CFLAGS) -c $^
@$(CC) $(CPPFLAGS) $(CFLAGS) -fPIC -c $^
@$(AR) rcs liblz4.a lz4.o lz4hc.o lz4frame.o xxhash.o
@echo compiling dynamic library $(LIBVER)
@$(CC) $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) -shared $^ -fPIC $(SONAME_FLAGS) -o $@.$(SHARED_EXT_VER)
@@ -112,6 +111,7 @@ uninstall:
@[ -f $(DESTDIR)$(LIBDIR)/liblz4.a ] && rm -f $(DESTDIR)$(LIBDIR)/liblz4.a
@[ -f $(DESTDIR)$(INCLUDEDIR)/lz4.h ] && rm -f $(DESTDIR)$(INCLUDEDIR)/lz4.h
@[ -f $(DESTDIR)$(INCLUDEDIR)/lz4hc.h ] && rm -f $(DESTDIR)$(INCLUDEDIR)/lz4hc.h
@[ -f $(DESTDIR)$(INCLUDEDIR)/lz4frame.h ] && rm -f $(DESTDIR)$(INCLUDEDIR)/lz4frame.h
@echo lz4 libraries successfully uninstalled
endif

19
lib/README.md Normal file
View File

@@ -0,0 +1,19 @@
LZ4 - Library Files
================================
This directory contains many files, but you don't necessarily need them all.
If you want to integrate LZ4 compression/decompression into your program, you basically need to include "**lz4.c**" and "**lz4.h**" only.
If you want more compression, at the cost of compression speed (but preserving decompression speed), you will also have to include "**lz4hc.c**" and "**lz4hc.h**". Note that lz4hc needs lz4 to work properly.
Next level, if you want to produce files or data streams compatible with lz4 utility, you will have to use and include "**lz4frame.c**" and **lz4frame.h**". This library encapsulate lz4-compressed blocks into the official interoperable frame format. In order to work properly, lz4frame needs lz4 and lz4hc, and also "**xxhash.c**" and "**xxhash.h**", which provide the error detection algorithm.
A more complex "lz4frame_static.h" is also provided, although its usage is not recommended. It contains definitions which are not guaranteed to remain stable within future versions. Use only if you don't plan to update your lz4 version.
The other files are not source code. There are :
- LICENSE : contains the BSD license text
- Makefile : script to compile or install lz4 library (static or dynamic)
- liblz4.pc.in : for pkg-config (make install)

687
lib/lz4.c
View File

File diff suppressed because it is too large Load Diff

235
lib/lz4.h
View File

@@ -1,7 +1,8 @@
/*
LZ4 - Fast LZ compression algorithm
Header File
Copyright (C) 2011-2014, Yann Collet.
Copyright (C) 2011-2015, Yann Collet.
BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
Redistribution and use in source and binary forms, with or without
@@ -28,7 +29,7 @@
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
You can contact the author at :
- LZ4 source repository : http://code.google.com/p/lz4/
- LZ4 source repository : https://github.com/Cyan4973/lz4
- LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c
*/
#pragma once
@@ -38,22 +39,22 @@ extern "C" {
#endif
/*
* lz4.h provides raw compression format functions, for optimal performance and integration into programs.
* If you need to generate data using an inter-operable format (respecting the framing specification),
* please use lz4frame.h instead.
* lz4.h provides block compression functions, and gives full buffer control to programmer.
* If you need to generate inter-operable compressed data (respecting LZ4 frame specification),
* and can let the library handle its own memory, please use lz4frame.h instead.
*/
/**************************************
Version
* Version
**************************************/
#define LZ4_VERSION_MAJOR 1 /* for breaking interface changes */
#define LZ4_VERSION_MINOR 5 /* for new (non-breaking) interface capabilities */
#define LZ4_VERSION_MINOR 7 /* for new (non-breaking) interface capabilities */
#define LZ4_VERSION_RELEASE 0 /* for tweaks, bug-fixes, or development */
#define LZ4_VERSION_NUMBER (LZ4_VERSION_MAJOR *100*100 + LZ4_VERSION_MINOR *100 + LZ4_VERSION_RELEASE)
int LZ4_versionNumber (void);
/**************************************
Tuning parameter
* Tuning parameter
**************************************/
/*
* LZ4_MEMORY_USAGE :
@@ -66,77 +67,90 @@ int LZ4_versionNumber (void);
/**************************************
Simple Functions
* Simple Functions
**************************************/
int LZ4_compress (const char* source, char* dest, int sourceSize);
int LZ4_compress_default(const char* source, char* dest, int sourceSize, int maxDestSize);
int LZ4_decompress_safe (const char* source, char* dest, int compressedSize, int maxDecompressedSize);
/*
LZ4_compress() :
Compresses 'sourceSize' bytes from 'source' into 'dest'.
Destination buffer must be already allocated,
and must be sized to handle worst cases situations (input data not compressible)
Worst case size evaluation is provided by function LZ4_compressBound()
inputSize : Max supported value is LZ4_MAX_INPUT_SIZE
return : the number of bytes written in buffer dest
or 0 if the compression fails
LZ4_compress_default() :
Compresses 'sourceSize' bytes from buffer 'source'
into already allocated 'dest' buffer of size 'maxDestSize'.
Compression is guaranteed to succeed if 'maxDestSize' >= LZ4_compressBound(sourceSize).
It also runs faster, so it's a recommended setting.
If the function cannot compress 'source' into a more limited 'dest' budget,
compression stops *immediately*, and the function result is zero.
As a consequence, 'dest' content is not valid.
This function never writes outside 'dest' buffer, nor read outside 'source' buffer.
sourceSize : Max supported value is LZ4_MAX_INPUT_VALUE
maxDestSize : full or partial size of buffer 'dest' (which must be already allocated)
return : the number of bytes written into buffer 'dest' (necessarily <= maxOutputSize)
or 0 if compression fails
LZ4_decompress_safe() :
compressedSize : is obviously the source size
maxDecompressedSize : is the size of the destination buffer, which must be already allocated.
return : the number of bytes decompressed into the destination buffer (necessarily <= maxDecompressedSize)
If the destination buffer is not large enough, decoding will stop and output an error code (<0).
compressedSize : is the precise full size of the compressed block.
maxDecompressedSize : is the size of destination buffer, which must be already allocated.
return : the number of bytes decompressed into destination buffer (necessarily <= maxDecompressedSize)
If destination buffer is not large enough, decoding will stop and output an error code (<0).
If the source stream is detected malformed, the function will stop decoding and return a negative result.
This function is protected against buffer overflow exploits,
and never writes outside of output buffer, nor reads outside of input buffer.
It is also protected against malicious data packets.
This function is protected against buffer overflow exploits, including malicious data packets.
It never writes outside output buffer, nor reads outside input buffer.
*/
/**************************************
Advanced Functions
* Advanced Functions
**************************************/
#define LZ4_MAX_INPUT_SIZE 0x7E000000 /* 2 113 929 216 bytes */
#define LZ4_COMPRESSBOUND(isize) ((unsigned int)(isize) > (unsigned int)LZ4_MAX_INPUT_SIZE ? 0 : (isize) + ((isize)/255) + 16)
#define LZ4_COMPRESSBOUND(isize) ((unsigned)(isize) > (unsigned)LZ4_MAX_INPUT_SIZE ? 0 : (isize) + ((isize)/255) + 16)
/*
LZ4_compressBound() :
Provides the maximum size that LZ4 compression may output in a "worst case" scenario (input data not compressible)
This function is primarily useful for memory allocation purposes (output buffer size).
This function is primarily useful for memory allocation purposes (destination buffer size).
Macro LZ4_COMPRESSBOUND() is also provided for compilation-time evaluation (stack memory allocation for example).
isize : is the input size. Max supported value is LZ4_MAX_INPUT_SIZE
return : maximum output size in a "worst case" scenario
or 0, if input size is too large ( > LZ4_MAX_INPUT_SIZE)
Note that LZ4_compress_default() compress faster when dest buffer size is >= LZ4_compressBound(srcSize)
inputSize : max supported value is LZ4_MAX_INPUT_SIZE
return : maximum output size in a "worst case" scenario
or 0, if input size is too large ( > LZ4_MAX_INPUT_SIZE)
*/
int LZ4_compressBound(int isize);
int LZ4_compressBound(int inputSize);
/*
LZ4_compress_fast() :
Same as LZ4_compress_default(), but allows to select an "acceleration" factor.
The larger the acceleration value, the faster the algorithm, but also the lesser the compression.
It's a trade-off. It can be fine tuned, with each successive value providing roughly +~3% to speed.
An acceleration value of "1" is the same as regular LZ4_compress_default()
Values <= 0 will be replaced by ACCELERATION_DEFAULT (see lz4.c), which is 1.
*/
int LZ4_compress_fast (const char* source, char* dest, int sourceSize, int maxDestSize, int acceleration);
/*
LZ4_compress_limitedOutput() :
Compress 'sourceSize' bytes from 'source' into an output buffer 'dest' of maximum size 'maxOutputSize'.
If it cannot achieve it, compression will stop, and result of the function will be zero.
This saves time and memory on detecting non-compressible (or barely compressible) data.
This function never writes outside of provided output buffer.
sourceSize : Max supported value is LZ4_MAX_INPUT_VALUE
maxOutputSize : is the size of the destination buffer (which must be already allocated)
return : the number of bytes written in buffer 'dest'
or 0 if compression fails
*/
int LZ4_compress_limitedOutput (const char* source, char* dest, int sourceSize, int maxOutputSize);
/*
LZ4_compress_withState() :
Same compression functions, but using an externally allocated memory space to store compression state.
LZ4_compress_fast_extState() :
Same compression function, just using an externally allocated memory space to store compression state.
Use LZ4_sizeofState() to know how much memory must be allocated,
and then, provide it as 'void* state' to compression functions.
and allocate it on 8-bytes boundaries (using malloc() typically).
Then, provide it as 'void* state' to compression function.
*/
int LZ4_sizeofState(void);
int LZ4_compress_withState (void* state, const char* source, char* dest, int inputSize);
int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize);
int LZ4_compress_fast_extState (void* state, const char* source, char* dest, int inputSize, int maxDestSize, int acceleration);
/*
LZ4_compress_destSize() :
Reverse the logic, by compressing as much data as possible from 'source' buffer
into already allocated buffer 'dest' of size 'targetDestSize'.
This function either compresses the entire 'source' content into 'dest' if it's large enough,
or fill 'dest' buffer completely with as much data as possible from 'source'.
*sourceSizePtr : will be modified to indicate how many bytes where read from 'source' to fill 'dest'.
New value is necessarily <= old value.
return : Nb bytes written into 'dest' (necessarily <= targetDestSize)
or 0 if compression fails
*/
int LZ4_compress_destSize (const char* source, char* dest, int* sourceSizePtr, int targetDestSize);
/*
@@ -152,7 +166,6 @@ LZ4_decompress_fast() :
*/
int LZ4_decompress_fast (const char* source, char* dest, int originalSize);
/*
LZ4_decompress_safe_partial() :
This function decompress a compressed block of size 'compressedSize' at position 'source'
@@ -169,9 +182,8 @@ int LZ4_decompress_safe_partial (const char* source, char* dest, int compressedS
/***********************************************
Streaming Compression Functions
* Streaming Compression Functions
***********************************************/
#define LZ4_STREAMSIZE_U64 ((1 << (LZ4_MEMORY_USAGE-3)) + 4)
#define LZ4_STREAMSIZE (LZ4_STREAMSIZE_U64 * sizeof(long long))
/*
@@ -187,7 +199,7 @@ typedef struct { long long table[LZ4_STREAMSIZE_U64]; } LZ4_stream_t;
* LZ4_resetStream
* Use this function to init an allocated LZ4_stream_t structure
*/
void LZ4_resetStream (LZ4_stream_t* LZ4_streamPtr);
void LZ4_resetStream (LZ4_stream_t* streamPtr);
/*
* LZ4_createStream will allocate and initialize an LZ4_stream_t structure
@@ -196,7 +208,7 @@ void LZ4_resetStream (LZ4_stream_t* LZ4_streamPtr);
* They are more future proof, in case of a change of LZ4_stream_t size.
*/
LZ4_stream_t* LZ4_createStream(void);
int LZ4_freeStream (LZ4_stream_t* LZ4_streamPtr);
int LZ4_freeStream (LZ4_stream_t* streamPtr);
/*
* LZ4_loadDict
@@ -205,36 +217,31 @@ int LZ4_freeStream (LZ4_stream_t* LZ4_streamPtr);
* Loading a size of 0 is allowed.
* Return : dictionary size, in bytes (necessarily <= 64 KB)
*/
int LZ4_loadDict (LZ4_stream_t* LZ4_streamPtr, const char* dictionary, int dictSize);
int LZ4_loadDict (LZ4_stream_t* streamPtr, const char* dictionary, int dictSize);
/*
* LZ4_compress_continue
* Compress data block 'source', using blocks compressed before as dictionary to improve compression ratio
* Previous data blocks are assumed to still be present at their previous location.
* LZ4_compress_fast_continue
* Compress buffer content 'src', using data from previously compressed blocks as dictionary to improve compression ratio.
* Important : Previous data blocks are assumed to still be present and unmodified !
* 'dst' buffer must be already allocated.
* If maxDstSize >= LZ4_compressBound(srcSize), compression is guaranteed to succeed, and runs faster.
* If not, and if compressed data cannot fit into 'dst' buffer size, compression stops, and function returns a zero.
*/
int LZ4_compress_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize);
/*
* LZ4_compress_limitedOutput_continue
* Same as before, but also specify a maximum target compressed size (maxOutputSize)
* If objective cannot be met, compression exits, and returns a zero.
*/
int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize, int maxOutputSize);
int LZ4_compress_fast_continue (LZ4_stream_t* streamPtr, const char* src, char* dst, int srcSize, int maxDstSize, int acceleration);
/*
* LZ4_saveDict
* If previously compressed data block is not guaranteed to remain available at its memory location
* save it into a safer place (char* safeBuffer)
* Note : you don't need to call LZ4_loadDict() afterwards,
* dictionary is immediately usable, you can therefore call again LZ4_compress_continue()
* Return : dictionary size in bytes, or 0 if error
* Note : any dictSize > 64 KB will be interpreted as 64KB.
* dictionary is immediately usable, you can therefore call LZ4_compress_fast_continue()
* Return : saved dictionary size in bytes (necessarily <= dictSize), or 0 if error
*/
int LZ4_saveDict (LZ4_stream_t* LZ4_streamPtr, char* safeBuffer, int dictSize);
int LZ4_saveDict (LZ4_stream_t* streamPtr, char* safeBuffer, int dictSize);
/************************************************
Streaming Decompression Functions
* Streaming Decompression Functions
************************************************/
#define LZ4_STREAMDECODESIZE_U64 4
@@ -265,8 +272,18 @@ int LZ4_setStreamDecode (LZ4_streamDecode_t* LZ4_streamDecode, const char* dicti
*_continue() :
These decoding functions allow decompression of multiple blocks in "streaming" mode.
Previously decoded blocks *must* remain available at the memory position where they were decoded (up to 64 KB)
If this condition is not possible, save the relevant part of decoded data into a safe buffer,
and indicate where is its new address using LZ4_setStreamDecode()
In the case of a ring buffers, decoding buffer must be either :
- Exactly same size as encoding buffer, with same update rule (block boundaries at same positions)
In which case, the decoding & encoding ring buffer can have any size, including very small ones ( < 64 KB).
- Larger than encoding buffer, by a minimum of maxBlockSize more bytes.
maxBlockSize is implementation dependent. It's the maximum size you intend to compress into a single block.
In which case, encoding and decoding buffers do not need to be synchronized,
and encoding ring buffer can have any size, including small ones ( < 64 KB).
- _At least_ 64 KB + 8 bytes + maxBlockSize.
In which case, encoding and decoding buffers do not need to be synchronized,
and encoding ring buffer can have any size, including larger than decoding buffer.
Whenever these conditions are not possible, save the last 64KB of decoded data into a safe buffer,
and indicate where it is saved using LZ4_setStreamDecode()
*/
int LZ4_decompress_safe_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* source, char* dest, int compressedSize, int maxDecompressedSize);
int LZ4_decompress_fast_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* source, char* dest, int originalSize);
@@ -276,8 +293,8 @@ int LZ4_decompress_fast_continue (LZ4_streamDecode_t* LZ4_streamDecode, const ch
Advanced decoding functions :
*_usingDict() :
These decoding functions work the same as
a combination of LZ4_setDictDecode() followed by LZ4_decompress_x_continue()
They are stand-alone and don't use nor update an LZ4_streamDecode_t structure.
a combination of LZ4_setStreamDecode() followed by LZ4_decompress_x_continue()
They are stand-alone. They don't need nor update an LZ4_streamDecode_t structure.
*/
int LZ4_decompress_safe_usingDict (const char* source, char* dest, int compressedSize, int maxDecompressedSize, const char* dictStart, int dictSize);
int LZ4_decompress_fast_usingDict (const char* source, char* dest, int originalSize, const char* dictStart, int dictSize);
@@ -285,29 +302,57 @@ int LZ4_decompress_fast_usingDict (const char* source, char* dest, int originalS
/**************************************
Obsolete Functions
* Obsolete Functions
**************************************/
/*
Obsolete decompression functions
These function names are deprecated and should no longer be used.
They are only provided here for compatibility with older user programs.
- LZ4_uncompress is the same as LZ4_decompress_fast
- LZ4_uncompress_unknownOutputSize is the same as LZ4_decompress_safe
These function prototypes are now disabled; uncomment them if you really need them.
It is highly recommended to stop using these functions and migrate to newer ones */
/* Deprecate Warnings */
/* Should these warnings messages be a problem,
it is generally possible to disable them,
with -Wno-deprecated-declarations for gcc
or _CRT_SECURE_NO_WARNINGS in Visual for example.
You can also define LZ4_DEPRECATE_WARNING_DEFBLOCK. */
#ifndef LZ4_DEPRECATE_WARNING_DEFBLOCK
# define LZ4_DEPRECATE_WARNING_DEFBLOCK
# define LZ4_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__)
# if (LZ4_GCC_VERSION >= 405) || defined(__clang__)
# define LZ4_DEPRECATED(message) __attribute__((deprecated(message)))
# elif (LZ4_GCC_VERSION >= 301)
# define LZ4_DEPRECATED(message) __attribute__((deprecated))
# elif defined(_MSC_VER)
# define LZ4_DEPRECATED(message) __declspec(deprecated(message))
# else
# pragma message("WARNING: You need to implement LZ4_DEPRECATED for this compiler")
# define LZ4_DEPRECATED(message)
# endif
#endif /* LZ4_DEPRECATE_WARNING_DEFBLOCK */
/* Obsolete compression functions */
/* These functions are planned to start generate warnings by r131 approximately */
int LZ4_compress (const char* source, char* dest, int sourceSize);
int LZ4_compress_limitedOutput (const char* source, char* dest, int sourceSize, int maxOutputSize);
int LZ4_compress_withState (void* state, const char* source, char* dest, int inputSize);
int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize);
int LZ4_compress_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize);
int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize, int maxOutputSize);
/* Obsolete decompression functions */
/* These function names are completely deprecated and must no longer be used.
They are only provided here for compatibility with older programs.
- LZ4_uncompress is the same as LZ4_decompress_fast
- LZ4_uncompress_unknownOutputSize is the same as LZ4_decompress_safe
These function prototypes are now disabled; uncomment them only if you really need them.
It is highly recommended to stop using these prototypes and migrate to maintained ones */
/* int LZ4_uncompress (const char* source, char* dest, int outputSize); */
/* int LZ4_uncompress_unknownOutputSize (const char* source, char* dest, int isize, int maxOutputSize); */
/* Obsolete streaming functions; use new streaming interface whenever possible */
void* LZ4_create (const char* inputBuffer);
int LZ4_sizeofStreamState(void);
int LZ4_resetStreamState(void* state, const char* inputBuffer);
char* LZ4_slideInputBuffer (void* state);
LZ4_DEPRECATED("use LZ4_createStream() instead") void* LZ4_create (char* inputBuffer);
LZ4_DEPRECATED("use LZ4_createStream() instead") int LZ4_sizeofStreamState(void);
LZ4_DEPRECATED("use LZ4_resetStream() instead") int LZ4_resetStreamState(void* state, char* inputBuffer);
LZ4_DEPRECATED("use LZ4_saveDict() instead") char* LZ4_slideInputBuffer (void* state);
/* Obsolete streaming decoding functions */
int LZ4_decompress_safe_withPrefix64k (const char* source, char* dest, int compressedSize, int maxOutputSize);
int LZ4_decompress_fast_withPrefix64k (const char* source, char* dest, int originalSize);
LZ4_DEPRECATED("use LZ4_decompress_safe_usingDict() instead") int LZ4_decompress_safe_withPrefix64k (const char* src, char* dst, int compressedSize, int maxDstSize);
LZ4_DEPRECATED("use LZ4_decompress_fast_usingDict() instead") int LZ4_decompress_fast_withPrefix64k (const char* src, char* dst, int originalSize);
#if defined (__cplusplus)

771
lib/lz4frame.c
View File

File diff suppressed because it is too large Load Diff

163
lib/lz4frame.h
View File

@@ -28,13 +28,12 @@
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
You can contact the author at :
- LZ4 source repository : http://code.google.com/p/lz4/
- LZ4 source mirror : https://github.com/Cyan4973/lz4
- LZ4 source repository : https://github.com/Cyan4973/lz4
- LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c
*/
/* LZ4F is a stand-alone API to create LZ4-compressed frames
* fully conformant to specification v1.4.1.
* fully conformant to specification v1.5.1.
* All related operations, including memory management, are handled by the library.
* You don't need lz4.h when using lz4frame.h.
* */
@@ -46,7 +45,7 @@ extern "C" {
#endif
/**************************************
Includes
* Includes
**************************************/
#include <stddef.h> /* size_t */
@@ -63,22 +62,66 @@ const char* LZ4F_getErrorName(LZ4F_errorCode_t code); /* return error code str
/**************************************
* Frame compression types
* ************************************/
typedef enum { LZ4F_default=0, max64KB=4, max256KB=5, max1MB=6, max4MB=7 } blockSizeID_t;
typedef enum { blockLinked=0, blockIndependent} blockMode_t;
typedef enum { noContentChecksum=0, contentChecksumEnabled } contentChecksum_t;
//#define LZ4F_DISABLE_OBSOLETE_ENUMS
#ifndef LZ4F_DISABLE_OBSOLETE_ENUMS
# define LZ4F_OBSOLETE_ENUM(x) ,x
#else
# define LZ4F_OBSOLETE_ENUM(x)
#endif
typedef enum {
LZ4F_default=0,
LZ4F_max64KB=4,
LZ4F_max256KB=5,
LZ4F_max1MB=6,
LZ4F_max4MB=7
LZ4F_OBSOLETE_ENUM(max64KB = LZ4F_max64KB)
LZ4F_OBSOLETE_ENUM(max256KB = LZ4F_max256KB)
LZ4F_OBSOLETE_ENUM(max1MB = LZ4F_max1MB)
LZ4F_OBSOLETE_ENUM(max4MB = LZ4F_max4MB)
} LZ4F_blockSizeID_t;
typedef enum {
LZ4F_blockLinked=0,
LZ4F_blockIndependent
LZ4F_OBSOLETE_ENUM(blockLinked = LZ4F_blockLinked)
LZ4F_OBSOLETE_ENUM(blockIndependent = LZ4F_blockIndependent)
} LZ4F_blockMode_t;
typedef enum {
LZ4F_noContentChecksum=0,
LZ4F_contentChecksumEnabled
LZ4F_OBSOLETE_ENUM(noContentChecksum = LZ4F_noContentChecksum)
LZ4F_OBSOLETE_ENUM(contentChecksumEnabled = LZ4F_contentChecksumEnabled)
} LZ4F_contentChecksum_t;
typedef enum {
LZ4F_frame=0,
LZ4F_skippableFrame
LZ4F_OBSOLETE_ENUM(skippableFrame = LZ4F_skippableFrame)
} LZ4F_frameType_t;
#ifndef LZ4F_DISABLE_OBSOLETE_ENUMS
typedef LZ4F_blockSizeID_t blockSizeID_t;
typedef LZ4F_blockMode_t blockMode_t;
typedef LZ4F_frameType_t frameType_t;
typedef LZ4F_contentChecksum_t contentChecksum_t;
#endif
typedef struct {
blockSizeID_t blockSizeID; /* max64KB, max256KB, max1MB, max4MB ; 0 == default */
blockMode_t blockMode; /* blockLinked, blockIndependent ; 0 == default */
contentChecksum_t contentChecksumFlag; /* noContentChecksum, contentChecksumEnabled ; 0 == default */
unsigned reserved[5];
LZ4F_blockSizeID_t blockSizeID; /* max64KB, max256KB, max1MB, max4MB ; 0 == default */
LZ4F_blockMode_t blockMode; /* blockLinked, blockIndependent ; 0 == default */
LZ4F_contentChecksum_t contentChecksumFlag; /* noContentChecksum, contentChecksumEnabled ; 0 == default */
LZ4F_frameType_t frameType; /* LZ4F_frame, skippableFrame ; 0 == default */
unsigned long long contentSize; /* Size of uncompressed (original) content ; 0 == unknown */
unsigned reserved[2]; /* must be zero for forward compatibility */
} LZ4F_frameInfo_t;
typedef struct {
LZ4F_frameInfo_t frameInfo;
unsigned compressionLevel; /* 0 == default (fast mode); values above 16 count as 16 */
unsigned autoFlush; /* 1 == always flush : reduce need for tmp buffer */
unsigned reserved[4];
int compressionLevel; /* 0 == default (fast mode); values above 16 count as 16; values below 0 count as 0 */
unsigned autoFlush; /* 1 == always flush (reduce need for tmp buffer) */
unsigned reserved[4]; /* must be zero for forward compatibility */
} LZ4F_preferences_t;
@@ -89,7 +132,7 @@ size_t LZ4F_compressFrameBound(size_t srcSize, const LZ4F_preferences_t* prefere
size_t LZ4F_compressFrame(void* dstBuffer, size_t dstMaxSize, const void* srcBuffer, size_t srcSize, const LZ4F_preferences_t* preferencesPtr);
/* LZ4F_compressFrame()
* Compress an entire srcBuffer into a valid LZ4 frame, as defined by specification v1.4.1.
* Compress an entire srcBuffer into a valid LZ4 frame, as defined by specification v1.5.1
* The most important rule is that dstBuffer MUST be large enough (dstMaxSize) to ensure compression completion even in worst case.
* You can get the minimum value of dstMaxSize by using LZ4F_compressFrameBound()
* If this condition is not respected, LZ4F_compressFrame() will fail (result is an errorCode)
@@ -101,9 +144,9 @@ size_t LZ4F_compressFrame(void* dstBuffer, size_t dstMaxSize, const void* srcBuf
/**********************************
* Advanced compression functions
* ********************************/
typedef void* LZ4F_compressionContext_t;
* Advanced compression functions
**********************************/
typedef struct LZ4F_cctx_s* LZ4F_compressionContext_t; /* must be aligned on 8-bytes */
typedef struct {
unsigned stableSrc; /* 1 == src content will remain available on future calls to LZ4F_compress(); avoid saving src content within tmp buffer as future dictionary */
@@ -113,8 +156,8 @@ typedef struct {
/* Resource Management */
#define LZ4F_VERSION 100
LZ4F_errorCode_t LZ4F_createCompressionContext(LZ4F_compressionContext_t* LZ4F_compressionContextPtr, unsigned version);
LZ4F_errorCode_t LZ4F_freeCompressionContext(LZ4F_compressionContext_t LZ4F_compressionContext);
LZ4F_errorCode_t LZ4F_createCompressionContext(LZ4F_compressionContext_t* cctxPtr, unsigned version);
LZ4F_errorCode_t LZ4F_freeCompressionContext(LZ4F_compressionContext_t cctx);
/* LZ4F_createCompressionContext() :
* The first thing to do is to create a compressionContext object, which will be used in all compression operations.
* This is achieved using LZ4F_createCompressionContext(), which takes as argument a version and an LZ4F_preferences_t structure.
@@ -127,60 +170,63 @@ LZ4F_errorCode_t LZ4F_freeCompressionContext(LZ4F_compressionContext_t LZ4F_comp
/* Compression */
size_t LZ4F_compressBegin(LZ4F_compressionContext_t compressionContext, void* dstBuffer, size_t dstMaxSize, const LZ4F_preferences_t* preferencesPtr);
size_t LZ4F_compressBegin(LZ4F_compressionContext_t cctx, void* dstBuffer, size_t dstMaxSize, const LZ4F_preferences_t* prefsPtr);
/* LZ4F_compressBegin() :
* will write the frame header into dstBuffer.
* dstBuffer must be large enough to accommodate a header (dstMaxSize). Maximum header size is 19 bytes.
* dstBuffer must be large enough to accommodate a header (dstMaxSize). Maximum header size is 15 bytes.
* The LZ4F_preferences_t structure is optional : you can provide NULL as argument, all preferences will then be set to default.
* The result of the function is the number of bytes written into dstBuffer for the header
* or an error code (can be tested using LZ4F_isError())
*/
size_t LZ4F_compressBound(size_t srcSize, const LZ4F_preferences_t* preferencesPtr);
size_t LZ4F_compressBound(size_t srcSize, const LZ4F_preferences_t* prefsPtr);
/* LZ4F_compressBound() :
* Provides the minimum size of Dst buffer given srcSize to handle worst case situations.
* preferencesPtr is optional : you can provide NULL as argument, all preferences will then be set to default.
* Note that different preferences will produce in different results.
* Different preferences can produce different results.
* prefsPtr is optional : you can provide NULL as argument, all preferences will then be set to cover worst case.
* This function includes frame termination cost (4 bytes, or 8 if frame checksum is enabled)
*/
size_t LZ4F_compressUpdate(LZ4F_compressionContext_t compressionContext, void* dstBuffer, size_t dstMaxSize, const void* srcBuffer, size_t srcSize, const LZ4F_compressOptions_t* compressOptionsPtr);
size_t LZ4F_compressUpdate(LZ4F_compressionContext_t cctx, void* dstBuffer, size_t dstMaxSize, const void* srcBuffer, size_t srcSize, const LZ4F_compressOptions_t* cOptPtr);
/* LZ4F_compressUpdate()
* LZ4F_compressUpdate() can be called repetitively to compress as much data as necessary.
* The most important rule is that dstBuffer MUST be large enough (dstMaxSize) to ensure compression completion even in worst case.
* If this condition is not respected, LZ4F_compress() will fail (result is an errorCode)
* You can get the minimum value of dstMaxSize by using LZ4F_compressBound()
* You can get the minimum value of dstMaxSize by using LZ4F_compressBound().
* If this condition is not respected, LZ4F_compress() will fail (result is an errorCode).
* LZ4F_compressUpdate() doesn't guarantee error recovery, so you have to reset compression context when an error occurs.
* The LZ4F_compressOptions_t structure is optional : you can provide NULL as argument.
* The result of the function is the number of bytes written into dstBuffer : it can be zero, meaning input data was just buffered.
* The function outputs an error code if it fails (can be tested using LZ4F_isError())
*/
size_t LZ4F_flush(LZ4F_compressionContext_t compressionContext, void* dstBuffer, size_t dstMaxSize, const LZ4F_compressOptions_t* compressOptionsPtr);
size_t LZ4F_flush(LZ4F_compressionContext_t cctx, void* dstBuffer, size_t dstMaxSize, const LZ4F_compressOptions_t* cOptPtr);
/* LZ4F_flush()
* Should you need to create compressed data immediately, without waiting for a block to be filled,
* you can call LZ4_flush(), which will immediately compress any remaining data buffered within compressionContext.
* The LZ4F_compressOptions_t structure is optional : you can provide NULL as argument.
* Should you need to generate compressed data immediately, without waiting for the current block to be filled,
* you can call LZ4_flush(), which will immediately compress any remaining data buffered within cctx.
* Note that dstMaxSize must be large enough to ensure the operation will be successful.
* LZ4F_compressOptions_t structure is optional : you can provide NULL as argument.
* The result of the function is the number of bytes written into dstBuffer
* (it can be zero, this means there was no data left within compressionContext)
* (it can be zero, this means there was no data left within cctx)
* The function outputs an error code if it fails (can be tested using LZ4F_isError())
*/
size_t LZ4F_compressEnd(LZ4F_compressionContext_t compressionContext, void* dstBuffer, size_t dstMaxSize, const LZ4F_compressOptions_t* compressOptionsPtr);
size_t LZ4F_compressEnd(LZ4F_compressionContext_t cctx, void* dstBuffer, size_t dstMaxSize, const LZ4F_compressOptions_t* cOptPtr);
/* LZ4F_compressEnd()
* When you want to properly finish the compressed frame, just call LZ4F_compressEnd().
* It will flush whatever data remained within compressionContext (like LZ4_flush())
* but also properly finalize the frame, with an endMark and a checksum.
* The result of the function is the number of bytes written into dstBuffer (necessarily >= 4 (endMark size))
* The result of the function is the number of bytes written into dstBuffer (necessarily >= 4 (endMark), or 8 if optional frame checksum is enabled)
* The function outputs an error code if it fails (can be tested using LZ4F_isError())
* The LZ4F_compressOptions_t structure is optional : you can provide NULL as argument.
* compressionContext can then be used again, starting with LZ4F_compressBegin().
* A successful call to LZ4F_compressEnd() makes cctx available again for next compression task.
*/
/***********************************
* Decompression functions
* *********************************/
* Decompression functions
***********************************/
typedef void* LZ4F_decompressionContext_t;
typedef struct LZ4F_dctx_s* LZ4F_decompressionContext_t; /* must be aligned on 8-bytes */
typedef struct {
unsigned stableDst; /* guarantee that decompressed data will still be there on next function calls (avoid storage into tmp buffers) */
@@ -190,38 +236,42 @@ typedef struct {
/* Resource management */
LZ4F_errorCode_t LZ4F_createDecompressionContext(LZ4F_decompressionContext_t* ctxPtr, unsigned version);
LZ4F_errorCode_t LZ4F_freeDecompressionContext(LZ4F_decompressionContext_t ctx);
LZ4F_errorCode_t LZ4F_createDecompressionContext(LZ4F_decompressionContext_t* dctxPtr, unsigned version);
LZ4F_errorCode_t LZ4F_freeDecompressionContext(LZ4F_decompressionContext_t dctx);
/* LZ4F_createDecompressionContext() :
* The first thing to do is to create a decompressionContext object, which will be used in all decompression operations.
* The first thing to do is to create an LZ4F_decompressionContext_t object, which will be used in all decompression operations.
* This is achieved using LZ4F_createDecompressionContext().
* The version provided MUST be LZ4F_VERSION. It is intended to track potential version differences between different binaries.
* The version provided MUST be LZ4F_VERSION. It is intended to track potential breaking differences between different versions.
* The function will provide a pointer to a fully allocated and initialized LZ4F_decompressionContext_t object.
* If the result LZ4F_errorCode_t is not OK_NoError, there was an error during context creation.
* Object can release its memory using LZ4F_freeDecompressionContext();
* The result is an errorCode, which can be tested using LZ4F_isError().
* dctx memory can be released using LZ4F_freeDecompressionContext();
* The result of LZ4F_freeDecompressionContext() is indicative of the current state of decompressionContext when being released.
* That is, it should be == 0 if decompression has been completed fully and correctly.
*/
/* Decompression */
size_t LZ4F_getFrameInfo(LZ4F_decompressionContext_t ctx,
size_t LZ4F_getFrameInfo(LZ4F_decompressionContext_t dctx,
LZ4F_frameInfo_t* frameInfoPtr,
const void* srcBuffer, size_t* srcSizePtr);
/* LZ4F_getFrameInfo()
* This function decodes frame header information, such as blockSize.
* It is optional : you could start by calling directly LZ4F_decompress() instead.
* The objective is to extract header information without starting decompression, typically for allocation purposes.
* LZ4F_getFrameInfo() can also be used *after* starting decompression, on a valid LZ4F_decompressionContext_t.
* This function decodes frame header information (such as max blockSize, frame checksum, etc.).
* Its usage is optional : you can start by calling directly LZ4F_decompress() instead.
* The objective is to extract frame header information, typically for allocation purposes.
* LZ4F_getFrameInfo() can also be used anytime *after* starting decompression, on any valid LZ4F_decompressionContext_t.
* The result is *copied* into an existing LZ4F_frameInfo_t structure which must be already allocated.
* The number of bytes read from srcBuffer will be provided within *srcSizePtr (necessarily <= original value).
* You are expected to resume decompression from where it stopped (srcBuffer + *srcSizePtr)
* The function result is an hint of how many srcSize bytes LZ4F_decompress() expects for next call,
* or an error code which can be tested using LZ4F_isError().
* or an error code which can be tested using LZ4F_isError()
* (typically, when there is not enough src bytes to fully decode the frame header)
* You are expected to resume decompression from where it stopped (srcBuffer + *srcSizePtr)
*/
size_t LZ4F_decompress(LZ4F_decompressionContext_t ctx,
size_t LZ4F_decompress(LZ4F_decompressionContext_t dctx,
void* dstBuffer, size_t* dstSizePtr,
const void* srcBuffer, size_t* srcSizePtr,
const LZ4F_decompressOptions_t* optionsPtr);
const LZ4F_decompressOptions_t* dOptPtr);
/* LZ4F_decompress()
* Call this function repetitively to regenerate data compressed within srcBuffer.
* The function will attempt to decode *srcSizePtr bytes from srcBuffer, into dstBuffer of maximum size *dstSizePtr.
@@ -241,12 +291,13 @@ size_t LZ4F_decompress(LZ4F_decompressionContext_t ctx,
* Schematically, it's the size of the current (or remaining) compressed block + header of next block.
* Respecting the hint provides some boost to performance, since it does skip intermediate buffers.
* This is just a hint, you can always provide any srcSize you want.
* When a frame is fully decoded, the function result will be 0. (no more data expected)
* When a frame is fully decoded, the function result will be 0 (no more data expected).
* If decompression failed, function result is an error code, which can be tested using LZ4F_isError().
*
* After a frame is fully decoded, dctx can be used again to decompress another frame.
*/
#if defined (__cplusplus)
}
#endif

29
lib/lz4frame_static.h
View File

@@ -29,8 +29,7 @@
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
You can contact the author at :
- LZ4 source repository : http://code.google.com/p/lz4/
- LZ4 source mirror : https://github.com/Cyan4973/lz4
- LZ4 source repository : https://github.com/Cyan4973/lz4
- LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c
*/
@@ -41,9 +40,17 @@ extern "C" {
#endif
/* lz4frame_static.h should be used solely in the context of static linking.
* It contains definitions which may still change overtime.
* Never use it in the context of DLL linking.
* */
/**************************************
* Includes
**************************************/
#include "lz4frame.h"
/**************************************
* Error management
* ************************************/
@@ -51,22 +58,24 @@ extern "C" {
ITEM(OK_NoError) ITEM(ERROR_GENERIC) \
ITEM(ERROR_maxBlockSize_invalid) ITEM(ERROR_blockMode_invalid) ITEM(ERROR_contentChecksumFlag_invalid) \
ITEM(ERROR_compressionLevel_invalid) \
ITEM(ERROR_headerVersion_wrong) ITEM(ERROR_blockChecksum_unsupported) ITEM(ERROR_reservedFlag_set) \
ITEM(ERROR_allocation_failed) \
ITEM(ERROR_srcSize_tooLarge) ITEM(ERROR_dstMaxSize_tooSmall) \
ITEM(ERROR_frameHeader_incomplete) ITEM(ERROR_frameType_unknown) ITEM(ERROR_frameSize_wrong) \
ITEM(ERROR_srcPtr_wrong) \
ITEM(ERROR_decompressionFailed) \
ITEM(ERROR_checksum_invalid) \
ITEM(ERROR_headerChecksum_invalid) ITEM(ERROR_contentChecksum_invalid) \
ITEM(ERROR_maxCode)
#define LZ4F_GENERATE_ENUM(ENUM) ENUM,
//#define LZ4F_DISABLE_OLD_ENUMS
#ifndef LZ4F_DISABLE_OLD_ENUMS
#define LZ4F_GENERATE_ENUM(ENUM) LZ4F_##ENUM, ENUM = LZ4F_##ENUM,
#else
#define LZ4F_GENERATE_ENUM(ENUM) LZ4F_##ENUM,
#endif
typedef enum { LZ4F_LIST_ERRORS(LZ4F_GENERATE_ENUM) } LZ4F_errorCodes; /* enum is exposed, to handle specific errors; compare function result to -enum value */
/**************************************
Includes
**************************************/
#include "lz4frame.h"
#if defined (__cplusplus)
}
#endif

245
lib/lz4hc.c
View File

@@ -1,52 +1,53 @@
/*
LZ4 HC - High Compression Mode of LZ4
Copyright (C) 2011-2014, Yann Collet.
BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
LZ4 HC - High Compression Mode of LZ4
Copyright (C) 2011-2015, Yann Collet.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
You can contact the author at :
- LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
- LZ4 source repository : http://code.google.com/p/lz4/
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
You can contact the author at :
- LZ4 source repository : https://github.com/Cyan4973/lz4
- LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c
*/
/**************************************
Tuning Parameter
* Tuning Parameter
**************************************/
static const int LZ4HC_compressionLevel_default = 8;
static const int LZ4HC_compressionLevel_default = 9;
/**************************************
Includes
* Includes
**************************************/
#include "lz4hc.h"
/**************************************
Local Compiler Options
* Local Compiler Options
**************************************/
#if defined(__GNUC__)
# pragma GCC diagnostic ignored "-Wunused-function"
@@ -58,18 +59,18 @@ static const int LZ4HC_compressionLevel_default = 8;
/**************************************
Common LZ4 definition
* Common LZ4 definition
**************************************/
#define LZ4_COMMONDEFS_ONLY
#include "lz4.c"
/**************************************
Local Constants
* Local Constants
**************************************/
#define DICTIONARY_LOGSIZE 16
#define MAXD (1<<DICTIONARY_LOGSIZE)
#define MAXD_MASK ((U32)(MAXD - 1))
#define MAXD_MASK (MAXD - 1)
#define HASH_LOG (DICTIONARY_LOGSIZE-1)
#define HASHTABLESIZE (1 << HASH_LOG)
@@ -81,36 +82,36 @@ static const int g_maxCompressionLevel = 16;
/**************************************
Local Types
* Local Types
**************************************/
typedef struct
{
U32 hashTable[HASHTABLESIZE];
U32 hashTable[HASHTABLESIZE];
U16 chainTable[MAXD];
const BYTE* end; /* next block here to continue on current prefix */
const BYTE* base; /* All index relative to this position */
const BYTE* dictBase; /* alternate base for extDict */
const BYTE* inputBuffer;/* deprecated */
BYTE* inputBuffer; /* deprecated */
U32 dictLimit; /* below that point, need extDict */
U32 lowLimit; /* below that point, no more dict */
U32 nextToUpdate;
U32 nextToUpdate; /* index from which to continue dictionary update */
U32 compressionLevel;
} LZ4HC_Data_Structure;
/**************************************
Local Macros
* Local Macros
**************************************/
#define HASH_FUNCTION(i) (((i) * 2654435761U) >> ((MINMATCH*8)-HASH_LOG))
#define DELTANEXT(p) chainTable[(size_t)(p) & MAXD_MASK]
#define GETNEXT(p) ((p) - (size_t)DELTANEXT(p))
//#define DELTANEXTU16(p) chainTable[(p) & MAXD_MASK] /* flexible, MAXD dependent */
#define DELTANEXTU16(p) chainTable[(U16)(p)] /* faster */
static U32 LZ4HC_hashPtr(const void* ptr) { return HASH_FUNCTION(LZ4_read32(ptr)); }
/**************************************
HC Compression
* HC Compression
**************************************/
static void LZ4HC_init (LZ4HC_Data_Structure* hc4, const BYTE* start)
{
@@ -118,7 +119,6 @@ static void LZ4HC_init (LZ4HC_Data_Structure* hc4, const BYTE* start)
MEM_INIT(hc4->chainTable, 0xFF, sizeof(hc4->chainTable));
hc4->nextToUpdate = 64 KB;
hc4->base = start - 64 KB;
hc4->inputBuffer = start;
hc4->end = start;
hc4->dictBase = start - 64 KB;
hc4->dictLimit = 64 KB;
@@ -140,7 +140,7 @@ FORCE_INLINE void LZ4HC_Insert (LZ4HC_Data_Structure* hc4, const BYTE* ip)
U32 h = LZ4HC_hashPtr(base+idx);
size_t delta = idx - HashTable[h];
if (delta>MAX_DISTANCE) delta = MAX_DISTANCE;
chainTable[idx & 0xFFFF] = (U16)delta;
DELTANEXTU16(idx) = (U16)delta;
HashTable[h] = idx;
idx++;
}
@@ -196,7 +196,7 @@ FORCE_INLINE int LZ4HC_InsertAndFindBestMatch (LZ4HC_Data_Structure* hc4, /* I
if (mlt > ml) { ml = mlt; *matchpos = base + matchIndex; } /* virtual matchpos */
}
}
matchIndex -= chainTable[matchIndex & 0xFFFF];
matchIndex -= DELTANEXTU16(matchIndex);
}
return (int)ml;
@@ -205,9 +205,9 @@ FORCE_INLINE int LZ4HC_InsertAndFindBestMatch (LZ4HC_Data_Structure* hc4, /* I
FORCE_INLINE int LZ4HC_InsertAndGetWiderMatch (
LZ4HC_Data_Structure* hc4,
const BYTE* ip,
const BYTE* iLowLimit,
const BYTE* iHighLimit,
const BYTE* const ip,
const BYTE* const iLowLimit,
const BYTE* const iHighLimit,
int longest,
const BYTE** matchpos,
const BYTE** startpos,
@@ -217,9 +217,9 @@ FORCE_INLINE int LZ4HC_InsertAndGetWiderMatch (
U32* const HashTable = hc4->hashTable;
const BYTE* const base = hc4->base;
const U32 dictLimit = hc4->dictLimit;
const BYTE* const lowPrefixPtr = base + dictLimit;
const U32 lowLimit = (hc4->lowLimit + 64 KB > (U32)(ip-base)) ? hc4->lowLimit : (U32)(ip - base) - (64 KB - 1);
const BYTE* const dictBase = hc4->dictBase;
const BYTE* match;
U32 matchIndex;
int nbAttempts = maxNbAttempts;
int delta = (int)(ip-iLowLimit);
@@ -234,42 +234,46 @@ FORCE_INLINE int LZ4HC_InsertAndGetWiderMatch (
nbAttempts--;
if (matchIndex >= dictLimit)
{
match = base + matchIndex;
if (*(iLowLimit + longest) == *(match - delta + longest))
if (LZ4_read32(match) == LZ4_read32(ip))
const BYTE* matchPtr = base + matchIndex;
if (*(iLowLimit + longest) == *(matchPtr - delta + longest))
if (LZ4_read32(matchPtr) == LZ4_read32(ip))
{
const BYTE* startt = ip;
const BYTE* tmpMatch = match;
const BYTE* const matchEnd = ip + MINMATCH + LZ4_count(ip+MINMATCH, match+MINMATCH, iHighLimit);
int mlt = MINMATCH + LZ4_count(ip+MINMATCH, matchPtr+MINMATCH, iHighLimit);
int back = 0;
while ((startt>iLowLimit) && (tmpMatch > iLowLimit) && (startt[-1] == tmpMatch[-1])) {startt--; tmpMatch--;}
while ((ip+back>iLowLimit)
&& (matchPtr+back > lowPrefixPtr)
&& (ip[back-1] == matchPtr[back-1]))
back--;
if ((matchEnd-startt) > longest)
mlt -= back;
if (mlt > longest)
{
longest = (int)(matchEnd-startt);
*matchpos = tmpMatch;
*startpos = startt;
longest = (int)mlt;
*matchpos = matchPtr+back;
*startpos = ip+back;
}
}
}
else
{
match = dictBase + matchIndex;
if (LZ4_read32(match) == LZ4_read32(ip))
const BYTE* matchPtr = dictBase + matchIndex;
if (LZ4_read32(matchPtr) == LZ4_read32(ip))
{
size_t mlt;
int back=0;
const BYTE* vLimit = ip + (dictLimit - matchIndex);
if (vLimit > iHighLimit) vLimit = iHighLimit;
mlt = LZ4_count(ip+MINMATCH, match+MINMATCH, vLimit) + MINMATCH;
mlt = LZ4_count(ip+MINMATCH, matchPtr+MINMATCH, vLimit) + MINMATCH;
if ((ip+mlt == vLimit) && (vLimit < iHighLimit))
mlt += LZ4_count(ip+mlt, base+dictLimit, iHighLimit);
while ((ip+back > iLowLimit) && (matchIndex+back > lowLimit) && (ip[back-1] == match[back-1])) back--;
while ((ip+back > iLowLimit) && (matchIndex+back > lowLimit) && (ip[back-1] == matchPtr[back-1])) back--;
mlt -= back;
if ((int)mlt > longest) { longest = (int)mlt; *matchpos = base + matchIndex + back; *startpos = ip+back; }
}
}
matchIndex -= chainTable[matchIndex & 0xFFFF];
matchIndex -= DELTANEXTU16(matchIndex);
}
return longest;
@@ -531,69 +535,38 @@ _Search3:
}
int LZ4_compressHC2(const char* source, char* dest, int inputSize, int compressionLevel)
{
LZ4HC_Data_Structure ctx;
LZ4HC_init(&ctx, (const BYTE*)source);
return LZ4HC_compress_generic (&ctx, source, dest, inputSize, 0, compressionLevel, noLimit);
}
int LZ4_compressHC(const char* source, char* dest, int inputSize) { return LZ4_compressHC2(source, dest, inputSize, 0); }
int LZ4_compressHC2_limitedOutput(const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel)
{
LZ4HC_Data_Structure ctx;
LZ4HC_init(&ctx, (const BYTE*)source);
return LZ4HC_compress_generic (&ctx, source, dest, inputSize, maxOutputSize, compressionLevel, limitedOutput);
}
int LZ4_compressHC_limitedOutput(const char* source, char* dest, int inputSize, int maxOutputSize)
{
return LZ4_compressHC2_limitedOutput(source, dest, inputSize, maxOutputSize, 0);
}
/*****************************
* Using external allocation
* ***************************/
int LZ4_sizeofStateHC(void) { return sizeof(LZ4HC_Data_Structure); }
int LZ4_compressHC2_withStateHC (void* state, const char* source, char* dest, int inputSize, int compressionLevel)
int LZ4_compress_HC_extStateHC (void* state, const char* src, char* dst, int srcSize, int maxDstSize, int compressionLevel)
{
if (((size_t)(state)&(sizeof(void*)-1)) != 0) return 0; /* Error : state is not aligned for pointers (32 or 64 bits) */
LZ4HC_init ((LZ4HC_Data_Structure*)state, (const BYTE*)source);
return LZ4HC_compress_generic (state, source, dest, inputSize, 0, compressionLevel, noLimit);
LZ4HC_init ((LZ4HC_Data_Structure*)state, (const BYTE*)src);
if (maxDstSize < LZ4_compressBound(srcSize))
return LZ4HC_compress_generic (state, src, dst, srcSize, maxDstSize, compressionLevel, limitedOutput);
else
return LZ4HC_compress_generic (state, src, dst, srcSize, maxDstSize, compressionLevel, noLimit);
}
int LZ4_compressHC_withStateHC (void* state, const char* source, char* dest, int inputSize)
{ return LZ4_compressHC2_withStateHC (state, source, dest, inputSize, 0); }
int LZ4_compressHC2_limitedOutput_withStateHC (void* state, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel)
int LZ4_compress_HC(const char* src, char* dst, int srcSize, int maxDstSize, int compressionLevel)
{
if (((size_t)(state)&(sizeof(void*)-1)) != 0) return 0; /* Error : state is not aligned for pointers (32 or 64 bits) */
LZ4HC_init ((LZ4HC_Data_Structure*)state, (const BYTE*)source);
return LZ4HC_compress_generic (state, source, dest, inputSize, maxOutputSize, compressionLevel, limitedOutput);
LZ4HC_Data_Structure state;
return LZ4_compress_HC_extStateHC(&state, src, dst, srcSize, maxDstSize, compressionLevel);
}
int LZ4_compressHC_limitedOutput_withStateHC (void* state, const char* source, char* dest, int inputSize, int maxOutputSize)
{ return LZ4_compressHC2_limitedOutput_withStateHC (state, source, dest, inputSize, maxOutputSize, 0); }
/**************************************
* Streaming Functions
* ************************************/
* Streaming Functions
**************************************/
/* allocation */
LZ4_streamHC_t* LZ4_createStreamHC(void) { return (LZ4_streamHC_t*)malloc(sizeof(LZ4_streamHC_t)); }
int LZ4_freeStreamHC (LZ4_streamHC_t* LZ4_streamHCPtr) { free(LZ4_streamHCPtr); return 0; }
int LZ4_freeStreamHC (LZ4_streamHC_t* LZ4_streamHCPtr) { free(LZ4_streamHCPtr); return 0; }
/* initialization */
void LZ4_resetStreamHC (LZ4_streamHC_t* LZ4_streamHCPtr, int compressionLevel)
{
LZ4_STATIC_ASSERT(sizeof(LZ4HC_Data_Structure) <= LZ4_STREAMHCSIZE); /* if compilation fails here, LZ4_STREAMHCSIZE must be increased */
LZ4_STATIC_ASSERT(sizeof(LZ4HC_Data_Structure) <= sizeof(LZ4_streamHC_t)); /* if compilation fails here, LZ4_STREAMHCSIZE must be increased */
((LZ4HC_Data_Structure*)LZ4_streamHCPtr)->base = NULL;
((LZ4HC_Data_Structure*)LZ4_streamHCPtr)->compressionLevel = (unsigned)compressionLevel;
}
@@ -646,14 +619,15 @@ static int LZ4_compressHC_continue_generic (LZ4HC_Data_Structure* ctxPtr,
}
/* Check if blocks follow each other */
if ((const BYTE*)source != ctxPtr->end) LZ4HC_setExternalDict(ctxPtr, (const BYTE*)source);
if ((const BYTE*)source != ctxPtr->end)
LZ4HC_setExternalDict(ctxPtr, (const BYTE*)source);
/* Check overlapping input/dictionary space */
{
const BYTE* sourceEnd = (const BYTE*) source + inputSize;
const BYTE* dictBegin = ctxPtr->dictBase + ctxPtr->lowLimit;
const BYTE* dictEnd = ctxPtr->dictBase + ctxPtr->dictLimit;
if ((sourceEnd > dictBegin) && ((BYTE*)source < dictEnd))
if ((sourceEnd > dictBegin) && ((const BYTE*)source < dictEnd))
{
if (sourceEnd > dictEnd) sourceEnd = dictEnd;
ctxPtr->lowLimit = (U32)(sourceEnd - ctxPtr->dictBase);
@@ -664,14 +638,12 @@ static int LZ4_compressHC_continue_generic (LZ4HC_Data_Structure* ctxPtr,
return LZ4HC_compress_generic (ctxPtr, source, dest, inputSize, maxOutputSize, ctxPtr->compressionLevel, limit);
}
int LZ4_compressHC_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize)
int LZ4_compress_HC_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize, int maxOutputSize)
{
return LZ4_compressHC_continue_generic ((LZ4HC_Data_Structure*)LZ4_streamHCPtr, source, dest, inputSize, 0, noLimit);
}
int LZ4_compressHC_limitedOutput_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize, int maxOutputSize)
{
return LZ4_compressHC_continue_generic ((LZ4HC_Data_Structure*)LZ4_streamHCPtr, source, dest, inputSize, maxOutputSize, limitedOutput);
if (maxOutputSize < LZ4_compressBound(inputSize))
return LZ4_compressHC_continue_generic ((LZ4HC_Data_Structure*)LZ4_streamHCPtr, source, dest, inputSize, maxOutputSize, limitedOutput);
else
return LZ4_compressHC_continue_generic ((LZ4HC_Data_Structure*)LZ4_streamHCPtr, source, dest, inputSize, maxOutputSize, noLimit);
}
@@ -684,7 +656,7 @@ int LZ4_saveDictHC (LZ4_streamHC_t* LZ4_streamHCPtr, char* safeBuffer, int dictS
if (dictSize > 64 KB) dictSize = 64 KB;
if (dictSize < 4) dictSize = 0;
if (dictSize > prefixSize) dictSize = prefixSize;
memcpy(safeBuffer, streamPtr->end - dictSize, dictSize);
memmove(safeBuffer, streamPtr->end - dictSize, dictSize);
{
U32 endIndex = (U32)(streamPtr->end - streamPtr->base);
streamPtr->end = (const BYTE*)safeBuffer + dictSize;
@@ -698,21 +670,39 @@ int LZ4_saveDictHC (LZ4_streamHC_t* LZ4_streamHCPtr, char* safeBuffer, int dictS
/***********************************
* Deprecated Functions
***********************************/
* Deprecated Functions
***********************************/
/* Deprecated compression functions */
/* These functions are planned to start generate warnings by r131 approximately */
int LZ4_compressHC(const char* src, char* dst, int srcSize) { return LZ4_compress_HC (src, dst, srcSize, LZ4_compressBound(srcSize), 0); }
int LZ4_compressHC_limitedOutput(const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compress_HC(src, dst, srcSize, maxDstSize, 0); }
int LZ4_compressHC2(const char* src, char* dst, int srcSize, int cLevel) { return LZ4_compress_HC (src, dst, srcSize, LZ4_compressBound(srcSize), cLevel); }
int LZ4_compressHC2_limitedOutput(const char* src, char* dst, int srcSize, int maxDstSize, int cLevel) { return LZ4_compress_HC(src, dst, srcSize, maxDstSize, cLevel); }
int LZ4_compressHC_withStateHC (void* state, const char* src, char* dst, int srcSize) { return LZ4_compress_HC_extStateHC (state, src, dst, srcSize, LZ4_compressBound(srcSize), 0); }
int LZ4_compressHC_limitedOutput_withStateHC (void* state, const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compress_HC_extStateHC (state, src, dst, srcSize, maxDstSize, 0); }
int LZ4_compressHC2_withStateHC (void* state, const char* src, char* dst, int srcSize, int cLevel) { return LZ4_compress_HC_extStateHC(state, src, dst, srcSize, LZ4_compressBound(srcSize), cLevel); }
int LZ4_compressHC2_limitedOutput_withStateHC (void* state, const char* src, char* dst, int srcSize, int maxDstSize, int cLevel) { return LZ4_compress_HC_extStateHC(state, src, dst, srcSize, maxDstSize, cLevel); }
int LZ4_compressHC_continue (LZ4_streamHC_t* ctx, const char* src, char* dst, int srcSize) { return LZ4_compress_HC_continue (ctx, src, dst, srcSize, LZ4_compressBound(srcSize)); }
int LZ4_compressHC_limitedOutput_continue (LZ4_streamHC_t* ctx, const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compress_HC_continue (ctx, src, dst, srcSize, maxDstSize); }
/* Deprecated streaming functions */
/* These functions currently generate deprecation warnings */
int LZ4_sizeofStreamStateHC(void) { return LZ4_STREAMHCSIZE; }
int LZ4_resetStreamStateHC(void* state, const char* inputBuffer)
int LZ4_resetStreamStateHC(void* state, char* inputBuffer)
{
if ((((size_t)state) & (sizeof(void*)-1)) != 0) return 1; /* Error : pointer is not aligned for pointer (32 or 64 bits) */
LZ4HC_init((LZ4HC_Data_Structure*)state, (const BYTE*)inputBuffer);
((LZ4HC_Data_Structure*)state)->inputBuffer = (BYTE*)inputBuffer;
return 0;
}
void* LZ4_createHC (const char* inputBuffer)
void* LZ4_createHC (char* inputBuffer)
{
void* hc4 = ALLOCATOR(1, sizeof(LZ4HC_Data_Structure));
LZ4HC_init ((LZ4HC_Data_Structure*)hc4, (const BYTE*)inputBuffer);
((LZ4HC_Data_Structure*)hc4)->inputBuffer = (BYTE*)inputBuffer;
return hc4;
}
@@ -722,17 +712,6 @@ int LZ4_freeHC (void* LZ4HC_Data)
return (0);
}
/*
int LZ4_compressHC_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize)
{
return LZ4HC_compress_generic (LZ4HC_Data, source, dest, inputSize, 0, 0, noLimit);
}
int LZ4_compressHC_limitedOutput_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int maxOutputSize)
{
return LZ4HC_compress_generic (LZ4HC_Data, source, dest, inputSize, maxOutputSize, 0, limitedOutput);
}
*/
int LZ4_compressHC2_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int compressionLevel)
{
return LZ4HC_compress_generic (LZ4HC_Data, source, dest, inputSize, 0, compressionLevel, noLimit);

195
lib/lz4hc.h
View File

@@ -1,7 +1,7 @@
/*
LZ4 HC - High Compression Mode of LZ4
Header File
Copyright (C) 2011-2014, Yann Collet.
Copyright (C) 2011-2015, Yann Collet.
BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
Redistribution and use in source and binary forms, with or without
@@ -28,8 +28,8 @@
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
You can contact the author at :
- LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
- LZ4 source repository : http://code.google.com/p/lz4/
- LZ4 source repository : https://github.com/Cyan4973/lz4
- LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c
*/
#pragma once
@@ -38,135 +38,150 @@
extern "C" {
#endif
/*****************************
* Includes
*****************************/
#include <stddef.h> /* size_t */
int LZ4_compressHC (const char* source, char* dest, int inputSize);
/**************************************
* Block Compression
**************************************/
int LZ4_compress_HC (const char* src, char* dst, int srcSize, int maxDstSize, int compressionLevel);
/*
LZ4_compressHC :
return : the number of bytes in compressed buffer dest
or 0 if compression fails.
note : destination buffer must be already allocated.
To avoid any problem, size it to handle worst cases situations (input data not compressible)
Worst case size evaluation is provided by function LZ4_compressBound() (see "lz4.h")
LZ4_compress_HC :
Destination buffer 'dst' must be already allocated.
Compression completion is guaranteed if 'dst' buffer is sized to handle worst circumstances (data not compressible)
Worst size evaluation is provided by function LZ4_compressBound() (see "lz4.h")
srcSize : Max supported value is LZ4_MAX_INPUT_SIZE (see "lz4.h")
compressionLevel : Recommended values are between 4 and 9, although any value between 0 and 16 will work.
0 means "use default value" (see lz4hc.c).
Values >16 behave the same as 16.
return : the number of bytes written into buffer 'dst'
or 0 if compression fails.
*/
int LZ4_compressHC_limitedOutput (const char* source, char* dest, int inputSize, int maxOutputSize);
/*
LZ4_compress_limitedOutput() :
Compress 'inputSize' bytes from 'source' into an output buffer 'dest' of maximum size 'maxOutputSize'.
If it cannot achieve it, compression will stop, and result of the function will be zero.
This function never writes outside of provided output buffer.
inputSize : Max supported value is 1 GB
maxOutputSize : is maximum allowed size into the destination buffer (which must be already allocated)
return : the number of output bytes written in buffer 'dest'
or 0 if compression fails.
*/
int LZ4_compressHC2 (const char* source, char* dest, int inputSize, int compressionLevel);
int LZ4_compressHC2_limitedOutput (const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel);
/*
Same functions as above, but with programmable 'compressionLevel'.
Recommended values are between 4 and 9, although any value between 0 and 16 will work.
'compressionLevel'==0 means use default 'compressionLevel' value.
Values above 16 behave the same as 16.
Equivalent variants exist for all other compression functions below.
*/
/* Note :
Decompression functions are provided within LZ4 source code (see "lz4.h") (BSD license)
*/
/**************************************
Using an external allocation
**************************************/
int LZ4_sizeofStateHC(void);
int LZ4_compressHC_withStateHC (void* state, const char* source, char* dest, int inputSize);
int LZ4_compressHC_limitedOutput_withStateHC (void* state, const char* source, char* dest, int inputSize, int maxOutputSize);
int LZ4_compressHC2_withStateHC (void* state, const char* source, char* dest, int inputSize, int compressionLevel);
int LZ4_compressHC2_limitedOutput_withStateHC(void* state, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel);
int LZ4_compress_HC_extStateHC(void* state, const char* src, char* dst, int srcSize, int maxDstSize, int compressionLevel);
/*
These functions are provided should you prefer to allocate memory for compression tables with your own allocation methods.
To know how much memory must be allocated for the compression tables, use :
int LZ4_sizeofStateHC();
LZ4_compress_HC_extStateHC() :
Use this function if you prefer to manually allocate memory for compression tables.
To know how much memory must be allocated for the compression tables, use :
int LZ4_sizeofStateHC();
Note that tables must be aligned for pointer (32 or 64 bits), otherwise compression will fail (return code 0).
Allocated memory must be aligned on 8-bytes boundaries (which a normal malloc() will do properly).
The allocated memory can be provided to the compression functions using 'void* state' parameter.
LZ4_compress_withStateHC() and LZ4_compress_limitedOutput_withStateHC() are equivalent to previously described functions.
They just use the externally allocated memory for state instead of allocating their own (on stack, or on heap).
The allocated memory can then be provided to the compression functions using 'void* state' parameter.
LZ4_compress_HC_extStateHC() is equivalent to previously described function.
It just uses externally allocated memory for stateHC.
*/
/**************************************
Experimental Streaming Functions
* Streaming Compression
**************************************/
#define LZ4_STREAMHCSIZE_U64 32774
#define LZ4_STREAMHCSIZE (LZ4_STREAMHCSIZE_U64 * sizeof(unsigned long long))
typedef struct { unsigned long long table[LZ4_STREAMHCSIZE_U64]; } LZ4_streamHC_t;
#define LZ4_STREAMHCSIZE 262192
#define LZ4_STREAMHCSIZE_SIZET (LZ4_STREAMHCSIZE / sizeof(size_t))
typedef struct { size_t table[LZ4_STREAMHCSIZE_SIZET]; } LZ4_streamHC_t;
/*
LZ4_streamHC_t
This structure allows static allocation of LZ4 HC streaming state.
State must then be initialized using LZ4_resetStreamHC() before first use.
LZ4_streamHC_t
This structure allows static allocation of LZ4 HC streaming state.
State must then be initialized using LZ4_resetStreamHC() before first use.
Static allocation should only be used with statically linked library.
If you want to use LZ4 as a DLL, please use construction functions below, which are more future-proof.
Static allocation should only be used in combination with static linking.
If you want to use LZ4 as a DLL, please use construction functions below, which are future-proof.
*/
LZ4_streamHC_t* LZ4_createStreamHC(void);
int LZ4_freeStreamHC (LZ4_streamHC_t* LZ4_streamHCPtr);
int LZ4_freeStreamHC (LZ4_streamHC_t* streamHCPtr);
/*
These functions create and release memory for LZ4 HC streaming state.
Newly created states are already initialized.
Existing state space can be re-used anytime using LZ4_resetStreamHC().
If you use LZ4 as a DLL, please use these functions instead of direct struct allocation,
to avoid size mismatch between different versions.
These functions create and release memory for LZ4 HC streaming state.
Newly created states are already initialized.
Existing state space can be re-used anytime using LZ4_resetStreamHC().
If you use LZ4 as a DLL, use these functions instead of static structure allocation,
to avoid size mismatch between different versions.
*/
void LZ4_resetStreamHC (LZ4_streamHC_t* LZ4_streamHCPtr, int compressionLevel);
int LZ4_loadDictHC (LZ4_streamHC_t* LZ4_streamHCPtr, const char* dictionary, int dictSize);
void LZ4_resetStreamHC (LZ4_streamHC_t* streamHCPtr, int compressionLevel);
int LZ4_loadDictHC (LZ4_streamHC_t* streamHCPtr, const char* dictionary, int dictSize);
int LZ4_compressHC_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize);
int LZ4_compressHC_limitedOutput_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize, int maxOutputSize);
int LZ4_compress_HC_continue (LZ4_streamHC_t* streamHCPtr, const char* src, char* dst, int srcSize, int maxDstSize);
int LZ4_saveDictHC (LZ4_streamHC_t* LZ4_streamHCPtr, char* safeBuffer, int maxDictSize);
int LZ4_saveDictHC (LZ4_streamHC_t* streamHCPtr, char* safeBuffer, int maxDictSize);
/*
These functions compress data in successive blocks of any size, using previous blocks as dictionary.
One key assumption is that each previous block will remain read-accessible while compressing next block.
These functions compress data in successive blocks of any size, using previous blocks as dictionary.
One key assumption is that previous blocks (up to 64 KB) remain read-accessible while compressing next blocks.
There is an exception for ring buffers, which can be smaller 64 KB.
Such case is automatically detected and correctly handled by LZ4_compress_HC_continue().
Before starting compression, state must be properly initialized, using LZ4_resetStreamHC().
A first "fictional block" can then be designated as initial dictionary, using LZ4_loadDictHC() (Optional).
Before starting compression, state must be properly initialized, using LZ4_resetStreamHC().
A first "fictional block" can then be designated as initial dictionary, using LZ4_loadDictHC() (Optional).
Then, use LZ4_compressHC_continue() or LZ4_compressHC_limitedOutput_continue() to compress each successive block.
They work like usual LZ4_compressHC() or LZ4_compressHC_limitedOutput(), but use previous memory blocks to improve compression.
Previous memory blocks (including initial dictionary when present) must remain accessible and unmodified during compression.
Then, use LZ4_compress_HC_continue() to compress each successive block.
It works like LZ4_compress_HC(), but use previous memory blocks as dictionary to improve compression.
Previous memory blocks (including initial dictionary when present) must remain accessible and unmodified during compression.
As a reminder, size 'dst' buffer to handle worst cases, using LZ4_compressBound(), to ensure success of compression operation.
If, for any reason, previous data block can't be preserved in memory during next compression block,
you must save it to a safer memory space,
using LZ4_saveDictHC().
If, for any reason, previous data blocks can't be preserved unmodified in memory during next compression block,
you must save it to a safer memory space, using LZ4_saveDictHC().
Return value of LZ4_saveDictHC() is the size of dictionary effectively saved into 'safeBuffer'.
*/
/**************************************
* Deprecated Streaming Functions
* ************************************/
/* Note : these streaming functions follows the older model, and should no longer be used */
void* LZ4_createHC (const char* inputBuffer);
char* LZ4_slideInputBufferHC (void* LZ4HC_Data);
int LZ4_freeHC (void* LZ4HC_Data);
* Deprecated Functions
**************************************/
/* Deprecate Warnings */
/* Should these warnings messages be a problem,
it is generally possible to disable them,
with -Wno-deprecated-declarations for gcc
or _CRT_SECURE_NO_WARNINGS in Visual for example.
You can also define LZ4_DEPRECATE_WARNING_DEFBLOCK. */
#ifndef LZ4_DEPRECATE_WARNING_DEFBLOCK
# define LZ4_DEPRECATE_WARNING_DEFBLOCK
# define LZ4_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__)
# if (LZ4_GCC_VERSION >= 405) || defined(__clang__)
# define LZ4_DEPRECATED(message) __attribute__((deprecated(message)))
# elif (LZ4_GCC_VERSION >= 301)
# define LZ4_DEPRECATED(message) __attribute__((deprecated))
# elif defined(_MSC_VER)
# define LZ4_DEPRECATED(message) __declspec(deprecated(message))
# else
# pragma message("WARNING: You need to implement LZ4_DEPRECATED for this compiler")
# define LZ4_DEPRECATED(message)
# endif
#endif // LZ4_DEPRECATE_WARNING_DEFBLOCK
int LZ4_compressHC2_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int compressionLevel);
int LZ4_compressHC2_limitedOutput_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel);
/* compression functions */
/* these functions are planned to trigger warning messages by r131 approximately */
int LZ4_compressHC (const char* source, char* dest, int inputSize);
int LZ4_compressHC_limitedOutput (const char* source, char* dest, int inputSize, int maxOutputSize);
int LZ4_compressHC2 (const char* source, char* dest, int inputSize, int compressionLevel);
int LZ4_compressHC2_limitedOutput (const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel);
int LZ4_compressHC_withStateHC (void* state, const char* source, char* dest, int inputSize);
int LZ4_compressHC_limitedOutput_withStateHC (void* state, const char* source, char* dest, int inputSize, int maxOutputSize);
int LZ4_compressHC2_withStateHC (void* state, const char* source, char* dest, int inputSize, int compressionLevel);
int LZ4_compressHC2_limitedOutput_withStateHC(void* state, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel);
int LZ4_compressHC_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize);
int LZ4_compressHC_limitedOutput_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize, int maxOutputSize);
int LZ4_sizeofStreamStateHC(void);
int LZ4_resetStreamStateHC(void* state, const char* inputBuffer);
/* Streaming functions following the older model; should no longer be used */
LZ4_DEPRECATED("use LZ4_createStreamHC() instead") void* LZ4_createHC (char* inputBuffer);
LZ4_DEPRECATED("use LZ4_saveDictHC() instead") char* LZ4_slideInputBufferHC (void* LZ4HC_Data);
LZ4_DEPRECATED("use LZ4_freeStreamHC() instead") int LZ4_freeHC (void* LZ4HC_Data);
LZ4_DEPRECATED("use LZ4_compress_HC_continue() instead") int LZ4_compressHC2_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int compressionLevel);
LZ4_DEPRECATED("use LZ4_compress_HC_continue() instead") int LZ4_compressHC2_limitedOutput_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel);
LZ4_DEPRECATED("use LZ4_createStreamHC() instead") int LZ4_sizeofStreamStateHC(void);
LZ4_DEPRECATED("use LZ4_resetStreamHC() instead") int LZ4_resetStreamStateHC(void* state, char* inputBuffer);
#if defined (__cplusplus)

186
lib/xxhash.c
View File

@@ -28,15 +28,13 @@ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
You can contact the author at :
- xxHash source repository : http://code.google.com/p/xxhash/
- xxHash source mirror : https://github.com/Cyan4973/xxHash
- public discussion board : https://groups.google.com/forum/#!forum/lz4c
- xxHash source repository : https://github.com/Cyan4973/xxHash
*/
/**************************************
* Tuning parameters
***************************************/
**************************************/
/* Unaligned memory access is automatically enabled for "common" CPU, such as x86.
* For others CPU, the compiler will be more cautious, and insert extra code to ensure aligned access is respected.
* If you know your target CPU supports unaligned memory access, you want to force this option manually to improve performance.
@@ -94,10 +92,7 @@ static void* XXH_malloc(size_t s) { return malloc(s); }
static void XXH_free (void* p) { free(p); }
/* for memcpy() */
#include <string.h>
static void* XXH_memcpy(void* dest, const void* src, size_t size)
{
return memcpy(dest,src,size);
}
static void* XXH_memcpy(void* dest, const void* src, size_t size) { return memcpy(dest,src,size); }
/**************************************
@@ -105,51 +100,36 @@ static void* XXH_memcpy(void* dest, const void* src, size_t size)
***************************************/
#if defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */
# include <stdint.h>
typedef uint8_t BYTE;
typedef uint16_t U16;
typedef uint32_t U32;
typedef int32_t S32;
typedef uint64_t U64;
typedef uint8_t BYTE;
typedef uint16_t U16;
typedef uint32_t U32;
typedef int32_t S32;
typedef uint64_t U64;
#else
typedef unsigned char BYTE;
typedef unsigned short U16;
typedef unsigned int U32;
typedef signed int S32;
typedef unsigned long long U64;
typedef unsigned char BYTE;
typedef unsigned short U16;
typedef unsigned int U32;
typedef signed int S32;
typedef unsigned long long U64;
#endif
#if defined(__GNUC__) && !defined(XXH_USE_UNALIGNED_ACCESS)
# define _PACKED __attribute__ ((packed))
#else
# define _PACKED
#endif
#if !defined(XXH_USE_UNALIGNED_ACCESS) && !defined(__GNUC__)
# ifdef __IBMC__
# pragma pack(1)
# else
# pragma pack(push, 1)
# endif
#endif
typedef struct _U32_S
static U32 XXH_read32(const void* memPtr)
{
U32 v;
} _PACKED U32_S;
typedef struct _U64_S
U32 val32;
memcpy(&val32, memPtr, 4);
return val32;
}
static U64 XXH_read64(const void* memPtr)
{
U64 v;
} _PACKED U64_S;
#if !defined(XXH_USE_UNALIGNED_ACCESS) && !defined(__GNUC__)
# pragma pack(pop)
#endif
#define A32(x) (((U32_S *)(x))->v)
#define A64(x) (((U64_S *)(x))->v)
U64 val64;
memcpy(&val64, memPtr, 8);
return val64;
}
/*****************************************
/******************************************
* Compiler-specific Functions and Macros
******************************************/
#define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__)
@@ -191,7 +171,55 @@ static U64 XXH_swap64 (U64 x)
#endif
/**************************************
/***************************************
* Architecture Macros
***************************************/
typedef enum { XXH_bigEndian=0, XXH_littleEndian=1 } XXH_endianess;
#ifndef XXH_CPU_LITTLE_ENDIAN /* XXH_CPU_LITTLE_ENDIAN can be defined externally, for example using a compiler switch */
static const int one = 1;
# define XXH_CPU_LITTLE_ENDIAN (*(const char*)(&one))
#endif
/*****************************
* Memory reads
*****************************/
typedef enum { XXH_aligned, XXH_unaligned } XXH_alignment;
FORCE_INLINE U32 XXH_readLE32_align(const void* ptr, XXH_endianess endian, XXH_alignment align)
{
if (align==XXH_unaligned)
return endian==XXH_littleEndian ? XXH_read32(ptr) : XXH_swap32(XXH_read32(ptr));
else
return endian==XXH_littleEndian ? *(const U32*)ptr : XXH_swap32(*(const U32*)ptr);
}
FORCE_INLINE U32 XXH_readLE32(const void* ptr, XXH_endianess endian)
{
return XXH_readLE32_align(ptr, endian, XXH_unaligned);
}
FORCE_INLINE U64 XXH_readLE64_align(const void* ptr, XXH_endianess endian, XXH_alignment align)
{
if (align==XXH_unaligned)
return endian==XXH_littleEndian ? XXH_read64(ptr) : XXH_swap64(XXH_read64(ptr));
else
return endian==XXH_littleEndian ? *(const U64*)ptr : XXH_swap64(*(const U64*)ptr);
}
FORCE_INLINE U64 XXH_readLE64(const void* ptr, XXH_endianess endian)
{
return XXH_readLE64_align(ptr, endian, XXH_unaligned);
}
/***************************************
* Macros
***************************************/
#define XXH_STATIC_ASSERT(c) { enum { XXH_static_assert = 1/(!!(c)) }; } /* use only *after* variable declarations */
/***************************************
* Constants
***************************************/
#define PRIME32_1 2654435761U
@@ -207,55 +235,7 @@ static U64 XXH_swap64 (U64 x)
#define PRIME64_5 2870177450012600261ULL
/***************************************
* Architecture Macros
****************************************/
typedef enum { XXH_bigEndian=0, XXH_littleEndian=1 } XXH_endianess;
#ifndef XXH_CPU_LITTLE_ENDIAN /* XXH_CPU_LITTLE_ENDIAN can be defined externally, for example using a compiler switch */
static const int one = 1;
# define XXH_CPU_LITTLE_ENDIAN (*(char*)(&one))
#endif
/**************************************
* Macros
***************************************/
#define XXH_STATIC_ASSERT(c) { enum { XXH_static_assert = 1/(!!(c)) }; } /* use only *after* variable declarations */
/****************************
* Memory reads
*****************************/
typedef enum { XXH_aligned, XXH_unaligned } XXH_alignment;
FORCE_INLINE U32 XXH_readLE32_align(const void* ptr, XXH_endianess endian, XXH_alignment align)
{
if (align==XXH_unaligned)
return endian==XXH_littleEndian ? A32(ptr) : XXH_swap32(A32(ptr));
else
return endian==XXH_littleEndian ? *(U32*)ptr : XXH_swap32(*(U32*)ptr);
}
FORCE_INLINE U32 XXH_readLE32(const void* ptr, XXH_endianess endian)
{
return XXH_readLE32_align(ptr, endian, XXH_unaligned);
}
FORCE_INLINE U64 XXH_readLE64_align(const void* ptr, XXH_endianess endian, XXH_alignment align)
{
if (align==XXH_unaligned)
return endian==XXH_littleEndian ? A64(ptr) : XXH_swap64(A64(ptr));
else
return endian==XXH_littleEndian ? *(U64*)ptr : XXH_swap64(*(U64*)ptr);
}
FORCE_INLINE U64 XXH_readLE64(const void* ptr, XXH_endianess endian)
{
return XXH_readLE64_align(ptr, endian, XXH_unaligned);
}
/****************************
/*****************************
* Simple Hash Functions
*****************************/
FORCE_INLINE U32 XXH32_endian_align(const void* input, size_t len, U32 seed, XXH_endianess endian, XXH_alignment align)
@@ -335,7 +315,7 @@ FORCE_INLINE U32 XXH32_endian_align(const void* input, size_t len, U32 seed, XXH
}
unsigned int XXH32 (const void* input, size_t len, unsigned seed)
unsigned XXH32 (const void* input, size_t len, unsigned seed)
{
#if 0
/* Simple version, good for code maintenance, but unfortunately slow for small inputs */
@@ -347,7 +327,7 @@ unsigned int XXH32 (const void* input, size_t len, unsigned seed)
XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN;
# if !defined(XXH_USE_UNALIGNED_ACCESS)
if ((((size_t)input) & 3) == 0) /* Input is aligned, let's leverage the speed advantage */
if ((((size_t)input) & 3) == 0) /* Input is 4-bytes aligned, leverage the speed benefit */
{
if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT)
return XXH32_endian_align(input, len, seed, XXH_littleEndian, XXH_aligned);
@@ -504,7 +484,7 @@ unsigned long long XXH64 (const void* input, size_t len, unsigned long long seed
}
/****************************************************
* Advanced Hash Functions
* Advanced Hash Functions
****************************************************/
/*** Allocation ***/
@@ -688,9 +668,9 @@ XXH_errorcode XXH32_update (XXH32_state_t* state_in, const void* input, size_t l
FORCE_INLINE U32 XXH32_digest_endian (const XXH32_state_t* state_in, XXH_endianess endian)
{
XXH_istate32_t* state = (XXH_istate32_t*) state_in;
const XXH_istate32_t* state = (const XXH_istate32_t*) state_in;
const BYTE * p = (const BYTE*)state->mem32;
BYTE* bEnd = (BYTE*)(state->mem32) + state->memsize;
const BYTE* bEnd = (const BYTE*)(state->mem32) + state->memsize;
U32 h32;
if (state->total_len >= 16)
@@ -842,9 +822,9 @@ XXH_errorcode XXH64_update (XXH64_state_t* state_in, const void* input, size_t l
FORCE_INLINE U64 XXH64_digest_endian (const XXH64_state_t* state_in, XXH_endianess endian)
{
XXH_istate64_t * state = (XXH_istate64_t *) state_in;
const XXH_istate64_t * state = (const XXH_istate64_t *) state_in;
const BYTE * p = (const BYTE*)state->mem64;
BYTE* bEnd = (BYTE*)state->mem64 + state->memsize;
const BYTE* bEnd = (const BYTE*)state->mem64 + state->memsize;
U64 h64;
if (state->total_len >= 32)

23
lib/xxhash.h
View File

@@ -1,7 +1,8 @@
/*
xxHash - Extremely Fast Hash algorithm
Header File
Copyright (C) 2012-2014, Yann Collet.
Copyright (C) 2012-2015, Yann Collet.
BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
Redistribution and use in source and binary forms, with or without
@@ -28,7 +29,7 @@
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
You can contact the author at :
- xxHash source repository : http://code.google.com/p/xxhash/
- xxHash source repository : https://github.com/Cyan4973/xxHash
*/
/* Notice extracted from xxHash homepage :
@@ -55,6 +56,12 @@ SHA1-32 0.28 GB/s 10
Q.Score is a measure of quality of the hash function.
It depends on successfully passing SMHasher test set.
10 is a perfect score.
A 64-bits version, named XXH64, is available since r35.
It offers much better speed, but for 64-bits applications only.
Name Speed on 64 bits Speed on 32 bits
XXH64 13.8 GB/s 1.9 GB/s
XXH32 6.8 GB/s 6.0 GB/s
*/
#pragma once
@@ -65,20 +72,15 @@ extern "C" {
/*****************************
Includes
* Definitions
*****************************/
#include <stddef.h> /* size_t */
/*****************************
Type
*****************************/
typedef enum { XXH_OK=0, XXH_ERROR } XXH_errorcode;
/*****************************
Simple Hash Functions
* Simple Hash Functions
*****************************/
unsigned int XXH32 (const void* input, size_t length, unsigned seed);
@@ -93,12 +95,13 @@ XXH32() :
Speed on Core 2 Duo @ 3 GHz (single thread, SMHasher benchmark) : 5.4 GB/s
XXH64() :
Calculate the 64-bits hash of sequence of length "len" stored at memory address "input".
Faster on 64-bits systems. Slower on 32-bits systems.
*/
/*****************************
Advanced Hash Functions
* Advanced Hash Functions
*****************************/
typedef struct { long long ll[ 6]; } XXH32_state_t;
typedef struct { long long ll[11]; } XXH64_state_t;

77
lz4_block_format.txt → lz4_Block_format.md
View File

@@ -1,31 +1,33 @@
LZ4 Format Description
Last revised: 2012-02-27
Author : Y. Collet
LZ4 Block Format Description
============================
Last revised: 2015-05-07.
Author : Yann Collet
This small specification intents to provide enough information
to anyone willing to produce LZ4-compatible compressed data blocks
This specification is intended for developers
willing to produce LZ4-compatible compressed data blocks
using any programming language.
LZ4 is an LZ77-type compressor with a fixed, byte-oriented encoding.
The most important design principle behind LZ4 is simplicity.
It helps to create an easy to read and maintain source code.
It also helps later on for optimisations, compactness, and speed.
There is no entropy encoder backend nor framing layer.
The latter is assumed to be handled by other parts of the system.
There is no entropy encoder back-end nor framing layer.
The latter is assumed to be handled by other parts of the system (see [LZ4 Frame format]).
This design is assumed to favor simplicity and speed.
It helps later on for optimizations, compactness, and features.
This document only describes the format,
not how the LZ4 compressor nor decompressor actually work.
This document describes only the block format,
not how the compressor nor decompressor actually work.
The correctness of the decompressor should not depend
on implementation details of the compressor, and vice versa.
[LZ4 Frame format]: LZ4_Frame_format.md
-- Compressed block format --
Compressed block format
-----------------------
An LZ4 compressed block is composed of sequences.
Schematically, a sequence is a suite of literals, followed by a match copy.
A sequence is a suite of literals (not-compressed bytes),
followed by a match copy.
Each sequence starts with a token.
The token is a one byte value, separated into two 4-bits fields.
@@ -34,27 +36,30 @@ Therefore each field ranges from 0 to 15.
The first field uses the 4 high-bits of the token.
It provides the length of literals to follow.
(Note : a literal is a not-compressed byte).
If the field value is 0, then there is no literal.
If it is 15, then we need to add some more bytes to indicate the full length.
Each additionnal byte then represent a value from 0 to 255,
Each additional byte then represent a value from 0 to 255,
which is added to the previous value to produce a total length.
When the byte value is 255, another byte is output.
There can be any number of bytes following the token. There is no "size limit".
(Sidenote this is why a not-compressible input block is expanded by 0.4%).
(Side note : this is why a not-compressible input block is expanded by 0.4%).
Example 1 : A length of 48 will be represented as :
- 15 : value for the 4-bits High field
- 33 : (=48-15) remaining length to reach 48
- 15 : value for the 4-bits High field
- 33 : (=48-15) remaining length to reach 48
Example 2 : A length of 280 will be represented as :
- 15 : value for the 4-bits High field
- 255 : following byte is maxed, since 280-15 >= 255
- 10 : (=280 - 15 - 255) ) remaining length to reach 280
- 15 : value for the 4-bits High field
- 255 : following byte is maxed, since 280-15 >= 255
- 10 : (=280 - 15 - 255) ) remaining length to reach 280
Example 3 : A length of 15 will be represented as :
- 15 : value for the 4-bits High field
- 0 : (=15-15) yes, the zero must be output
- 15 : value for the 4-bits High field
- 0 : (=15-15) yes, the zero must be output
Following the token and optional length bytes, are the literals themselves.
They are exactly as numerous as previously decoded (length of literals).
@@ -64,7 +69,8 @@ It's possible that there are zero literal.
Following the literals is the match copy operation.
It starts by the offset.
This is a 2 bytes value, in little endian format.
This is a 2 bytes value, in little endian format
(the 1st byte is the "low" byte, the 2nd one is the "high" byte).
The offset represents the position of the match to be copied from.
1 means "current position - 1 byte".
@@ -82,7 +88,7 @@ we output additional bytes, one at a time, with values ranging from 0 to 255.
They are added to total to provide the final match length.
A 255 value means there is another byte to read and add.
There is no limit to the number of optional bytes that can be output this way.
(This points towards a maximum achievable compression ratio of ~250).
(This points towards a maximum achievable compression ratio of about 250).
With the offset and the matchlength,
the decoder can now proceed to copy the data from the already decoded buffer.
@@ -90,13 +96,15 @@ On decoding the matchlength, we reach the end of the compressed sequence,
and therefore start another one.
-- Parsing restrictions --
Parsing restrictions
-----------------------
There are specific parsing rules to respect in order to remain compatible
with assumptions made by the decoder :
1) The last 5 bytes are always literals
2) The last match must start at least 12 bytes before end of block
Consequently, a block with less than 13 bytes cannot be compressed.
1. The last 5 bytes are always literals
2. The last match must start at least 12 bytes before end of block.
Consequently, a block with less than 13 bytes cannot be compressed.
These rules are in place to ensure that the decoder
will never read beyond the input buffer, nor write beyond the output buffer.
@@ -104,8 +112,8 @@ Note that the last sequence is also incomplete,
and stops right after literals.
-- Additional notes --
Additional notes
-----------------------
There is no assumption nor limits to the way the compressor
searches and selects matches within the source data block.
It could be a fast scan, a multi-probe, a full search using BST,
@@ -117,4 +125,3 @@ or full optimal parsing.
All these trade-off offer distinctive speed/memory/compression advantages.
Whatever the method used by the compressor, its result will be decodable
by any LZ4 decoder if it follows the format specification described above.

385
lz4_Frame_format.md Normal file
View File

@@ -0,0 +1,385 @@
LZ4 Frame Format Description
============================
###Notices
Copyright (c) 2013-2015 Yann Collet
Permission is granted to copy and distribute this document
for any purpose and without charge,
including translations into other languages
and incorporation into compilations,
provided that the copyright notice and this notice are preserved,
and that any substantive changes or deletions from the original
are clearly marked.
Distribution of this document is unlimited.
###Version
1.5.1 (31/03/2015)
Introduction
------------
The purpose of this document is to define a lossless compressed data format,
that is independent of CPU type, operating system,
file system and character set, suitable for
File compression, Pipe and streaming compression
using the [LZ4 algorithm](http://www.lz4.info).
The data can be produced or consumed,
even for an arbitrarily long sequentially presented input data stream,
using only an a priori bounded amount of intermediate storage,
and hence can be used in data communications.
The format uses the LZ4 compression method,
and optional [xxHash-32 checksum method](https://github.com/Cyan4973/xxHash),
for detection of data corruption.
The data format defined by this specification
does not attempt to allow random access to compressed data.
This specification is intended for use by implementers of software
to compress data into LZ4 format and/or decompress data from LZ4 format.
The text of the specification assumes a basic background in programming
at the level of bits and other primitive data representations.
Unless otherwise indicated below,
a compliant compressor must produce data sets
that conform to the specifications presented here.
It doesn’t need to support all options though.
A compliant decompressor must be able to decompress
at least one working set of parameters
that conforms to the specifications presented here.
It may also ignore checksums.
Whenever it does not support a specific parameter within the compressed stream,
it must produce a non-ambiguous error code
and associated error message explaining which parameter is unsupported.
General Structure of LZ4 Frame format
-------------------------------------
| MagicNb | F. Descriptor | Block | (...) | EndMark | C. Checksum |
|:-------:|:-------------:| ----- | ----- | ------- | ----------- |
| 4 bytes | 3-11 bytes | | | 4 bytes | 4 bytes |
__Magic Number__
4 Bytes, Little endian format.
Value : 0x184D2204
__Frame Descriptor__
3 to 11 Bytes, to be detailed in the next part.
Most important part of the spec.
__Data Blocks__
To be detailed later on.
That’s where compressed data is stored.
__EndMark__
The flow of blocks ends when the last data block has a size of “0”.
The size is expressed as a 32-bits value.
__Content Checksum__
Content Checksum verify that the full content has been decoded correctly.
The content checksum is the result
of [xxh32() hash function](https://github.com/Cyan4973/xxHash)
digesting the original (decoded) data as input, and a seed of zero.
Content checksum is only present when its associated flag
is set in the frame descriptor.
Content Checksum validates the result,
that all blocks were fully transmitted in the correct order and without error,
and also that the encoding/decoding process itself generated no distortion.
Its usage is recommended.
__Frame Concatenation__
In some circumstances, it may be preferable to append multiple frames,
for example in order to add new data to an existing compressed file
without re-framing it.
In such case, each frame has its own set of descriptor flags.
Each frame is considered independent.
The only relation between frames is their sequential order.
The ability to decode multiple concatenated frames
within a single stream or file
is left outside of this specification.
As an example, the reference lz4 command line utility behavior is
to decode all concatenated frames in their sequential order.
Frame Descriptor
----------------
| FLG | BD | (Content Size) | HC |
| ------- | ------- |:--------------:| ------- |
| 1 byte | 1 byte | 0 - 8 bytes | 1 byte |
The descriptor uses a minimum of 3 bytes,
and up to 11 bytes depending on optional parameters.
__FLG byte__
| BitNb | 7-6 | 5 | 4 | 3 | 2 | 1-0 |
| ------- | ------- | ------- | --------- | ------- | --------- | -------- |
|FieldName| Version | B.Indep | B.Checksum| C.Size | C.Checksum|*Reserved*|
__BD byte__
| BitNb | 7 | 6-5-4 | 3-2-1-0 |
| ------- | -------- | ------------ | -------- |
|FieldName|*Reserved*| Block MaxSize|*Reserved*|
In the tables, bit 7 is highest bit, while bit 0 is lowest.
__Version Number__
2-bits field, must be set to “01”.
Any other value cannot be decoded by this version of the specification.
Other version numbers will use different flag layouts.
__Block Independence flag__
If this flag is set to “1”, blocks are independent.
If this flag is set to “0”, each block depends on previous ones
(up to LZ4 window size, which is 64 KB).
In such case, it’s necessary to decode all blocks in sequence.
Block dependency improves compression ratio, especially for small blocks.
On the other hand, it makes direct jumps or multi-threaded decoding impossible.
__Block checksum flag__
If this flag is set, each data block will be followed by a 4-bytes checksum,
calculated by using the xxHash-32 algorithm on the raw (compressed) data block.
The intention is to detect data corruption (storage or transmission errors)
immediately, before decoding.
Block checksum usage is optional.
__Content Size flag__
If this flag is set, the uncompressed size of data included within the frame
will be present as an 8 bytes unsigned little endian value, after the flags.
Content Size usage is optional.
__Content checksum flag__
If this flag is set, a content checksum will be appended after the EndMark.
Recommended value : “1” (content checksum is present)
__Block Maximum Size__
This information is intended to help the decoder allocate memory.
Size here refers to the original (uncompressed) data size.
Block Maximum Size is one value among the following table :
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| --- | --- | --- | --- | ----- | ------ | ---- | ---- |
| N/A | N/A | N/A | N/A | 64 KB | 256 KB | 1 MB | 4 MB |
The decoder may refuse to allocate block sizes above a (system-specific) size.
Unused values may be used in a future revision of the spec.
A decoder conformant to the current version of the spec
is only able to decode blocksizes defined in this spec.
__Reserved bits__
Value of reserved bits **must** be 0 (zero).
Reserved bit might be used in a future version of the specification,
typically enabling new optional features.
If this happens, a decoder respecting the current version of the specification
shall not be able to decode such a frame.
__Content Size__
This is the original (uncompressed) size.
This information is optional, and only present if the associated flag is set.
Content size is provided using unsigned 8 Bytes, for a maximum of 16 HexaBytes.
Format is Little endian.
This value is informational, typically for display or memory allocation.
It can be skipped by a decoder, or used to validate content correctness.
__Header Checksum__
One-byte checksum of combined descriptor fields, including optional ones.
The value is the second byte of xxh32() : ` (xxh32()>>8) & 0xFF `
using zero as a seed,
and the full Frame Descriptor as an input
(including optional fields when they are present).
A wrong checksum indicates an error in the descriptor.
Header checksum is informational and can be skipped.
Data Blocks
-----------
| Block Size | data | (Block Checksum) |
|:----------:| ------ |:----------------:|
| 4 bytes | | 0 - 4 bytes |
__Block Size__
This field uses 4-bytes, format is little-endian.
The highest bit is “1” if data in the block is uncompressed.
The highest bit is “0” if data in the block is compressed by LZ4.
All other bits give the size, in bytes, of the following data block
(the size does not include the block checksum if present).
Block Size shall never be larger than Block Maximum Size.
Such a thing could happen for incompressible source data.
In such case, such a data block shall be passed in uncompressed format.
__Data__
Where the actual data to decode stands.
It might be compressed or not, depending on previous field indications.
Uncompressed size of Data can be any size, up to “block maximum size”.
Note that data block is not necessarily full :
an arbitrary “flush” may happen anytime. Any block can be “partially filled”.
__Block checksum__
Only present if the associated flag is set.
This is a 4-bytes checksum value, in little endian format,
calculated by using the xxHash-32 algorithm on the raw (undecoded) data block,
and a seed of zero.
The intention is to detect data corruption (storage or transmission errors)
before decoding.
Block checksum is cumulative with Content checksum.
Skippable Frames
----------------
| Magic Number | Frame Size | User Data |
|:------------:|:----------:| --------- |
| 4 bytes | 4 bytes | |
Skippable frames allow the integration of user-defined data
into a flow of concatenated frames.
Its design is pretty straightforward,
with the sole objective to allow the decoder to quickly skip
over user-defined data and continue decoding.
For the purpose of facilitating identification,
it is discouraged to start a flow of concatenated frames with a skippable frame.
If there is a need to start such a flow with some user data
encapsulated into a skippable frame,
it’s recommended to start with a zero-byte LZ4 frame
followed by a skippable frame.
This will make it easier for file type identifiers.
__Magic Number__
4 Bytes, Little endian format.
Value : 0x184D2A5X, which means any value from 0x184D2A50 to 0x184D2A5F.
All 16 values are valid to identify a skippable frame.
__Frame Size__
This is the size, in bytes, of the following User Data
(without including the magic number nor the size field itself).
4 Bytes, Little endian format, unsigned 32-bits.
This means User Data can’t be bigger than (2^32-1) Bytes.
__User Data__
User Data can be anything. Data will just be skipped by the decoder.
Legacy frame
------------
The Legacy frame format was defined into the initial versions of “LZ4Demo”.
Newer compressors should not use this format anymore, as it is too restrictive.
Main characteristics of the legacy format :
- Fixed block size : 8 MB.
- All blocks must be completely filled, except the last one.
- All blocks are always compressed, even when compression is detrimental.
- The last block is detected either because
it is followed by the “EOF” (End of File) mark,
or because it is followed by a known Frame Magic Number.
- No checksum
- Convention is Little endian
| MagicNb | B.CSize | CData | B.CSize | CData | (...) | EndMark |
| ------- | ------- | ----- | ------- | ----- | ------- | ------- |
| 4 bytes | 4 bytes | CSize | 4 bytes | CSize | x times | EOF |
__Magic Number__
4 Bytes, Little endian format.
Value : 0x184C2102
__Block Compressed Size__
This is the size, in bytes, of the following compressed data block.
4 Bytes, Little endian format.
__Data__
Where the actual compressed data stands.
Data is always compressed, even when compression is detrimental.
__EndMark__
End of legacy frame is implicit only.
It must be followed by a standard EOF (End Of File) signal,
wether it is a file or a stream.
Alternatively, if the frame is followed by a valid Frame Magic Number,
it is considered completed.
It makes legacy frames compatible with frame concatenation.
Any other value will be interpreted as a block size,
and trigger an error if it does not fit within acceptable range.
Version changes
---------------
1.5.1 : changed format to MarkDown compatible
1.5 : removed Dictionary ID from specification
1.4.1 : changed wording from “stream” to “frame”
1.4 : added skippable streams, re-added stream checksum
1.3 : modified header checksum
1.2 : reduced choice of “block size”, to postpone decision on “dynamic size of BlockSize Field”.
1.1 : optional fields are now part of the descriptor
1.0 : changed “block size” specification, adding a compressed/uncompressed flag
0.9 : reduced scale of “block maximum size” table
0.8 : removed : high compression flag
0.7 : removed : stream checksum
0.6 : settled : stream size uses 8 bytes, endian convention is little endian
0.5: added copyright notice
0.4 : changed format to Google Doc compatible OpenDocument

219
programs/Makefile
View File

@@ -19,32 +19,32 @@
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
#
# You can contact the author at :
# - LZ4 source repository : http://code.google.com/p/lz4/
# - LZ4 forum froup : https://groups.google.com/forum/#!forum/lz4c
# - LZ4 source repository : https://github.com/Cyan4973/lz4
# - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c
# ##########################################################################
# lz4 : Command Line Utility, supporting gzip-like arguments
# lz4c : CLU, supporting also legacy lz4demo arguments
# lz4c32: Same as lz4c, but forced to compile in 32-bits mode
# fuzzer : Test tool, to check lz4 integrity on target platform
# fuzzer32: Same as fuzzer, but forced to compile in 32-bits mode
# frametest : Test tool, to check lz4frame integrity on target platform
# frametest32: Same as frametest, but forced to compile in 32-bits mode
# fullbench : Precisely measure speed for each LZ4 function variant
# fullbench32: Same as fullbench, but forced to compile in 32-bits mode
# datagen : generates synthetic data samples for tests & benchmarks
# ##########################################################################
RELEASE?= r126
RELEASE?= r130
DESTDIR?=
PREFIX ?= /usr
PREFIX ?= /usr/local
CFLAGS ?= -O3
CFLAGS += -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wstrict-prototypes -pedantic -DLZ4_VERSION=\"$(RELEASE)\"
FLAGS = -I../lib $(CPPFLAGS) $(CFLAGS) $(LDFLAGS)
CFLAGS += -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-qual -Wcast-align -Wstrict-prototypes -pedantic -DLZ4_VERSION=\"$(RELEASE)\"
FLAGS := -I../lib $(CPPFLAGS) $(CFLAGS) $(LDFLAGS)
BINDIR=$(PREFIX)/bin
MANDIR=$(PREFIX)/share/man/man1
LZ4DIR=../lib
TEST_FILES = COPYING
TEST_TARGETS=test-native
BINDIR := $(PREFIX)/bin
MANDIR := $(PREFIX)/share/man/man1
LZ4DIR := ../lib
# Define *.exe as extension for Windows systems
@@ -58,12 +58,19 @@ endif
# Select test target for Travis CI's Build Matrix
TRAVIS_TARGET=$(LZ4_TRAVIS_CI_ENV)
TRAVIS_TARGET:= $(LZ4_TRAVIS_CI_ENV)
TEST_FILES := COPYING
TEST_TARGETS := test-native
FUZZER_TIME := -T9mn
default: lz4 lz4c
default: lz4
all: lz4 lz4c lz4c32 fullbench fullbench32 fuzzer fuzzer32 frametest frametest32 datagen
m32: lz4c32 fullbench32 fuzzer32 frametest32
bins: lz4 lz4c fullbench fuzzer frametest datagen
all: bins m32
lz4: $(LZ4DIR)/lz4.c $(LZ4DIR)/lz4hc.c $(LZ4DIR)/lz4frame.c $(LZ4DIR)/xxhash.c bench.c lz4io.c lz4cli.c
$(CC) $(FLAGS) $^ -o $@$(EXT)
@@ -92,12 +99,11 @@ frametest: $(LZ4DIR)/lz4frame.c $(LZ4DIR)/lz4.c $(LZ4DIR)/lz4hc.c $(LZ4DIR)/xxha
frametest32: $(LZ4DIR)/lz4frame.c $(LZ4DIR)/lz4.c $(LZ4DIR)/lz4hc.c $(LZ4DIR)/xxhash.c frametest.c
$(CC) -m32 $(FLAGS) $^ -o $@$(EXT)
datagen : datagen.c
datagen : datagen.c datagencli.c
$(CC) $(FLAGS) $^ -o $@$(EXT)
clean:
@rm -f core *.o *.test \
@rm -f core *.o *.test tmp* \
lz4$(EXT) lz4c$(EXT) lz4c32$(EXT) \
fullbench$(EXT) fullbench32$(EXT) \
fuzzer$(EXT) fuzzer32$(EXT) \
@@ -113,22 +119,26 @@ ifneq (,$(filter $(shell uname),Linux Darwin GNU/kFreeBSD GNU))
install: lz4 lz4c
@echo Installing binaries
@install -d -m 755 $(DESTDIR)$(BINDIR)/ $(DESTDIR)$(MANDIR)/
@install -m 755 lz4$(EXT) $(DESTDIR)$(BINDIR)/lz4$(EXT)
@ln -sf lz4$(EXT) $(DESTDIR)$(BINDIR)/lz4cat
@install -m 755 lz4c$(EXT) $(DESTDIR)$(BINDIR)/lz4c$(EXT)
@install -m 755 lz4 $(DESTDIR)$(BINDIR)/lz4
@ln -sf lz4 $(DESTDIR)$(BINDIR)/lz4cat
@ln -sf lz4 $(DESTDIR)$(BINDIR)/unlz4
@install -m 755 lz4c $(DESTDIR)$(BINDIR)/lz4c
@echo Installing man pages
@install -m 644 lz4.1 $(DESTDIR)$(MANDIR)/lz4.1
@install -m 644 lz4c.1 $(DESTDIR)$(MANDIR)/lz4c.1
@install -m 644 lz4cat.1 $(DESTDIR)$(MANDIR)/lz4cat.1
@ln -sf lz4.1 $(DESTDIR)$(MANDIR)/lz4c.1
@ln -sf lz4.1 $(DESTDIR)$(MANDIR)/lz4cat.1
@ln -sf lz4.1 $(DESTDIR)$(MANDIR)/unlz4.1
@echo lz4 installation completed
uninstall:
rm -f $(DESTDIR)$(BINDIR)/lz4cat
[ -x $(DESTDIR)$(BINDIR)/lz4$(EXT) ] && rm -f $(DESTDIR)$(BINDIR)/lz4$(EXT)
[ -x $(DESTDIR)$(BINDIR)/lz4c$(EXT) ] && rm -f $(DESTDIR)$(BINDIR)/lz4c$(EXT)
rm -f $(DESTDIR)$(BINDIR)/unlz4
[ -x $(DESTDIR)$(BINDIR)/lz4 ] && rm -f $(DESTDIR)$(BINDIR)/lz4
[ -x $(DESTDIR)$(BINDIR)/lz4c ] && rm -f $(DESTDIR)$(BINDIR)/lz4c
[ -f $(DESTDIR)$(MANDIR)/lz4.1 ] && rm -f $(DESTDIR)$(MANDIR)/lz4.1
[ -f $(DESTDIR)$(MANDIR)/lz4c.1 ] && rm -f $(DESTDIR)$(MANDIR)/lz4c.1
[ -f $(DESTDIR)$(MANDIR)/lz4cat.1 ] && rm -f $(DESTDIR)$(MANDIR)/lz4cat.1
rm -f $(DESTDIR)$(MANDIR)/lz4c.1
rm -f $(DESTDIR)$(MANDIR)/lz4cat.1
rm -f $(DESTDIR)$(MANDIR)/unlz4.1
@echo lz4 programs successfully uninstalled
test: test-lz4 test-lz4c test-frametest test-fullbench test-fuzzer test-mem
@@ -139,13 +149,56 @@ test-all: test test32
test-travis: $(TRAVIS_TARGET)
test-lz4: lz4 datagen
./datagen -g16KB | ./lz4 -9 | ./lz4 -vdq > $(VOID)
./datagen | ./lz4 | ./lz4 -vdq > $(VOID)
./datagen -g6M -p100 | ./lz4 -9BD | ./lz4 -vdq > $(VOID)
./datagen -g256MB | ./lz4 -vqB4D | ./lz4 -vdq > $(VOID)
./datagen -g6GB | ./lz4 -vqB5D | ./lz4 -vdq > $(VOID)
# test frame concatenation with null-length frame
test-lz4-sparse: lz4 datagen
@echo "\n ---- test sparse file support ----"
./datagen -g5M -P100 > tmpSrc
./lz4 -B4D tmpSrc | ./lz4 -dv --sparse > tmpB4
diff -s tmpSrc tmpB4
./lz4 -B5D tmpSrc | ./lz4 -dv --sparse > tmpB5
diff -s tmpSrc tmpB5
./lz4 -B6D tmpSrc | ./lz4 -dv --sparse > tmpB6
diff -s tmpSrc tmpB6
./lz4 -B7D tmpSrc | ./lz4 -dv --sparse > tmpB7
diff -s tmpSrc tmpB7
./lz4 tmpSrc | ./lz4 -dv --no-sparse > tmpNoSparse
diff -s tmpSrc tmpNoSparse
ls -ls tmp*
./datagen -s1 -g1200007 -P100 | ./lz4 | ./lz4 -dv --sparse > tmpOdd # Odd size file (to not finish on an exact nb of blocks)
./datagen -s1 -g1200007 -P100 | diff -s - tmpOdd
ls -ls tmpOdd
@rm tmp*
@echo "\n Compatibility with Console :"
echo "Hello World 1 !" | ./lz4 | ./lz4 -d -c
echo "Hello World 2 !" | ./lz4 | ./lz4 -d | cat
echo "Hello World 3 !" | ./lz4 --no-frame-crc | ./lz4 -d -c
@echo "\n Compatibility with Append :"
./datagen -P100 -g1M > tmp1M
cat tmp1M > tmp2M
cat tmp1M >> tmp2M
./lz4 -B5 -v tmp1M tmpC
./lz4 -d -v tmpC tmpR
./lz4 -d -v tmpC >> tmpR
ls -ls tmp*
diff tmp2M tmpR
@rm tmp*
test-lz4-contentSize: lz4 datagen
@echo "\n ---- test original size support ----"
./datagen -g15M > tmp
./lz4 -v tmp | ./lz4 -t
./lz4 -v --content-size tmp | ./lz4 -d > tmp2
diff -s tmp tmp2
# test large size [2-4] GB
@./datagen -g3G -P100 | ./lz4 | ./lz4 --decompress --force --sparse - tmp
@ls -ls tmp
./lz4 --quiet --content-size tmp | ./lz4 --verbose --decompress --force --sparse - tmp2
@ls -ls tmp2
@rm tmp*
test-lz4-frame-concatenation: lz4 datagen
@echo "\n ---- test frame concatenation ----"
@echo -n > empty.test
@echo hi > nonempty.test
cat nonempty.test empty.test nonempty.test > orig.test
@@ -156,49 +209,101 @@ test-lz4: lz4 datagen
sdiff orig.test result.test
@rm *.test
@echo frame concatenation test completed
# test frame concatenation with null-length frame
test-lz4-multiple: lz4 datagen
@echo "\n ---- test multiple files ----"
@./datagen -s1 > tmp1 2> $(VOID)
@./datagen -s2 -g100K > tmp2 2> $(VOID)
@./datagen -s3 -g1M > tmp3 2> $(VOID)
./lz4 -f -m tmp*
ls -ls tmp*
rm tmp1 tmp2 tmp3
./lz4 -df -m *.lz4
ls -ls tmp*
./lz4 -f -m tmp1 notHere tmp2; echo $$?
@rm tmp*
test-lz4-basic: lz4 datagen
@echo "\n ---- test lz4 basic compression/decompression ----"
./datagen -g0 | ./lz4 -v | ./lz4 -t
./datagen -g16KB | ./lz4 -9 | ./lz4 -t
./datagen -g20KB > tmpSrc
./lz4 < tmpSrc | ./lz4 -d > tmpRes
diff -q tmpSrc tmpRes
./lz4 --no-frame-crc < tmpSrc | ./lz4 -d > tmpRes
diff -q tmpSrc tmpRes
./datagen | ./lz4 | ./lz4 -t
./datagen -g6M -P99 | ./lz4 -9BD | ./lz4 -t
./datagen -g17M | ./lz4 -9v | ./lz4 -qt
./datagen -g33M | ./lz4 --no-frame-crc | ./lz4 -t
./datagen -g256MB | ./lz4 -vqB4D | ./lz4 -t
./datagen -g6GB | ./lz4 -vqB5D | ./lz4 -qt
./datagen -g6GB | ./lz4 -vq9BD | ./lz4 -qt
@rm tmp*
test-lz4: lz4 datagen test-lz4-basic test-lz4-multiple test-lz4-sparse test-lz4-contentSize test-lz4-frame-concatenation
@echo "\n ---- test pass-through ----"
./datagen | ./lz4 -tf
test-lz4c: lz4c datagen
./datagen -g256MB | ./lz4c -l -v | ./lz4c -vdq > $(VOID)
@echo "\n ---- test lz4c version ----"
./datagen -g256MB | ./lz4c -l -v | ./lz4c -t
test-lz4c32: lz4 lz4c32 lz4 datagen
./datagen -g16KB | ./lz4c32 -9 | ./lz4c32 -vdq > $(VOID)
./datagen -g16KB | ./lz4c32 -9 | ./lz4 -vdq > $(VOID)
./datagen | ./lz4c32 | ./lz4c32 -vdq > $(VOID)
./datagen | ./lz4c32 | ./lz4 -vdq > $(VOID)
./datagen -g256MB | ./lz4c32 -vqB4D | ./lz4c32 -vdq > $(VOID)
./datagen -g256MB | ./lz4c32 -vqB4D | ./lz4 -vdq > $(VOID)
./datagen -g6GB | ./lz4c32 -vqB5D | ./lz4c32 -vdq > $(VOID)
test-interop-32-64: lz4 lz4c32 datagen
@echo "\n ---- test interoperability 32-bits -vs- 64 bits ----"
./datagen -g16KB | ./lz4c32 -9 | ./lz4 -t
./datagen -P10 | ./lz4 -9B4 | ./lz4c32 -t
./datagen | ./lz4c32 | ./lz4 -t
./datagen -g1M | ./lz4 -3B5 | ./lz4c32 -t
./datagen -g256MB | ./lz4c32 -vqB4D | ./lz4 -qt
./datagen -g1G -P90 | ./lz4 | ./lz4c32 -t
./datagen -g6GB | ./lz4c32 -vq9BD | ./lz4 -qt
test-lz4c32-basic: lz4c32 datagen
@echo "\n ---- test lz4c32 32-bits version ----"
./datagen -g16KB | ./lz4c32 -9 | ./lz4c32 -t
./datagen | ./lz4c32 | ./lz4c32 -t
./datagen -g256MB | ./lz4c32 -vqB4D | ./lz4c32 -qt
./datagen -g6GB | ./lz4c32 -vqB5D | ./lz4c32 -qt
test-lz4c32: test-lz4c32-basic test-interop-32-64
test-fullbench: fullbench
./fullbench --no-prompt $(TEST_FILES)
./fullbench --no-prompt $(NB_LOOPS) $(TEST_FILES)
test-fullbench32: fullbench32
./fullbench32 --no-prompt $(TEST_FILES)
./fullbench32 --no-prompt $(NB_LOOPS) $(TEST_FILES)
test-fuzzer: fuzzer
./fuzzer
./fuzzer $(FUZZER_TIME)
test-fuzzer32: fuzzer32
./fuzzer32
./fuzzer32 $(FUZZER_TIME)
test-frametest: frametest
./frametest
./frametest $(FUZZER_TIME)
test-frametest32: frametest32
./frametest32
./frametest32 $(FUZZER_TIME)
test-mem: lz4 datagen fuzzer frametest
test-mem: lz4 datagen fuzzer frametest fullbench
@echo "\n ---- valgrind tests : memory analyzer ----"
valgrind --leak-check=yes --error-exitcode=1 ./datagen -g50M > $(VOID)
./datagen -g16KB > tmp
valgrind --leak-check=yes ./lz4 -9 -BD -f tmp /dev/null
valgrind --leak-check=yes --error-exitcode=1 ./lz4 -9 -BD -f tmp $(VOID)
./datagen -g16KB -s2 > tmp2
./datagen -g16KB -s3 > tmp3
valgrind --leak-check=yes --error-exitcode=1 ./lz4 --force --multiple tmp tmp2 tmp3
./datagen -g16MB > tmp
valgrind --leak-check=yes ./lz4 -9 -B5D -f tmp /dev/null
valgrind --leak-check=yes --error-exitcode=1 ./lz4 -9 -B5D -f tmp tmp2
valgrind --leak-check=yes --error-exitcode=1 ./lz4 -t tmp2
valgrind --leak-check=yes --error-exitcode=1 ./lz4 -bi1 tmp
valgrind --leak-check=yes --error-exitcode=1 ./fullbench -i1 tmp tmp2
./datagen -g256MB > tmp
valgrind --leak-check=yes ./lz4 -B4D -f -vq tmp /dev/null
rm tmp
valgrind --leak-check=yes ./fuzzer -i64 -t1
valgrind --leak-check=yes ./frametest -i256
valgrind --leak-check=yes --error-exitcode=1 ./lz4 -B4D -f -vq tmp $(VOID)
rm tmp*
valgrind --leak-check=yes --error-exitcode=1 ./fuzzer -i64 -t1
valgrind --leak-check=yes --error-exitcode=1 ./frametest -i256
test-mem32: lz4c32 datagen
# unfortunately, valgrind doesn't seem to work with non-native binary. If someone knows how to do a valgrind-test on a 32-bits exe with a 64-bits system...

78
programs/bench.c
View File

@@ -1,6 +1,7 @@
/*
bench.c - Demo program to benchmark open-source compression algorithm
Copyright (C) Yann Collet 2012-2014
bench.c - Demo program to benchmark open-source compression algorithms
Copyright (C) Yann Collet 2012-2015
GPL v2 License
This program is free software; you can redistribute it and/or modify
@@ -18,16 +19,18 @@
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
You can contact the author at :
- LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
- LZ4 source repository : http://code.google.com/p/lz4/
- LZ4 source repository : https://github.com/Cyan4973/lz4
- LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c
*/
/**************************************
* Compiler Options
***************************************/
/* Disable some Visual warning messages */
#define _CRT_SECURE_NO_WARNINGS
#define _CRT_SECURE_NO_DEPRECATE /* VS2005 */
#if defined(_MSC_VER) || defined(_WIN32)
# define _CRT_SECURE_NO_WARNINGS
# define _CRT_SECURE_NO_DEPRECATE /* VS2005 */
# define BMK_LEGACY_TIMER 1 /* S_ISREG & gettimeofday() are not supported by MSVC */
#endif
/* Unix Large Files support (>4GB) */
#define _FILE_OFFSET_BITS 64
@@ -37,11 +40,6 @@
# define _LARGEFILE64_SOURCE
#endif
/* S_ISREG & gettimeofday() are not supported by MSVC */
#if defined(_MSC_VER) || defined(_WIN32)
# define BMK_LEGACY_TIMER 1
#endif
/**************************************
* Includes
@@ -60,9 +58,9 @@
#include "lz4.h"
#define COMPRESSOR0 LZ4_compress_local
static int LZ4_compress_local(const char* src, char* dst, int size, int clevel) { (void)clevel; return LZ4_compress(src, dst, size); }
static int LZ4_compress_local(const char* src, char* dst, int srcSize, int dstSize, int clevel) { (void)clevel; return LZ4_compress_default(src, dst, srcSize, dstSize); }
#include "lz4hc.h"
#define COMPRESSOR1 LZ4_compressHC2
#define COMPRESSOR1 LZ4_compress_HC
#define DEFAULTCOMPRESSOR COMPRESSOR0
#include "xxhash.h"
@@ -123,8 +121,8 @@ struct chunkParameters
struct compressionParameters
{
int (*compressionFunction)(const char*, char*, int, int);
int (*decompressionFunction)(const char*, char*, int);
int (*compressionFunction)(const char* src, char* dst, int srcSize, int dstSize, int cLevel);
int (*decompressionFunction)(const char* src, char* dst, int dstSize);
};
@@ -141,15 +139,15 @@ static int chunkSize = DEFAULT_CHUNKSIZE;
static int nbIterations = NBLOOPS;
static int BMK_pause = 0;
void BMK_SetBlocksize(int bsize) { chunkSize = bsize; }
void BMK_setBlocksize(int bsize) { chunkSize = bsize; }
void BMK_SetNbIterations(int nbLoops)
void BMK_setNbIterations(int nbLoops)
{
nbIterations = nbLoops;
DISPLAY("- %i iterations -\n", nbIterations);
}
void BMK_SetPause(void) { BMK_pause = 1; }
void BMK_setPause(void) { BMK_pause = 1; }
/*********************************************************
@@ -206,16 +204,21 @@ static size_t BMK_findMaxMem(U64 requiredMem)
while (!testmem)
{
requiredMem -= step;
if (requiredMem > step) requiredMem -= step;
else requiredMem >>= 1;
testmem = (BYTE*) malloc ((size_t)requiredMem);
}
free (testmem);
return (size_t) (requiredMem - step);
/* keep some space available */
if (requiredMem > step) requiredMem -= step;
else requiredMem >>= 1;
return (size_t)requiredMem;
}
static U64 BMK_GetFileSize(char* infilename)
static U64 BMK_GetFileSize(const char* infilename)
{
int r;
#if defined(_MSC_VER)
@@ -234,7 +237,7 @@ static U64 BMK_GetFileSize(char* infilename)
* Public function
**********************************************************/
int BMK_benchFile(char** fileNamesTable, int nbFiles, int cLevel)
int BMK_benchFiles(const char** fileNamesTable, int nbFiles, int cLevel)
{
int fileIdx=0;
char* orig_buff;
@@ -265,7 +268,7 @@ int BMK_benchFile(char** fileNamesTable, int nbFiles, int cLevel)
while (fileIdx<nbFiles)
{
FILE* inFile;
char* inFileName;
const char* inFileName;
U64 inFileSize;
size_t benchedSize;
int nbChunks;
@@ -278,15 +281,13 @@ int BMK_benchFile(char** fileNamesTable, int nbFiles, int cLevel)
/* Check file existence */
inFileName = fileNamesTable[fileIdx++];
inFile = fopen( inFileName, "rb" );
if (inFile==NULL)
{
DISPLAY( "Pb opening %s\n", inFileName);
return 11;
}
if (inFile==NULL) { DISPLAY( "Pb opening %s\n", inFileName); return 11; }
/* Memory allocation & restrictions */
inFileSize = BMK_GetFileSize(inFileName);
if (inFileSize==0) { DISPLAY( "file is empty\n"); fclose(inFile); return 11; }
benchedSize = (size_t) BMK_findMaxMem(inFileSize * 2) / 2;
if (benchedSize==0) { DISPLAY( "not enough memory\n"); fclose(inFile); return 11; }
if ((U64)benchedSize > inFileSize) benchedSize = (size_t)inFileSize;
if (benchedSize < inFileSize)
{
@@ -295,12 +296,11 @@ int BMK_benchFile(char** fileNamesTable, int nbFiles, int cLevel)
/* Alloc */
chunkP = (struct chunkParameters*) malloc(((benchedSize / (size_t)chunkSize)+1) * sizeof(struct chunkParameters));
orig_buff = (char*)malloc((size_t )benchedSize);
orig_buff = (char*)malloc((size_t)benchedSize);
nbChunks = (int) ((int)benchedSize / chunkSize) + 1;
maxCompressedChunkSize = LZ4_compressBound(chunkSize);
compressedBuffSize = nbChunks * maxCompressedChunkSize;
compressedBuffer = (char*)malloc((size_t )compressedBuffSize);
compressedBuffer = (char*)malloc((size_t)compressedBuffSize);
if (!orig_buff || !compressedBuffer)
{
@@ -371,11 +371,12 @@ int BMK_benchFile(char** fileNamesTable, int nbFiles, int cLevel)
while(BMK_GetMilliSpan(milliTime) < TIMELOOP)
{
for (chunkNb=0; chunkNb<nbChunks; chunkNb++)
chunkP[chunkNb].compressedSize = compP.compressionFunction(chunkP[chunkNb].origBuffer, chunkP[chunkNb].compressedBuffer, chunkP[chunkNb].origSize, cLevel);
chunkP[chunkNb].compressedSize = compP.compressionFunction(chunkP[chunkNb].origBuffer, chunkP[chunkNb].compressedBuffer, chunkP[chunkNb].origSize, maxCompressedChunkSize, cLevel);
nbLoops++;
}
milliTime = BMK_GetMilliSpan(milliTime);
nbLoops += !nbLoops; /* avoid division by zero */
if ((double)milliTime < fastestC*nbLoops) fastestC = (double)milliTime/nbLoops;
cSize=0; for (chunkNb=0; chunkNb<nbChunks; chunkNb++) cSize += chunkP[chunkNb].compressedSize;
ratio = (double)cSize/(double)benchedSize*100.;
@@ -397,8 +398,9 @@ int BMK_benchFile(char** fileNamesTable, int nbFiles, int cLevel)
}
milliTime = BMK_GetMilliSpan(milliTime);
nbLoops += !nbLoops; /* avoid division by zero */
if ((double)milliTime < fastestD*nbLoops) fastestD = (double)milliTime/nbLoops;
DISPLAY("%1i-%-14.14s : %9i -> %9i (%5.2f%%),%7.1f MB/s ,%7.1f MB/s\r", loopNb, inFileName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / fastestC / 1000., (double)benchedSize / fastestD / 1000.);
DISPLAY("%1i-%-14.14s : %9i -> %9i (%5.2f%%),%7.1f MB/s ,%7.1f MB/s \r", loopNb, inFileName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / fastestC / 1000., (double)benchedSize / fastestD / 1000.);
/* CRC Checking */
crcCheck = XXH32(orig_buff, (unsigned int)benchedSize,0);
@@ -408,9 +410,9 @@ int BMK_benchFile(char** fileNamesTable, int nbFiles, int cLevel)
if (crcOrig==crcCheck)
{
if (ratio<100.)
DISPLAY("%-16.16s : %9i -> %9i (%5.2f%%),%7.1f MB/s ,%7.1f MB/s\n", inFileName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / fastestC / 1000., (double)benchedSize / fastestD / 1000.);
DISPLAY("%-16.16s : %9i -> %9i (%5.2f%%),%7.1f MB/s ,%7.1f MB/s \n", inFileName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / fastestC / 1000., (double)benchedSize / fastestD / 1000.);
else
DISPLAY("%-16.16s : %9i -> %9i (%5.1f%%),%7.1f MB/s ,%7.1f MB/s \n", inFileName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / fastestC / 1000., (double)benchedSize / fastestD / 1000.);
DISPLAY("%-16.16s : %9i -> %9i (%5.1f%%),%7.1f MB/s ,%7.1f MB/s \n", inFileName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / fastestC / 1000., (double)benchedSize / fastestD / 1000.);
}
totals += benchedSize;
totalz += cSize;
@@ -426,7 +428,7 @@ int BMK_benchFile(char** fileNamesTable, int nbFiles, int cLevel)
if (nbFiles > 1)
DISPLAY("%-16.16s :%10llu ->%10llu (%5.2f%%), %6.1f MB/s , %6.1f MB/s\n", " TOTAL", (long long unsigned int)totals, (long long unsigned int)totalz, (double)totalz/(double)totals*100., (double)totals/totalc/1000., (double)totals/totald/1000.);
if (BMK_pause) { DISPLAY("\npress enter...\n"); getchar(); }
if (BMK_pause) { DISPLAY("\npress enter...\n"); (void)getchar(); }
return 0;
}

23
programs/bench.h
View File

@@ -1,6 +1,6 @@
/*
bench.h - Demo program to benchmark open-source compression algorithm
Copyright (C) Yann Collet 2012-2014
Copyright (C) Yann Collet 2012-2015
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
@@ -17,26 +17,17 @@
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
You can contact the author at :
- LZ4 source repository : http://code.google.com/p/lz4/
- LZ4 public forum : https://group.google.com/forum/#!forum/lz4c
- LZ4 source repository : https://github.com/Cyan4973/lz4
- LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c
*/
#pragma once
#if defined (__cplusplus)
extern "C" {
#endif
/* Main function */
int BMK_benchFile(char** fileNamesTable, int nbFiles, int cLevel);
int BMK_benchFiles(const char** fileNamesTable, int nbFiles, int cLevel);
/* Set Parameters */
void BMK_SetBlocksize(int bsize);
void BMK_SetNbIterations(int nbLoops);
void BMK_SetPause(void);
void BMK_setBlocksize(int bsize);
void BMK_setNbIterations(int nbLoops);
void BMK_setPause(void);
#if defined (__cplusplus)
}
#endif

347
programs/datagen.c
View File

@@ -19,26 +19,20 @@
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
You can contact the author at :
- LZ4 source repository : http://code.google.com/p/lz4
- LZ4 source mirror : https://github.com/Cyan4973/lz4
- LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c
- ZSTD source repository : https://github.com/Cyan4973/zstd
- Public forum : https://groups.google.com/forum/#!forum/lz4c
*/
/**************************************
Remove Visual warning messages
* Includes
**************************************/
#define _CRT_SECURE_NO_WARNINGS // fgets
#include <stdlib.h> /* malloc */
#include <stdio.h> /* FILE, fwrite */
#include <string.h> /* memcpy */
/**************************************
Includes
**************************************/
#include <stdio.h> // fgets, sscanf
#include <string.h> // strcmp
/**************************************
Basic Types
* Basic Types
**************************************/
#if defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */
# include <stdint.h>
@@ -57,230 +51,173 @@
/**************************************
Constants
* OS-specific Includes
**************************************/
#ifndef LZ4_VERSION
# define LZ4_VERSION "r125"
#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(_WIN32) || defined(__CYGWIN__)
# include <fcntl.h> /* _O_BINARY */
# include <io.h> /* _setmode, _isatty */
# define SET_BINARY_MODE(file) _setmode(_fileno(file), _O_BINARY)
#else
# define SET_BINARY_MODE(file)
#endif
#define KB *(1 <<10)
#define MB *(1 <<20)
#define GB *(1U<<30)
#define CDG_SIZE_DEFAULT (64 KB)
#define CDG_SEED_DEFAULT 0
#define CDG_COMPRESSIBILITY_DEFAULT 50
/**************************************
* Constants
**************************************/
#define KB *(1 <<10)
#define PRIME1 2654435761U
#define PRIME2 2246822519U
/**************************************
Macros
**************************************/
#define DISPLAY(...) fprintf(stderr, __VA_ARGS__)
#define DISPLAYLEVEL(l, ...) if (displayLevel>=l) { DISPLAY(__VA_ARGS__); }
/**************************************
Local Parameters
**************************************/
static unsigned no_prompt = 0;
static char* programName;
static unsigned displayLevel = 2;
/*********************************************************
functions
* Local Functions
*********************************************************/
#define CDG_rotl32(x,r) ((x << r) | (x >> (32 - r)))
static unsigned int CDG_rand(U32* src)
#define RDG_rotl32(x,r) ((x << r) | (x >> (32 - r)))
static unsigned int RDG_rand(U32* src)
{
U32 rand32 = *src;
rand32 *= PRIME1;
rand32 += PRIME2;
rand32 = CDG_rotl32(rand32, 13);
rand32 ^= PRIME2;
rand32 = RDG_rotl32(rand32, 13);
*src = rand32;
return rand32;
}
#define CDG_RAND15BITS ((CDG_rand(seed) >> 3) & 32767)
#define CDG_RANDLENGTH ( ((CDG_rand(seed) >> 7) & 3) ? (CDG_rand(seed) % 14) : (CDG_rand(seed) & 511) + 15)
#define CDG_RANDCHAR (((CDG_rand(seed) >> 9) & 63) + '0')
static void CDG_generate(U64 size, U32* seed, double proba)
#define LTSIZE 8192
#define LTMASK (LTSIZE-1)
static void* RDG_createLiteralDistrib(double ld)
{
BYTE fullbuff[32 KB + 128 KB + 1];
BYTE* buff = fullbuff + 32 KB;
U64 total=0;
U32 P32 = (U32)(32768 * proba);
U32 pos=1;
U32 genBlockSize = 128 KB;
BYTE* lt = (BYTE*)malloc(LTSIZE);
U32 i = 0;
BYTE character = '0';
BYTE firstChar = '(';
BYTE lastChar = '}';
// Build initial prefix
fullbuff[0] = CDG_RANDCHAR;
while (pos<32 KB)
if (ld==0.0)
{
// Select : Literal (char) or Match (within 32K)
if (CDG_RAND15BITS < P32)
character = 0;
firstChar = 0;
lastChar =255;
}
while (i<LTSIZE)
{
U32 weight = (U32)((double)(LTSIZE - i) * ld) + 1;
U32 end;
if (weight + i > LTSIZE) weight = LTSIZE-i;
end = i + weight;
while (i < end) lt[i++] = character;
character++;
if (character > lastChar) character = firstChar;
}
return lt;
}
static char RDG_genChar(U32* seed, const void* ltctx)
{
const BYTE* lt = (const BYTE*)ltctx;
U32 id = RDG_rand(seed) & LTMASK;
return lt[id];
}
#define RDG_DICTSIZE (32 KB)
#define RDG_RAND15BITS ((RDG_rand(seed) >> 3) & 32767)
#define RDG_RANDLENGTH ( ((RDG_rand(seed) >> 7) & 7) ? (RDG_rand(seed) & 15) : (RDG_rand(seed) & 511) + 15)
void RDG_genBlock(void* buffer, size_t buffSize, size_t prefixSize, double matchProba, void* litTable, unsigned* seedPtr)
{
BYTE* buffPtr = (BYTE*)buffer;
const U32 matchProba32 = (U32)(32768 * matchProba);
size_t pos = prefixSize;
void* ldctx = litTable;
U32* seed = seedPtr;
/* special case */
while (matchProba >= 1.0)
{
size_t size0 = RDG_rand(seed) & 3;
size0 = (size_t)1 << (16 + size0 * 2);
size0 += RDG_rand(seed) & (size0-1); /* because size0 is power of 2*/
if (buffSize < pos + size0)
{
// Copy (within 64K)
U32 d;
int ref;
int length = CDG_RANDLENGTH + 4;
U32 offset = CDG_RAND15BITS + 1;
if (offset > pos) offset = pos;
ref = pos - offset;
memset(buffPtr+pos, 0, buffSize-pos);
return;
}
memset(buffPtr+pos, 0, size0);
pos += size0;
buffPtr[pos-1] = RDG_genChar(seed, ldctx);
}
/* init */
if (pos==0) buffPtr[0] = RDG_genChar(seed, ldctx), pos=1;
/* Generate compressible data */
while (pos < buffSize)
{
/* Select : Literal (char) or Match (within 32K) */
if (RDG_RAND15BITS < matchProba32)
{
/* Copy (within 32K) */
size_t match;
size_t d;
int length = RDG_RANDLENGTH + 4;
U32 offset = RDG_RAND15BITS + 1;
if (offset > pos) offset = (U32)pos;
match = pos - offset;
d = pos + length;
while (pos < d) fullbuff[pos++] = fullbuff[ref++];
if (d > buffSize) d = buffSize;
while (pos < d) buffPtr[pos++] = buffPtr[match++];
}
else
{
// Literal (noise)
U32 d = pos + CDG_RANDLENGTH;
while (pos < d) fullbuff[pos++] = CDG_RANDCHAR;
/* Literal (noise) */
size_t d;
size_t length = RDG_RANDLENGTH;
d = pos + length;
if (d > buffSize) d = buffSize;
while (pos < d) buffPtr[pos++] = RDG_genChar(seed, ldctx);
}
}
}
// Generate compressible data
pos = 0;
void RDG_genBuffer(void* buffer, size_t size, double matchProba, double litProba, unsigned seed)
{
void* ldctx;
if (litProba==0.0) litProba = matchProba / 4.5;
ldctx = RDG_createLiteralDistrib(litProba);
RDG_genBlock(buffer, size, 0, matchProba, ldctx, &seed);
free(ldctx);
}
#define RDG_BLOCKSIZE (128 KB)
void RDG_genOut(unsigned long long size, double matchProba, double litProba, unsigned seed)
{
BYTE buff[RDG_DICTSIZE + RDG_BLOCKSIZE];
U64 total = 0;
size_t genBlockSize = RDG_BLOCKSIZE;
void* ldctx;
/* init */
if (litProba==0.0) litProba = matchProba / 4.5;
ldctx = RDG_createLiteralDistrib(litProba);
SET_BINARY_MODE(stdout);
/* Generate dict */
RDG_genBlock(buff, RDG_DICTSIZE, 0, matchProba, ldctx, &seed);
/* Generate compressible data */
while (total < size)
{
if (size-total < 128 KB) genBlockSize = (U32)(size-total);
RDG_genBlock(buff, RDG_DICTSIZE+RDG_BLOCKSIZE, RDG_DICTSIZE, matchProba, ldctx, &seed);
if (size-total < RDG_BLOCKSIZE) genBlockSize = (size_t)(size-total);
total += genBlockSize;
buff[genBlockSize] = 0;
pos = 0;
while (pos<genBlockSize)
{
// Select : Literal (char) or Match (within 32K)
if (CDG_RAND15BITS < P32)
{
// Copy (within 64K)
int ref;
U32 d;
int length = CDG_RANDLENGTH + 4;
U32 offset = CDG_RAND15BITS + 1;
if (pos + length > genBlockSize ) length = genBlockSize - pos;
ref = pos - offset;
d = pos + length;
while (pos < d) buff[pos++] = buff[ref++];
}
else
{
// Literal (noise)
U32 d;
int length = CDG_RANDLENGTH;
if (pos + length > genBlockSize) length = genBlockSize - pos;
d = pos + length;
while (pos < d) buff[pos++] = CDG_RANDCHAR;
}
}
// output datagen
pos=0;
for (;pos+512<=genBlockSize;pos+=512)
printf("%512.512s", buff+pos);
for (;pos<genBlockSize;pos++) printf("%c", buff[pos]);
// Regenerate prefix
memcpy(fullbuff, buff + 96 KB, 32 KB);
}
}
int CDG_usage(void)
{
DISPLAY( "Compressible data generator\n");
DISPLAY( "Usage :\n");
DISPLAY( " %s [size] [args]\n", programName);
DISPLAY( "\n");
DISPLAY( "Arguments :\n");
DISPLAY( " -g# : generate # data (default:%i)\n", CDG_SIZE_DEFAULT);
DISPLAY( " -s# : Select seed (default:%i)\n", CDG_SEED_DEFAULT);
DISPLAY( " -p# : Select compressibility in %% (default:%i%%)\n", CDG_COMPRESSIBILITY_DEFAULT);
DISPLAY( " -h : display help and exit\n");
return 0;
}
int main(int argc, char** argv)
{
int argNb;
int proba = CDG_COMPRESSIBILITY_DEFAULT;
U64 size = CDG_SIZE_DEFAULT;
U32 seed = CDG_SEED_DEFAULT;
// Check command line
programName = argv[0];
for(argNb=1; argNb<argc; argNb++)
{
char* argument = argv[argNb];
if(!argument) continue; // Protection if argument empty
// Decode command (note : aggregated commands are allowed)
if (*argument=='-')
{
if (!strcmp(argument, "--no-prompt")) { no_prompt=1; continue; }
argument++;
while (*argument!=0)
{
switch(*argument)
{
case 'h':
return CDG_usage();
case 'g':
argument++;
size=0;
while ((*argument>='0') && (*argument<='9'))
{
size *= 10;
size += *argument - '0';
argument++;
}
if (*argument=='K') { size <<= 10; argument++; }
if (*argument=='M') { size <<= 20; argument++; }
if (*argument=='G') { size <<= 30; argument++; }
if (*argument=='B') { argument++; }
break;
case 's':
argument++;
seed=0;
while ((*argument>='0') && (*argument<='9'))
{
seed *= 10;
seed += *argument - '0';
argument++;
}
break;
case 'p':
argument++;
proba=0;
while ((*argument>='0') && (*argument<='9'))
{
proba *= 10;
proba += *argument - '0';
argument++;
}
if (proba<0) proba=0;
if (proba>100) proba=100;
break;
case 'v':
displayLevel = 4;
argument++;
break;
default: ;
}
}
}
fwrite(buff, 1, genBlockSize, stdout);
/* update dict */
memcpy(buff, buff + RDG_BLOCKSIZE, RDG_DICTSIZE);
}
// Get Seed
DISPLAYLEVEL(4, "Data Generator %s \n", LZ4_VERSION);
DISPLAYLEVEL(3, "Seed = %u \n", seed);
if (proba!=CDG_COMPRESSIBILITY_DEFAULT) DISPLAYLEVEL(3, "Compressibility : %i%%\n", proba);
CDG_generate(size, &seed, ((double)proba) / 100);
return 0;
free(ldctx);
}

40
programs/datagen.h Normal file
View File

@@ -0,0 +1,40 @@
/*
datagen.h - compressible data generator header
Copyright (C) Yann Collet 2012-2015
GPL v2 License
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
You can contact the author at :
- ZSTD source repository : https://github.com/Cyan4973/zstd
- Public forum : https://groups.google.com/forum/#!forum/lz4c
*/
#include <stddef.h> /* size_t */
void RDG_genOut(unsigned long long size, double matchProba, double litProba, unsigned seed);
void RDG_genBuffer(void* buffer, size_t size, double matchProba, double litProba, unsigned seed);
/* RDG_genOut
Generate 'size' bytes of compressible data into stdout.
Compressibility can be controlled using 'matchProba'.
'LitProba' is optional, and affect variability of bytes. If litProba==0.0, default value is used.
Generated data can be selected using 'seed'.
If (matchProba, litProba and seed) are equal, the function always generate the same content.
RDG_genBuffer
Same as RDG_genOut, but generate data into provided buffer
*/

193
programs/datagencli.c Normal file
View File

@@ -0,0 +1,193 @@
/*
datagencli.c
compressible data command line generator
Copyright (C) Yann Collet 2012-2015
GPL v2 License
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
You can contact the author at :
- ZSTD source repository : https://github.com/Cyan4973/zstd
- Public forum : https://groups.google.com/forum/#!forum/lz4c
*/
/**************************************
* Includes
**************************************/
#include <stdio.h> /* fprintf, stderr */
#include "datagen.h" /* RDG_generate */
/**************************************
* Basic Types
**************************************/
#if defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */
# include <stdint.h>
typedef uint8_t BYTE;
typedef uint16_t U16;
typedef uint32_t U32;
typedef int32_t S32;
typedef uint64_t U64;
#else
typedef unsigned char BYTE;
typedef unsigned short U16;
typedef unsigned int U32;
typedef signed int S32;
typedef unsigned long long U64;
#endif
/**************************************
* Constants
**************************************/
#ifndef ZSTD_VERSION
# define ZSTD_VERSION "r1"
#endif
#define KB *(1 <<10)
#define MB *(1 <<20)
#define GB *(1U<<30)
#define SIZE_DEFAULT (64 KB)
#define SEED_DEFAULT 0
#define COMPRESSIBILITY_DEFAULT 50
/**************************************
* Macros
**************************************/
#define DISPLAY(...) fprintf(stderr, __VA_ARGS__)
#define DISPLAYLEVEL(l, ...) if (displayLevel>=l) { DISPLAY(__VA_ARGS__); }
static unsigned displayLevel = 2;
/*********************************************************
* Command line
*********************************************************/
static int usage(char* programName)
{
DISPLAY( "Compressible data generator\n");
DISPLAY( "Usage :\n");
DISPLAY( " %s [size] [args]\n", programName);
DISPLAY( "\n");
DISPLAY( "Arguments :\n");
DISPLAY( " -g# : generate # data (default:%i)\n", SIZE_DEFAULT);
DISPLAY( " -s# : Select seed (default:%i)\n", SEED_DEFAULT);
DISPLAY( " -P# : Select compressibility in %% (default:%i%%)\n", COMPRESSIBILITY_DEFAULT);
DISPLAY( " -h : display help and exit\n");
DISPLAY( "Special values :\n");
DISPLAY( " -P0 : generate incompressible noise\n");
DISPLAY( " -P100 : generate sparse files\n");
return 0;
}
int main(int argc, char** argv)
{
int argNb;
double proba = (double)COMPRESSIBILITY_DEFAULT / 100;
double litProba = 0.0;
U64 size = SIZE_DEFAULT;
U32 seed = SEED_DEFAULT;
char* programName;
/* Check command line */
programName = argv[0];
for(argNb=1; argNb<argc; argNb++)
{
char* argument = argv[argNb];
if(!argument) continue; /* Protection if argument empty */
/* Handle commands. Aggregated commands are allowed */
if (*argument=='-')
{
argument++;
while (*argument!=0)
{
switch(*argument)
{
case 'h':
return usage(programName);
case 'g':
argument++;
size=0;
while ((*argument>='0') && (*argument<='9'))
{
size *= 10;
size += *argument - '0';
argument++;
}
if (*argument=='K') { size <<= 10; argument++; }
if (*argument=='M') { size <<= 20; argument++; }
if (*argument=='G') { size <<= 30; argument++; }
if (*argument=='B') { argument++; }
break;
case 's':
argument++;
seed=0;
while ((*argument>='0') && (*argument<='9'))
{
seed *= 10;
seed += *argument - '0';
argument++;
}
break;
case 'P':
argument++;
proba=0.0;
while ((*argument>='0') && (*argument<='9'))
{
proba *= 10;
proba += *argument - '0';
argument++;
}
if (proba>100.) proba=100.;
proba /= 100.;
break;
case 'L': /* hidden argument : Literal distribution probability */
argument++;
litProba=0.;
while ((*argument>='0') && (*argument<='9'))
{
litProba *= 10;
litProba += *argument - '0';
argument++;
}
if (litProba>100.) litProba=100.;
litProba /= 100.;
break;
case 'v':
displayLevel = 4;
argument++;
break;
default:
return usage(programName);
}
}
}
}
DISPLAYLEVEL(4, "Data Generator %s \n", ZSTD_VERSION);
DISPLAYLEVEL(3, "Seed = %u \n", seed);
if (proba!=COMPRESSIBILITY_DEFAULT) DISPLAYLEVEL(3, "Compressibility : %i%%\n", (U32)(proba*100));
RDG_genOut(size, proba, litProba, seed);
DISPLAYLEVEL(1, "\n");
return 0;
}

406
programs/frametest.c
View File

@@ -1,6 +1,7 @@
/*
frameTest - test tool for lz4frame
Copyright (C) Yann Collet 2014
Copyright (C) Yann Collet 2014-2015
GPL v2 License
This program is free software; you can redistribute it and/or modify
@@ -18,38 +19,43 @@
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
You can contact the author at :
- LZ4 source repository : http://code.google.com/p/lz4/
- LZ4 source repository : https://github.com/Cyan4973/lz4
- LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c
*/
/**************************************
Compiler specific
* Compiler specific
**************************************/
#define _CRT_SECURE_NO_WARNINGS // fgets
#ifdef _MSC_VER /* Visual Studio */
# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */
# pragma warning(disable : 4146) /* disable: C4146: minus unsigned expression */
#endif
#define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__)
#ifdef __GNUC__
# pragma GCC diagnostic ignored "-Wmissing-braces" /* GCC bug 53119 : doesn't accept { 0 } as initializer (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119) */
# pragma GCC diagnostic ignored "-Wmissing-field-initializers" /* GCC bug 53119 : doesn't accept { 0 } as initializer (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119) */
/* S_ISREG & gettimeofday() are not supported by MSVC */
#if defined(_MSC_VER) || defined(_WIN32)
# define FUZ_LEGACY_TIMER 1
#endif
/**************************************
Includes
* Includes
**************************************/
#include <stdlib.h> // free
#include <stdio.h> // fgets, sscanf
#include <sys/timeb.h> // timeb
#include <string.h> // strcmp
#include <stdlib.h> /* malloc, free */
#include <stdio.h> /* fprintf */
#include <string.h> /* strcmp */
#include "lz4frame_static.h"
#include "xxhash.h" // XXH64
#include "xxhash.h" /* XXH64 */
/* Use ftime() if gettimeofday() is not available on your target */
#if defined(FUZ_LEGACY_TIMER)
# include <sys/timeb.h> /* timeb, ftime */
#else
# include <sys/time.h> /* gettimeofday */
#endif
/**************************************
Basic Types
* Basic Types
**************************************/
#if defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */
# include <stdint.h>
@@ -67,13 +73,26 @@ typedef unsigned long long U64;
#endif
/* unoptimized version; solves endianess & alignment issues */
static void FUZ_writeLE32 (void* dstVoidPtr, U32 value32)
{
BYTE* dstPtr = (BYTE*)dstVoidPtr;
dstPtr[0] = (BYTE)value32;
dstPtr[1] = (BYTE)(value32 >> 8);
dstPtr[2] = (BYTE)(value32 >> 16);
dstPtr[3] = (BYTE)(value32 >> 24);
}
/**************************************
Constants
* Constants
**************************************/
#ifndef LZ4_VERSION
# define LZ4_VERSION ""
#endif
#define LZ4F_MAGIC_SKIPPABLE_START 0x184D2A50U
#define KB *(1U<<10)
#define MB *(1U<<20)
#define GB *(1U<<30)
@@ -87,7 +106,7 @@ static const U32 prime2 = 2246822519U;
/**************************************
Macros
* Macros
**************************************/
#define DISPLAY(...) fprintf(stderr, __VA_ARGS__)
#define DISPLAYLEVEL(l, ...) if (displayLevel>=l) { DISPLAY(__VA_ARGS__); }
@@ -100,7 +119,7 @@ static U32 g_time = 0;
/*****************************************
Local Parameters
* Local Parameters
*****************************************/
static U32 no_prompt = 0;
static char* programName;
@@ -109,8 +128,10 @@ static U32 pause = 0;
/*********************************************************
Fuzzer functions
* Fuzzer functions
*********************************************************/
#if defined(FUZ_LEGACY_TIMER)
static U32 FUZ_GetMilliStart(void)
{
struct timeb tb;
@@ -120,6 +141,19 @@ static U32 FUZ_GetMilliStart(void)
return nCount;
}
#else
static U32 FUZ_GetMilliStart(void)
{
struct timeval tv;
U32 nCount;
gettimeofday(&tv, NULL);
nCount = (U32) (tv.tv_usec/1000 + (tv.tv_sec & 0xfffff) * 1000);
return nCount;
}
#endif
static U32 FUZ_GetMilliSpan(U32 nTimeStart)
{
@@ -151,15 +185,15 @@ static void FUZ_fillCompressibleNoiseBuffer(void* buffer, unsigned bufferSize, d
unsigned pos = 0;
U32 P32 = (U32)(32768 * proba);
// First Byte
/* First Byte */
BBuffer[pos++] = (BYTE)(FUZ_rand(seed));
while (pos < bufferSize)
{
// Select : Literal (noise) or copy (within 64K)
/* Select : Literal (noise) or copy (within 64K) */
if (FUZ_RAND15BITS < P32)
{
// Copy (within 64K)
/* Copy (within 64K) */
unsigned match, end;
unsigned length = FUZ_RANDLENGTH + 4;
unsigned offset = FUZ_RAND15BITS + 1;
@@ -171,7 +205,7 @@ static void FUZ_fillCompressibleNoiseBuffer(void* buffer, unsigned bufferSize, d
}
else
{
// Literal (noise)
/* Literal (noise) */
unsigned end;
unsigned length = FUZ_RANDLENGTH;
if (pos + length > bufferSize) length = bufferSize - pos;
@@ -203,18 +237,20 @@ int basicTests(U32 seed, double compressibility)
void* decodedBuffer;
U32 randState = seed;
size_t cSize, testSize;
LZ4F_preferences_t prefs = { 0 };
LZ4F_decompressionContext_t dCtx;
LZ4F_preferences_t prefs;
LZ4F_decompressionContext_t dCtx = NULL;
LZ4F_compressionContext_t cctx = NULL;
U64 crcOrig;
// Create compressible test buffer
/* Create compressible test buffer */
memset(&prefs, 0, sizeof(prefs));
CNBuffer = malloc(COMPRESSIBLE_NOISE_LENGTH);
compressedBuffer = malloc(LZ4F_compressFrameBound(COMPRESSIBLE_NOISE_LENGTH, NULL));
decodedBuffer = malloc(COMPRESSIBLE_NOISE_LENGTH);
FUZ_fillCompressibleNoiseBuffer(CNBuffer, COMPRESSIBLE_NOISE_LENGTH, compressibility, &randState);
crcOrig = XXH64(CNBuffer, COMPRESSIBLE_NOISE_LENGTH, 1);
// Trivial tests : one-step frame
/* Trivial tests : one-step frame */
testSize = COMPRESSIBLE_NOISE_LENGTH;
DISPLAYLEVEL(3, "Using NULL preferences : \n");
cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, NULL), CNBuffer, testSize, NULL);
@@ -236,16 +272,63 @@ int basicTests(U32 seed, double compressibility)
DISPLAYLEVEL(3, "Single Block : \n");
errorCode = LZ4F_decompress(dCtx, decodedBuffer, &decodedBufferSize, compressedBuffer, &compressedBufferSize, NULL);
if (LZ4F_isError(errorCode)) goto _output_error;
crcDest = XXH64(decodedBuffer, COMPRESSIBLE_NOISE_LENGTH, 1);
if (crcDest != crcOrig) goto _output_error;
DISPLAYLEVEL(3, "Regenerated %i bytes \n", (int)decodedBufferSize);
DISPLAYLEVEL(4, "Reusing decompression context \n");
{
size_t iSize = compressedBufferSize - 4;
const BYTE* cBuff = (const BYTE*) compressedBuffer;
DISPLAYLEVEL(3, "Missing last 4 bytes : ");
errorCode = LZ4F_decompress(dCtx, decodedBuffer, &decodedBufferSize, cBuff, &iSize, NULL);
if (LZ4F_isError(errorCode)) goto _output_error;
if (!errorCode) goto _output_error;
DISPLAYLEVEL(3, "indeed, request %u bytes \n", (unsigned)errorCode);
cBuff += iSize;
iSize = errorCode;
errorCode = LZ4F_decompress(dCtx, decodedBuffer, &decodedBufferSize, cBuff, &iSize, NULL);
if (errorCode != 0) goto _output_error;
crcDest = XXH64(decodedBuffer, COMPRESSIBLE_NOISE_LENGTH, 1);
if (crcDest != crcOrig) goto _output_error;
}
{
size_t oSize = 0;
size_t iSize = 0;
LZ4F_frameInfo_t fi;
DISPLAYLEVEL(3, "Start by feeding 0 bytes, to get next input size : ");
errorCode = LZ4F_decompress(dCtx, NULL, &oSize, ip, &iSize, NULL);
if (LZ4F_isError(errorCode)) goto _output_error;
DISPLAYLEVEL(3, " %u \n", (unsigned)errorCode);
DISPLAYLEVEL(3, "get FrameInfo on null input : ");
errorCode = LZ4F_getFrameInfo(dCtx, &fi, ip, &iSize);
if (errorCode != (size_t)-LZ4F_ERROR_frameHeader_incomplete) goto _output_error;
DISPLAYLEVEL(3, " correctly failed : %s \n", LZ4F_getErrorName(errorCode));
DISPLAYLEVEL(3, "get FrameInfo on not enough input : ");
iSize = 6;
errorCode = LZ4F_getFrameInfo(dCtx, &fi, ip, &iSize);
if (errorCode != (size_t)-LZ4F_ERROR_frameHeader_incomplete) goto _output_error;
DISPLAYLEVEL(3, " correctly failed : %s \n", LZ4F_getErrorName(errorCode));
ip += iSize;
DISPLAYLEVEL(3, "get FrameInfo on enough input : ");
iSize = 15 - iSize;
errorCode = LZ4F_getFrameInfo(dCtx, &fi, ip, &iSize);
if (LZ4F_isError(errorCode)) goto _output_error;
DISPLAYLEVEL(3, " correctly decoded \n");
ip += iSize;
}
DISPLAYLEVEL(3, "Byte after byte : \n");
while (ip < iend)
{
size_t oSize = oend-op;
size_t iSize = 1;
//DISPLAY("%7i \n", (int)(ip-(BYTE*)compressedBuffer));
errorCode = LZ4F_decompress(dCtx, op, &oSize, ip, &iSize, NULL);
if (LZ4F_isError(errorCode)) goto _output_error;
op += oSize;
@@ -253,28 +336,28 @@ int basicTests(U32 seed, double compressibility)
}
crcDest = XXH64(decodedBuffer, COMPRESSIBLE_NOISE_LENGTH, 1);
if (crcDest != crcOrig) goto _output_error;
DISPLAYLEVEL(3, "Regenerated %i bytes \n", (int)decodedBufferSize);
DISPLAYLEVEL(3, "Regenerated %u/%u bytes \n", (unsigned)(op-(BYTE*)decodedBuffer), COMPRESSIBLE_NOISE_LENGTH);
errorCode = LZ4F_freeDecompressionContext(dCtx);
if (LZ4F_isError(errorCode)) goto _output_error;
}
DISPLAYLEVEL(3, "Using 64 KB block : \n");
prefs.frameInfo.blockSizeID = max64KB;
prefs.frameInfo.contentChecksumFlag = contentChecksumEnabled;
prefs.frameInfo.blockSizeID = LZ4F_max64KB;
prefs.frameInfo.contentChecksumFlag = LZ4F_contentChecksumEnabled;
cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs);
if (LZ4F_isError(cSize)) goto _output_error;
DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize);
DISPLAYLEVEL(3, "without checksum : \n");
prefs.frameInfo.contentChecksumFlag = noContentChecksum;
prefs.frameInfo.contentChecksumFlag = LZ4F_noContentChecksum;
cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs);
if (LZ4F_isError(cSize)) goto _output_error;
DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize);
DISPLAYLEVEL(3, "Using 256 KB block : \n");
prefs.frameInfo.blockSizeID = max256KB;
prefs.frameInfo.contentChecksumFlag = contentChecksumEnabled;
prefs.frameInfo.blockSizeID = LZ4F_max256KB;
prefs.frameInfo.contentChecksumFlag = LZ4F_contentChecksumEnabled;
cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs);
if (LZ4F_isError(cSize)) goto _output_error;
DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize);
@@ -314,42 +397,165 @@ int basicTests(U32 seed, double compressibility)
}
DISPLAYLEVEL(3, "without checksum : \n");
prefs.frameInfo.contentChecksumFlag = noContentChecksum;
prefs.frameInfo.contentChecksumFlag = LZ4F_noContentChecksum;
cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs);
if (LZ4F_isError(cSize)) goto _output_error;
DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize);
DISPLAYLEVEL(3, "Using 1 MB block : \n");
prefs.frameInfo.blockSizeID = max1MB;
prefs.frameInfo.contentChecksumFlag = contentChecksumEnabled;
prefs.frameInfo.blockSizeID = LZ4F_max1MB;
prefs.frameInfo.contentChecksumFlag = LZ4F_contentChecksumEnabled;
cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs);
if (LZ4F_isError(cSize)) goto _output_error;
DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize);
DISPLAYLEVEL(3, "without checksum : \n");
prefs.frameInfo.contentChecksumFlag = noContentChecksum;
prefs.frameInfo.contentChecksumFlag = LZ4F_noContentChecksum;
cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs);
if (LZ4F_isError(cSize)) goto _output_error;
DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize);
DISPLAYLEVEL(3, "Using 4 MB block : \n");
prefs.frameInfo.blockSizeID = max4MB;
prefs.frameInfo.contentChecksumFlag = contentChecksumEnabled;
prefs.frameInfo.blockSizeID = LZ4F_max4MB;
prefs.frameInfo.contentChecksumFlag = LZ4F_contentChecksumEnabled;
cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs);
if (LZ4F_isError(cSize)) goto _output_error;
DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize);
DISPLAYLEVEL(3, "without checksum : \n");
prefs.frameInfo.contentChecksumFlag = noContentChecksum;
prefs.frameInfo.contentChecksumFlag = LZ4F_noContentChecksum;
cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs);
if (LZ4F_isError(cSize)) goto _output_error;
DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize);
{
size_t errorCode;
BYTE* const ostart = (BYTE*)compressedBuffer;
BYTE* op = ostart;
errorCode = LZ4F_createCompressionContext(&cctx, LZ4F_VERSION);
if (LZ4F_isError(errorCode)) goto _output_error;
DISPLAYLEVEL(3, "compress without frameSize : \n");
memset(&(prefs.frameInfo), 0, sizeof(prefs.frameInfo));
errorCode = LZ4F_compressBegin(cctx, compressedBuffer, testSize, &prefs);
if (LZ4F_isError(errorCode)) goto _output_error;
op += errorCode;
errorCode = LZ4F_compressUpdate(cctx, op, LZ4F_compressBound(testSize, &prefs), CNBuffer, testSize, NULL);
if (LZ4F_isError(errorCode)) goto _output_error;
op += errorCode;
errorCode = LZ4F_compressEnd(cctx, compressedBuffer, testSize, NULL);
if (LZ4F_isError(errorCode)) goto _output_error;
DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)(op-ostart));
DISPLAYLEVEL(3, "compress with frameSize : \n");
prefs.frameInfo.contentSize = testSize;
op = ostart;
errorCode = LZ4F_compressBegin(cctx, compressedBuffer, testSize, &prefs);
if (LZ4F_isError(errorCode)) goto _output_error;
op += errorCode;
errorCode = LZ4F_compressUpdate(cctx, op, LZ4F_compressBound(testSize, &prefs), CNBuffer, testSize, NULL);
if (LZ4F_isError(errorCode)) goto _output_error;
op += errorCode;
errorCode = LZ4F_compressEnd(cctx, compressedBuffer, testSize, NULL);
if (LZ4F_isError(errorCode)) goto _output_error;
DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)(op-ostart));
DISPLAYLEVEL(3, "compress with wrong frameSize : \n");
prefs.frameInfo.contentSize = testSize+1;
op = ostart;
errorCode = LZ4F_compressBegin(cctx, compressedBuffer, testSize, &prefs);
if (LZ4F_isError(errorCode)) goto _output_error;
op += errorCode;
errorCode = LZ4F_compressUpdate(cctx, op, LZ4F_compressBound(testSize, &prefs), CNBuffer, testSize, NULL);
if (LZ4F_isError(errorCode)) goto _output_error;
op += errorCode;
errorCode = LZ4F_compressEnd(cctx, op, testSize, NULL);
if (LZ4F_isError(errorCode)) { DISPLAYLEVEL(3, "Error correctly detected : %s \n", LZ4F_getErrorName(errorCode)); }
else
goto _output_error;
errorCode = LZ4F_freeCompressionContext(cctx);
if (LZ4F_isError(errorCode)) goto _output_error;
cctx = NULL;
}
DISPLAYLEVEL(3, "Skippable frame test : \n");
{
size_t decodedBufferSize = COMPRESSIBLE_NOISE_LENGTH;
unsigned maxBits = FUZ_highbit((U32)decodedBufferSize);
BYTE* op = (BYTE*)decodedBuffer;
BYTE* const oend = (BYTE*)decodedBuffer + COMPRESSIBLE_NOISE_LENGTH;
BYTE* ip = (BYTE*)compressedBuffer;
BYTE* iend = (BYTE*)compressedBuffer + cSize + 8;
LZ4F_errorCode_t errorCode = LZ4F_createDecompressionContext(&dCtx, LZ4F_VERSION);
if (LZ4F_isError(errorCode)) goto _output_error;
/* generate skippable frame */
FUZ_writeLE32(ip, LZ4F_MAGIC_SKIPPABLE_START);
FUZ_writeLE32(ip+4, (U32)cSize);
DISPLAYLEVEL(3, "random segment sizes : \n");
while (ip < iend)
{
unsigned nbBits = FUZ_rand(&randState) % maxBits;
size_t iSize = (FUZ_rand(&randState) & ((1<<nbBits)-1)) + 1;
size_t oSize = oend-op;
if (iSize > (size_t)(iend-ip)) iSize = iend-ip;
errorCode = LZ4F_decompress(dCtx, op, &oSize, ip, &iSize, NULL);
if (LZ4F_isError(errorCode)) goto _output_error;
op += oSize;
ip += iSize;
}
DISPLAYLEVEL(3, "Skipped %i bytes \n", (int)decodedBufferSize);
/* generate zero-size skippable frame */
DISPLAYLEVEL(3, "zero-size skippable frame\n");
ip = (BYTE*)compressedBuffer;
op = (BYTE*)decodedBuffer;
FUZ_writeLE32(ip, LZ4F_MAGIC_SKIPPABLE_START+1);
FUZ_writeLE32(ip+4, 0);
iend = ip+8;
while (ip < iend)
{
unsigned nbBits = FUZ_rand(&randState) % maxBits;
size_t iSize = (FUZ_rand(&randState) & ((1<<nbBits)-1)) + 1;
size_t oSize = oend-op;
if (iSize > (size_t)(iend-ip)) iSize = iend-ip;
errorCode = LZ4F_decompress(dCtx, op, &oSize, ip, &iSize, NULL);
if (LZ4F_isError(errorCode)) goto _output_error;
op += oSize;
ip += iSize;
}
DISPLAYLEVEL(3, "Skipped %i bytes \n", (int)(ip - (BYTE*)compressedBuffer - 8));
DISPLAYLEVEL(3, "Skippable frame header complete in first call \n");
ip = (BYTE*)compressedBuffer;
op = (BYTE*)decodedBuffer;
FUZ_writeLE32(ip, LZ4F_MAGIC_SKIPPABLE_START+2);
FUZ_writeLE32(ip+4, 10);
iend = ip+18;
while (ip < iend)
{
size_t iSize = 10;
size_t oSize = 10;
if (iSize > (size_t)(iend-ip)) iSize = iend-ip;
errorCode = LZ4F_decompress(dCtx, op, &oSize, ip, &iSize, NULL);
if (LZ4F_isError(errorCode)) goto _output_error;
op += oSize;
ip += iSize;
}
DISPLAYLEVEL(3, "Skipped %i bytes \n", (int)(ip - (BYTE*)compressedBuffer - 8));
}
DISPLAY("Basic tests completed \n");
_end:
free(CNBuffer);
free(compressedBuffer);
free(decodedBuffer);
LZ4F_freeDecompressionContext(dCtx); dCtx = NULL;
LZ4F_freeCompressionContext(cctx); cctx = NULL;
return testResult;
_output_error:
@@ -362,8 +568,8 @@ _output_error:
static void locateBuffDiff(const void* buff1, const void* buff2, size_t size, unsigned nonContiguous)
{
int p=0;
BYTE* b1=(BYTE*)buff1;
BYTE* b2=(BYTE*)buff2;
const BYTE* b1=(const BYTE*)buff1;
const BYTE* b2=(const BYTE*)buff2;
if (nonContiguous)
{
DISPLAY("Non-contiguous output test (%i bytes)\n", (int)size);
@@ -376,7 +582,7 @@ static void locateBuffDiff(const void* buff1, const void* buff2, size_t size, un
static const U32 srcDataLength = 9 MB; /* needs to be > 2x4MB to test large blocks */
int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressibility)
int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressibility, U32 duration)
{
unsigned testResult = 0;
unsigned testNb = 0;
@@ -387,11 +593,16 @@ int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressi
LZ4F_decompressionContext_t dCtx = NULL;
LZ4F_compressionContext_t cCtx = NULL;
size_t result;
const U32 startTime = FUZ_GetMilliStart();
XXH64_state_t xxh64;
# define CHECK(cond, ...) if (cond) { DISPLAY("Error => "); DISPLAY(__VA_ARGS__); \
DISPLAY(" (seed %u, test nb %u) \n", seed, testNb); goto _output_error; }
// Create buffers
/* Init */
duration *= 1000;
/* Create buffers */
result = LZ4F_createDecompressionContext(&dCtx, LZ4F_VERSION);
CHECK(LZ4F_isError(result), "Allocation failed (error %i)", (int)result);
result = LZ4F_createCompressionContext(&cCtx, LZ4F_VERSION);
@@ -400,43 +611,56 @@ int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressi
CHECK(srcBuffer==NULL, "srcBuffer Allocation failed");
compressedBuffer = malloc(LZ4F_compressFrameBound(srcDataLength, NULL));
CHECK(compressedBuffer==NULL, "compressedBuffer Allocation failed");
decodedBuffer = malloc(srcDataLength);
decodedBuffer = calloc(1, srcDataLength); /* calloc avoids decodedBuffer being considered "garbage" by scan-build */
CHECK(decodedBuffer==NULL, "decodedBuffer Allocation failed");
FUZ_fillCompressibleNoiseBuffer(srcBuffer, srcDataLength, compressibility, &coreRand);
// jump to requested testNb
for (testNb =0; testNb < startTest; testNb++) (void)FUZ_rand(&coreRand); // sync randomizer
/* jump to requested testNb */
for (testNb =0; (testNb < startTest); testNb++) (void)FUZ_rand(&coreRand); // sync randomizer
// main fuzzer loop
for ( ; testNb < nbTests; testNb++)
/* main fuzzer test loop */
for ( ; (testNb < nbTests) || (duration > FUZ_GetMilliSpan(startTime)) ; testNb++)
{
U32 randState = coreRand ^ prime1;
unsigned BSId = 4 + (FUZ_rand(&randState) & 3);
unsigned BMId = FUZ_rand(&randState) & 1;
unsigned CCflag = FUZ_rand(&randState) & 1;
unsigned autoflush = (FUZ_rand(&randState) & 7) == 2;
LZ4F_preferences_t prefs = { 0 };
LZ4F_compressOptions_t cOptions = { 0 };
LZ4F_decompressOptions_t dOptions = { 0 };
LZ4F_preferences_t prefs;
LZ4F_compressOptions_t cOptions;
LZ4F_decompressOptions_t dOptions;
unsigned nbBits = (FUZ_rand(&randState) % (FUZ_highbit(srcDataLength-1) - 1)) + 1;
size_t srcSize = (FUZ_rand(&randState) & ((1<<nbBits)-1)) + 1;
size_t srcStart = FUZ_rand(&randState) % (srcDataLength - srcSize);
U64 frameContentSize = ((FUZ_rand(&randState) & 0xF) == 1) ? srcSize : 0;
size_t cSize;
U64 crcOrig, crcDecoded;
LZ4F_preferences_t* prefsPtr = &prefs;
(void)FUZ_rand(&coreRand); // update rand seed
prefs.frameInfo.blockMode = (blockMode_t)BMId;
prefs.frameInfo.blockSizeID = (blockSizeID_t)BSId;
prefs.frameInfo.contentChecksumFlag = (contentChecksum_t)CCflag;
(void)FUZ_rand(&coreRand); /* update seed */
memset(&prefs, 0, sizeof(prefs));
memset(&cOptions, 0, sizeof(cOptions));
memset(&dOptions, 0, sizeof(dOptions));
prefs.frameInfo.blockMode = (LZ4F_blockMode_t)BMId;
prefs.frameInfo.blockSizeID = (LZ4F_blockSizeID_t)BSId;
prefs.frameInfo.contentChecksumFlag = (LZ4F_contentChecksum_t)CCflag;
prefs.frameInfo.contentSize = frameContentSize;
prefs.autoFlush = autoflush;
prefs.compressionLevel = FUZ_rand(&randState) % 5;
if ((FUZ_rand(&randState)&0xF) == 1) prefsPtr = NULL;
if ((FUZ_rand(&randState) & 0xF) == 1) prefsPtr = NULL;
DISPLAYUPDATE(2, "\r%5u ", testNb);
crcOrig = XXH64((BYTE*)srcBuffer+srcStart, (U32)srcSize, 1);
crcOrig = XXH64((BYTE*)srcBuffer+srcStart, srcSize, 1);
if ((FUZ_rand(&randState)&0xF) == 2)
if ((FUZ_rand(&randState) & 0xFFF) == 0)
{
/* create a skippable frame (rare case) */
BYTE* op = (BYTE*)compressedBuffer;
FUZ_writeLE32(op, LZ4F_MAGIC_SKIPPABLE_START + (FUZ_rand(&randState) & 15));
FUZ_writeLE32(op+4, (U32)srcSize);
cSize = srcSize+8;
}
else if ((FUZ_rand(&randState) & 0xF) == 2)
{
cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(srcSize, prefsPtr), (char*)srcBuffer + srcStart, srcSize, prefsPtr);
CHECK(LZ4F_isError(cSize), "LZ4F_compressFrame failed : error %i (%s)", (int)cSize, LZ4F_getErrorName(cSize));
@@ -483,10 +707,12 @@ int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressi
const BYTE* const iend = ip + cSize;
BYTE* op = (BYTE*)decodedBuffer;
BYTE* const oend = op + srcDataLength;
size_t totalOut = 0;
unsigned maxBits = FUZ_highbit((U32)cSize);
unsigned nonContiguousDst = (FUZ_rand(&randState) & 3) == 1;
nonContiguousDst += FUZ_rand(&randState) & nonContiguousDst; /* 0=>0; 1=>1,2 */
XXH64_reset(&xxh64, 1);
if (maxBits < 3) maxBits = 3;
while (ip < iend)
{
unsigned nbBitsI = (FUZ_rand(&randState) % (maxBits-1)) + 1;
@@ -497,22 +723,24 @@ int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressi
if (oSize > (size_t)(oend-op)) oSize = oend-op;
dOptions.stableDst = FUZ_rand(&randState) & 1;
if (nonContiguousDst==2) dOptions.stableDst = 0;
//if (ip == compressedBuffer+62073) DISPLAY("oSize : %i : pos %i \n", (int)oSize, (int)(op-(BYTE*)decodedBuffer));
result = LZ4F_decompress(dCtx, op, &oSize, ip, &iSize, &dOptions);
//if (op+oSize >= (BYTE*)decodedBuffer+94727) DISPLAY("iSize : %i : pos %i \n", (int)iSize, (int)(ip-(BYTE*)compressedBuffer));
//if ((int)result<0) DISPLAY("iSize : %i : pos %i \n", (int)iSize, (int)(ip-(BYTE*)compressedBuffer));
if (result == (size_t)-ERROR_checksum_invalid) locateBuffDiff((BYTE*)srcBuffer+srcStart, decodedBuffer, srcSize, nonContiguousDst);
if (result == (size_t)-LZ4F_ERROR_contentChecksum_invalid)
locateBuffDiff((BYTE*)srcBuffer+srcStart, decodedBuffer, srcSize, nonContiguousDst);
CHECK(LZ4F_isError(result), "Decompression failed (error %i:%s)", (int)result, LZ4F_getErrorName((LZ4F_errorCode_t)result));
XXH64_update(&xxh64, op, (U32)oSize);
totalOut += oSize;
op += oSize;
ip += iSize;
op += nonContiguousDst;
if (nonContiguousDst==2) op = decodedBuffer; // overwritten destination
if (nonContiguousDst==2) op = (BYTE*)decodedBuffer; /* overwritten destination */
}
CHECK(result != 0, "Frame decompression failed (error %i)", (int)result);
crcDecoded = XXH64_digest(&xxh64);
if (crcDecoded != crcOrig) locateBuffDiff((BYTE*)srcBuffer+srcStart, decodedBuffer, srcSize, nonContiguousDst);
CHECK(crcDecoded != crcOrig, "Decompression corruption");
if (totalOut) /* otherwise, it's a skippable frame */
{
crcDecoded = XXH64_digest(&xxh64);
if (crcDecoded != crcOrig) locateBuffDiff((BYTE*)srcBuffer+srcStart, decodedBuffer, srcSize, nonContiguousDst);
CHECK(crcDecoded != crcOrig, "Decompression corruption");
}
}
}
@@ -528,7 +756,7 @@ _end:
if (pause)
{
DISPLAY("press enter to finish \n");
getchar();
(void)getchar();
}
return testResult;
@@ -545,6 +773,7 @@ int FUZ_usage(void)
DISPLAY( "\n");
DISPLAY( "Arguments :\n");
DISPLAY( " -i# : Nb of tests (default:%u) \n", nbTestsDefault);
DISPLAY( " -T# : Duration of tests, in seconds (default: use Nb of tests) \n");
DISPLAY( " -s# : Select seed (default:prompt user)\n");
DISPLAY( " -t# : Select starting test number (default:0)\n");
DISPLAY( " -p# : Select compressibility in %% (default:%i%%)\n", FUZ_COMPRESSIBILITY_DEFAULT);
@@ -563,16 +792,17 @@ int main(int argc, char** argv)
int testNb = 0;
int proba = FUZ_COMPRESSIBILITY_DEFAULT;
int result=0;
U32 duration=0;
// Check command line
/* Check command line */
programName = argv[0];
for(argNb=1; argNb<argc; argNb++)
{
char* argument = argv[argNb];
if(!argument) continue; // Protection if argument empty
if(!argument) continue; /* Protection if argument empty */
// Decode command (note : aggregated commands are allowed)
/* Decode command (note : aggregated commands are allowed) */
if (argument[0]=='-')
{
if (!strcmp(argument, "--no-prompt"))
@@ -605,7 +835,7 @@ int main(int argc, char** argv)
case 'i':
argument++;
nbTests=0;
nbTests=0; duration=0;
while ((*argument>='0') && (*argument<='9'))
{
nbTests *= 10;
@@ -613,6 +843,32 @@ int main(int argc, char** argv)
argument++;
}
break;
case 'T':
argument++;
nbTests = 0; duration = 0;
for (;;)
{
switch(*argument)
{
case 'm': duration *= 60; argument++; continue;
case 's':
case 'n': argument++; continue;
case '0':
case '1':
case '2':
case '3':
case '4':
case '5':
case '6':
case '7':
case '8':
case '9': duration *= 10; duration += *argument++ - '0'; continue;
}
break;
}
break;
case 's':
argument++;
seed=0;
@@ -654,7 +910,7 @@ int main(int argc, char** argv)
}
}
// Get Seed
/* Get Seed */
printf("Starting lz4frame tester (%i-bits, %s)\n", (int)(sizeof(size_t)*8), LZ4_VERSION);
if (!seedset) seed = FUZ_GetMilliStart() % 10000;
@@ -665,5 +921,5 @@ int main(int argc, char** argv)
if (testNb==0) result = basicTests(seed, ((double)proba) / 100);
if (result) return 1;
return fuzzerTests(seed, nbTests, testNb, ((double)proba) / 100);
return fuzzerTests(seed, nbTests, testNb, ((double)proba) / 100, duration);
}

714
programs/fullbench.c
View File

File diff suppressed because it is too large Load Diff

378
programs/fuzzer.c
View File

@@ -19,13 +19,12 @@
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
You can contact the author at :
- LZ4 source repository : http://code.google.com/p/lz4
- LZ4 source mirror : https://github.com/Cyan4973/lz4
- LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c
*/
/**************************************
* Remove Visual warning messages
* Compiler options
**************************************/
#ifdef _MSC_VER /* Visual Studio */
# define _CRT_SECURE_NO_WARNINGS /* fgets */
@@ -34,21 +33,32 @@
# pragma warning(disable : 4310) /* disable: C4310: constant char value > 127 */
#endif
/* S_ISREG & gettimeofday() are not supported by MSVC */
#if defined(_MSC_VER) || defined(_WIN32)
# define FUZ_LEGACY_TIMER 1
#endif
/**************************************
* Includes
* Includes
**************************************/
#include <stdlib.h>
#include <stdio.h> /* fgets, sscanf */
#include <sys/timeb.h> /* timeb */
#include <string.h> /* strcmp */
#include "lz4.h"
#include "lz4hc.h"
#include "xxhash.h"
/* Use ftime() if gettimeofday() is not available on your target */
#if defined(FUZ_LEGACY_TIMER)
# include <sys/timeb.h> /* timeb, ftime */
#else
# include <sys/time.h> /* gettimeofday */
#endif
/**************************************
* Basic Types
* Basic Types
**************************************/
#if defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */
# include <stdint.h>
@@ -67,7 +77,7 @@ typedef unsigned long long U64;
/**************************************
* Constants
* Constants
**************************************/
#ifndef LZ4_VERSION
# define LZ4_VERSION ""
@@ -88,7 +98,7 @@ typedef unsigned long long U64;
/*****************************************
* Macros
* Macros
*****************************************/
#define DISPLAY(...) fprintf(stderr, __VA_ARGS__)
#define DISPLAYLEVEL(l, ...) if (g_displayLevel>=l) { DISPLAY(__VA_ARGS__); }
@@ -98,8 +108,10 @@ static U32 g_time = 0;
/*********************************************************
Fuzzer functions
* Fuzzer functions
*********************************************************/
#if defined(FUZ_LEGACY_TIMER)
static U32 FUZ_GetMilliStart(void)
{
struct timeb tb;
@@ -109,6 +121,20 @@ static U32 FUZ_GetMilliStart(void)
return nCount;
}
#else
static U32 FUZ_GetMilliStart(void)
{
struct timeval tv;
U32 nCount;
gettimeofday(&tv, NULL);
nCount = (U32) (tv.tv_usec/1000 + (tv.tv_sec & 0xfffff) * 1000);
return nCount;
}
#endif
static U32 FUZ_GetMilliSpan(U32 nTimeStart)
{
U32 nCurrent = FUZ_GetMilliStart();
@@ -127,10 +153,10 @@ static U32 FUZ_rand(U32* src)
{
U32 rand32 = *src;
rand32 *= PRIME1;
rand32 += PRIME2;
rand32 ^= PRIME2;
rand32 = FUZ_rotl32(rand32, 13);
*src = rand32;
return rand32 >> 3;
return rand32;
}
@@ -142,27 +168,28 @@ static void FUZ_fillCompressibleNoiseBuffer(void* buffer, size_t bufferSize, dou
size_t pos = 0;
U32 P32 = (U32)(32768 * proba);
// First Byte
BBuffer[pos++] = (BYTE)(FUZ_rand(seed));
/* First Bytes */
while (pos < 20)
BBuffer[pos++] = (BYTE)(FUZ_rand(seed));
while (pos < bufferSize)
{
// Select : Literal (noise) or copy (within 64K)
/* Select : Literal (noise) or copy (within 64K) */
if (FUZ_RAND15BITS < P32)
{
// Copy (within 64K)
/* Copy (within 64K) */
size_t match, d;
size_t length = FUZ_RANDLENGTH + 4;
size_t offset = FUZ_RAND15BITS + 1;
if (offset > pos) offset = pos;
while (offset > pos) offset >>= 1;
d = pos + length;
if (d > bufferSize) d = bufferSize;
while (d > bufferSize) d = bufferSize;
match = pos - offset;
while (pos < d) BBuffer[pos++] = BBuffer[match++];
}
else
{
// Literal (noise)
/* Literal (noise) */
size_t d;
size_t length = FUZ_RANDLENGTH;
d = pos + length;
@@ -177,13 +204,13 @@ static void FUZ_fillCompressibleNoiseBuffer(void* buffer, size_t bufferSize, dou
#define BLOCKSIZE_I134 (32 MB)
static int FUZ_AddressOverflow(void)
{
char* buffers[MAX_NB_BUFF_I134+1] = {0};
char* buffers[MAX_NB_BUFF_I134+1];
int i, nbBuff=0;
int highAddress = 0;
printf("Overflow tests : ");
// Only possible in 32-bits
/* Only possible in 32-bits */
if (sizeof(void*)==8)
{
printf("64 bits mode : no overflow \n");
@@ -273,7 +300,17 @@ static void FUZ_displayUpdate(unsigned testNb)
}
static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const double compressibility)
static void FUZ_findDiff(const void* buff1, const void* buff2)
{
const BYTE* b1 = (const BYTE*)buff1;
const BYTE* b2 = (const BYTE*)buff2;
size_t i=0;
while (b1[i]==b2[i]) i++;
DISPLAY("Wrong Byte at position %u\n", (unsigned)i);
}
static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double compressibility, U32 duration)
{
unsigned long long bytes = 0;
unsigned long long cbytes = 0;
@@ -286,33 +323,35 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do
int ret;
unsigned cycleNb;
# define FUZ_CHECKTEST(cond, ...) if (cond) { printf("Test %u : ", testNb); printf(__VA_ARGS__); \
printf(" (seed %u, cycle %u) \n", seed, cycleNb); goto _output_error; }
printf(" (seed %u, cycle %u) \n", seed, cycleNb); goto _output_error; }
# define FUZ_DISPLAYTEST { testNb++; g_displayLevel<3 ? 0 : printf("%2u\b\b", testNb); if (g_displayLevel==4) fflush(stdout); }
void* stateLZ4 = malloc(LZ4_sizeofState());
void* stateLZ4HC = malloc(LZ4_sizeofStateHC());
void* LZ4continue;
LZ4_stream_t LZ4dict;
LZ4_streamHC_t LZ4dictHC;
U32 crcOrig, crcCheck;
U32 coreRandState = seed;
U32 randState = coreRandState ^ PRIME3;
int result = 0;
const U32 startTime = FUZ_GetMilliStart();
// init
/* init */
memset(&LZ4dict, 0, sizeof(LZ4dict));
duration *= 1000;
// Create compressible test buffer
/* Create compressible test buffer */
CNBuffer = malloc(COMPRESSIBLE_NOISE_LENGTH);
FUZ_fillCompressibleNoiseBuffer(CNBuffer, COMPRESSIBLE_NOISE_LENGTH, compressibility, &randState);
compressedBuffer = (char*)malloc(LZ4_compressBound(FUZ_MAX_BLOCK_SIZE));
decodedBuffer = (char*)malloc(FUZ_MAX_DICT_SIZE + FUZ_MAX_BLOCK_SIZE);
// move to startCycle
/* move to startCycle */
for (cycleNb = 0; cycleNb < startCycle; cycleNb++)
{
(void)FUZ_rand(&coreRandState);
if (0) // some problems related to dictionary re-use; in this case, enable this loop
if (0) /* some problems are related to dictionary re-use; in this case, enable this loop */
{
int dictSize, blockSize, blockStart;
char* dict;
@@ -334,8 +373,8 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do
}
}
// Test loop
for (cycleNb = startCycle; cycleNb < nbCycles; cycleNb++)
/* Main test loop */
for (cycleNb = startCycle; (cycleNb < nbCycles) || (FUZ_GetMilliSpan(startTime) < duration) ; cycleNb++)
{
U32 testNb = 0;
char* dict;
@@ -347,7 +386,7 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do
(void)FUZ_rand(&coreRandState);
randState = coreRandState ^ PRIME3;
// Select block to test
/* Select block to test */
blockSize = FUZ_rand(&randState) % FUZ_MAX_BLOCK_SIZE;
blockStart = FUZ_rand(&randState) % (COMPRESSIBLE_NOISE_LENGTH - blockSize);
dictSize = FUZ_rand(&randState) % FUZ_MAX_DICT_SIZE;
@@ -357,23 +396,58 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do
/* Compression tests */
// Test compression HC
/* Test compression destSize */
FUZ_DISPLAYTEST;
{
int srcSize = blockSize;
int targetSize = srcSize * ((FUZ_rand(&randState) & 127)+1) >> 7;
char endCheck = FUZ_rand(&randState) & 255;
compressedBuffer[targetSize] = endCheck;
ret = LZ4_compress_destSize(block, compressedBuffer, &srcSize, targetSize);
FUZ_CHECKTEST(ret > targetSize, "LZ4_compress_destSize() result larger than dst buffer !");
FUZ_CHECKTEST(compressedBuffer[targetSize] != endCheck, "LZ4_compress_destSize() overwrite dst buffer !");
FUZ_CHECKTEST(srcSize > blockSize, "LZ4_compress_destSize() fed more than src buffer !");
DISPLAY("destSize : %7i/%7i; content%7i/%7i ", ret, targetSize, srcSize, blockSize);
if (targetSize>0)
{
FUZ_CHECKTEST((ret==0), "LZ4_compress_destSize() compression failed");
/* check correctness */
FUZ_DISPLAYTEST;
crcOrig = XXH32(block, srcSize, 0);
compressedSize = ret;
endCheck = FUZ_rand(&randState) & 255;
decodedBuffer[srcSize] = endCheck;
ret = LZ4_decompress_safe(compressedBuffer, decodedBuffer, compressedSize, srcSize);
FUZ_CHECKTEST(ret<0, "LZ4_decompress_safe() failed on data compressed by LZ4_compress_destSize");
FUZ_CHECKTEST(ret!=srcSize, "LZ4_decompress_safe() failed : did not fully decompressed data");
FUZ_CHECKTEST(decodedBuffer[srcSize] != endCheck, "LZ4_decompress_safe() overwrite dst buffer !");
crcCheck = XXH32(decodedBuffer, srcSize, 0);
FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe() corrupted decoded data");
DISPLAY(" OK \n");
}
else
DISPLAY(" \n");
}
/* Test compression HC */
FUZ_DISPLAYTEST;
ret = LZ4_compressHC(block, compressedBuffer, blockSize);
FUZ_CHECKTEST(ret==0, "LZ4_compressHC() failed");
HCcompressedSize = ret;
// Test compression HC using external state
/* Test compression HC using external state */
FUZ_DISPLAYTEST;
ret = LZ4_compressHC_withStateHC(stateLZ4HC, block, compressedBuffer, blockSize);
FUZ_CHECKTEST(ret==0, "LZ4_compressHC_withStateHC() failed");
// Test compression using external state
/* Test compression using external state */
FUZ_DISPLAYTEST;
ret = LZ4_compress_withState(stateLZ4, block, compressedBuffer, blockSize);
FUZ_CHECKTEST(ret==0, "LZ4_compress_withState() failed");
// Test compression
/* Test compression */
FUZ_DISPLAYTEST;
ret = LZ4_compress(block, compressedBuffer, blockSize);
FUZ_CHECKTEST(ret==0, "LZ4_compress() failed");
@@ -383,7 +457,7 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do
crcOrig = XXH32(block, blockSize, 0);
// Test decoding with output size being exactly what's necessary => must work
/* Test decoding with output size being exactly what's necessary => must work */
FUZ_DISPLAYTEST;
ret = LZ4_decompress_fast(compressedBuffer, decodedBuffer, blockSize);
FUZ_CHECKTEST(ret<0, "LZ4_decompress_fast failed despite correct space");
@@ -391,19 +465,19 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do
crcCheck = XXH32(decodedBuffer, blockSize, 0);
FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_fast corrupted decoded data");
// Test decoding with one byte missing => must fail
/* Test decoding with one byte missing => must fail */
FUZ_DISPLAYTEST;
decodedBuffer[blockSize-1] = 0;
ret = LZ4_decompress_fast(compressedBuffer, decodedBuffer, blockSize-1);
FUZ_CHECKTEST(ret>=0, "LZ4_decompress_fast should have failed, due to Output Size being too small");
FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_fast overrun specified output buffer");
// Test decoding with one byte too much => must fail
/* Test decoding with one byte too much => must fail */
FUZ_DISPLAYTEST;
ret = LZ4_decompress_fast(compressedBuffer, decodedBuffer, blockSize+1);
FUZ_CHECKTEST(ret>=0, "LZ4_decompress_fast should have failed, due to Output Size being too large");
// Test decoding with output size exactly what's necessary => must work
/* Test decoding with output size exactly what's necessary => must work */
FUZ_DISPLAYTEST;
decodedBuffer[blockSize] = 0;
ret = LZ4_decompress_safe(compressedBuffer, decodedBuffer, compressedSize, blockSize);
@@ -517,16 +591,18 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do
/* Compress using dictionary */
FUZ_DISPLAYTEST;
LZ4continue = LZ4_create (dict);
LZ4_compress_continue ((LZ4_stream_t*)LZ4continue, dict, compressedBuffer, dictSize); // Just to fill hash tables
blockContinueCompressedSize = LZ4_compress_continue ((LZ4_stream_t*)LZ4continue, block, compressedBuffer, blockSize);
FUZ_CHECKTEST(blockContinueCompressedSize==0, "LZ4_compress_continue failed");
free (LZ4continue);
{
LZ4_stream_t LZ4_stream;
LZ4_resetStream(&LZ4_stream);
LZ4_compress_continue (&LZ4_stream, dict, compressedBuffer, dictSize); /* Just to fill hash tables */
blockContinueCompressedSize = LZ4_compress_continue (&LZ4_stream, block, compressedBuffer, blockSize);
FUZ_CHECKTEST(blockContinueCompressedSize==0, "LZ4_compress_continue failed");
}
/* Decompress with dictionary as prefix */
FUZ_DISPLAYTEST;
memcpy(decodedBuffer, dict, dictSize);
ret = LZ4_decompress_fast_withPrefix64k(compressedBuffer, decodedBuffer+dictSize, blockSize);
ret = LZ4_decompress_fast_usingDict(compressedBuffer, decodedBuffer+dictSize, blockSize, decodedBuffer, dictSize);
FUZ_CHECKTEST(ret!=blockContinueCompressedSize, "LZ4_decompress_fast_withPrefix64k did not read all compressed block input");
crcCheck = XXH32(decodedBuffer+dictSize, blockSize, 0);
if (crcCheck!=crcOrig)
@@ -539,14 +615,14 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do
FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_fast_withPrefix64k corrupted decoded data (dict %i)", dictSize);
FUZ_DISPLAYTEST;
ret = LZ4_decompress_safe_withPrefix64k(compressedBuffer, decodedBuffer+dictSize, blockContinueCompressedSize, blockSize);
FUZ_CHECKTEST(ret!=blockSize, "LZ4_decompress_safe_withPrefix64k did not regenerate original data");
ret = LZ4_decompress_safe_usingDict(compressedBuffer, decodedBuffer+dictSize, blockContinueCompressedSize, blockSize, decodedBuffer, dictSize);
FUZ_CHECKTEST(ret!=blockSize, "LZ4_decompress_safe_usingDict did not regenerate original data");
crcCheck = XXH32(decodedBuffer+dictSize, blockSize, 0);
FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe_withPrefix64k corrupted decoded data");
FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe_usingDict corrupted decoded data");
// Compress using External dictionary
/* Compress using External dictionary */
FUZ_DISPLAYTEST;
dict -= (FUZ_rand(&randState) & 0xF) + 1; // Separation, so it is an ExtDict
dict -= (FUZ_rand(&randState) & 0xF) + 1; /* Separation, so it is an ExtDict */
if (dict < (char*)CNBuffer) dict = (char*)CNBuffer;
LZ4_loadDict(&LZ4dict, dict, dictSize);
blockContinueCompressedSize = LZ4_compress_continue(&LZ4dict, block, compressedBuffer, blockSize);
@@ -555,7 +631,7 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do
FUZ_DISPLAYTEST;
LZ4_loadDict(&LZ4dict, dict, dictSize);
ret = LZ4_compress_limitedOutput_continue(&LZ4dict, block, compressedBuffer, blockSize, blockContinueCompressedSize-1);
FUZ_CHECKTEST(ret>0, "LZ4_compress_limitedOutput_continue using ExtDict should fail : one missing byte for output buffer");
FUZ_CHECKTEST(ret>0, "LZ4_compress_limitedOutput_continue using ExtDict should fail : one missing byte for output buffer : %i written, %i buffer", ret, blockContinueCompressedSize);
FUZ_DISPLAYTEST;
LZ4_loadDict(&LZ4dict, dict, dictSize);
@@ -563,7 +639,7 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do
FUZ_CHECKTEST(ret!=blockContinueCompressedSize, "LZ4_compress_limitedOutput_compressed size is different (%i != %i)", ret, blockContinueCompressedSize);
FUZ_CHECKTEST(ret<=0, "LZ4_compress_limitedOutput_continue should work : enough size available within output buffer");
// Decompress with dictionary as external
/* Decompress with dictionary as external */
FUZ_DISPLAYTEST;
decodedBuffer[blockSize] = 0;
ret = LZ4_decompress_fast_usingDict(compressedBuffer, decodedBuffer, blockSize, dict, dictSize);
@@ -571,11 +647,7 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do
FUZ_CHECKTEST(decodedBuffer[blockSize], "LZ4_decompress_fast_usingDict overrun specified output buffer size")
crcCheck = XXH32(decodedBuffer, blockSize, 0);
if (crcCheck!=crcOrig)
{
int i=0;
while (block[i]==decodedBuffer[i]) i++;
printf("Wrong Byte at position %i/%i\n", i, blockSize);
}
FUZ_findDiff(block, decodedBuffer);
FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_fast_usingDict corrupted decoded data (dict %i)", dictSize);
FUZ_DISPLAYTEST;
@@ -589,7 +661,7 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do
FUZ_DISPLAYTEST;
decodedBuffer[blockSize-1] = 0;
ret = LZ4_decompress_fast_usingDict(compressedBuffer, decodedBuffer, blockSize-1, dict, dictSize);
FUZ_CHECKTEST(ret>=0, "LZ4_decompress_fast_withDict should have failed : wrong original size (-1 byte)");
FUZ_CHECKTEST(ret>=0, "LZ4_decompress_fast_usingDict should have failed : wrong original size (-1 byte)");
FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_fast_usingDict overrun specified output buffer size");
FUZ_DISPLAYTEST;
@@ -610,9 +682,9 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do
}
}
// Compress HC using External dictionary
/* Compress HC using External dictionary */
FUZ_DISPLAYTEST;
dict -= (FUZ_rand(&randState) & 7); // even bigger separation
dict -= (FUZ_rand(&randState) & 7); /* even bigger separation */
if (dict < (char*)CNBuffer) dict = (char*)CNBuffer;
LZ4_resetStreamHC (&LZ4dictHC, FUZ_rand(&randState) & 0x7);
LZ4_loadDictHC(&LZ4dictHC, dict, dictSize);
@@ -637,31 +709,27 @@ static int FUZ_test(U32 seed, const U32 nbCycles, const U32 startCycle, const do
FUZ_CHECKTEST(decodedBuffer[blockSize], "LZ4_decompress_safe_usingDict overrun specified output buffer size")
crcCheck = XXH32(decodedBuffer, blockSize, 0);
if (crcCheck!=crcOrig)
{
int i=0;
while (block[i]==decodedBuffer[i]) i++;
printf("Wrong Byte at position %i/%i\n", i, blockSize);
}
FUZ_findDiff(block, decodedBuffer);
FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe_usingDict corrupted decoded data");
// ***** End of tests *** //
// Fill stats
/* ***** End of tests *** */
/* Fill stats */
bytes += blockSize;
cbytes += compressedSize;
hcbytes += HCcompressedSize;
ccbytes += blockContinueCompressedSize;
}
if (nbCycles<=1) nbCycles = cycleNb; /* end by time */
bytes += !bytes; /* avoid division by 0 */
printf("\r%7u /%7u - ", cycleNb, nbCycles);
printf("all tests completed successfully \n");
printf("compression ratio: %0.3f%%\n", (double)cbytes/bytes*100);
printf("HC compression ratio: %0.3f%%\n", (double)hcbytes/bytes*100);
printf("ratio with dict: %0.3f%%\n", (double)ccbytes/bytes*100);
// unalloc
/* release memory */
{
int result = 0;
_exit:
free(CNBuffer);
free(compressedBuffer);
@@ -692,13 +760,13 @@ static void FUZ_unitTests(void)
char ringBuffer[ringBufferSize];
U32 randState = 1;
// Init
/* Init */
FUZ_fillCompressibleNoiseBuffer(testInput, testInputSize, 0.50, &randState);
// 32-bits address space overflow test
/* 32-bits address space overflow test */
FUZ_AddressOverflow();
// LZ4 streaming tests
/* LZ4 streaming tests */
{
LZ4_stream_t* statePtr;
LZ4_stream_t streamingState;
@@ -706,12 +774,12 @@ static void FUZ_unitTests(void)
U64 crcNew;
int result;
// Allocation test
/* Allocation test */
statePtr = LZ4_createStream();
FUZ_CHECKTEST(statePtr==NULL, "LZ4_createStream() allocation failed");
LZ4_freeStream(statePtr);
// simple compression test
/* simple compression test */
crcOrig = XXH64(testInput, testCompressedSize, 0);
LZ4_resetStream(&streamingState);
result = LZ4_compress_limitedOutput_continue(&streamingState, testInput, testCompressed, testCompressedSize, testCompressedSize-1);
@@ -722,7 +790,7 @@ static void FUZ_unitTests(void)
crcNew = XXH64(testVerify, testCompressedSize, 0);
FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() decompression corruption");
// ring buffer test
/* ring buffer test */
{
XXH64_state_t xxhOrig;
XXH64_state_t xxhNew;
@@ -753,10 +821,10 @@ static void FUZ_unitTests(void)
FUZ_CHECKTEST(result!=(int)messageSize, "ringBuffer : LZ4_decompress_safe() test failed");
XXH64_update(&xxhNew, testVerify + dNext, messageSize);
crcNew = crcOrig = XXH64_digest(&xxhNew);
crcNew = XXH64_digest(&xxhNew);
FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() decompression corruption");
// prepare next message
/* prepare next message */
iNext += messageSize;
rNext += messageSize;
dNext += messageSize;
@@ -767,21 +835,20 @@ static void FUZ_unitTests(void)
}
}
// LZ4 HC streaming tests
/* LZ4 HC streaming tests */
{
LZ4_streamHC_t* sp;
LZ4_streamHC_t sHC;
//XXH64_state_t xxh;
U64 crcOrig;
U64 crcNew;
int result;
// Allocation test
/* Allocation test */
sp = LZ4_createStreamHC();
FUZ_CHECKTEST(sp==NULL, "LZ4_createStreamHC() allocation failed");
LZ4_freeStreamHC(sp);
// simple compression test
/* simple HC compression test */
crcOrig = XXH64(testInput, testCompressedSize, 0);
LZ4_resetStreamHC(&sHC, 0);
result = LZ4_compressHC_limitedOutput_continue(&sHC, testInput, testCompressed, testCompressedSize, testCompressedSize-1);
@@ -792,7 +859,7 @@ static void FUZ_unitTests(void)
crcNew = XXH64(testVerify, testCompressedSize, 0);
FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() decompression corruption");
// simple dictionary compression test
/* simple dictionary HC compression test */
crcOrig = XXH64(testInput + 64 KB, testCompressedSize, 0);
LZ4_resetStreamHC(&sHC, 0);
LZ4_loadDictHC(&sHC, testInput, 64 KB);
@@ -804,7 +871,7 @@ static void FUZ_unitTests(void)
crcNew = XXH64(testVerify, testCompressedSize, 0);
FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() simple dictionary decompression test : corruption");
// multiple HC compression test with dictionary
/* multiple HC compression test with dictionary */
{
int result1, result2;
int segSize = testCompressedSize / 2;
@@ -824,7 +891,7 @@ static void FUZ_unitTests(void)
FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() dictionary decompression corruption");
}
// remote dictionary HC compression test
/* remote dictionary HC compression test */
crcOrig = XXH64(testInput + 64 KB, testCompressedSize, 0);
LZ4_resetStreamHC(&sHC, 0);
LZ4_loadDictHC(&sHC, testInput, 32 KB);
@@ -836,7 +903,7 @@ static void FUZ_unitTests(void)
crcNew = XXH64(testVerify, testCompressedSize, 0);
FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe_usingDict() decompression corruption");
// multiple HC compression with ext. dictionary
/* multiple HC compression with ext. dictionary */
{
XXH64_state_t crcOrigState;
XXH64_state_t crcNewState;
@@ -885,7 +952,7 @@ static void FUZ_unitTests(void)
}
}
// ring buffer test
/* ring buffer test */
{
XXH64_state_t xxhOrig;
XXH64_state_t xxhNew;
@@ -916,7 +983,7 @@ static void FUZ_unitTests(void)
FUZ_CHECKTEST(result!=(int)messageSize, "ringBuffer : LZ4_decompress_safe() test failed");
XXH64_update(&xxhNew, testVerify + dNext, messageSize);
crcNew = crcOrig = XXH64_digest(&xxhNew);
crcNew = XXH64_digest(&xxhNew);
FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() decompression corruption");
// prepare next message
@@ -929,24 +996,51 @@ static void FUZ_unitTests(void)
}
}
// small decoder-side ring buffer test
/* small decoder-side ring buffer test */
{
XXH64_state_t xxhOrig;
XXH64_state_t xxhNew;
LZ4_streamDecode_t decodeState;
const U32 maxMessageSizeLog = 10;
const U32 maxMessageSizeLog = 12;
const U32 maxMessageSizeMask = (1<<maxMessageSizeLog) - 1;
U32 messageSize = (FUZ_rand(&randState) & maxMessageSizeMask) + 1;
U32 messageSize;
U32 totalMessageSize = 0;
U32 iNext = 0;
U32 dNext = 0;
const U32 dBufferSize = 64 KB + maxMessageSizeMask;
const U32 dBufferSize = 64 KB;
XXH64_reset(&xxhOrig, 0);
XXH64_reset(&xxhNew, 0);
LZ4_resetStreamHC(&sHC, 0);
LZ4_setStreamDecode(&decodeState, NULL, 0);
#define BSIZE1 65537
#define BSIZE2 16435
/* first block */
messageSize = BSIZE1;
XXH64_update(&xxhOrig, testInput + iNext, messageSize);
crcOrig = XXH64_digest(&xxhOrig);
result = LZ4_compressHC_limitedOutput_continue(&sHC, testInput + iNext, testCompressed, messageSize, testCompressedSize-ringBufferSize);
FUZ_CHECKTEST(result==0, "LZ4_compressHC_limitedOutput_continue() compression failed");
result = LZ4_decompress_safe_continue(&decodeState, testCompressed, testVerify + dNext, result, messageSize);
FUZ_CHECKTEST(result!=(int)messageSize, "64K D.ringBuffer : LZ4_decompress_safe() test failed");
XXH64_update(&xxhNew, testVerify + dNext, messageSize);
crcNew = XXH64_digest(&xxhNew);
FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() decompression corruption");
/* prepare next message */
dNext += messageSize;
totalMessageSize += messageSize;
messageSize = BSIZE2;
iNext = 132000;
memcpy(testInput + iNext, testInput + 8, messageSize);
if (dNext > dBufferSize) dNext = 0;
while (totalMessageSize < 9 MB)
{
XXH64_update(&xxhOrig, testInput + iNext, messageSize);
@@ -956,76 +1050,22 @@ static void FUZ_unitTests(void)
FUZ_CHECKTEST(result==0, "LZ4_compressHC_limitedOutput_continue() compression failed");
result = LZ4_decompress_safe_continue(&decodeState, testCompressed, testVerify + dNext, result, messageSize);
FUZ_CHECKTEST(result!=(int)messageSize, "ringBuffer : LZ4_decompress_safe() test failed");
FUZ_CHECKTEST(result!=(int)messageSize, "64K D.ringBuffer : LZ4_decompress_safe() test failed");
XXH64_update(&xxhNew, testVerify + dNext, messageSize);
crcNew = crcOrig = XXH64_digest(&xxhNew);
FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() decompression corruption");
crcNew = XXH64_digest(&xxhNew);
if (crcOrig != crcNew)
FUZ_findDiff(testInput + iNext, testVerify + dNext);
FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() decompression corruption during small decoder-side ring buffer test");
// prepare next message
/* prepare next message */
dNext += messageSize;
totalMessageSize += messageSize;
messageSize = (FUZ_rand(&randState) & maxMessageSizeMask) + 1;
iNext = (FUZ_rand(&randState) & 65535);
if (dNext + messageSize > dBufferSize) dNext = 0;
if (dNext > dBufferSize) dNext = 0;
}
}
// long stream test ; Warning : very long test !
if (1)
{
XXH64_state_t crcOrigState;
XXH64_state_t crcNewState;
const U64 totalTestSize = 6ULL << 30;
U64 totalTestDone = 0;
size_t oldStart = 0;
size_t oldSize = 0;
U32 segNb = 1;
DISPLAY("Long HC streaming test (%u MB)\n", (U32)(totalTestSize >> 20));
LZ4_resetStreamHC(&sHC, 0);
XXH64_reset(&crcOrigState, 0);
XXH64_reset(&crcNewState, 0);
while (totalTestDone < totalTestSize)
{
size_t testSize = (FUZ_rand(&randState) & 65535) + 1;
size_t testStart = FUZ_rand(&randState) & 65535;
FUZ_displayUpdate((U32)(totalTestDone >> 20));
if (testStart == oldStart + oldSize) // Corner case not covered by this test (LZ4_decompress_safe_usingDict() limitation)
testStart++;
XXH64_update(&crcOrigState, testInput + testStart, testSize);
crcOrig = XXH64_digest(&crcOrigState);
result = LZ4_compressHC_limitedOutput_continue(&sHC, testInput + testStart, testCompressed, (int)testSize, LZ4_compressBound((int)testSize));
FUZ_CHECKTEST(result==0, "LZ4_compressHC_limitedOutput_continue() dictionary compression failed : result = %i", result);
result = LZ4_decompress_safe_usingDict(testCompressed, testVerify, result, (int)testSize, testInput + oldStart, (int)oldSize);
FUZ_CHECKTEST(result!=(int)testSize, "LZ4_decompress_safe_usingDict() dictionary decompression part %u failed", segNb);
XXH64_update(&crcNewState, testVerify, testSize);
crcNew = XXH64_digest(&crcNewState);
if (crcOrig!=crcNew)
{
size_t c=0;
while (testVerify[c] == testInput[testStart+c]) c++;
DISPLAY("Bad decompression at %u / %u \n", (U32)c, (U32)testSize);
}
FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe_usingDict() part %u corruption", segNb);
oldStart = testStart;
oldSize = testSize;
totalTestDone += testSize;
segNb ++;
}
DISPLAY("\r");
}
}
printf("All unit tests completed successfully \n");
@@ -1042,6 +1082,7 @@ static int FUZ_usage(char* programName)
DISPLAY( "\n");
DISPLAY( "Arguments :\n");
DISPLAY( " -i# : Nb of tests (default:%i) \n", NB_ATTEMPTS);
DISPLAY( " -T# : Duration of tests, in seconds (default: use Nb of tests) \n");
DISPLAY( " -s# : Select seed (default:prompt user)\n");
DISPLAY( " -t# : Select starting test number (default:0)\n");
DISPLAY( " -P# : Select compressibility in %% (default:%i%%)\n", FUZ_COMPRESSIBILITY_DEFAULT);
@@ -1062,8 +1103,9 @@ int main(int argc, char** argv)
int proba = FUZ_COMPRESSIBILITY_DEFAULT;
int pause = 0;
char* programName = argv[0];
U32 duration = 0;
// Check command line
/* Check command line */
for(argNb=1; argNb<argc; argNb++)
{
char* argument = argv[argNb];
@@ -1095,7 +1137,7 @@ int main(int argc, char** argv)
case 'i':
argument++;
nbTests=0;
nbTests = 0; duration = 0;
while ((*argument>='0') && (*argument<='9'))
{
nbTests *= 10;
@@ -1104,6 +1146,31 @@ int main(int argc, char** argv)
}
break;
case 'T':
argument++;
nbTests = 0; duration = 0;
for (;;)
{
switch(*argument)
{
case 'm': duration *= 60; argument++; continue;
case 's':
case 'n': argument++; continue;
case '0':
case '1':
case '2':
case '3':
case '4':
case '5':
case '6':
case '7':
case '8':
case '9': duration *= 10; duration += *argument++ - '0'; continue;
}
break;
}
break;
case 's':
argument++;
seed=0; seedset=1;
@@ -1144,7 +1211,6 @@ int main(int argc, char** argv)
}
}
// Get Seed
printf("Starting LZ4 fuzzer (%i-bits, %s)\n", (int)(sizeof(size_t)*8), LZ4_VERSION);
if (!seedset) seed = FUZ_GetMilliStart() % 10000;
@@ -1156,11 +1222,11 @@ int main(int argc, char** argv)
if (nbTests<=0) nbTests=1;
{
int result = FUZ_test(seed, nbTests, testNb, ((double)proba) / 100);
int result = FUZ_test(seed, nbTests, testNb, ((double)proba) / 100, duration);
if (pause)
{
DISPLAY("press enter ... \n");
getchar();
(void)getchar();
}
return result;
}

211
programs/lz4.1
View File

@@ -1,66 +1,173 @@
\"
\" lz4.1: This is a manual page for 'lz4' program. This file is part of the
\" lz4 <https://code.google.com/p/lz4/> project.
\" Author: Yann Collet
\"
\" No hyphenation
.hy 0
.nr HY 0
.TH lz4 "1" "2014-02-27" "lz4" "User Commands"
.TH lz4 "1" "2015-03-21" "lz4" "User Commands"
.SH NAME
\fBlz4\fR - Extremely fast compression algorithm
\fBlz4, unlz4, lz4cat\fR \- Compress or decompress .lz4 files
.SH SYNOPSIS
.TP 5
\fBlz4\fR [\fBOPTIONS\fR] [-|INPUT-FILE] <OUTPUT-FILE>
.PP
.B unlz4
is equivalent to
.BR "lz4 \-d"
.br
.B lz4cat
is equivalent to
.BR "lz4 \-dc"
.br
.PP
When writing scripts that need to decompress files,
it is recommended to always use the name
.B lz4
with appropriate arguments
.RB ( "lz4 \-d"
or
.BR "lz4 \-dc" )
instead of the names
.B unlz4
and
.BR lz4cat .
.SH DESCRIPTION
.PP
\fBlz4\fR is an extremely fast lossless compression algorithm. It is based on
the \fBLZ77\fR family of compression scheme. At the compression speed of 400
MB/s per core, \fBlz4\fR is also scalable with multi-core CPUs. It features
an extremely fast decoder, with speed in multiple GB/s per core, typically
reaching the RAM speed limits on multi-core systems. \fBlz4\fR supports
following options
\fBlz4\fR is an extremely fast lossless compression algorithm,
based on \fBbyte-aligned LZ77\fR family of compression scheme.
\fBlz4\fR offers compression speeds of 400 MB/s per core, linearly scalable with multi-core CPUs.
It features an extremely fast decoder, with speed in multiple GB/s per core,
typically reaching RAM speed limit on multi-core systems.
.B lz4
supports a command line syntax similar to
.BR gzip (1).
The native file format is the
.B .lz4
format.
.SS "Concatenation of .lz4 files"
It is possible to concatenate
.B .lz4
files as is.
.B lz4
will decompress such files as if they were a single
.B .lz4
file. For example:
lz4 file1 > foo.lz4
lz4 file2 >> foo.lz4
then
lz4cat foo.lz4
is equivalent to :
cat file1 file2
.PP
.SH OPTIONS
.
.SS "Short commands concatenation"
In some cases, some options can be expressed using short command
.B "-x"
or long command
.B "--long-word" .
Short commands can be concatenated together. For example,
.B "-d -c"
is equivalent to
.B "-dc" .
Long commands cannot be concatenated.
They must be clearly separated by a space.
.SS "Multiple commands"
When multiple contradictory commands are issued on a same command line,
only the latest one will be applied.
.
.SS "Operation mode"
.TP
.BR \-z ", " \-\-compress
Compress.
This is the default operation mode
when no operation mode option is specified ,
no other operation mode is implied from the command name
(for example,
.B unlz4
implies
.B \-\-decompress ),
nor from the input file name
(for example, a file extension
.B .lz4
implies
.B \-\-decompress
by default).
.B -z
can also be used to force compression of an already compressed
.B .lz4
file.
.TP
.BR \-d ", " \-\-decompress ", " \-\-uncompress
Decompress.
.B --decompress
is also the default operation when the input filename has an
.B .lz4
extensionq
.TP
.BR \-t ", " \-\-test
Test the integrity of compressed
.B .lz4
files.
The decompressed data is discarded.
No files are created nor removed.
.
.SS "Operation modifiers"
.TP
.B \-1
fast compression (default)
.TP
.B \-9
high compression
.TP
.B \-d
decompression
.BR \-f ", " --[no-]force
This option has several effects:
.RS
.IP \(bu 3
If the target file already exists,
overwrite it without prompting.
.IP \(bu 3
When used with
.B \-\-decompress
and
.B lz4
cannot recognize the type of the source file,
copy the source file as is to standard output.
This allows
.B lz4cat
.B \-\-force
to be used like
.BR cat (1)
for files that have not been compressed with
.BR lz4 .
.RE
.TP
.B \-f
overwrite output without prompting
.TP
.B \-h/\-H
display help/long help and exit
.TP
.B \-V
display Version number and exit
.TP
.B \-v
verbose mode
.TP
.B \-q
suppress warnings; specify twice to suppress errors too
.TP
.B \-c
.BR \-c ", " \--stdout ", " \--to-stdout
force write to standard output, even if it is the console
.TP
.B \-t
test compressed file integrity
.TP
.B \-z
force compression
.TP
.B \-l
use Legacy format (useful for Linux Kernel compression)
.BR \-m ", " \--multiple
Multiple file names.
By default, the second filename is used as the output filename for the compressed file.
With
.B -m
, you can specify any number of input filenames. Each of them will be compressed
independently, and the resulting name of the compressed file will be
.B filename.lz4
.
.TP
.B \-B#
block size [4-7](default : 7)
@@ -69,11 +176,37 @@ following options
.B \-BD
block dependency (improve compression ratio)
.TP
.B \-BX
enable block checksum (default:disabled)
.B \--[no-]frame-crc
select frame checksum (default:enabled)
.TP
.B \-Sx
disable stream checksum (default:enabled)
.B \--[no-]content-size
header includes original size (default:not present)
Note : this option can only be activated when the original size can be determined,
hence for a file. It won't work with unknown source size, such as stdin or pipe.
.TP
.B \--[no-]sparse
sparse file support (default:enabled)
.TP
.B \-l
use Legacy format (useful for Linux Kernel compression)
.
.SS "Other options"
.TP
.BR \-v ", " --verbose
verbose mode
.TP
.BR \-q ", " --quiet
suppress warnings; specify twice to suppress errors too
.TP
.B \-h/\-H
display help/long help and exit
.TP
.BR \-V ", " \--version
display Version number and exit
.TP
.BR \-k ", " \--keep
Don't delete source file.
This is default behavior anyway, so this option is just for compatibility with gzip/xz.
.TP
.B \-b
benchmark file(s)
@@ -82,7 +215,7 @@ following options
iteration loops [1-9](default : 3), benchmark mode only
.SH BUGS
Report bugs at:- https://code.google.com/p/lz4/
Report bugs at: https://github.com/Cyan4973/lz4
.SH AUTHOR
Yann Collet

33
programs/lz4c.1
View File

@@ -1,33 +0,0 @@
\"
\" lz4c.1: This is a manual page for 'lz4c' program. This file is part of the
\" lz4 <https://code.google.com/p/lz4/> project.
\"
\" No hyphenation
.hy 0
.nr HY 0
.TH lz4c "1" "2014-04-15" "lz4c" "User Commands"
.SH NAME
\fBlz4\fR - Extremely fast compression algorithm
.SH SYNOPSIS
.TP 5
\fBlz4c\fR [\fBOPTIONS\fR] [-|INPUT-FILE] <OUTPUT-FILE>
.SH DESCRIPTION
.PP
\fBlz4c\fR is the legacy version of \fBlz4\fR.
As such, it supports older supplementary legacy commands.
\fBlz4c\fR is now deprecated.
It is recommended to use \fBlz4\fR instead whenever possible.
To get a list of commands specific to lz4c, do :
lz4c -h
.SH BUGS
Report bugs at:- https://code.google.com/p/lz4/
.SH AUTHOR
Yann Collet

32
programs/lz4cat.1
View File

@@ -1,32 +0,0 @@
\"
\" lz4cat.1: This is a manual page for 'lz4cat' program. This file is part of
\" the lz4 <https://code.google.com/p/lz4/> project.
\"
\" No hyphenation
.hy 0
.nr HY 0
.TH lz4cat "1" "2014-06-20" "lz4cat" "User Commands"
.SH NAME
\fBlz4cat\fR - Utility based on LZ4
.SH SYNOPSIS
.TP 5
\fBlz4cat\fR [\fBOPTIONS\fR] [-|INPUT-FILE]
.SH DESCRIPTION
.PP
\fBlz4cat\fR is an utility based on \fBlz4\fR, an extremely fast lossless compression algorithm.
\fBlz4cat\fR decompress input file or stream, redirecting its output to the console.
It is equivalent to \fBlz4 -cd\fR,
Available options are the same as \fBlz4\fR ones (man lz4).
.SH BUGS
Report bugs at:- https://code.google.com/p/lz4/
.SH AUTHOR
Yann Collet

175
programs/lz4cli.c
View File

@@ -1,6 +1,6 @@
/*
LZ4cli - LZ4 Command Line Interface
Copyright (C) Yann Collet 2011-2014
Copyright (C) Yann Collet 2011-2015
GPL v2 License
@@ -19,7 +19,7 @@
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
You can contact the author at :
- LZ4 source repository : http://code.google.com/p/lz4/
- LZ4 source repository : https://github.com/Cyan4973/lz4
- LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c
*/
/*
@@ -59,23 +59,20 @@
#include <stdlib.h> /* exit, calloc, free */
#include <string.h> /* strcmp, strlen */
#include "bench.h" /* BMK_benchFile, BMK_SetNbIterations, BMK_SetBlocksize, BMK_SetPause */
#include "lz4io.h"
#include "lz4io.h" /* LZ4IO_compressFilename, LZ4IO_decompressFilename, LZ4IO_compressMultipleFilenames */
/****************************
* OS-specific Includes
*****************************/
#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(_WIN32) || defined(__CYGWIN__)
# include <fcntl.h> /* _O_BINARY */
# include <io.h> /* _setmode, _isatty */
#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(_WIN32)
# include <io.h> /* _isatty */
# ifdef __MINGW32__
int _fileno(FILE *stream); /* MINGW somehow forgets to include this prototype into <stdio.h> */
# endif
# define SET_BINARY_MODE(file) _setmode(_fileno(file), _O_BINARY)
# define IS_CONSOLE(stdStream) _isatty(_fileno(stdStream))
#else
# include <unistd.h> /* isatty */
# define SET_BINARY_MODE(file)
# define IS_CONSOLE(stdStream) isatty(fileno(stdStream))
#endif
@@ -85,12 +82,13 @@
******************************/
#define COMPRESSOR_NAME "LZ4 command line interface"
#ifndef LZ4_VERSION
# define LZ4_VERSION "r126"
# define LZ4_VERSION "r128"
#endif
#define AUTHOR "Yann Collet"
#define WELCOME_MESSAGE "*** %s %i-bits %s, by %s (%s) ***\n", COMPRESSOR_NAME, (int)(sizeof(void*)*8), LZ4_VERSION, AUTHOR, __DATE__
#define LZ4_EXTENSION ".lz4"
#define LZ4_CAT "lz4cat"
#define LZ4CAT "lz4cat"
#define UNLZ4 "unlz4"
#define KB *(1U<<10)
#define MB *(1U<<20)
@@ -104,7 +102,7 @@
***************************************/
#define DISPLAY(...) fprintf(stderr, __VA_ARGS__)
#define DISPLAYLEVEL(l, ...) if (displayLevel>=l) { DISPLAY(__VA_ARGS__); }
static unsigned displayLevel = 2; /* 0 : no display ; 1: errors ; 2 : + result + interaction + warnings ; 3 : + progression; 4 : + information */
static unsigned displayLevel = 2; /* 0 : no display ; 1: errors only ; 2 : downgradable normal ; 3 : non-downgradable normal; 4 : + information */
/**************************************
@@ -136,10 +134,10 @@ static char* programName;
#define EXTENDED_FORMAT
#define DEFAULT_COMPRESSOR LZ4IO_compressFilename
#define DEFAULT_DECOMPRESSOR LZ4IO_decompressFilename
int LZ4IO_compressFilename_Legacy(char* input_filename, char* output_filename, int compressionlevel); /* hidden function */
int LZ4IO_compressFilename_Legacy(const char* input_filename, const char* output_filename, int compressionlevel); /* hidden function */
/****************************
/*****************************
* Functions
*****************************/
static int usage(void)
@@ -170,11 +168,14 @@ static int usage_advanced(void)
DISPLAY( " -q : suppress warnings; specify twice to suppress errors too\n");
DISPLAY( " -c : force write to standard output, even if it is the console\n");
DISPLAY( " -t : test compressed file integrity\n");
DISPLAY( " -m : multiple input files (implies automatic output filenames)\n");
DISPLAY( " -l : compress using Legacy format (Linux kernel compression)\n");
DISPLAY( " -B# : Block size [4-7](default : 7)\n");
DISPLAY( " -BD : Block dependency (improve compression ratio)\n");
/* DISPLAY( " -BX : enable block checksum (default:disabled)\n"); *//* Option currently inactive */
DISPLAY( " -Sx : disable stream checksum (default:enabled)\n");
DISPLAY( "--no-frame-crc : disable stream checksum (default:enabled)\n");
DISPLAY( "--content-size : compressed frame includes original size (default:not present)\n");
DISPLAY( "--[no-]sparse : sparse file support (default:enabled)\n");
DISPLAY( "Benchmark arguments :\n");
DISPLAY( " -b : benchmark file(s)\n");
DISPLAY( " -i# : iteration loops [1-9](default : 3), benchmark mode only\n");
@@ -184,7 +185,6 @@ static int usage_advanced(void)
DISPLAY( " -c1 : high compression\n");
DISPLAY( " -hc : high compression\n");
DISPLAY( " -y : overwrite output without prompting \n");
DISPLAY( " -s : suppress warnings \n");
#endif /* ENABLE_LZ4C_LEGACY_OPTIONS */
EXTENDED_HELP;
return 0;
@@ -251,7 +251,7 @@ static int badusage(void)
static void waitEnter(void)
{
DISPLAY("Press enter to continue...\n");
getchar();
(void)getchar();
}
@@ -261,25 +261,29 @@ int main(int argc, char** argv)
cLevel=0,
decode=0,
bench=0,
filenamesStart=2,
legacy_format=0,
forceStdout=0,
forceCompress=0,
main_pause=0;
char* input_filename=0;
char* output_filename=0;
main_pause=0,
multiple_inputs=0,
operationResult=0;
const char* input_filename=0;
const char* output_filename=0;
char* dynNameSpace=0;
const char** inFileNames = NULL;
unsigned ifnIdx=0;
char nullOutput[] = NULL_OUTPUT;
char extension[] = LZ4_EXTENSION;
int blockSize;
int blockSize;
/* Init */
programName = argv[0];
LZ4IO_setOverwrite(0);
blockSize = LZ4IO_setBlockSizeID(LZ4_BLOCKSIZEID_DEFAULT);
/* lz4cat behavior */
if (!strcmp(programName, LZ4_CAT)) { decode=1; forceStdout=1; output_filename=stdoutmark; displayLevel=1; }
/* lz4cat predefined behavior */
if (!strcmp(programName, LZ4CAT)) { decode=1; forceStdout=1; output_filename=stdoutmark; displayLevel=1; }
if (!strcmp(programName, UNLZ4)) { decode=1; }
/* command switches */
for(i=1; i<argc; i++)
@@ -288,7 +292,29 @@ int main(int argc, char** argv)
if(!argument) continue; /* Protection if argument empty */
/* Decode command (note : aggregated commands are allowed) */
/* long commands (--long-word) */
if (!strcmp(argument, "--compress")) { forceCompress = 1; continue; }
if ((!strcmp(argument, "--decompress"))
|| (!strcmp(argument, "--uncompress"))) { decode = 1; continue; }
if (!strcmp(argument, "--multiple")) { multiple_inputs = 1; if (inFileNames==NULL) inFileNames = (const char**)malloc(argc * sizeof(char*)); continue; }
if (!strcmp(argument, "--test")) { decode = 1; LZ4IO_setOverwrite(1); output_filename=nulmark; continue; }
if (!strcmp(argument, "--force")) { LZ4IO_setOverwrite(1); continue; }
if (!strcmp(argument, "--no-force")) { LZ4IO_setOverwrite(0); continue; }
if ((!strcmp(argument, "--stdout"))
|| (!strcmp(argument, "--to-stdout"))) { forceStdout=1; output_filename=stdoutmark; displayLevel=1; continue; }
if (!strcmp(argument, "--frame-crc")) { LZ4IO_setStreamChecksumMode(1); continue; }
if (!strcmp(argument, "--no-frame-crc")) { LZ4IO_setStreamChecksumMode(0); continue; }
if (!strcmp(argument, "--content-size")) { LZ4IO_setContentSize(1); continue; }
if (!strcmp(argument, "--no-content-size")) { LZ4IO_setContentSize(0); continue; }
if (!strcmp(argument, "--sparse")) { LZ4IO_setSparseFile(2); continue; }
if (!strcmp(argument, "--no-sparse")) { LZ4IO_setSparseFile(0); continue; }
if (!strcmp(argument, "--verbose")) { displayLevel=4; continue; }
if (!strcmp(argument, "--quiet")) { if (displayLevel) displayLevel--; continue; }
if (!strcmp(argument, "--version")) { DISPLAY(WELCOME_MESSAGE); return 0; }
if (!strcmp(argument, "--keep")) { continue; } /* keep source file (default anyway; just for xz/lzma compatibility) */
/* Short commands (note : aggregated short commands are allowed) */
if (argument[0]=='-')
{
/* '-' means stdin/stdout */
@@ -308,7 +334,6 @@ int main(int argc, char** argv)
if ((argument[0]=='c') && (argument[1]=='1')) { cLevel=9; argument++; continue; } /* -c1 (high compression) */
if ((argument[0]=='h') && (argument[1]=='c')) { cLevel=9; argument++; continue; } /* -hc (high compression) */
if (*argument=='y') { LZ4IO_setOverwrite(1); continue; } /* -y (answer 'yes' to overwrite permission) */
if (*argument=='s') { displayLevel=1; continue; } /* -s (silent mode) */
#endif /* ENABLE_LZ4C_LEGACY_OPTIONS */
if ((*argument>='0') && (*argument<='9'))
@@ -353,7 +378,7 @@ int main(int argc, char** argv)
case 'v': displayLevel=4; break;
/* Quiet mode */
case 'q': displayLevel--; break;
case 'q': if (displayLevel) displayLevel--; break;
/* keep source file (default anyway, so useless) (for xz/lzma compatibility) */
case 'k': break;
@@ -372,36 +397,46 @@ int main(int argc, char** argv)
{
int B = argument[1] - '0';
blockSize = LZ4IO_setBlockSizeID(B);
BMK_SetBlocksize(blockSize);
BMK_setBlocksize(blockSize);
argument++;
break;
}
case 'D': LZ4IO_setBlockMode(LZ4IO_blockLinked); argument++; break;
case 'X': LZ4IO_setBlockChecksumMode(1); argument ++; break; /* currently disables */
case 'X': LZ4IO_setBlockChecksumMode(1); argument ++; break; /* currently disabled */
default : exitBlockProperties=1;
}
if (exitBlockProperties) break;
}
break;
/* Modify Stream properties */
case 'S': if (argument[1]=='x') { LZ4IO_setStreamChecksumMode(0); argument++; break; } else { badusage(); }
/* Benchmark */
case 'b': bench=1; break;
case 'b': bench=1; multiple_inputs=1;
if (inFileNames == NULL)
inFileNames = (const char**) malloc(argc * sizeof(char*));
break;
/* Treat non-option args as input files. See https://code.google.com/p/lz4/issues/detail?id=151 */
case 'm': multiple_inputs=1;
if (inFileNames == NULL)
inFileNames = (const char**) malloc(argc * sizeof(char*));
break;
/* Modify Nb Iterations (benchmark only) */
case 'i':
if ((argument[1] >='1') && (argument[1] <='9'))
{
int iters = argument[1] - '0';
BMK_SetNbIterations(iters);
argument++;
unsigned iters = 0;
while ((argument[1] >='0') && (argument[1] <='9'))
{
iters *= 10;
iters += argument[1] - '0';
argument++;
}
BMK_setNbIterations(iters);
}
break;
/* Pause at the end (hidden option) */
case 'p': main_pause=1; BMK_SetPause(); break;
case 'p': main_pause=1; BMK_setPause(); break;
/* Specific commands for customized versions */
EXTENDED_ARGUMENTS;
@@ -413,29 +448,41 @@ int main(int argc, char** argv)
continue;
}
/* first provided filename is input */
if (!input_filename) { input_filename=argument; filenamesStart=i; continue; }
/* Store in *inFileNames[] if -m is used. */
if (multiple_inputs) { inFileNames[ifnIdx++]=argument; continue; }
/* second provided filename is output */
/* Store first non-option arg in input_filename to preserve original cli logic. */
if (!input_filename) { input_filename=argument; continue; }
/* Second non-option arg in output_filename to preserve original cli logic. */
if (!output_filename)
{
output_filename=argument;
if (!strcmp (output_filename, nullOutput)) output_filename = nulmark;
continue;
}
/* 3rd non-option arg should not exist */
DISPLAYLEVEL(1, "Warning : %s won't be used ! Do you want multiple input files (-m) ? \n", argument);
}
DISPLAYLEVEL(3, WELCOME_MESSAGE);
if (!decode) DISPLAYLEVEL(4, "Blocks size : %i KB\n", blockSize>>10);
/* No input filename ==> use stdin */
if (multiple_inputs) input_filename = inFileNames[0], output_filename = (const char*)(inFileNames[0]);
if(!input_filename) { input_filename=stdinmark; }
/* Check if input or output are defined as console; trigger an error in this case */
/* Check if input is defined as console; trigger an error in this case */
if (!strcmp(input_filename, stdinmark) && IS_CONSOLE(stdin) ) badusage();
/* Check if benchmark is selected */
if (bench) return BMK_benchFile(argv+filenamesStart, argc-filenamesStart, cLevel);
if (bench)
{
int bmkResult = BMK_benchFiles(inFileNames, ifnIdx, cLevel);
free((void*)inFileNames);
return bmkResult;
}
/* No output filename ==> try to select one automatically (when possible) */
while (!output_filename)
@@ -450,9 +497,9 @@ int main(int argc, char** argv)
{
size_t l = strlen(input_filename);
dynNameSpace = (char*)calloc(1,l+5);
strcpy(dynNameSpace, input_filename);
strcat(dynNameSpace, LZ4_EXTENSION);
output_filename = dynNameSpace;
strcpy(output_filename, input_filename);
strcpy(output_filename+l, LZ4_EXTENSION);
DISPLAYLEVEL(2, "Compressed filename will be : %s \n", output_filename);
break;
}
@@ -461,12 +508,12 @@ int main(int argc, char** argv)
size_t outl;
size_t inl = strlen(input_filename);
dynNameSpace = (char*)calloc(1,inl+1);
output_filename = dynNameSpace;
strcpy(output_filename, input_filename);
strcpy(dynNameSpace, input_filename);
outl = inl;
if (inl>4)
while ((outl >= inl-4) && (input_filename[outl] == extension[outl-inl+4])) output_filename[outl--]=0;
while ((outl >= inl-4) && (input_filename[outl] == extension[outl-inl+4])) dynNameSpace[outl--]=0;
if (outl != inl-5) { DISPLAYLEVEL(1, "Cannot determine an output filename\n"); badusage(); }
output_filename = dynNameSpace;
DISPLAYLEVEL(2, "Decoding file %s \n", output_filename);
}
}
@@ -474,28 +521,40 @@ int main(int argc, char** argv)
/* Check if output is defined as console; trigger an error in this case */
if (!strcmp(output_filename,stdoutmark) && IS_CONSOLE(stdout) && !forceStdout) badusage();
/* No warning message in pure pipe mode (stdin + stdout) */
/* Downgrade notification level in pure pipe mode (stdin + stdout) and multiple file mode */
if (!strcmp(input_filename, stdinmark) && !strcmp(output_filename,stdoutmark) && (displayLevel==2)) displayLevel=1;
if ((multiple_inputs) && (displayLevel==2)) displayLevel=1;
/* IO Stream/File */
LZ4IO_setNotificationLevel(displayLevel);
if (decode) DEFAULT_DECOMPRESSOR(input_filename, output_filename);
if (decode)
{
if (multiple_inputs)
operationResult = LZ4IO_decompressMultipleFilenames(inFileNames, ifnIdx, LZ4_EXTENSION);
else
DEFAULT_DECOMPRESSOR(input_filename, output_filename);
}
else
{
/* compression is default action */
if (legacy_format)
{
DISPLAYLEVEL(3, "! Generating compressed LZ4 using Legacy format (deprecated) ! \n");
LZ4IO_compressFilename_Legacy(input_filename, output_filename, cLevel);
}
/* compression is default action */
if (legacy_format)
{
DISPLAYLEVEL(3, "! Generating compressed LZ4 using Legacy format (deprecated) ! \n");
LZ4IO_compressFilename_Legacy(input_filename, output_filename, cLevel);
}
else
{
if (multiple_inputs)
operationResult = LZ4IO_compressMultipleFilenames(inFileNames, ifnIdx, LZ4_EXTENSION, cLevel);
else
{
DEFAULT_COMPRESSOR(input_filename, output_filename, cLevel);
}
DEFAULT_COMPRESSOR(input_filename, output_filename, cLevel);
}
}
if (main_pause) waitEnter();
free(dynNameSpace);
free((void*)inFileNames);
if (operationResult != 0) return operationResult;
return 0;
}

873
programs/lz4io.c
View File

File diff suppressed because it is too large Load Diff

30
programs/lz4io.h
View File

@@ -1,6 +1,6 @@
/*
LZ4io.h - LZ4 File/Stream Interface
Copyright (C) Yann Collet 2011-2014
Copyright (C) Yann Collet 2011-2015
GPL v2 License
This program is free software; you can redistribute it and/or modify
@@ -18,7 +18,7 @@
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
You can contact the author at :
- LZ4 source repository : http://code.google.com/p/lz4/
- LZ4 source repository : https://github.com/Cyan4973/lz4
- LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c
*/
/*
@@ -29,17 +29,18 @@
- The license of this source file is GPLv2.
*/
#pragma once
/* ************************************************** */
/* Special input/output values */
/* ************************************************** */
#define NULL_OUTPUT "null"
static char stdinmark[] = "stdin";
static char stdoutmark[] = "stdout";
static char const stdinmark[] = "stdin";
static char const stdoutmark[] = "stdout";
#ifdef _WIN32
static char nulmark[] = "nul";
static char const nulmark[] = "nul";
#else
static char nulmark[] = "/dev/null";
static char const nulmark[] = "/dev/null";
#endif
@@ -47,10 +48,13 @@ static char nulmark[] = "/dev/null";
/* ****************** Functions ********************* */
/* ************************************************** */
int LZ4IO_compressFilename (char* input_filename, char* output_filename, int compressionlevel);
int LZ4IO_decompressFilename(char* input_filename, char* output_filename);
int LZ4IO_compressFilename (const char* input_filename, const char* output_filename, int compressionlevel);
int LZ4IO_decompressFilename(const char* input_filename, const char* output_filename);
int LZ4IO_compressMultipleFilenames(const char** inFileNamesTable, int ifntSize, const char* suffix, int compressionlevel);
int LZ4IO_decompressMultipleFilenames(const char** inFileNamesTable, int ifntSize, const char* suffix);
/* ************************************************** */
/* ****************** Parameters ******************** */
/* ************************************************** */
@@ -67,11 +71,17 @@ int LZ4IO_setBlockSizeID(int blockSizeID);
typedef enum { LZ4IO_blockLinked=0, LZ4IO_blockIndependent} LZ4IO_blockMode_t;
int LZ4IO_setBlockMode(LZ4IO_blockMode_t blockMode);
/* Default setting : no checksum */
/* Default setting : no block checksum */
int LZ4IO_setBlockChecksumMode(int xxhash);
/* Default setting : checksum enabled */
/* Default setting : stream checksum enabled */
int LZ4IO_setStreamChecksumMode(int xxhash);
/* Default setting : 0 (no notification) */
int LZ4IO_setNotificationLevel(int level);
/* Default setting : 0 (disabled) */
int LZ4IO_setSparseFile(int enable);
/* Default setting : 0 (disabled) */
int LZ4IO_setContentSize(int enable);

40
test/Makefile Normal file
View File

@@ -0,0 +1,40 @@
# ##########################################################################
# LZ4 tests - Makefile
# Copyright (C) Takayuki Matsuoka - Yann Collet 2011-2015
#
# GPL v2 License
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
#
# You can contact the author at :
# - LZ4 source repository : https://github.com/Cyan4973/lz4
# - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c
# ##########################################################################
# versionstest : Compatibility test between LZ4 versions stored on Github (r116+)
# ##########################################################################
PYTHON?= python3
TESTDIR := lz4test
default: all
all: versionstest
versionstest:
$(PYTHON) test-lz4-versions.py
clean:
@rm -fR $(TESTDIR)
@echo Cleaning completed

147
test/test-lz4-versions.py Normal file
View File

@@ -0,0 +1,147 @@
#!/usr/bin/env python3
import glob
import subprocess
import filecmp
import os
import shutil
import sys
import hashlib
repo_url = 'https://github.com/Cyan4973/lz4.git'
tmp_dir_name = 'test/lz4test'
make_cmd = 'make'
git_cmd = 'git'
test_dat_src = 'README.md'
test_dat = 'test_dat'
head = 'r999'
def proc(cmd_args, pipe=True, dummy=False):
if dummy:
return
if pipe:
subproc = subprocess.Popen(cmd_args,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
else:
subproc = subprocess.Popen(cmd_args)
return subproc.communicate()
def make(args, pipe=True):
return proc([make_cmd] + args, pipe)
def git(args, pipe=True):
return proc([git_cmd] + args, pipe)
def get_git_tags():
stdout, stderr = git(['tag', '-l', 'r[0-9][0-9][0-9]'])
tags = stdout.decode('utf-8').split()
return tags
# http://stackoverflow.com/a/19711609/2132223
def sha1_of_file(filepath):
with open(filepath, 'rb') as f:
return hashlib.sha1(f.read()).hexdigest()
if __name__ == '__main__':
error_code = 0
base_dir = os.getcwd() + '/..' # /path/to/lz4
tmp_dir = base_dir + '/' + tmp_dir_name # /path/to/lz4/test/lz4test
clone_dir = tmp_dir + '/' + 'lz4' # /path/to/lz4/test/lz4test/lz4
programs_dir = base_dir + '/programs' # /path/to/lz4/programs
os.makedirs(tmp_dir, exist_ok=True)
# since Travis clones limited depth, we should clone full repository
if not os.path.isdir(clone_dir):
git(['clone', repo_url, clone_dir])
shutil.copy2(base_dir + '/' + test_dat_src, tmp_dir + '/' + test_dat)
# Retrieve all release tags
print('Retrieve all release tags :')
os.chdir(clone_dir)
tags = [head] + get_git_tags()
print(tags);
# Build all release lz4c and lz4c32
for tag in tags:
os.chdir(base_dir)
dst_lz4c = '{}/lz4c.{}' .format(tmp_dir, tag) # /path/to/lz4/test/lz4test/lz4c.<TAG>
dst_lz4c32 = '{}/lz4c32.{}'.format(tmp_dir, tag) # /path/to/lz4/test/lz4test/lz4c32.<TAG>
if not os.path.isfile(dst_lz4c) or not os.path.isfile(dst_lz4c32) or tag == head:
if tag != head:
r_dir = '{}/{}'.format(tmp_dir, tag) # /path/to/lz4/test/lz4test/<TAG>
os.makedirs(r_dir, exist_ok=True)
os.chdir(clone_dir)
git(['--work-tree=' + r_dir, 'checkout', tag, '--', '.'], False)
os.chdir(r_dir + '/programs') # /path/to/lz4/lz4test/<TAG>/programs
make(['clean', 'lz4c', 'lz4c32'], False)
else:
os.chdir(programs_dir)
make(['lz4c', 'lz4c32'], False)
shutil.copy2('lz4c', dst_lz4c)
shutil.copy2('lz4c32', dst_lz4c32)
# Compress test.dat by all released lz4c and lz4c32
print('Compress test.dat by all released lz4c and lz4c32')
os.chdir(tmp_dir)
for lz4 in glob.glob("*.lz4"):
os.remove(lz4)
for tag in tags:
proc(['./lz4c.' + tag, '-1fz', test_dat, test_dat + '_1_64_' + tag + '.lz4'])
proc(['./lz4c.' + tag, '-9fz', test_dat, test_dat + '_9_64_' + tag + '.lz4'])
proc(['./lz4c32.' + tag, '-1fz', test_dat, test_dat + '_1_32_' + tag + '.lz4'])
proc(['./lz4c32.' + tag, '-9fz', test_dat, test_dat + '_9_32_' + tag + '.lz4'])
print('Full list of compressed files')
lz4s = sorted(glob.glob('*.lz4'))
for lz4 in lz4s:
print(lz4 + ' : ' + repr(os.path.getsize(lz4)))
# Remove duplicated .lz4 files
print('')
print('Duplicated files')
lz4s = sorted(glob.glob('*.lz4'))
for i, lz4 in enumerate(lz4s):
if not os.path.isfile(lz4):
continue
for j in range(i+1, len(lz4s)):
lz4t = lz4s[j]
if not os.path.isfile(lz4t):
continue
if filecmp.cmp(lz4, lz4t):
os.remove(lz4t)
print('{} == {}'.format(lz4, lz4t))
print('Enumerate only different compressed files')
lz4s = sorted(glob.glob('*.lz4'))
for lz4 in lz4s:
print(lz4 + ' : ' + repr(os.path.getsize(lz4)) + ', ' + sha1_of_file(lz4))
# Decompress remained .lz4 files by all released lz4c and lz4c32
print('Decompression tests and verifications')
lz4s = sorted(glob.glob('*.lz4'))
for dec in glob.glob("*.dec"):
os.remove(dec)
for lz4 in lz4s:
print(lz4, end=" ")
for tag in tags:
print(tag, end=" ")
proc(['./lz4c.' + tag, '-df', lz4, lz4 + '_d64_' + tag + '.dec'])
proc(['./lz4c32.' + tag, '-df', lz4, lz4 + '_d32_' + tag + '.dec'])
print(' OK') # well, here, decompression has worked; but file is not yet verified
# Compare all '.dec' files with test_dat
decs = glob.glob('*.dec')
for dec in decs:
if not filecmp.cmp(dec, test_dat):
print('ERR : ' + dec)
error_code = 1
else:
print('OK : ' + dec)
os.remove(dec)
if error_code != 0:
print('ERROR')
sys.exit(error_code)

159
visual/2012/datagen/datagen.vcxproj Normal file
View File

@@ -0,0 +1,159 @@
<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup Label="ProjectConfigurations">
<ProjectConfiguration Include="Debug|Win32">
<Configuration>Debug</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Debug|x64">
<Configuration>Debug</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|Win32">
<Configuration>Release</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|x64">
<Configuration>Release</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
</ItemGroup>
<PropertyGroup Label="Globals">
<ProjectGuid>{D745AE2F-596A-403A-9B91-81A8C6779243}</ProjectGuid>
<Keyword>Win32Proj</Keyword>
<RootNamespace>datagen</RootNamespace>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
<ImportGroup Label="ExtensionSettings">
</ImportGroup>
<ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="PropertySheets">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="PropertySheets">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<PropertyGroup Label="UserMacros" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<LinkIncremental>true</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<LinkIncremental>true</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<LinkIncremental>false</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<LinkIncremental>false</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<ClCompile>
<PrecompiledHeader>
</PrecompiledHeader>
<WarningLevel>Level4</WarningLevel>
<Optimization>Disabled</Optimization>
<PreprocessorDefinitions>WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<ClCompile>
<PrecompiledHeader>
</PrecompiledHeader>
<WarningLevel>Level4</WarningLevel>
<Optimization>Disabled</Optimization>
<PreprocessorDefinitions>WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<ClCompile>
<WarningLevel>Level4</WarningLevel>
<PrecompiledHeader>
</PrecompiledHeader>
<Optimization>MaxSpeed</Optimization>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<PreprocessorDefinitions>WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<ClCompile>
<WarningLevel>Level4</WarningLevel>
<PrecompiledHeader>
</PrecompiledHeader>
<Optimization>MaxSpeed</Optimization>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<PreprocessorDefinitions>WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
</Link>
</ItemDefinitionGroup>
<ItemGroup>
<ClCompile Include="..\..\..\programs\datagen.c" />
<ClCompile Include="..\..\..\programs\datagencli.c" />
</ItemGroup>
<ItemGroup>
<ClInclude Include="..\..\..\programs\datagen.h" />
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
<ImportGroup Label="ExtensionTargets">
</ImportGroup>
</Project>

30
visual/2012/datagen/datagen.vcxproj.filters Normal file
View File

@@ -0,0 +1,30 @@
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<Filter Include="Fichiers sources">
<UniqueIdentifier>{4FC737F1-C7A5-4376-A066-2A32D752A2FF}</UniqueIdentifier>
<Extensions>cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx</Extensions>
</Filter>
<Filter Include="Fichiers d%27en-tête">
<UniqueIdentifier>{93995380-89BD-4b04-88EB-625FBE52EBFB}</UniqueIdentifier>
<Extensions>h;hpp;hxx;hm;inl;inc;xsd</Extensions>
</Filter>
<Filter Include="Fichiers de ressources">
<UniqueIdentifier>{67DA6AB6-F800-4c08-8B7A-83BB121AAD01}</UniqueIdentifier>
<Extensions>rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms</Extensions>
</Filter>
</ItemGroup>
<ItemGroup>
<ClCompile Include="..\..\..\programs\datagen.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
<ClCompile Include="..\..\..\programs\datagencli.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
</ItemGroup>
<ItemGroup>
<ClInclude Include="..\..\..\programs\datagen.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
</ItemGroup>
</Project>

166
visual/2012/frametest/frametest.vcxproj Normal file
View File

@@ -0,0 +1,166 @@
<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup Label="ProjectConfigurations">
<ProjectConfiguration Include="Debug|Win32">
<Configuration>Debug</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Debug|x64">
<Configuration>Debug</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|Win32">
<Configuration>Release</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|x64">
<Configuration>Release</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
</ItemGroup>
<PropertyGroup Label="Globals">
<ProjectGuid>{39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}</ProjectGuid>
<Keyword>Win32Proj</Keyword>
<RootNamespace>frametest</RootNamespace>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
<ImportGroup Label="ExtensionSettings">
</ImportGroup>
<ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="PropertySheets">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="PropertySheets">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<PropertyGroup Label="UserMacros" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<LinkIncremental>true</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<LinkIncremental>true</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<LinkIncremental>false</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<LinkIncremental>false</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<ClCompile>
<PrecompiledHeader>
</PrecompiledHeader>
<WarningLevel>Level4</WarningLevel>
<Optimization>Disabled</Optimization>
<PreprocessorDefinitions>WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<ClCompile>
<PrecompiledHeader>
</PrecompiledHeader>
<WarningLevel>Level4</WarningLevel>
<Optimization>Disabled</Optimization>
<PreprocessorDefinitions>WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<ClCompile>
<WarningLevel>Level4</WarningLevel>
<PrecompiledHeader>
</PrecompiledHeader>
<Optimization>MaxSpeed</Optimization>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<PreprocessorDefinitions>WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<ClCompile>
<WarningLevel>Level4</WarningLevel>
<PrecompiledHeader>
</PrecompiledHeader>
<Optimization>MaxSpeed</Optimization>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<PreprocessorDefinitions>WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
</Link>
</ItemDefinitionGroup>
<ItemGroup>
<ClCompile Include="..\..\..\lib\lz4.c" />
<ClCompile Include="..\..\..\lib\lz4frame.c" />
<ClCompile Include="..\..\..\lib\lz4hc.c" />
<ClCompile Include="..\..\..\lib\xxhash.c" />
<ClCompile Include="..\..\..\programs\frametest.c" />
</ItemGroup>
<ItemGroup>
<ClInclude Include="..\..\..\lib\lz4.h" />
<ClInclude Include="..\..\..\lib\lz4frame.h" />
<ClInclude Include="..\..\..\lib\lz4frame_static.h" />
<ClInclude Include="..\..\..\lib\lz4hc.h" />
<ClInclude Include="..\..\..\lib\xxhash.h" />
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
<ImportGroup Label="ExtensionTargets">
</ImportGroup>
</Project>

51
visual/2012/frametest/frametest.vcxproj.filters Normal file
View File

@@ -0,0 +1,51 @@
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<Filter Include="Fichiers sources">
<UniqueIdentifier>{4FC737F1-C7A5-4376-A066-2A32D752A2FF}</UniqueIdentifier>
<Extensions>cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx</Extensions>
</Filter>
<Filter Include="Fichiers d%27en-tête">
<UniqueIdentifier>{93995380-89BD-4b04-88EB-625FBE52EBFB}</UniqueIdentifier>
<Extensions>h;hpp;hxx;hm;inl;inc;xsd</Extensions>
</Filter>
<Filter Include="Fichiers de ressources">
<UniqueIdentifier>{67DA6AB6-F800-4c08-8B7A-83BB121AAD01}</UniqueIdentifier>
<Extensions>rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms</Extensions>
</Filter>
</ItemGroup>
<ItemGroup>
<ClCompile Include="..\..\..\lib\lz4.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
<ClCompile Include="..\..\..\lib\lz4frame.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
<ClCompile Include="..\..\..\lib\lz4hc.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
<ClCompile Include="..\..\..\lib\xxhash.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
<ClCompile Include="..\..\..\programs\frametest.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
</ItemGroup>
<ItemGroup>
<ClInclude Include="..\..\..\lib\lz4.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
<ClInclude Include="..\..\..\lib\lz4frame.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
<ClInclude Include="..\..\..\lib\lz4frame_static.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
<ClInclude Include="..\..\..\lib\lz4hc.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
<ClInclude Include="..\..\..\lib\xxhash.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
</ItemGroup>
</Project>

166
visual/2012/fullbench/fullbench.vcxproj Normal file
View File

@@ -0,0 +1,166 @@
<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup Label="ProjectConfigurations">
<ProjectConfiguration Include="Debug|Win32">
<Configuration>Debug</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Debug|x64">
<Configuration>Debug</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|Win32">
<Configuration>Release</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|x64">
<Configuration>Release</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
</ItemGroup>
<PropertyGroup Label="Globals">
<ProjectGuid>{6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}</ProjectGuid>
<Keyword>Win32Proj</Keyword>
<RootNamespace>fullbench</RootNamespace>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
<ImportGroup Label="ExtensionSettings">
</ImportGroup>
<ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="PropertySheets">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="PropertySheets">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<PropertyGroup Label="UserMacros" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<LinkIncremental>true</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<LinkIncremental>true</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<LinkIncremental>false</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<LinkIncremental>false</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<ClCompile>
<PrecompiledHeader>
</PrecompiledHeader>
<WarningLevel>Level4</WarningLevel>
<Optimization>Disabled</Optimization>
<PreprocessorDefinitions>WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<ClCompile>
<PrecompiledHeader>
</PrecompiledHeader>
<WarningLevel>Level4</WarningLevel>
<Optimization>Disabled</Optimization>
<PreprocessorDefinitions>WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<ClCompile>
<WarningLevel>Level4</WarningLevel>
<PrecompiledHeader>
</PrecompiledHeader>
<Optimization>MaxSpeed</Optimization>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<PreprocessorDefinitions>WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<ClCompile>
<WarningLevel>Level4</WarningLevel>
<PrecompiledHeader>
</PrecompiledHeader>
<Optimization>MaxSpeed</Optimization>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<PreprocessorDefinitions>WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
</Link>
</ItemDefinitionGroup>
<ItemGroup>
<ClCompile Include="..\..\..\lib\lz4.c" />
<ClCompile Include="..\..\..\lib\lz4frame.c" />
<ClCompile Include="..\..\..\lib\lz4hc.c" />
<ClCompile Include="..\..\..\lib\xxhash.c" />
<ClCompile Include="..\..\..\programs\fullbench.c" />
</ItemGroup>
<ItemGroup>
<ClInclude Include="..\..\..\lib\lz4.h" />
<ClInclude Include="..\..\..\lib\lz4frame.h" />
<ClInclude Include="..\..\..\lib\lz4frame_static.h" />
<ClInclude Include="..\..\..\lib\lz4hc.h" />
<ClInclude Include="..\..\..\lib\xxhash.h" />
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
<ImportGroup Label="ExtensionTargets">
</ImportGroup>
</Project>

51
visual/2012/fullbench/fullbench.vcxproj.filters Normal file
View File

@@ -0,0 +1,51 @@
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<Filter Include="Fichiers sources">
<UniqueIdentifier>{4FC737F1-C7A5-4376-A066-2A32D752A2FF}</UniqueIdentifier>
<Extensions>cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx</Extensions>
</Filter>
<Filter Include="Fichiers d%27en-tête">
<UniqueIdentifier>{93995380-89BD-4b04-88EB-625FBE52EBFB}</UniqueIdentifier>
<Extensions>h;hpp;hxx;hm;inl;inc;xsd</Extensions>
</Filter>
<Filter Include="Fichiers de ressources">
<UniqueIdentifier>{67DA6AB6-F800-4c08-8B7A-83BB121AAD01}</UniqueIdentifier>
<Extensions>rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms</Extensions>
</Filter>
</ItemGroup>
<ItemGroup>
<ClCompile Include="..\..\..\lib\lz4.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
<ClCompile Include="..\..\..\lib\lz4frame.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
<ClCompile Include="..\..\..\lib\lz4hc.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
<ClCompile Include="..\..\..\lib\xxhash.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
<ClCompile Include="..\..\..\programs\fullbench.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
</ItemGroup>
<ItemGroup>
<ClInclude Include="..\..\..\lib\lz4.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
<ClInclude Include="..\..\..\lib\lz4frame.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
<ClInclude Include="..\..\..\lib\lz4frame_static.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
<ClInclude Include="..\..\..\lib\lz4hc.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
<ClInclude Include="..\..\..\lib\xxhash.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
</ItemGroup>
</Project>

163
visual/2012/fuzzer/fuzzer.vcxproj Normal file
View File

@@ -0,0 +1,163 @@
<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup Label="ProjectConfigurations">
<ProjectConfiguration Include="Debug|Win32">
<Configuration>Debug</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Debug|x64">
<Configuration>Debug</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|Win32">
<Configuration>Release</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|x64">
<Configuration>Release</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
</ItemGroup>
<PropertyGroup Label="Globals">
<ProjectGuid>{18B9F1A7-9C66-4352-898B-30804DADE0FD}</ProjectGuid>
<Keyword>Win32Proj</Keyword>
<RootNamespace>fuzzer</RootNamespace>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
<ImportGroup Label="ExtensionSettings">
</ImportGroup>
<ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="PropertySheets">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="PropertySheets">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<PropertyGroup Label="UserMacros" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<LinkIncremental>true</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<LinkIncremental>true</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<LinkIncremental>false</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<LinkIncremental>false</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<ClCompile>
<PrecompiledHeader>
</PrecompiledHeader>
<WarningLevel>Level4</WarningLevel>
<Optimization>Disabled</Optimization>
<PreprocessorDefinitions>WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<ClCompile>
<PrecompiledHeader>
</PrecompiledHeader>
<WarningLevel>Level4</WarningLevel>
<Optimization>Disabled</Optimization>
<PreprocessorDefinitions>WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<ClCompile>
<WarningLevel>Level4</WarningLevel>
<PrecompiledHeader>
</PrecompiledHeader>
<Optimization>MaxSpeed</Optimization>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<PreprocessorDefinitions>WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<ClCompile>
<WarningLevel>Level4</WarningLevel>
<PrecompiledHeader>
</PrecompiledHeader>
<Optimization>MaxSpeed</Optimization>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<PreprocessorDefinitions>WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
</Link>
</ItemDefinitionGroup>
<ItemGroup>
<ClCompile Include="..\..\..\lib\lz4.c" />
<ClCompile Include="..\..\..\lib\lz4hc.c" />
<ClCompile Include="..\..\..\lib\xxhash.c" />
<ClCompile Include="..\..\..\programs\fuzzer.c" />
</ItemGroup>
<ItemGroup>
<ClInclude Include="..\..\..\lib\lz4.h" />
<ClInclude Include="..\..\..\lib\lz4hc.h" />
<ClInclude Include="..\..\..\lib\xxhash.h" />
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
<ImportGroup Label="ExtensionTargets">
</ImportGroup>
</Project>

42
visual/2012/fuzzer/fuzzer.vcxproj.filters Normal file
View File

@@ -0,0 +1,42 @@
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<Filter Include="Fichiers sources">
<UniqueIdentifier>{4FC737F1-C7A5-4376-A066-2A32D752A2FF}</UniqueIdentifier>
<Extensions>cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx</Extensions>
</Filter>
<Filter Include="Fichiers d%27en-tête">
<UniqueIdentifier>{93995380-89BD-4b04-88EB-625FBE52EBFB}</UniqueIdentifier>
<Extensions>h;hpp;hxx;hm;inl;inc;xsd</Extensions>
</Filter>
<Filter Include="Fichiers de ressources">
<UniqueIdentifier>{67DA6AB6-F800-4c08-8B7A-83BB121AAD01}</UniqueIdentifier>
<Extensions>rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms</Extensions>
</Filter>
</ItemGroup>
<ItemGroup>
<ClCompile Include="..\..\..\programs\fuzzer.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
<ClCompile Include="..\..\..\lib\lz4.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
<ClCompile Include="..\..\..\lib\lz4hc.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
<ClCompile Include="..\..\..\lib\xxhash.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
</ItemGroup>
<ItemGroup>
<ClInclude Include="..\..\..\lib\lz4.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
<ClInclude Include="..\..\..\lib\lz4hc.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
<ClInclude Include="..\..\..\lib\xxhash.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
</ItemGroup>
</Project>

66
visual/2012/lz4.sln Normal file
View File

@@ -0,0 +1,66 @@

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio Express 2012 for Windows Desktop
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "lz4", "lz4\lz4.vcxproj", "{E30329AC-0057-4FE0-8FDA-7F650D398C4C}"
EndProject
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "fuzzer", "fuzzer\fuzzer.vcxproj", "{18B9F1A7-9C66-4352-898B-30804DADE0FD}"
EndProject
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "fullbench", "fullbench\fullbench.vcxproj", "{6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}"
EndProject
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "frametest", "frametest\frametest.vcxproj", "{39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}"
EndProject
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "datagen", "datagen\datagen.vcxproj", "{D745AE2F-596A-403A-9B91-81A8C6779243}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Win32 = Debug|Win32
Debug|x64 = Debug|x64
Release|Win32 = Release|Win32
Release|x64 = Release|x64
EndGlobalSection
GlobalSection(ProjectConfigurationPlatforms) = postSolution
{E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Debug|Win32.ActiveCfg = Debug|Win32
{E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Debug|Win32.Build.0 = Debug|Win32
{E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Debug|x64.ActiveCfg = Debug|x64
{E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Debug|x64.Build.0 = Debug|x64
{E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Release|Win32.ActiveCfg = Release|Win32
{E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Release|Win32.Build.0 = Release|Win32
{E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Release|x64.ActiveCfg = Release|x64
{E30329AC-0057-4FE0-8FDA-7F650D398C4C}.Release|x64.Build.0 = Release|x64
{18B9F1A7-9C66-4352-898B-30804DADE0FD}.Debug|Win32.ActiveCfg = Debug|Win32
{18B9F1A7-9C66-4352-898B-30804DADE0FD}.Debug|Win32.Build.0 = Debug|Win32
{18B9F1A7-9C66-4352-898B-30804DADE0FD}.Debug|x64.ActiveCfg = Debug|x64
{18B9F1A7-9C66-4352-898B-30804DADE0FD}.Debug|x64.Build.0 = Debug|x64
{18B9F1A7-9C66-4352-898B-30804DADE0FD}.Release|Win32.ActiveCfg = Release|Win32
{18B9F1A7-9C66-4352-898B-30804DADE0FD}.Release|Win32.Build.0 = Release|Win32
{18B9F1A7-9C66-4352-898B-30804DADE0FD}.Release|x64.ActiveCfg = Release|x64
{18B9F1A7-9C66-4352-898B-30804DADE0FD}.Release|x64.Build.0 = Release|x64
{6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Debug|Win32.ActiveCfg = Debug|Win32
{6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Debug|Win32.Build.0 = Debug|Win32
{6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Debug|x64.ActiveCfg = Debug|x64
{6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Debug|x64.Build.0 = Debug|x64
{6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Release|Win32.ActiveCfg = Release|Win32
{6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Release|Win32.Build.0 = Release|Win32
{6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Release|x64.ActiveCfg = Release|x64
{6A4DF4EF-C77F-43C6-8901-DDCD20879E4E}.Release|x64.Build.0 = Release|x64
{39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Debug|Win32.ActiveCfg = Debug|Win32
{39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Debug|Win32.Build.0 = Debug|Win32
{39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Debug|x64.ActiveCfg = Debug|x64
{39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Debug|x64.Build.0 = Debug|x64
{39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Release|Win32.ActiveCfg = Release|Win32
{39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Release|Win32.Build.0 = Release|Win32
{39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Release|x64.ActiveCfg = Release|x64
{39AD6ECC-8BAD-4368-95E4-A1AA2F077BB7}.Release|x64.Build.0 = Release|x64
{D745AE2F-596A-403A-9B91-81A8C6779243}.Debug|Win32.ActiveCfg = Debug|Win32
{D745AE2F-596A-403A-9B91-81A8C6779243}.Debug|Win32.Build.0 = Debug|Win32
{D745AE2F-596A-403A-9B91-81A8C6779243}.Debug|x64.ActiveCfg = Debug|x64
{D745AE2F-596A-403A-9B91-81A8C6779243}.Debug|x64.Build.0 = Debug|x64
{D745AE2F-596A-403A-9B91-81A8C6779243}.Release|Win32.ActiveCfg = Release|Win32
{D745AE2F-596A-403A-9B91-81A8C6779243}.Release|Win32.Build.0 = Release|Win32
{D745AE2F-596A-403A-9B91-81A8C6779243}.Release|x64.ActiveCfg = Release|x64
{D745AE2F-596A-403A-9B91-81A8C6779243}.Release|x64.Build.0 = Release|x64
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
EndGlobalSection
EndGlobal

170
visual/2012/lz4/lz4.vcxproj Normal file
View File

@@ -0,0 +1,170 @@
<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup Label="ProjectConfigurations">
<ProjectConfiguration Include="Debug|Win32">
<Configuration>Debug</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Debug|x64">
<Configuration>Debug</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|Win32">
<Configuration>Release</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|x64">
<Configuration>Release</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
</ItemGroup>
<PropertyGroup Label="Globals">
<ProjectGuid>{E30329AC-0057-4FE0-8FDA-7F650D398C4C}</ProjectGuid>
<Keyword>Win32Proj</Keyword>
<RootNamespace>lz4</RootNamespace>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<PlatformToolset>v110</PlatformToolset>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
<ImportGroup Label="ExtensionSettings">
</ImportGroup>
<ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="PropertySheets">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="PropertySheets">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<PropertyGroup Label="UserMacros" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<LinkIncremental>true</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<LinkIncremental>true</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<LinkIncremental>false</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<LinkIncremental>false</LinkIncremental>
<IncludePath>$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath);</IncludePath>
</PropertyGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<ClCompile>
<PrecompiledHeader>
</PrecompiledHeader>
<WarningLevel>Level4</WarningLevel>
<Optimization>Disabled</Optimization>
<PreprocessorDefinitions>WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<ClCompile>
<PrecompiledHeader>
</PrecompiledHeader>
<WarningLevel>Level4</WarningLevel>
<Optimization>Disabled</Optimization>
<PreprocessorDefinitions>WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<ClCompile>
<WarningLevel>Level4</WarningLevel>
<PrecompiledHeader>
</PrecompiledHeader>
<Optimization>MaxSpeed</Optimization>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<PreprocessorDefinitions>WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<ClCompile>
<WarningLevel>Level4</WarningLevel>
<PrecompiledHeader>
</PrecompiledHeader>
<Optimization>MaxSpeed</Optimization>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<PreprocessorDefinitions>WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<TreatWarningAsError>true</TreatWarningAsError>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
</Link>
</ItemDefinitionGroup>
<ItemGroup>
<ClInclude Include="..\..\..\lib\lz4.h" />
<ClInclude Include="..\..\..\lib\lz4frame.h" />
<ClInclude Include="..\..\..\lib\lz4frame_static.h" />
<ClInclude Include="..\..\..\lib\lz4hc.h" />
<ClInclude Include="..\..\..\lib\xxhash.h" />
<ClInclude Include="..\..\..\programs\bench.h" />
<ClInclude Include="..\..\..\programs\lz4io.h" />
</ItemGroup>
<ItemGroup>
<ClCompile Include="..\..\..\lib\lz4.c" />
<ClCompile Include="..\..\..\lib\lz4frame.c" />
<ClCompile Include="..\..\..\lib\lz4hc.c" />
<ClCompile Include="..\..\..\lib\xxhash.c" />
<ClCompile Include="..\..\..\programs\bench.c" />
<ClCompile Include="..\..\..\programs\lz4cli.c" />
<ClCompile Include="..\..\..\programs\lz4io.c" />
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
<ImportGroup Label="ExtensionTargets">
</ImportGroup>
</Project>

63
visual/2012/lz4/lz4.vcxproj.filters Normal file
View File

@@ -0,0 +1,63 @@
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<Filter Include="Fichiers sources">
<UniqueIdentifier>{4FC737F1-C7A5-4376-A066-2A32D752A2FF}</UniqueIdentifier>
<Extensions>cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx</Extensions>
</Filter>
<Filter Include="Fichiers d%27en-tête">
<UniqueIdentifier>{93995380-89BD-4b04-88EB-625FBE52EBFB}</UniqueIdentifier>
<Extensions>h;hpp;hxx;hm;inl;inc;xsd</Extensions>
</Filter>
<Filter Include="Fichiers de ressources">
<UniqueIdentifier>{67DA6AB6-F800-4c08-8B7A-83BB121AAD01}</UniqueIdentifier>
<Extensions>rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms</Extensions>
</Filter>
</ItemGroup>
<ItemGroup>
<ClInclude Include="..\..\..\programs\lz4io.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
<ClInclude Include="..\..\..\programs\bench.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
<ClInclude Include="..\..\..\lib\xxhash.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
<ClInclude Include="..\..\..\lib\lz4hc.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
<ClInclude Include="..\..\..\lib\lz4frame_static.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
<ClInclude Include="..\..\..\lib\lz4frame.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
<ClInclude Include="..\..\..\lib\lz4.h">
<Filter>Fichiers d%27en-tête</Filter>
</ClInclude>
</ItemGroup>
<ItemGroup>
<ClCompile Include="..\..\..\programs\lz4io.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
<ClCompile Include="..\..\..\programs\lz4cli.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
<ClCompile Include="..\..\..\programs\bench.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
<ClCompile Include="..\..\..\lib\xxhash.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
<ClCompile Include="..\..\..\lib\lz4hc.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
<ClCompile Include="..\..\..\lib\lz4frame.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
<ClCompile Include="..\..\..\lib\lz4.c">
<Filter>Fichiers sources</Filter>
</ClCompile>
</ItemGroup>
</Project>